PyCon 2011

PyCon logoEvery year, I try to attend the PyCon con­fer­ence dat­ing back to to IPC7 or 8, I don’t recall. That was last cen­tury. This year, it’s once again in Atlanta, GA, and while it doesn’t make for the most excit­ing scenery, it’s a very func­tional arrange­ment. Just a few min­utes ago, Jesse Noller announced the full list of talks, and it’s look­ing like a great set again this year. Con­grat­u­la­tions to every­one who was accepted, and sorry that even more great talks couldn’t be fit into the sched­ule. I’m sure it was a dif­fi­cult decision.

So, I’ve already paid, sched­uled, booked and oth­er­wise pop­u­lated a TripIt plan for this March. Hope to see every­one there.

The myth of the social network

The ever inter­est­ing Jere­miah Owyang has a post up on his blog about what to do if your boss “friends” you on Face­book. Now, it was made over a year ago, but I just saw it when he re-posted a link to it on the twit­ter­verse. It brings up the inter­est­ing idea of what options are avail­able in such a sit­u­a­tion, and while I pre­fer to believe that a man­ager would never “friend” a sub­or­di­nate — it would be a gigan­tic abuse of power — it’s highly likely to hap­pen on a more and more fre­quent basis. While his advice is sound, even if I dis­agree with the con­clu­sion, I think it’s perched on a dan­ger­ously false assump­tion: that we have but one social network.

A Venn diagram of three social networks: work, friends and familyPer­haps I am more than a smidgen old school, but I do not believe that my life has but one dimen­sion to it. I do not believe that it has a sin­gle con­tin­u­ous social inter­ac­tion stream. For me, I try to keep things par­ti­tioned up and care­fully seg­re­gated for a few rea­sons I’ll go into. The major seg­ments are in the nerdi­li­cious Venn dia­gram to the right: work, friends and fam­ily. Now, some might scoff at the idea of keep­ing friends and fam­ily as two par­tially over­lap­ping (or par­tially con­nected, if we are to use the more cor­rect graph ter­mi­nol­ogy) spheres of social inter­ac­tion, but I believe that if we are hon­est with our­selves, that we will acknowl­edge that while we have friends and fam­ily, not all fam­ily are friends, and only some of your friends are fam­ily. It’s the nature of our rela­tion­ships. If I am truth­ful, I even sub­di­vide my friends into var­i­ous groups. While they might all be equal in my affec­tions, that does not imply that they are a fun­gi­ble resource. The var­i­ous sub­groups often are not really com­pat­i­ble, and while that is par­tially a result of my slightly schiz­o­phrenic nature, I believe that if we exam­ine closely, it is true for many.

The rea­sons for keep­ing these cir­cles only par­tially inter­sect­ing are numer­ous, but even­tu­ally boil down to sev­eral rough guides. First, I gen­er­ally keep my per­sonal and pro­fes­sional life sep­a­rate in order to be able to bet­ter man­age peo­ple at work, and also to keep the inevitable drama to a min­i­mum. Some might see this as harsh, as they trea­sure their “work friends”, but this is just my expe­ri­ence and per­sonal pref­er­ence. The sec­ond, and more dif­fi­cult to explain, is that some groups of friends will never mix like a good vinai­grette. Instead, they’ll stay sep­a­rated because of “lifestyle choices” that oth­ers might not approve of. Per­son­ally, I don’t care. Lastly, there are those that are sep­a­rate through noth­ing more than the fact that my con­nec­tion to a group of peo­ple is dif­fer­ent, or encom­passes a dif­fer­ent set of inter­ests, than another. It’s not a rank­ing of impor­tance — all friends are impor­tant — but sim­ply an acknowl­edg­ment of dif­fer­ences. With each of these groups I might share dif­fer­ent aspects of my life and my per­son­al­ity, and so there’s no one stream of thoughts that fits them all.

So there in lies the fal­lacy of “the social net­work”. There is not one, but a mul­ti­tude which over­lap and inter­link one another in ways that are dif­fi­cult to explain. So on Face­book, I might have fam­ily and friends, but I also have for­mer peers in school and oth­ers who I am friendly toward, but who might not quite rate the rather overused and abused label of friend. It is this sim­plis­tic per­cep­tion of our inter­con­nect­ed­ness that I believe is also the root cause of the col­lapse of pri­vacy engen­dered by sites like Face­book. Because there is but one term, friend, for every­one, we are encour­aged to debase the term and apply it indis­crim­i­nately to many for whom it is a poor moniker. The per­son I am to my cowork­ers, my fam­ily and my friends — not to men­tion acquain­tances — is not exactly the same. The empha­sis is dif­fer­ent, and the will­ing­ness to share is cer­tainly very different.

Where then is this ram­bling post going? I’m not sure, but I believe we have deval­ued the idea of friend­ship in the Face­book era, and allowed our­selves to pro­mote many to a level they are not yet deserv­ing of, and by doing so deval­ued those who are. Some sites pro­vide some mod­icum of con­trol, or more likely the appear­ance of con­trol, over what gets shared with who, but the real­ity is that the imple­men­ta­tions are so slip­shod, vari­able and unin­tu­itive that their use­ful­ness is min­i­mal to most people.

And since no blog post is com­plete with­out a wit­ti­cism, I leave you with a quote from The Incred­i­bles:

Mr. Incred­i­ble: You mean you killed off real heroes so that you could pre­tend to be one?
Syn­drome: Oh, I’m real. Real enough to defeat you! And I did it with­out your pre­cious gifts, your oh-so-special pow­ers. I’ll give them hero­ics. I’ll give them the most spec­tac­u­lar hero­ics the world has ever seen! And when I’m old and I’ve had my fun, I’ll sell my inven­tions so that every­one can have pow­ers. Every­one can be super! And when everyone’s super–
[chuck­les evilly]
Syn­drome: –no one will be.

The mythical golden paycheck of the Fed

There is a per­ni­cious myth that is con­sis­tently spread by many peo­ple that the “aver­age” Fed­eral employee is bathing in the excesses of gov­ern­ment largess and is over­paid. While this is likely to be true in some cases, the typ­i­cal posi­tion is gen­er­ally under­paid ver­sus the area and respon­si­bil­i­ties. To show how bad the sit­u­a­tion really is, I’ve taken two sets of numbers:

  • Sched­ule 9 (Locality-based Com­pa­ra­bil­ity Pay­ments) of Exec­u­tive Order 13561, which cov­ers Fed­eral civil­ian and mil­i­tary employ­ees pay scales begin­ning on Jan­u­ary 1, 2011. (PDF)
  • Kiplinger’s top 100 cities, includ­ing cost-of-living information

From this, I derived the rel­a­tive pay scales and cost-of-living. This is not in absolute num­bers, but ref­er­enced against 0, which is “Aver­age”, and is in per­cent­age points above and below aver­age. You can click on this chart to see:

A chart comparing the cost-of-living of various cities with the federal locality pay

First, it’s impor­tant to note that as best as I can under­stand, if you are not in one of the areas that receive spe­cial con­sid­er­a­tion, you still receive an 14.16% uplift in your pay. No, I don’t know why. It’s likely to hide a lot of other things, and con­tin­ues the gen­er­ally dis­hon­est account­ing of all branches of the US Gov­ern­ment. I’ve adjusted out this num­ber. Some things stand out in glar­ing relief. First, peo­ple in Hous­ton are mas­sively over­paid, even if you account for the fact they have to live in Hous­ton. In fact, while Houston’s cost-of-living is approx­i­mately 11% below aver­age, Fed­eral work­ers there receive over 28% (15% effec­tive) increase in pay, which means that they make nearly 26% more than they should. Con­versely, New York City employ­ees also see a 28% adjust­ment (15% effec­tive), but their cost of liv­ing is 3-4x what it is on aver­age. How the local­ity adjust­ment for Hous­ton, TX and New York City, NY can be effec­tively the same tells you how com­pletely bass ack­wards this sys­tem is.

As for those DC employ­ees? They’re not as bad off as New York, but they’re hardly the over­paid Hous­ton (and Dal­las) employ­ees, and in fact the gap is still nearly 30% under the aver­age. Put that in your tea cup and smoke it.

If you want to see my “work”, you can down­load the Num­bers spread­sheet here.

IntelliJ IDEA 10 performance on Mac OS X

The degree to which I hate Eclipse can not be under­stated. For me, it’s a truly loath­some piece of “soft­ware”. I tried out Net­beans but I found it amaz­ingly unsta­ble and slow. So, since I had been using PyCharm with some hap­pi­ness, and admir­ing RubyMine, I decided to pop for the “full enchi­lada” and drop the $249 for Intel­liJ IDEA. For­tu­nately, the nice peo­ple at Jet­Brains let me just pay the dif­fer­ence, and that couldn’t have been eas­ier. I see no rea­son to not pay for some­thing that I believe demon­strates equal or bet­ter value.

A large whaleOne thing I’ve noticed, though, since work­ing with the full Intel­liJ IDEA 10 instal­la­tion is that it was get­ting slower and often the spin­ning beach ball of doom would appear with dis­con­cert­ing reg­u­lar­ity. Doing some dig­ging online, I dis­cov­ered some point­ers that seem to have helped resolve the issue. I did a cou­ple things:

  • Moved to the 64-bit Java 6 runtime
  • Upped the mem­ory avail­able and early-allocated to the JVM

To do this, all you have to do is edit the Info.plist that’s located in /Applications/IntelliJ IDEA 10/Contents and change a cou­ple prop­er­ties. To move to the 64-bit run­time, you sim­ply need to edit Java:JVMArchs to place the x86_64 archi­tec­ture ahead of the i386 archi­tec­ture. That’s easy enough.

The sec­ond thing is upping the mem­ory allo­ca­tion, which can be done in the same prop­er­ties file by edit­ing Java:VMOptions and mak­ing it look like this:

-Xms512m -Xmx1024m -XX:MaxPermSize=512m -ea -Xbootclasspath/a:../lib/boot.jar

Now, this is on a machine that has 12GB of RAM, so I’m not sure how it’s going to be on my note­book with “only” 4GB. What amazes me is that in the old days, Emacs was seen as a huge beast of a pro­gram and often peo­ple would dread start­ing it up because it would take a long time and con­sume huge amounts of resources. Now, a GUI Emacs instance starts in a sec­ond and con­sumes (on my machine) about 64MB (100MB vir­tual) of RAM. This is com­pa­ra­ble to what Terminal.app uses. A sad, sad world.

Once I changed those set­tings, I’ve found that the IDEA is a lot faster and hasn’t shown me the beach ball yet. Still, some­thing bugs me that it needs that kind of resources. Though, when I have to write Java code, it does make my life 1000x eas­ier than before.

Umberto Eco on Wikileaks

Umberto Eco has long been one of my most favored authors. I’ve even tried to delve into his more schol­arly work, but it escapes me to a large extent. Now, he tack­les the absur­dity of the Wik­ileaks fiasco in an arti­cle in Presseu­rop:

But let’s turn to the more pro­found sig­nif­i­cance of what has occurred. For­merly, back in the days of Orwell, every power could be con­ceived of as a Big Brother watch­ing over its sub­jects’ every move. The Orwellian prophecy came com­pletely true once the pow­ers that be could mon­i­tor every phone call made by the cit­i­zen, every hotel he stayed in, every toll road he took and so on and so forth. The cit­i­zen became the total vic­tim of the watch­ful eye of the state. But when it tran­spires, as it has now, that even the crypts of state secrets are not beyond the hacker’s grasp, the sur­veil­lance ceases to work only one-way and becomes cir­cu­lar. The state has its eye on every cit­i­zen, but every cit­i­zen, or at least every hacker – the cit­i­zens’ self-appointed avenger – can pry into the state’s every secret.

Just so.

I’m Sorry

A man's sad face "dressed" as a puppy.I’m sorry for back­ing Barack Obama. I’m sorry I voted for him in the pri­maries. I’m sorry I shov­eled money into his cam­paign. It appears that I was swept up in a cult of per­son­al­ity, and while he is no-doubt one of the smartest Pres­i­dents in his­tory, he’s also one of, if not the, most spineless.

What a sad, sad thing. So much oppor­tu­nity and energy squan­dered on a man unwill­ing to stand up for much of any­thing. Pol­i­tics is like life in the ani­mal king­dom; peo­ple smell fear, they smell weak­ness. Unfor­tu­nately, the only scent that Pres­i­dent Obama gives off if weak­ness and a patho­log­i­cal desire to cave in the name of “bipar­ti­san­ship”. Even when the Democ­rats — who live up to the adage that they could “snatch defeat from the jaws of victory” — controlled both houses, they acted like a sub­servient party.

It’s just depress­ing. With the elec­tion of Tea Party sup­port­ers we now have a “per­fect storm” of stu­pid­ity, men­dac­ity and spinelessness.

(:summary (first clojure-conj))

As I’m writ­ing this, it’s the first time I’ve had hotel-provided Inter­net access since I arrived on Thurs­day. The rea­son I men­tion this is that it is, by far, the only neg­a­tive expe­ri­ence that I’ve had at the first Clo­jure Conj. It has been the best pro­gram­ming con­fer­ence I’ve been to in a long time. The amount of brain bend­ing infor­ma­tion and “food for thought” is stag­ger­ing for some­thing with 1 track over two days for a lan­guage that’s only been around for 3 years.

I think it’s fair to say that at this con­fer­ence, I’ve been one of the dumb­est peo­ple around and that makes me very happy. That may sound coun­ter­in­tu­itive, but it pro­vides a mas­sive amount of stim­u­la­tion and learn­ing oppor­tu­nity — things that I will be think­ing about for months. It also gives me hope for the indus­try that there are so many bril­liant peo­ple out there who are work­ing to define a new future for the indus­try in gen­eral and Clo­jure specifically

There wasn’t a sin­gle dud of a pre­sen­ta­tion, or pre­sen­ter, and while one or two weren’t really in my area of inter­est, I still found much to learn from them. This is a level of qual­ity that is dif­fi­cult to achieve, and the team behind the con­fer­ence deserve enor­mous credit. What amazed me was the abil­ity of pre­sen­ters to dis­till their thoughts into some core zen-like under­stand­ing and present it in a digestible fash­ion to the audi­ence. I won’t even get into the amaz­ing abil­ity of Rich Hickey to explain the most com­plex ideas in ways that every­one seems to under­stand. In many ways, he rep­re­sents the CS pro­fes­sor we all wished we had. Finally, where many con­fer­ences have the best con­tent in the hall­way, this one had great con­tent every­where, and it seemed like no mat­ter who you walked past, they were involved in some­thing interesting.

The con­fer­ence ended with a back­yard BBQ at Stu­art Halloway’s house, who was the MC of the con­fer­ence, and gen­er­ally kept all the cats cor­ralled with the help of a lot of tal­ented peo­ple. It was a beau­ti­ful end­ing to a great con­fer­ence, and one that has invig­o­rated my desire to do some­thing inter­est­ing. There’s likely to be many posts that fall out of things that are inspired by the con­fer­ence, but not until I’ve had some time to lay in my hammock.

In sum­mary: I can’t wait for next year.

Redis and Python, Pt 2 — The Characters

One of the great things about Redis is the rich­ness of its data struc­ture. I once heard some­thing call it a “data struc­ture in the cloud”, and that’s not com­pletely inac­cu­rate. Before delv­ing deeper into using Redis with Python, I wanted to cover the core data types that are the foun­da­tion of this data-structured approach. If you’ve only used mem­cached, then this is going to be a bit new to you.

Read More »

Redis and Python, Pt 1 — Enter stage left

Redis is a key-value store if you want to strip the idea down. A key-value store maps a key, usu­ally a string, to some “blob” of data, and gives you the basic com­mands you need to manip­u­late them. There are a few things that I think make it inter­est­ing, however:

Rich data structures
Unlike some tra­di­tional key-value stores, like mem­cached, Redis pro­vides a col­lec­tion of data struc­tures to work with. These include strings (like mem­cached), lists, sets, sorted sets and hashes. This sets it apart from a lot of the com­peti­tors, and for me, rep­re­sents its “killer fea­ture”. Strings can even be inter­preted as an inte­ger for some purposes.
Lib­eral license
The license of Redis is quite sim­ply the BSD License. This is — to my mind — the sim­plest, clean­est and “most free” license avail­able. Pretty much, you sim­ply can’t claim the work is yours, and you have to acknowl­edge the orig­i­nal owner. It is non-viral.
Sim­ple
Redis is sim­ple in a cou­ple of dif­fer­ent ways. First, it’s triv­ially sim­ple to setup, as we’ll see. Next, it’s portable across just about any plat­form. The pro­to­col between client and server is a sim­ple text-based pro­to­col, and finally the imple­men­ta­tion is clean and single-threaded while achiev­ing great performance.
Fast
It’s fast. It’s pos­si­ble there are other stores that might be mar­gin­ally faster, but Redis achieves 50 – 100,000 operations/second on rel­a­tively cheap hard­ware, while using only a sin­gle CPU.

More impor­tantly, Sal­va­tore San­fil­ippo, the main devel­oper describes Redis not as the tra­di­tional key-value store, but as a “data struc­tures server”, and I think that’s a great place to start. So, before delv­ing into Redis and Python, I think it’s impor­tant to dis­cuss some concepts.

Read More »

Alles für den Staat

Today, in the New York Times, there was another arti­cle about how the US gov­ern­ment wishes to make it eas­ier to estab­lish the National Panop­ti­con. This fol­lows a pre­vi­ous arti­cle that con­tained the amaz­ing open­ing paragraph:

Fed­eral law enforce­ment and national secu­rity offi­cials are prepar­ing to seek sweep­ing new reg­u­la­tions for the Inter­net, argu­ing that their abil­ity to wire­tap crim­i­nal and ter­ror­ism sus­pects is “going dark” as peo­ple increas­ingly com­mu­ni­cate online instead of by telephone.

Antique telephone That’s right, those damned med­dling kids are using some­thing besides a tele­phone, and all that juicy sex­ting is being missed by our glo­ri­ous law-enforcement appa­ra­tus. If only they could hear every­thing you said, through every chan­nel, in every form, with no pos­si­ble option for pri­vacy, we’d all be safe.

This insan­ity reminds me entirely too much of the Min­is­terium für Staatßicher­heit, or Stasi that ter­ror­ized East Ger­many from 1950 to 1991. The level of whole­sale and omnipresent obser­va­tion of our lives does not com­pare to what the Stasi achieved, and I do not meant to com­pare the two directly. I sim­ply mean that the aspi­ra­tions seem dis­turbingly sim­i­lar. Where the Stasi would drill a hole, the Admin­is­tra­tion seeks to keep the keys to your encryp­tion. In some ways, a more ter­ri­fy­ing thing as it is impos­si­ble to know if your com­mu­ni­ca­tion is being decrypted, whereas you can actu­ally find the holes in the wall.

Some have said that “if you have noth­ing to hide, why do you care?”, but that is a dis­hon­est, disin­gen­u­ous and insult­ingly juve­nile response. It is not I who need jus­tify my pri­vacy. I may sim­ply be dis­cussing the lat­est color swatches for my bed­room, but that does not in any way mean the state has the slight­est right to mon­i­tor them. That con­cern does not even begin to delve into the cor­rup­tion and abuse of power that is rife in any sit­u­a­tion such as this. Peo­ple often cite Lord Acton’s admo­ni­tion that:

Power tends to cor­rupt, and absolute power cor­rupts absolutely.

While that obser­va­tion is true, and unshak­able, it is this one that I feel most worthy:

The fate of every democ­racy, of every gov­ern­ment based on the sov­er­eignty of the peo­ple, depends on the choices it makes between these oppo­site prin­ci­ples, absolute power on the one hand, and on the other the restraints of legal­ity and the author­ity of tradition.

Remem­ber that, when peo­ple ask that you for­sake your own free­dom of thought and con­scious for the “greater good”. Peo­ple who do so have for­got­ten what it means to be an Amer­i­can, and what free­dom might actu­ally feel like. There are always ways to dis­cover infor­ma­tion that do not require the whole­sale slaugh­ter of pri­vacy and free­doms, but they also require a mod­icum of respect for the ideas and a large mea­sure of intel­li­gence to reframe the prob­lem. It is only those hun­gry for power who decide what they want — total sur­veil­lance — and then scheme at any cost to jus­tify it.

They who can give up essen­tial lib­erty to obtain a lit­tle tem­po­rary safety, deserve nei­ther lib­erty nor safety.
– Ben­jamin Franklin

As you think on this prob­lem, ask your­self “would I want my enemy to have this capa­bil­ity?” Some­day, it will be your enemy, or even the enemy of the entire coun­try. No sys­tem of such power is impen­e­tra­ble. No sys­tem of such scope can not be sub­verted for even more nefar­i­ous uses. No sys­tem of such com­plex­ity and reach is with­out enor­mous flaws.