Kevin's Weblog

We’re pretty obsessed with performance at Gilt Groupe. You can get a taste for what we’re dealing with, and how we’re dealing with it, from our recent presentation at RailsConf.

One of the techniques we’re using is to precompute what certain high-volume pages will look like at a given time in the future, and store the result as static HTML that we serve to the actual users at that time. For ease of initial development, and because there’s still a fair bit of business logic involved in determining which version of a particular page to serve, this was done inside our normal controller actions which look for a static file to serve, before falling back to generating it dynamically.

We’re now running on Rails 2.3 and, of course, Rails Metal is the new hotness in 2.3. I spent the last couple days looking into how much improvement in static file serving we would see by moving it into the Metal layer. Based on most of what I’ve read, I expected we might shave off a couple milliseconds. This expectation turned out to be dramatically wrong.

Metal components operate outside the realm of the usual Rails timing and logging components, so you don’t get any internal measurements of page performance. Instead, I fired up ab to measure the serving times externally. What I found for the page I was benchmarking was that the Metal implementation took about 5ms. The old controller action took 170ms. But, wait... the Rails logs were only reporting 8ms for that action. Something was fishy.

I started inserting timers at various places in the Rails stack, trying to figure out where the other 160ms was going. A little bit was routing logic and other miscellaneous overhead, but even setting a timer around the very entry points into the Rails request serving path, I was only seeing 15ms being spent. This was getting really puzzling, because at this point where a Rack response is returned to the web server, I expected things to look identical between Metal and ActionController. However, looking more closely at the response objects I discovered the critical difference. The Metal response returns an [String], while the controller returned an ActionController::Response.

I went into the Rails source and found the each method for ActionController::Response. Here it is:

   def each(&callback)
     if @body.respond_to?(:call)
       @writer = lambda { |x| callback.call(x) }
       @body.call(self, self)
     elsif @body.is_a?(String)
       @body.each_line(&callback)
     else
       @body.each(&callback)
     end

     @writer = callback
     @block.call(self) if @block
   end

The critical line is the case where the body is a String. The code iterates over each line in the response. Each line is written individually to the network socket. In the case of the particular page I was looking at, that was 1300 writes. Ouch.

To confirm this was the problem, I changed that line to

      yield @body

With the whole body being sent in a single write, ab reported 15ms per request, right in line with what I measured inside Rails.

1 line changed. 150ms gained. Not too bad.

This sort of performance pessimization we uncovered is particularly insidious because it’s completely invisible to all the usual Rails monitoring tools. It doesn’t show up in your logged response time; you won’t see it in NewRelic or TuneUp. The only way you’re going to find out about it is by running an external benchmarking tool. Of course, this is always a good idea, but it’s easy to forget to do it, because the tools that work inside the Rails ecosystem are so nice. But the lesson here is, if you’re working on performance optimizations, make sure to always get a second opinion.

I took a bit of a blogging hiatus. Not particularly that I ran out of things to say, but rather there were technical difficulties. The short version, because I’m sure nobody cares, is this: last fall I retired my last non-Intel Mac. My blogging workflow involves Emacs PSGML mode, and makes use of a couple obscure corners of that package. Somehow on Intel Macs, one of those corners was busted. Don’t ask me why. I vaguely intended to try to debug the problem, but never got around to it. Last week, I decided I had something valuable enough to say that I would — the horrors — undertake to write valid XHTML without editor support. And, low and behold, everything worked perfectly again. I haven’t updated Emacs. I’m speculating something in some software update fixed something that was the root cause of this rather obscure problem. But, yeah, I really don’t know. But, I’m back!

Yesterday, DHH made an argument that alpha male programmers aren’t keeping women out of the tech field. I’m of the opinion that he’s wrong and that his argument is flawed, and in a moment I’ll explain why, but let me get a few things out of the way so they don’t distract from the rest of my argument.

First, I don’t think that “alpha males” or the “rockstar” mentality are the only causes of under-representation of women in technology. As far as I can tell, the causes are many, varied, generally difficult to deal with, and in one or two cases may even be valid reasons why we might ultimately accept some degree of imbalance in the field. Second, this is not strictly a gender issue. Some men are also driven away by this behavior, although I think women are more likely to be; and also some “alpha males” happen to be female. Third, this is not a personal attack on DHH or any other individual, although some people might read parts of it in that way. But my goal here is that the range of individuals who find themselves uncomfortably reflected in what I say, but don’t simply reject it all out of hand, might view that discomfort as an opportunity for personal growth. Finally, I am certainly not claiming that I am perfect in this regard. I’ve made mistakes in the past; I will make them again. I simply hope that my friends will be kind enough to point out my mistakes to me, so that I can try to do better.

Okay, now that that’s all out of the way...

I first claim that DHH is wrong. My proof is empirical and anecdotal, but fortunately for me, I’m on the side of this debate that gets to use those techniques. I.e., I’m asserting existence of a phenomenon, rather than non-existence. I know numerous women who have been driven away by alpha male behavior. In some cases, they simply moved to a different team, or a different company. In other cases, they switched to non-programmer roles. And some left the industry entirely. I know this because they told me. They described specific individuals and specific behaviors which drove them away.

With some frequency in these debates, male programmers will claim that they don’t know any women who have left the field for this reason (or who have experienced sexism in the field, or who were offended by the example under discussion, or even just the milder claim that DHH makes, that no one has any idea what to do). I can explain this in only one of two ways: either they don’t know any women in the field to begin with or don’t talk with them beyond the minimal professional requirements, or women are not telling them because they are part of the problem. Perhaps it would be more effective if these women directly confronted the people causing the problem, but the fact of the matter is that most people, men and women, dislike conflict. We’re much more comfortable griping to a sympathetic friend than to the cause of our unhappiness. So consider me something akin to an anonymizing proxy. Without revealing names or too many of the specifics, please trust me when I say that almost every woman in the field experiences and complains about this.

Now I also said that DHH’s argument is flawed, and I will spend the rest of this post pointing out the various flaws I see.

DHH claims alpha males cannot be a problem in programming because the average male programmer is “meek, tame, and introverted” compared to other fields. First off, “alpha males” are by definition not average; they are the most dominant individuals of a group. And, it may even be possible that the general meekness or introversion of programmers makes it easier for a small set of individuals to dominate the interaction, rather than reaching a condition of detente between a group of uniformly assertive individuals. Second, presumably DHH does not interact with these people from other fields in a professional context. A point repeatedly stressed in this recent “pr0n star” controversy is that it’s not an issue of people being anti-sex or anti-porn; it’s about what’s appropriate in the workplace, or in a professional context. Standards for behavior in a social context are different. Third, he speaks in terms of whether these other men are more or less “R-rated”. This is not the point. Women are just as “R-rated” as men. They curse. They talk about sex (often more explicitly than men). The issue is not about whether we say or do adult things, it’s about whether we respect each other as human beings and whether we understand the societal norms of what is and is not appropriate in particular contexts. In fact, in this regard, I’ll defend DHH. Saying “fuck” (in the exclamatory, non-sexual usage) in the course of a technical presentation is not problematic in this day and age within the technology community. I think most of us swear freely in the course of struggling with a nasty bug or production problem. This is a normative expression of frustration within our community, and it does not oppress or disrespect other members of the community. (As far as I know. It’s possible that people just aren’t telling me that it upsets them when I curse at my monitor. If that’s the case, I hope someone will tell me.) Finally, DHH observes that these other fields have a more even mix of men and women. What he misses is that when the distribution is relatively equal it is generally easier and more comfortable for men to be men and women to be women. It is perhaps counterintuitive, but environments which are heavily skewed call for greater sensitivity to gender or other cultural differences simply because it is so easy to unintentionally create an oppressive or exclusionary atmosphere.

In the final paragraphs of his post, DHH suggests that somehow by respecting women we are squashing some other sort of “edge” and diversity in the community. I’m a little puzzled by what he means by this, and I’m sort of afraid that he thinks that being a heterosexual male who likes to look at scantily-clad women (or who openly admits as much) is somehow “edgy”. It’s not. By definition, hetero males like women; and it’s well established that men tend to be visually oriented. Pointing out that you fall in this category does not make you “diverse”, it makes you a completely typical representative of your gender and orientation. No one needs to be reminded of it.

Moreover, it might be true that maximal gains are had by pushing the edges (although I don’t think that one should naively assume that analogy from physics or economics applies to social endeavors), but for this to be work there has to be negative feedback when boundaries are crossed. If the edge-walkers want society to accept their behavior, they must be prepared to apologize, to make reparations, and to correct their course when they go over the line. This is the difference between a trend-setter and a sociopath.

There’s quite a bit more that I could say on this issue, but I fear this may be becoming too long already, and I think it’s probably best to focus only on the arguments presented in this particular post at the moment. To summarize things in a couple of sentences, the phenomenon of women being discouraged by alpha male behavior is real. You merely need to talk, and listen, to women in the field to verify this. (But you might have to earn some trust first.) Comparisons with men in other fields in non-professional settings do not have much relevance to the matter at hand. Claims that respecting the feelings and experiences of a minority group is damaging to the community overall are extraordinary and require extraordinary support. Being a thought leader and being offensive are two very different things.

It’s really quite discouraging that so much of this discussion still seems mired in the question of whether a problem even exists, or whether it is desirable and possible to address the problem. This lack of acceptance leads both to the explicit refusals to acknowledge the validity of the complaints of the offended, as well as the phenomena of false apologies and insincere claims that “I would help if only I knew how (and if it doesn’t require any actual change of behavior on my part)”. Male programmers need to pull their heads out of the sand. The evidence, both hard statistical data and anecdotal, is overwhelming. It also is not hard to find advice about what can be done differently. The hard part is moving from a vague desire for diversity and balance to serious, meaningful, sometimes painful self-examination and commitment to change and improvement. It’s not easy to admit flaws in yourself, to acknowledge when you’ve hurt another person, or to make a true apology. Change doesn’t happen overnight or simply because you say that you want it to. It takes work, but it’s an important part of being a human being and being a member of a community.

Recently we upgraded to Rails 2.3 at work. Upon doing so, we saw our application take an aggregate performance hit of about 50%. Spending a little quality time with ruby-prof, one thing that stood out as peculiar was an awful lot of time being spent creating and tending to Ruby Thread objects. Tracing up the call stack, these were coming from inside the memcached client.

Buried in the Rails 2.3 release notes is the innocuous looking statement:

The bundled memcached client has been updated to version 1.6.4.99.

What this fails to tell you is that this upgrade added a timeout option to the client which, as the RDoc will happily inform you, “is a major performance penalty in Ruby 1.8”. Moreover, the default value for the timeout is 0.5s, rather than nil.

Our application makes heavy use of caching, so we were getting massacred by this. To add insult to injury, our most heavily optimized pages were hit the hardest, since they used memcached the most. Adding :timeout => nil to our cache layer immediately restored the previous performance.

Lessons from this story: if you’re a Rails developer using memcached, go add that option to your code; if you’re a library author adding an option with significant performance costs, default to the old behavior; if you’re on the Ruby core team, please add an interface to select so that people don’t do silly things like use Threads to implement timeouts on socket reads.

I’m a huge fan of Norman’s previous book The Design of Everyday Things and I highly recommend it to anyone who builds things, physical or virtual. This book takes on some of the aspects of design beyond the functional or behavior, and discusses the other aspects of why people are attracted to certain objects. It’s quite interesting for the first 2/3s of the book or so, but then takes a strange hard left into a discussion of robots and why robots will need to have emotions too. Very strange. But, I would still recommend reading the earlier parts of the book.

It’s been a year and a half since my last political post. I’m a little surprised that it’s been quite that long, but not entirely shocked. I spent nearly all of this election cycle somewhat withdrawn from the whole thing; feeling disenfranchised from years of being told that my dissent was unpatriotic, and being disgusted by the amount of ugly politics-as-usual that was going on. I wasn’t spellbound by Obama like so many of my friends and neighbors, but slowly gained respect for his thoughfulness, general positivity, sense of context, and willingness to confront real, serious issues that most politicians shirk from.

So, it wasn’t until yesterday that I really started to feel a nervous excitement that maybe we really would see a change this year. By 5PM, I was compulsively reloading web pages, and when the counts hit 200 electoral votes with only the east coast reporting I got that “OMG it’s really happening” feeling. Around 7, I headed down to the Green Dragon with a friend to watch on the TV there. I think I started vibrating at some point shortly before they called it.

The whole room was silent for Obama’s speech. I’m pretty sure most of us cried. I think over the last couple years I’d resigned myself to only being able to have an effect on my local community. His speech inspired me to again turn my efforts to a larger sphere. My community will still be my primary, day-to-day focus; but I’ll definitely be more engaged at the national level as well. What Obama said is true: this election doesn’t change anything by itself, it just gives us the opportunity to change. This is a difficult time for our country, and it’s going to take hard work by many people to solve these problems. But there’s a huge population of people like me that had all but given up and are now re-energized, and I think we can really make some improvements.

Closing thoughts... To Obama: live your rhetoric and remember that you now represent all Americans, not just your supporters. To McCain: you won back some of my respect with your concession speech; please let this be a return to your old, pre-election self. To supporters of both: be as gracious in victory and defeat as your candidates were last night. Talk and listen to people you don’t agree with, and realize that your goals are largely the same, even when your methods differ. Find the places where you can work together.

Stay hopeful.

It shouldn’t need to be pointed out, but the root of ‘useful’ is ‘use’. If you don’t actually use an object, regardless of whether you abstractly feel it to be useful, perhaps it isn’t. There is a particular risk with objects that are both beautiful and useful that we fail to use them, whether because we feel they are too expensive or unique and irreplaceable to risk damage, or because we put them on display as works of art but then fail to reach for them when the need arises, or whatever excuse there may be.

Recently there are a handful of objects I’ve been treating this way: my pocket knives and my fountain pen. Yesterday, I actually did take the intentional action of putting a knife in my pocket, along with my wallet and keys, before heading out the door. Today I did some paperwork with my fountain pen. Hopefully this marks a return to these habits.

It’s now official that OSCON is leaving Portland and moving to San Jose next year. There’s a lot of unhappiness around here about it and a definite feeling that the stated reason (rising travel costs) isn’t the whole story.

Personally, I don’t think it’s clear that making the conference local for more people is a good thing. I’ve noted two years running that I felt like it was a very different experience to be living in Portland and attending the conference — you’re much less immersed in the conference. Increasing the number of people who are there for the sessions but don’t stick around for the evening social activities won’t necessarily improve the conference.

ZDNet thinks “Portland just doesn’t have the facilities to house a fast-growing conference covering a global audience”, but I’d like to hear more justification for that claim. OSCON doesn’t come close to filling the whole Oregon Convention Center.

Personally, this feels like the usual Silicon Valley bubble mentality, combined with a touch of corporate greed. I also have a fear, possibly unjustified, that this may also trigger a slip backwards on the “women in tech” front. My impression is that Portland has one of the best environments for trying to increase the participation of women in technology and open source, and I also have the impression that the Valley has one of the worst.

I’m trying really hard to look at this beyond the personal impact on me (because it certainly did just get less likely that I’ll be able to attend next year), and I’m hearing a lot of the same unhappiness from people outside Portland as well. I guess what I’d like is a bit more openness from O’Reilly about this. Because, really, openness is what OSCON is all about, right?

Have nothing in your house that you do not know to be useful, or believe to be beautiful.
William Morris

I’ve been reading Emotional Design, and the inclusion of this quote inspired me to try to take an inventory of my house using this criterion. Of course, ideally I’d have only objects that are both beautiful and useful, but initially I’ll just make it my goal to be rid of things that fail both and make some notes for future improvements.

This morning I tackled the easy rooms: the living room and dining room. No surprises there. Most things are both beautiful and useful, except for my futon and folding chairs which are only useful, and the plants and artwork which are only beautiful.

Now I wonder if I should make the kitchen or the office the next room. Either way, it’s going to be a much bigger task.

I’ve been tinkering a little more with the Beust Challenge, following up on my previous post. There are a couple significant new developments to report.

First, Hynek (Pichi) Vychodil wrote a faster version using a permutation generator.

Second, I also ported the first “CrazyBob” solution using a bit mask to Erlang.

Third, I discovered that my overly literal ports were actually slowing things down. The CrazyBob code uses an unspecified Listener class that receives the numbers in the series, and presumably computes the actual results from there. (Aside, I cannot actually benchmark against the Java code because I haven’t found what this implementation is.) I fudged this in my original ports by simply spawning a process, which then discarded all the messages it received. After I noticed that the code was pegging both of my CPUs, though, I realized that message passing might actually be the bottleneck in my code. Turns out this was the case, and removing the listener process and just computing the results in the main process actually sped things up substantially.

Finally, I got access to a machine with HiPE enabled.

So... here’s the results. First on my MacBook Pro, without HiPE:

The bitmask solution starts out the fastest, but loses out in the end to the more clever solutions. pichi edges out the crazybob solution by about a third.

Now on Linux 2.6 with HiPE:

And our new winner is... bitmask! HiPE barely helps the original brute-force solutions at all, while crazybob and pichi gain about a factor of two. bitmask, on the other hand, picks up an order of magnitude, and is now only a factor of 3 slower than the posted results for the Java crazybob solution (with unknown differences in hardware).

Conclusion: Erlang doesn’t suck!