Kevin's Weblog

Overview of LJ’s evolution from 1 server to ~90 currently including load balancing and distributed caching (memcached). 4 million users, 50 million dynamic page views / day.

Evolution:

• 1 shared server
• 1 dedicated server
• one web, one DB
• multiple web, one DB
• replicated DB
• more webs, more DB slaves, chaos, I/O, CPU contention

Scaling problem: every DB has to write, so eventually you lose no matter how many slaves you have.

Solution: user clusters. All user data stored in one cluster. Never join from multiple users. One global master keeps user metadata (user-cluster map)

But... still lots of points of failure. Add master-master configuration, 2 machines per cluster. Challenges: avoid corruption, unique id collisions, backup has a cold cache.

Custom-built load balancer. Using persistent connections, which helps with burstiness, guarantees connections are good. Paid users get to cut in line.

Replaced RAID with MogileFS. Lost the thread here for a bit...

Caching is important. Would be nice if it could all be handled by the DB, but there are limitations: 32 bit, application logic. Where else to cache? Mod_perl — too much memory use, bad hit rate. Shared memory — tricky, only works on one machine.

memcached is their Open Source, distributed caching system. APIs for many languages. Various machines have different free mem, provide some number of buckets. Hash request, pick a bucket. written in C, event-drived, single-threaded using libevent. custom memory allocator.

10 hosts, 28 instances, 30GB cached, 93% hit rate.

This session ended up standing room only, with me as one of the standing, so I couldn’t take notes at the time. This will be more recollections and commentary

This talk turned out to be about something a little different than I expected from the abstract. In particular, his definition of “enterprise” seemed to include only internal applications that don’t directly make money for the company (but where you can definitely lose money if it doesn’t work). I would have tended to include things like we do: high-performance, high-availibility web services clearly coupled to revenue.

Consequently, I found a number of things I disagreed with. For example, I like metadata a lot and it makes my life a lot easier. However, he seemed to advocate against config files and the like, in favor of keeping all the constants, configuration, and logic in your code.

I did agree with some things. Write tests. Keep methods clear and simple. Use OO (although, I’m not sure I’d go quite as far as he did).

A big area that I expected to be touched on when talking about “enterprise” systems was deployment and monitoring, but that wasn’t touched on at all.

Not there yet, but getting closer. Most of the design, including OO, is complete.

Pipelining operators <== and ==> make map grep max ... a little easier to read and understand. They’re lazy too!

‘fat comma’ makes pairs. Useful for named subroutine parameters, but there is also a new syntax: format $text, :indent[8] :ignore{ /\t/ }

slurp replaces local $/ nonsense

Enhanced sorting removes need to Schwarzian Transform and similar manuevers.

Real classes:

class DogTag {
  has Str  $.name;
  has Int  $.serial;
  has Str  @.record is rw;
  has Base $:posting is rw;
  ...
}

Accessor methods are created automatically. Attributes are read-only unless marked read-write. Private variables marked with ‘:’. Lots of implicit method call stuff, which I’m a little nervous about, but maybe it really is easier to read.

There is a concept of submethods which override a method, but only for that exact subclass, not for sub-subclasses. Supposedly this is good for initialization, but I didn’t quite follow the argument.

Can add “roles” to classes, which are sort of like macros that insert some attributes directly into the class. Good for component-based design. Can create a role without implementations to create interfaces.