Back in 2012, we talked about our foundation for this at Indeed:

Blog: http://engineering.indeed.com/blog/2012/11/logrepo-enabling-...

Talk: http://engineering.indeed.com/talks/logrepo-enabling-data-dr...

tl; dr: a human-readable log format that uses a sortable UID and arbitrary types/fields, captured via a log4j syslog-ng adapter, and aggregated to a central server for manual access and processing

Jobs4Pets: http://www.indeed.com/jobs4pets

Curious to hear opinions from people who've done service load distribution with hardware or software load balancers...