Hacker News new | past | comments | ask | show | jobs | submit login

Yes, and it works pretty solidly. We used to use syslog-ng, writing to named pipes, but you have to be sure that there is something reading from the pipe before you start writing to it, otherwise it blocks. You don't really want to use the syslog protocol (via the syslog(3) library call) for random logging because you may end up hitting the upper bounds of the log lines. I wanted to use rsyslog, mainly for the local buffering, but it ONLY seems to support the syslog format/protocol, including prefixing all lines with a date and time and a hostname.

We use scribe and we have a stdin2scribe program (python) that can be used to hook into any log output (like apache access and error logs). We have it set up in a two tier system, all systems that we'd want to log from run a "scribe leaf" on a port on localhost, and this forwards all logs to a "scribe aggregator" (behind a load balancer), with a buffering space on the local disk when the aggregator can not be contacted. It's a pretty solid system and I recommend it.

We also have services, and command line and library interfaces to those services, that let you grab all the logs that came in on a certain time frame, or tail all the data coming into the aggregators in real time (one of them is a wrapper around a more generic tool that just tags the logs, the wrapper takes grep-style filtering arguments and the output is pipped to a pretty printer).




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: