-
Notifications
You must be signed in to change notification settings - Fork 34
Recording counters
istatd can record counters reflecting the performance of the host it is running on. However, that is not its only function, or even its main function.
istatd listens for incoming connections on TCP port 8111 (default -- change with --stat-port.) On those connections, individual lines (separated with CRLF -- \r\n) of statistics are accepted. A line has one of three formats:
-
counter.namevalue -
counter.nametimestampvalue -
counter.nametimestampsumsumSquaredminmaxcount
The value, sum, sumSquared, min and max are floating-point values. timestamp and count are integer values, where timestamp is in UNIX epoch UTC time. The "value" of a bucket of data for the last format is sum divided by count. The reason you may want to break these apart is that, with sum, sum squared, and count, you can calculate the standard deviation of all data points that contribute to a particular counter for a particular time.
Names are formulated roughly as C-style identifiers separated by periods. Each period defines a new "branch" in a tree of counters. Some additional characters are allowed in counter names, that are not allowed in C identifiers, such as dashes.
Names can also contain carets. Carets are the same as periods, except the next caret in a name will "rewind" the name to the position of the previous caret, and substitute the next substring. This will generate many counters from a single sample. For example, if the counter name is cpu.load^host.jupiter^role.database^everything then the data on that line will be merged into three counters: cpu.load.host.jupiter, cpu.load.role.database, and cpu.load.everything.
See the wiki entry for Counter Types: events or gauges. Hint: you have to get this right the first time!
Additionally, istatd accepts the same kind of data in UDP packets that arrive on the same port as TCP. Each UDP packet can contain multiple lines/counter values, but must end with a proper linefeed.
There is no security against forged packets, other than whatever firewalls you have to protect your machines in general. To aid in this configuration, you can tell istatd to only bind to a certain IP address (such as a 10.x internal address, or the 127.0.0.1 localhost address) using --listen-address.
At IMVU, we run one istatd process on each host, in "agent" mode, where it doesn't store data on disk, but instead just forwards data to a master database. Any process on a host can then fling any counter value to localhost:8111 in a UDP packet (or TCP stream) and be reasonably certain that the counter event will be taken care of by the system.