How do I pre-process an infinite stream of text before writing it to a file?

Yes, use a daemon program that takes the stream as input, and does just what you described. I would recommend C instead of a script, as it has very straightforward input/output, and very low overhead.

Assuming you have an executable called 'capture' and a filtering program called 'filter', you can chain them together from a bash shell using

bash-prompt$ capture capture-params | filter

Anything that capture writes to stdout will be available to filter as input from stdin. It is a simple matter, from filter's point of view, of reading lines, and when the end ... size pattern is found, writing the output to an output file (or again to stdout). If you write to a stdout, you can redirect that to a file using

bash-prompt$ capture capture-params | filter > output-file.txt

You can get on the fly text processing with awk. You will need to learn the language but I use for similar tasks at live log parsing. I do tail -f file.log | awk -f myscript.awk

Each line will be analyzed through the awk script you create and with if-then-else, you can detect some words present in the line and activate other parts of awk code to parse the line differently or even run external programs.

By far, the most elegant application for what you are describing is using a low footprint round robin database. RRDtool is the OpenSource industry standard, high performance data logging and graphing.

Using a bash command you can input your data into the database, and should you choose to, graphing it is also very simple.

SEE:http://oss.oetiker.ch/rrdtool/gallery/index.en.html