key-value store for time series data? key-value store for time series data? database database

key-value store for time series data?


It sounds like MongoDB would be a very good fit. Updates and inserts are super fast, so you might want to create a document for every event, such as:

{   object: XYZ,   ts : new Date()}

Then you can index the ts field and queries will also be fast. (By the way, you can create multiple indexes on a single database.)

How to do your three queries:

retrieve all the data for object XYZ between time t1 and time t2

db.data.find({object : XYZ, ts : {$gt : t1, $lt : t2}})

do the above, but return one date point per day (first, last, closed to time t...)

// firstdb.data.find({object : XYZ, ts : {$gt : new Date(/* start of day */)}}).sort({ts : 1}).limit(1)// lastdb.data.find({object : XYZ, ts : {$lt : new Date(/* end of day */)}}).sort({ts : -1}).limit(1)

For closest to some time, you'd probably need a custom JavaScript function, but it's doable.

retrieve all data for all objects for a particular timestamp

db.data.find({ts : timestamp})

Feel free to ask on the user list if you have any questions, someone else might be able to think of an easier way of getting closest-to-a-time events.


This is why databases specific to time series data exist - relational databases simply aren't fast enough for large time series.

I've used Fame quite a lot at investment banks. It's very fast but I imagine very expensive. However if your application requires the speed it might be worth looking it.


There is an open source timeseries database under active development (.NET only for now) that I wrote. It can store massive amounts (terrabytes) of uniform data in a "binary flat file" fashion. All usage is stream-oriented (forward or reverse). We actively use it for the stock ticks storage and analysis at our company.

I am not sure this will be exactly what you need, but it will allow you to get the first two points - get values from t1 to t2 for any series (one series per file) or just take one data point.

https://code.google.com/p/timeseriesdb/

// Create a new file for MyStruct data.// Use BinCompressedFile<,> for compressed storage of deltasusing (var file = new BinSeriesFile<UtcDateTime, MyStruct>("data.bts")){   file.UniqueIndexes = true; // enforces index uniqueness   file.InitializeNewFile(); // create file and write header   file.AppendData(data); // append data (stream of ArraySegment<>)}// Read needed data.using (var file = (IEnumerableFeed<UtcDateTime, MyStrut>) BinaryFile.Open("data.bts", false)){    // Enumerate one item at a time maxitum 10 items starting at 2011-1-1    // (can also get one segment at a time with StreamSegments)    foreach (var val in file.Stream(new UtcDateTime(2011,1,1), maxItemCount = 10)        Console.WriteLine(val);}