Has anyone published a detailed comparison between different in-memory RDBMSs? [closed]
The information is scattered all over the web, but here's what I found out:
Introduction to database benchmarking
The first thing that you need to do is figure out a standard benchmark to use for comparing your in-memory database choices. A wiki page on Oracle gives an overview of the industry standard database benchmarks. If you can find results for the same industry standard benchmarks for TimesTen and MySql Cluster, you'll be able to compare the two. Finding a detailed comparison of all the in-memory databases is quite rare.
Summary: Since you mentioned OLTP, the industry standard benchmark that you're probably interested in is TPC-E which simulates the OLTP workload of a brokerage firm. Other industry standard benchmarks measure decision support and e-commerce database transactions.
Open source database benchmark kits
The Open Source Development Labs Database Test Suite is the most comprehensive open source implementation of those industry standard benchmarks I mentioned earlier. They have four main suite of tests numbered DBT1, DBT2, DBT3, DBT4. They haven't implemented the TPC-E benchmark, but the Database Test 2 (DBT2) is what you are looking for since it simulates OLTP transactions. Everyone simply refers to running the DBT2 benchmark instead of spelling it out as OSDLB DBT2.
Results
MySql
You'll need to register to get the white paper emailed to you, but here is the summary that I took from their site:
Ultimately, MySQL Cluster was able to reach 100,000 transactions per minute in an 8-node configuration. This showed impressive scaling improvements when compared to a 2-node cluster which was able to obtain 26,000 transactions per minute. These performance improvements were realized through the use of new Multi Core Intel Xeon servers based on the Intel Core T micro architecture.
Oracle
Unfortunately the DBT2 does not support TimesTen. However I found a whitepaper that gives detailed benchmarking information for TimesTen in an OLTP workload.
IBM
Other Resources
- In Memory DBMS section of a blog that focuses on databases.
- Transaction Processing Performance Council website - You can search their database for benchmarks from various hardware vendors using various DBMSes. I was unable to find any in memory DBMS benchmarks however.
VoltDB is an in-memory highly-scalable OLTP database that provides SQL/ACID and runs on commodity hardware in a shared-nothing cluster. It is open-source and exists in both a free/community (AGPLv3) version as well as a commercially supported version.
There is a post in the VoltDB forums discussing some of the other in-memory OLTP RDBMS solutions at http://community.voltdb.com/node/95. Feel free to review and add to that posting.
Sebastian Czechowski at BigDataMatters has a comparison of four different IMDBs, suggesting selects tend to be 5X faster, and inserts/deletes tend to be 2x-4x faster.
http://bigdatamatters.com/bigdatamatters/2009/12/oracle_ibm_database_comparison.html