Is it OK to query a MongoDB multiple times per request?
To answer the Q about $in....
I did some performance tests with the following scenario:
~24 million docs in a collection
Lookup 1 million of those documents based on a key (indexed)
Using CSharp driver from .NET
Results:
Querying 1 at a time, single threaded : 109s
Querying 1 at a time, multi threaded : 48s
Querying 100K at a time using $in, single threaded=20s
Querying 100K at a time using $in, multi threaded=9s
So noticeably better performance using a large $in (restricted to max query size).
Update:Following on from comments below about how $in performs with different chunk sizes (queries multi-threaded):
Querying 10 at a time (100000 batches) = 8.8s
Querying 100 at a time (10000 batches) = 4.32s
Querying 1000 at a time (1000 batches) = 4.31s
Querying 10000 at a time (100 batches) = 8.4s
Querying 100000 at a time (10 batches) = 9s (per original results above)
So there does look to be a sweet-spot for how many values to batch up in to an $in clause vs. the number of round trips