Mongodb - sharding - adding multiple shards at the same time

When you add shards, all sharded collections need to rebalance their data, which means moving chunks around until all the shards have roughly the same number of chunks. In the past (prior to 3.4) there was no parallelism when it came to chunk migrations, however now the number of parallel migrations that can happen on a cluster is directly related to the number of shards in the cluster. From the 3.4 patch notes:

for a sharded cluster with n shards, MongoDB can perform at most n/2 (rounded down) simultaneous chunk migrations

Right now that means you can do 2 simultaneous migrations, if you increase to 5 total shards you don't get any extra capacity. However, 6 shards gets you 3 simultaneous migrations and your final total of 8 gets you 4 simultaneous migrations.

In addition to having more migrations in flight at a time, adding more shards at once decreases the total number of migrations that have to take place (if you add one, rebalance; add another, rebalance etc. you will end up with more total migrations than just adding 4 shards and doing a single rebalance).

To illustrate, consider a few simple scenarios using a theoretical collection with 400 chunks and how many moves it takes to reach equilibrium each time:

Scenario 1 - Add a single shard at a time

1st addition: # of migrations = 802nd addition: # of migrations = 66 (22)3rd addition: # of migrations = 55 (28)4th addition: # of migrations = 50 (13)Total migrations = 251

Scenario 2 - Add 2 shards at a time

1st addition: # of migrations = 66 (22)2nd addition: # of migrations = 50 (13)Total migrations = 116

Scenario 3 - Add 4 shards immediately

Total migrations = 50 (13)

The figures in parentheses represent how many sequential migration operations have to happen if we assume maximum parallelization and assume all migrations take the same time. I think based on the above, in terms of migration throughput and efficiency, adding 4 immediately is the best course of action.

The downside, however, is the impact to your cluster. As you mention, migrations aren't free (though they are more efficient than they used to be, especially with the new WiredTiger defaults) and the more you have in flight, the more impact they will have. Therefore, as with most things, it's a trade off.

We also haven't even factored in what collections are largest, busiest, or have most chunks. There are many strategies you could adopt to minimize impact to performance for your users (you could disable balancing for your most important, busiest collections, do them last, but have the rest re-balance immediately, for example).

Hence, I can't give you a definitive answer, but if you consider all of the above, you should be able to come to the right decision for your data and your cluster.