MongoDB server freeze - large amount of collections

Creation of new collections has no point. It is an engineering flaw. This amount of collections surely will freeze db server because it is forced to walk over list of available collections and check a presence of collection in the list. Also, I believe, the are some operations like metadata updates involved.

From the operations log it is getting clear that building an index takes significant time

"query" : {        "createIndexes" : "_cf71_df2016-07_dt2016-11_tof2_groupct",        "indexes" : [             {                "key" : {                    "expireAt" : 1                },                "name" : "exAt",                "background" : true,                "expireAfterSeconds" : 0            }        ]    }

Also, things like acquiring locks take time, which are necessary for creation new namespace and index building. Again, parameters like "expireAfterSeconds" : 0 are not wise in such scale. Basically, you will have to run 1.4M timers every 60 seconds to find and prune expired records and ensure that index will be rebuilt (see those deletes on mongostat).

Please, consider changes toward reducing number of collections or splitting collections between deployments. Also you can drop indexes for inactive collections. Also, it would be wise to drop all empty & unused collections.

For immediate solution you have to find current bottlenecks: RAM, CPU or IOpS.You have extra RAM, you can give it to MongoDB, it will help to cache more and avoid unnecessary reads. Measure your IOpS via iostat to see how busy is your drive.