mongodb StatefulSet on kubernetes is not working anymore after kubernetes update
I have workaround this issue:
- adding a MongoDB container to the cluster to dump and restore the MongoDB data
- dumping the current database
- deleting the MongoDB instance
- recreating a new MongoDB instance
- restoring the data to the new instance
yes unfortunately this comes with a downtime
Looking for an answer drawing from credible and/or official sources.
One official source would be "Running MongoDB on Kubernetes with StatefulSets" (from 2017, so a few adaptation/evolution might be needed), but you seem to have follow it.
Your error message was seen 2 months ago in mongodb.org SERVER 37724
In 4.0, we did make a change to the journaling process in which it follows the oplog instead of the data files themselves. That has a chance to be what is happening here.
To test that, try and use a MongoDB 3.6, to see if the issue persists.