Inconsistent index distribution across shards in a MongoDB sharded cluster is more common than you think, and it occurs because the user creates indexes directly in the shards without using MongoS.
This issue can lead to migration failures during chunk migrations, where the system cannot transfer data between shards due to index mismatches.
A typical error message found in the shard log when this happens might look like this:
1 2 3 4 5 | {"t":{"$date":"2024-05-23T14:27:49.111+00:00"},"s":"I", "c":"SHARDING", "id":5087102, "ctx":"migrateThread", "msg":"Recipient failed to copy oplog entries for retryable writes and transactions from donor","attr": {"namespace":"test.cincomillones","migrationSessionId":"Shard_1_Shard_2_646884t4835b68a1af8781","fromShard": "Shard_1","error":"migrate failed: CannotCreateCollection: aborting, shard is missing 1 indexes and collection is not empty. Non-trivial index creation should be scheduled manually"}} |
The error indicates that the cluster can’t migrate chunks due to missing indexes on one or more shards. To resolve this, it’s necessary to ensure index consistency across all shards.
Steps to check and resolve this issue:
The first step is to connect to the PRIMARY node of each shard and inspect the indexes for the problematic collection. In this example, as we saw in the log file, the collection test.cincomillones has an index inconsistency.
It’s important to check the primary shard as a reference to see on which shard the indexes are missing.
Shard 1:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | shard1:PRIMARY> db.cincomillones.getIndexes() [ { "v": 2, "key": { "_id": 1 }, "name": "_id_" }, { "v": 2, "key": { "BetID": "hashed" }, "name": "BetID_hashed" }, { "v": 2, "key": { "MasterEventId": 1 }, "name": "MasterEventId_1" } ] |
Shard 2:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | shard2:PRIMARY> db.cincomillones.getIndexes() [ { "v": 2, "key": { "_id": 1 }, "name": "_id_" }, { "v": 2, "key": { "BetID": "hashed" }, "name": "BetID_hashed" } ] |
As shown, Shard 1 has an extra index (MasterEventId_1) compared to Shard 2. This index mismatch is causing the chunk migration failure.
To resolve this, we need to create the missing index on the shard that lacks it. In this case, we will add the MasterEventId index to Shard 2:
1 | shard2:PRIMARY> db.cincomillones.createIndex({ "MasterEventId": 1 }) |
After creating the missing index, it’s a good practice to verify that all shards have consistent indexes.
We can run the getIndexes again on all shards, or you can use the MongoDB built-in command shardedIndexConsistency to check this.
You need to run the following command on the PRIMARY configServer to check for index inconsistencies across shards:
1 | config:PRIMARY> db.serverStatus().shardedIndexConsistency |
This command will report any further inconsistencies, allowing you to ensure that all shards are in sync.
After the creation of the missing index, we will get none sharded collections with inconsistent indexes:
1 2 3 | configRepl:PRIMARY> db.serverStatus().shardedIndexConsistency { "numShardedCollectionsWithInconsistentIndexes" : NumberLong(0) } |
If you are running MongoDB 5.0, you can use an aggregation pipeline to find inconsistent indexes across shards as described here:
https://www.mongodb.com/docs/v5.0/tutorial/manage-indexes/#find-inconsistent-indexes-across-shards
Starting on MongoDB 7.0, you can use a new feature to check the inconsistent index:
https://www.mongodb.com/docs/manual/reference/inconsistency-type/InconsistentIndex/
This method returns a cursor with a batch of documents showing the inconsistencies found in the sharding metadata, i.e.:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | { cursor: { id: Long("0"), ns: "test.$cmd.aggregate", firstBatch: [ { type: "InconsistentIndex", description: "Found an index of a sharded collection that is inconsistent between different shards", details: { namespace: "test.authors", info: { missingFromShards: [ "shard-rs1" ], inconsistentProperties: [ ], indexName: "index1" } } } ], }, ok: 1 } |
Ensuring index consistency across shards is critical for the smooth operation of a MongoDB sharded cluster. Inconsistent indexes can block chunk migrations, leading to potential downtime or performance degradation.
By checking each shard’s indexes, fixing mismatches, and verifying consistency with the methods described, you can keep your cluster running smoothly.
For more details on index consistency checks, refer to MongoDB’s official documentation.:
https://www.mongodb.com/docs/v5.0/tutorial/manage-indexes/#find-inconsistent-indexes-across-shards
https://www.mongodb.com/docs/manual/reference/inconsistency-type/InconsistentIndex/
MongoDB Performance Tuning is a collection of insights, strategies, and best practices from Percona’s MongoDB experts. Use it to diagnose — and correct — the issues that may be affecting your database’s performance.
Download MongoDB Performance Tuning Guide
About the Author
Zelmar MicheliniZelmar, a MongoDB specialist, joined Percona in 2022 as a Support Engineer. With a sharp focus on performance optimization and a knack for troubleshooting, he's recognized for his expertise in MongoDB intricacies, delivering top-notch support.
