我们有一个mongoDb集群，有3个碎片，每个碎片是一个包含3个节点的副本集，我们使用的mongoDb版本是3.2.6。我们有一个大数据库，大小约为230 G，包含约5500个集合。我们发现约2300个集合不平衡，其他3200个集合均匀分布在3个碎片上。
下面是sh.status的结果（整个结果太大了，我只贴了一部分）：
mongos> sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("57557345fa5a196a00b7c77a")
}
shards:
{ "_id" : "shard1", "host" : "shard1/10.25.8.151:27018,10.25.8.159:27018" }
{ "_id" : "shard2", "host" : "shard2/10.25.2.6:27018,10.25.8.178:27018" }
{ "_id" : "shard3", "host" : "shard3/10.25.2.19:27018,10.47.102.176:27018" }
active mongoses:
"3.2.6" : 1
balancer:
Currently enabled: yes
Currently running: yes
Balancer lock taken at Sat Sep 03 2016 09:58:58 GMT+0800 (CST) by iZ23vbzyrjiZ:27017:1467949335:-2109714153:Balancer
Collections with active migrations:
bdtt.normal_20131017 started at Sun Sep 18 2016 17:03:11 GMT+0800 (CST)
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
1490 : Failed with error 'aborted', from shard2 to shard3
1490 : Failed with error 'aborted', from shard2 to shard1
14 : Failed with error 'data transfer error', from shard2 to shard1
databases:
{ "_id" : "bdtt", "primary" : "shard2", "partitioned" : true }
bdtt.normal_20160908
shard key: { "_id" : "hashed" }
unique: false
balancing: true
chunks:
shard2 142
too many chunks to print, use verbose if you want to force print
bdtt.normal_20160909
shard key: { "_id" : "hashed" }
unique: false
balancing: true
chunks:
shard1 36
shard2 42
shard3 46
too many chunks to print, use verbose if you want to force print
bdtt.normal_20160910
shard key: { "_id" : "hashed" }
unique: false
balancing: true
chunks:
shard1 34
shard2 32
shard3 32
too many chunks to print, use verbose if you want to force print
bdtt.normal_20160911
shard key: { "_id" : "hashed" }
unique: false
balancing: true
chunks:
shard1 30
shard2 32
shard3 32
too many chunks to print, use verbose if you want to force print
bdtt.normal_20160912
shard key: { "_id" : "hashed" }
unique: false
balancing: true
chunks:
shard2 126
too many chunks to print, use verbose if you want to force print
bdtt.normal_20160913
shard key: { "_id" : "hashed" }
unique: false
balancing: true
chunks:
shard2 118
too many chunks to print, use verbose if you want to force print
}集合“normal_20160913”不平衡，我将此集合的getShardDistribution（）结果发布如下：
mongos> db.normal_20160913.getShardDistribution()
Shard shard2 at shard2/10.25.2.6:27018,10.25.8.178:27018
data : 4.77GiB docs : 203776 chunks : 118
estimated data per chunk : 41.43MiB
estimated docs per chunk : 1726
Totals
data : 4.77GiB docs : 203776 chunks : 118
Shard shard2 contains 100% data, 100% docs in cluster, avg obj size on shard : 24KiBbalancer进程处于运行状态，chunksize为默认值（64 M）：
mongos> sh.isBalancerRunning()
true
mongos> use config
switched to db config
mongos> db.settings.find()
{ "_id" : "chunksize", "value" : NumberLong(64) }
{ "_id" : "balancer", "stopped" : false }而且我发现了很多moveChunk错误从mogos日志，这可能是为什么一些集合没有很好地平衡的原因，这里是他们的最新部分：
2016-09-19T14:25:25.427+0800 I SHARDING [conn37136926] moveChunk result: { ok: 0.0, errmsg: "Not starting chunk migration because another migration is already in progress", code: 117 }
2016-09-19T14:25:59.620+0800 I SHARDING [conn37136926] moveChunk result: { ok: 0.0, errmsg: "Not starting chunk migration because another migration is already in progress", code: 117 }
2016-09-19T14:25:59.644+0800 I SHARDING [conn37136926] moveChunk result: { ok: 0.0, errmsg: "Not starting chunk migration because another migration is already in progress", code: 117 }
2016-09-19T14:35:02.701+0800 I SHARDING [conn37136926] moveChunk result: { ok: 0.0, errmsg: "Not starting chunk migration because another migration is already in progress", code: 117 }
2016-09-19T14:35:02.728+0800 I SHARDING [conn37136926] moveChunk result: { ok: 0.0, errmsg: "Not starting chunk migration because another migration is already in progress", code: 117 }
2016-09-19T14:42:18.232+0800 I SHARDING [conn37136926] moveChunk result: { ok: 0.0, errmsg: "Not starting chunk migration because another migration is already in progress", code: 117 }
2016-09-19T14:42:18.256+0800 I SHARDING [conn37136926] moveChunk result: { ok: 0.0, errmsg: "Not starting chunk migration because another migration is already in progress", code: 117 }
2016-09-19T14:42:27.101+0800 I SHARDING [conn37136926] moveChunk result: { ok: 0.0, errmsg: "Not starting chunk migration because another migration is already in progress", code: 117 }
2016-09-19T14:42:27.112+0800 I SHARDING [conn37136926] moveChunk result: { ok: 0.0, errmsg: "Not starting chunk migration because another migration is already in progress", code: 117 }
2016-09-19T14:43:41.889+0800 I SHARDING [conn37136926] moveChunk result: { ok: 0.0, errmsg: "Not starting chunk migration because another migration is already in progress", code: 117 }我尝试手动使用moveChunk命令，它返回相同的错误：
mongos> sh.moveChunk("bdtt.normal_20160913", {_id:ObjectId("57d6d107edac9244b6048e65")}, "shard3")
{
"cause" : {
"ok" : 0,
"errmsg" : "Not starting chunk migration because another migration is already in progress",
"code" : 117
},
"code" : 117,
"ok" : 0,
"errmsg" : "move failed"
}我不确定是否创建了太多的集合而导致迁移不堪重负？每天大约会创建60-80个新集合。
我需要帮助在这里回答以下问题，任何提示将是伟大的：
1.为什么有些藏品不平衡，是不是和新增藏品数量大有关？
1.是否有任何命令可以检查正在处理的迁移作业的详细信息？我得到了很多错误日志，其中显示一些迁移作业正在运行，但我找不到哪个正在运行。
2条答案
按热度按时间dgenwo3n1#
回答我自己的问题：最后我们找到了根本原因，这是一个完全相同的问题与这个“MongoDB balancer timeout with delayed replica“，由异常副本集配置引起的。当这个问题发生时，我们的副本集配置如下：
副本集中有4个节点：一个主节点、一个从节点、一个仲裁器和一个24小时延迟的从节点。这使得3个节点成为多数，因为仲裁器没有数据存在，所以平衡器需要等待延迟的从节点以满足写入关注（确保接收器碎片已经接收到块）。
有几种方法可以解决这个问题。我们刚刚删除了仲裁器，平衡器现在工作正常。
ffscu2ro2#
我想推测一下，但我的猜测是，您的集合非常不平衡，目前正在通过块迁移进行平衡（可能需要很长时间）。因此，您的手动块迁移会排队，但不会立即执行。
以下几点可能会更清楚一些：
你可以选择在你的集合中做一些pre-splitting，预分割过程本质上是配置一个空集合来开始平衡，避免在第一时间出现不平衡，因为一旦它们不平衡，块迁移过程可能就不是你的朋友了。
另外，你可能需要重新检查你的碎片键，你可能在碎片键上做了一些错误的事情，导致了很多不平衡。
另外，你的数据量看起来并不太大，不足以保证分片配置。记住，除非你的数据量/工作集大小属性强迫你，否则永远不要做分片配置。因为分片不是免费的（你可能已经感觉到痛苦了）。