我在cassandra3.11.5中得到以下错误
LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@356cb07f) to class org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1709177197:Memory@[7e7a3b64b910..7e7a3b96b910) was not released before the reference was garbage collected
ERROR [CompactionExecutor:3763] 2020-11-02 02:34:53,316 CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:3763,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:180) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268) ~[apache-cassandra-3.11.5.jar:3.11.5]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.5.jar:3.11.5]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939) ~[na:1.8.0_60]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153) ~[apache-cassandra-3.11.5.jar:3.11.5]
... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_60]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936) ~[na:1.8.0_60]
... 24 common frames omitted
WARN [GossipTasks:1] 2020-11-02 02:34:53,302 FailureDetector.java:278 - Not marking nodes down due to local pause of 7468254535 > 5000000000
ERROR [Reference-Reaper] 2020-11-02 02:34:53,208 Ref.java:229 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@33fef449) to class org.apache.cassandra.io.util.FileHandle$Cleanup@1098210945:/data/cassandra/data/elvisevallogks/decision_detail-7351f6205f3811ea922ce1efeeba3e49/md-3695-big-Index.db was not released before the reference was garbage collected
ERROR [CompactionExecutor:3762] 2020-11-02 02:34:53,208 CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:3762,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:180) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268) ~[apache-cassandra-3.11.5.jar:3.11.5]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.5.jar:3.11.5]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939) ~[na:1.8.0_60]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153) ~[apache-cassandra-3.11.5.jar:3.11.5]
... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_60]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936) ~[na:1.8.0_60]
... 24 common frames omitted
INFO [Service Thread] 2020-11-02 02:34:53,877 StatusLogger.java:51 - MemtablePostFlush 0 0 3596 0 0这些是设置
# Cassandra-specfic parameters
# add
cassandra soft memlock unlimited
cassandra hard memlock unlimited
cassandra soft nproc 65535
# change
* soft nofile 655350
* hard nofile 655350
Cassandra过程
cat /proc/25586/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 0 bytes
Max resident set unlimited unlimited bytes
Max processes 65535 692334 processes
Max open files 655350 655350 files
Max locked memory unlimited unlimited bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 692334 692334 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us我在网上搜索了一下，推荐的设置是设置vm.max\u map\u count 1048575，并验证了该值的设置是否正确。堆大小设置为32 gb，并且计算机有171 gb的ram。
sysctl vm.max_map_count
vm.max_map_count = 1048575节点没有崩溃，但当我尝试连接时会随机超时。
1条答案
按热度按时间a1o7rhls1#
您可能需要检查并启用disk\u access\u模式，将默认的“standard”设置为“mmap\u index\u only”，它通过使用仅Map的索引文件来避免页面错误。页面错误有时会导致内存泄漏，建议根据集群利用率将堆硬编码为特定大小。
问题是，当堆的默认值为32或更高时，会对堆中的许多对象进行松散压缩，并且不会有很好的性能，因此，当您有机会滚动重新启动时，这两个更改将很好地结合在一起。
谢谢，甘尼什