NoSpamLogger.java最大内存使用量达到了Cassandra

Question

我拥有一个Cassandra的5节点集群，每个节点上的复制因子为3时约有650 GB的数据。我最近开始在/ var/log /卡桑德拉/ SYSTEM.LOG。

INFO [ReadStage-5] 2017年10月17日17：06：07887 NoSpamLogger.java:91 - 达到（1.000GiB），最大存储器使用不能分配1.000MiB的块

我试图增加file_cache_size_in_mb，但迟早会出现这种相同的错误。我曾试图为这个参数高达2GB，但无济于事。

当错误发生时，CPU使用率增加，读取延迟非常不稳定。我每半小时就会看到一次这种激增。请注意下面列表中的时间安排。

INFO [ReadStage-5] 2017-10-17 17：06：07,887 NoSpamLogger.java:91 - 达到的最大内存使用量（1.000GiB），无法分配1.000MiB的块大小 INFO [ReadStage-36] 10-17 17：36：09,807 NoSpamLogger.java:91 - 最大内存使用量达到（1.000GiB），无法分配1.000MiB块大小 INFO [ReadStage-15] 2017-10-17 18:05:56,003 NoSpamLogger.java： 91 - 最大内存使用量达到（2.000GiB），无法分配1.000MiB块大小 INFO [ReadStage-28] 2017-10-17 18:36:01,177 NoSpamLogger.java:91 - 最大内存使用量达到（2.000GiB），不能分配大块1.000MiB

我有两个表是按小时分区的，分区很大。防爆。以下是nodetool表它们的输出统计

Read Count: 4693453 
    Read Latency: 0.36752741680805157 ms. 
    Write Count: 561026 
    Write Latency: 0.03742310516803143 ms. 
    Pending Flushes: 0 
     Table: raw_data 
     SSTable count: 55 
     Space used (live): 594395754275 
     Space used (total): 594395754275 
     Space used by snapshots (total): 0 
     Off heap memory used (total): 360753372 
     SSTable Compression Ratio: 0.20022598072758296 
     Number of keys (estimate): 45163 
     Memtable cell count: 90441 
     Memtable data size: 685647925 
     Memtable off heap memory used: 0 
     Memtable switch count: 1 
     Local read count: 0 
     Local read latency: NaN ms 
     Local write count: 126710 
     Local write latency: 0.096 ms 
     Pending flushes: 0 
     Percent repaired: 52.99 
     Bloom filter false positives: 167775 
     Bloom filter false ratio: 0.16152 
     Bloom filter space used: 264448 
     Bloom filter off heap memory used: 264008 
     Index summary off heap memory used: 31060 
     Compression metadata off heap memory used: 360458304 
     Compacted partition minimum bytes: 51 
     **Compacted partition maximum bytes: 3449259151** 
     Compacted partition mean bytes: 16642499 
     Average live cells per slice (last five minutes): 1.0005435888450147 
     Maximum live cells per slice (last five minutes): 42 
     Average tombstones per slice (last five minutes): 1.0 
     Maximum tombstones per slice (last five minutes): 1 
     Dropped Mutations: 0 



    Read Count: 4712814 
    Read Latency: 0.3356051004771247 ms. 
    Write Count: 643718 
    Write Latency: 0.04168356951335834 ms. 
    Pending Flushes: 0 
     Table: customer_profile_history 
     SSTable count: 20 
     Space used (live): 9423364484 
     Space used (total): 9423364484 
     Space used by snapshots (total): 0 
     Off heap memory used (total): 6560008 
     SSTable Compression Ratio: 0.1744084338623116 
     Number of keys (estimate): 69 
     Memtable cell count: 35242 
     Memtable data size: 789595302 
     Memtable off heap memory used: 0 
     Memtable switch count: 1 
     Local read count: 2307 
     Local read latency: NaN ms 
     Local write count: 51772 
     Local write latency: 0.076 ms 
     Pending flushes: 0 
     Percent repaired: 0.0 
     Bloom filter false positives: 0 
     Bloom filter false ratio: 0.00000 
     Bloom filter space used: 384 
     Bloom filter off heap memory used: 224 
     Index summary off heap memory used: 400 
     Compression metadata off heap memory used: 6559384 
     Compacted partition minimum bytes: 20502 
     **Compacted partition maximum bytes: 4139110981** 
     Compacted partition mean bytes: 708736810 
     Average live cells per slice (last five minutes): NaN 
     Maximum live cells per slice (last five minutes): 0 
     Average tombstones per slice (last five minutes): NaN 
     Maximum tombstones per slice (last five minutes): 0 
     Dropped Mutations: 0 

这里所说：

cdsdb/raw_data histograms 
Percentile SSTables  Write Latency  Read Latency Partition Size  Cell Count 
           (micros)   (micros)   (bytes)     
50%    0.00    61.21    0.00   1955666    642 
75%    1.00    73.46    0.00   17436917    4768 
95%    3.00   105.78    0.00   107964792    24601 
98%    8.00   219.34    0.00   186563160    42510 
99%   12.00   315.85    0.00   268650950    61214 
Min    0.00    6.87    0.00    51     0 
Max   14.00   1358.10    0.00  3449259151   7007506 

cdsdb/customer_profile_history histograms 
Percentile SSTables  Write Latency  Read Latency Partition Size  Cell Count 
           (micros)   (micros)   (bytes)     
50%    0.00    73.46    0.00   223875792    61214 
75%    0.00    88.15    0.00   668489532   182785 
95%    0.00   152.32    0.00  1996099046   654949 
98%    0.00   785.94    0.00  3449259151   1358102 
99%    0.00   943.13    0.00  3449259151   1358102 
Min    0.00    24.60    0.00    5723     4 
Max    0.00   5839.59    0.00  5960319812   1955666 

可否请你建议了一种方法来缓解这个问题？

Answer 1

根据张贴的cfhistograms输出，分区是巨大的。

raw_data表的95％百分位数的分区大小为107MB，最大的 为3.44GB。 customer_profile_history的95％百分位数分区为 ，大小为1.99GB，最大为5.96GB。

这显然与您每隔半小时注意到的问题有关，因为这些巨大的分区被写入到sstable中。数据模型必须根据分区大小进行更改，最好使分区间隔为“分钟”而不是“小时”。所以2GB的分区将减少到33MB的分区。

推荐的分区大小是保持最大接近100MB。虽然理论上我们可以存储超过100MB，但性能会受到影响。请记住，每个分区的读取操作都是通过电线超过100MB的数据。在你的情况下，其超过2GB，因此所有的性能影响。