2013-10-08 67 views
2

我经历了停电杀害所有4个节点在我的卡桑德拉集群。我已经带回了所有的盒子,但是现在当我尝试启动Cassandra(method bin/Cassandra)时,在有一些日志重放之后,我收到以下消息。卡桑德拉例外:指数(1)必须小于尺寸(1)

Exception encountered during startup: index (1) must be less than size (1) 
INFO 10:11:00,479 CFS(Keyspace='system', ColumnFamily='peers') liveRatio is 13.10204081632653 (just-counted was 13.10204081632653). calculation took 18ms for 21 columns 

(这是刚刚结束,我可以发布更多的消息,如果有帮助)

想出了正在运行只读取一个,我认为我有复制因子3所以所有keyspaces没有其他人就无法工作。

任何想法是为如何做才能挽回这个局面。我有相当多的数据库中的有价值的信息。

卡桑德拉V1.2.6,

的CentOS V6.4

更新:

所有这三个节点死亡似乎提交日志的重播期间要失败。我已经在加粗他们得到的线。我最初的想法是删除该提交日志,并尝试重新启动..但我有点害怕后果!

INFO 10:46:48,600 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1379365208291.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208292.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208293.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208294.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208295.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208296.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208297.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208298.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208299.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208301.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208302.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208304.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208305.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208306.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208307.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208308.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208309.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208310.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208311.log, /var/lib/cassandra/commitlog/CommitLog-2-1381251642075.log, /var/lib/cassandra/commitlog/CommitLog-2-1381251731119.log 
INFO 10:46:48,614 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1379365208291.log 
INFO 10:46:49,598 GC for ParNew: 605 ms for 2 collections, 50055912 used; max is 1046937600 
INFO 10:46:51,775 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1379365208291.log 
INFO 10:46:51,776 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1379365208292.log 
INFO 10:46:53,995 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1379365208292.log 
INFO 10:46:53,995 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1379365208293.log 
**INFO 10:46:54,087 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1379365208293.log** 
ERROR 10:46:54,088 Exception encountered during startup 
java.lang.IndexOutOfBoundsException: index (1) must be less than size (1) 
    at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) 
    at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) 
    at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45) 
    at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:94) 
    at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76) 
    at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31) 
    at java.util.TreeMap.compare(TreeMap.java:1188) 
    at java.util.TreeMap.put(TreeMap.java:531) 
    at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:102) 
    at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:88) 
    at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114) 
    at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109) 
    at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:101) 
    at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:397) 
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:202) 
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:97) 
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146) 
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126) 
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:298) 
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:441) 
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:484) 
java.lang.IndexOutOfBoundsException: index (1) must be less than size (1) 
    at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) 
    at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) 
    at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45) 
    at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:94) 
    at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76) 
    at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31) 
    at java.util.TreeMap.compare(TreeMap.java:1188) 
    at java.util.TreeMap.put(TreeMap.java:531) 
    at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:102) 
    at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:88) 
    at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114) 
    at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109) 
    at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:101) 
    at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:397) 
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:202) 
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:97) 
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146) 
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126) 
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:298) 
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:441) 
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:484) 
Exception encountered during startup: index (1) must be less than size (1) 
INFO 10:46:54,110 CFS(Keyspace='system', ColumnFamily='peers') liveRatio is 13.10204081632653 (just-counted was 13.10204081632653). calculation took 19ms for 21 columns 
+0

删除提交日志意味着可能会丢失已写入,但不会刷新到Cassandra的任何数据。这似乎是一个[已知问题](https://issues.apache.org/jira/browse/CASSANDRA-5905)([本以及](https://issues.apache.org/jira/browse/CASSANDRA -5202))。 –

+0

是的,我认为可能是这样。为了将来的参考,我应该在常规基础上刷新节点,即nodetool flush? – gerrymcdev

+0

C *定期进行冲洗(离我头顶4小时) –

回答

2

我最好的理论是,这是删除并重新创建具有相同的名称(https://issues.apache.org/jira/browse/CASSANDRA-5905)表中的一个可能后果。我们的目标是修复2.1(https://issues.apache.org/jira/browse/CASSANDRA-5202);在此期间,更喜欢TRUNCATE丢弃/重新创建。

+0

谢谢@peter。这很有趣,因为我确实有这种情况。 我有hadoop运行到cassandra和他们需要目标输出表清除之前他们运行,这是通过删除和重新创建表完成的一些作业。 我最终通过删除受影响的文件(我知道我最有可能有一点数据丢失,但我们去了)得到所有死去的节点。我认为这些文件因为崩溃而被破坏,但它们并不是最后的日志文件,而是分散在各处。我想这会支持你的理论? – gerrymcdev

+0

是的,这确实适合。 – jbellis

0

虽然它已经很晚了,你必须解决这个问题为好。对于其他有类似问题的人,请尝试下面的步骤

sudo bash -c 'rm -rf /var/lib/cassandra/data/system/*' 

请记住,它仅用于开发目的。 该命令将删除您的所有数据。