分区后弹簧批量挂起

我正在处理200万条记录和20个网格大小，以供我的弹簧批量应用程序使用。对于100-200k的记录，它工作正常，但是在200万次分区后失败。分区后弹簧批量挂起

分区信息：

{ 
    partition - 20 = { 
     startKey = 1980865.0, 
     endKey = 2069192.0, 
     partitionSize = 72, 
     partitionId = 20, 
     partitionRecordCount = 88328 
    }, 
    partition - 10 = { 
     st 
     artKey = 938305.0, 
     endKey = 1042560.0, 
     partitionSize = 72, 
     partitionId = 10, 
     partitionRecordCount = 104256 
    }, 
    partition - 11 = { 
     startKey = 1042561.0, 
     endKey = 1146816.0, 
     partitionSize = 72, 
     partitionId = 11, 
     partitionRecordCount = 104256 
    }, 
    partition - 12 = { 
     startKey = 1146817.0, 
     endKey = 1251072.0, 
     partitionSize = 72, 
     partitionId = 12, 
     partitionRecordCount = 104256 
    }, 
    partition - 13 = { 
     startKey = 1251073.0, 
     endKey = 1355328.0, 
     partitionSize = 72, 
     partitionId = 13, 
     partitionRecordCount = 104256 
    }, 
    par 
    tition - 14 = { 
     startKey = 1355329.0, 
     endKey = 1459584.0, 
     partitionSize = 72, 
     partitionId = 14, 
     partitionRecordCount = 104256 
    }, 
    partition - 15 = { 
     startKey = 1459585.0, 
     end 
     Key = 1563840.0, 
     partitionSize = 72, 
     partitionId = 15, 
     partitionRecordCount = 104256 
    }, 
    partition - 16 = { 
     startKey = 1563841.0, 
     endKey = 1668096.0, 
     partitionSize = 72, 
     p 
     artitionId = 16, 
     partitionRecordCount = 104256 
    }, 
    partition - 17 = { 
     startKey = 1668097.0, 
     endKey = 1772352.0, 
     partitionSize = 72, 
     partitionId = 17, 
     partitionRecordCoun 
     t = 104256 
    }, 
    partition - 18 = { 
     startKey = 1772353.0, 
     endKey = 1876608.0, 
     partitionSize = 72, 
     partitionId = 18, 
     partitionRecordCount = 104256 
    }, 
    partition - 19 = { 
     startKey = 
      1876609.0, 
     endKey = 1980864.0, 
     partitionSize = 72, 
     partitionId = 19, 
     partitionRecordCount = 104256 
    }, 
    partition - 1 = { 
     startKey = 1.0, 
     endKey = 104256.0, 
     partitionSize 
      = 72, 
     partitionId = 1, 
     partitionRecordCount = 104256 
    }, 
    partition - 2 = { 
     startKey = 104257.0, 
     endKey = 208512.0, 
     partitionSize = 72, 
     partitionId = 2, 
     partitionRecordCou 
     nt = 104256 
    }, 
    partition - 3 = { 
     startKey = 208513.0, 
     endKey = 312768.0, 
     partitionSize = 72, 
     partitionId = 3, 
     partitionRecordCount = 104256 
    }, 
    partition - 4 = { 
     startKey = 3127 
      69.0, 
     endKey = 417024.0, 
     partitionSize = 72, 
     partitionId = 4, 
     partitionRecordCount = 104256 
    }, 
    partition - 5 = { 
     startKey = 417025.0, 
     endKey = 521280.0, 
     partitionSize = 7 
      2, 
     partitionId = 5, 
     partitionRecordCount = 104256 
    }, 
    partition - 6 = { 
     startKey = 521281.0, 
     endKey = 625536.0, 
     partitionSize = 72, 
     partitionId = 6, 
     partitionRecordCount 
      = 104256 
    }, 
    partition - 7 = { 
     startKey = 625537.0, 
     endKey = 729792.0, 
     partitionSize = 72, 
     partitionId = 7, 
     partitionRecordCount = 104256 
    }, 
    partition - 8 = { 
     startKey = 729793 
      .0, 
     endKey = 834048.0, 
     partitionSize = 72, 
     partitionId = 8, 
     partitionRecordCount = 104256 
    }, 
    partition - 9 = { 
     startKey = 834049.0, 
     endKey = 938304.0, 
     partitionSize = 72, 
     partitionId = 9, 
     partitionRecordCount = 104256 
    } 
} 
Total no. of Partitions: 20

最后登录它打印是创建的分区，然后控制从来没有进入作家。对于200k条记录来说，它很挣扎，所以我增加了连接池的大小，这解决了问题，但是它失败了200万行。

没有错误，只是挂在那里。

来源

2017-08-26 ViS

对于20个分区？你有没有分析应用程序？它挂在什么地方？我知道在使用大量分区导致应用程序看起来像挂起时，步骤执行存储在作业存储库中的方式往往存在问题，因为它们需要在工作人员启动之前存储。虽然... –

@MichaelMinella，我还没有完成分析，但是分区是早些时候20分钟，后来，我减少到5分钟。它仍然等待4-5小时，以便作者开始写作很少的文件，然后挂起或有时会因'通信链接失败'错误而中断。这个错误虽然从来没有伴随更少的数据。 – ViS

得到它的工作，分区查询是不正确的，再加上，文件写入没有线程。 – ViS

这可能与db有关。你可以检查应用程序是否能够使用写入凭据访问数据库。我注意到这种问题，分区已经完成，并且实际的处理被挂起。故障排除后，意识到它无法写入数据库。一旦db被修复，它运行良好。

来源

2017-11-20 18:31:37 Sowjanya

分区后弹簧批量挂起

回答

相关问题