为什么集群模式比Spark中的客户端模式慢得多？

Question

我有使用GraphFrame创建的15k图形对象。这些图很小（10-100个节点，每个图10-100个边）。我只需要在每个图上应用最短路径和某些其他算法。我目前的实现是使用for循环遍历所有15k图形。令人惊讶的是，集群模式比客户端模式慢得多。

我处理了10张图。以下是客户端模式和集群模式的性能指标。我试图找出为什么集群模式比客户端模式慢。我无法理解指标。

配置为群集（火花独立）

配置为客户端模式：

客户端模式：

Scheduling mode = FIFO 
Spark Context default degree of parallelism = 4 
Aggregated Spark stage metrics: 
numStages => 1019 
sum(numTasks) => 3263 
elapsedTime => 38675 (39 s) 
sum(stageDuration) => 19177 (19 s) 
sum(executorRunTime) => 41413 (41 s) 
sum(executorCpuTime) => 6305 (6 s) 
sum(executorDeserializeTime) => 6386 (6 s) 
sum(executorDeserializeCpuTime) => 3382 (3 s) 
sum(resultSerializationTime) => 82 (82 ms) 
sum(jvmGCTime) => 1171 (1 s) 
sum(shuffleFetchWaitTime) => 3 (3 ms) 
sum(shuffleWriteTime) => 603 (0.6 s) 
max(resultSize) => 23947 (23.0 KB) 
sum(numUpdatedBlockStatuses) => 1848 
sum(diskBytesSpilled) => 0 (0 Bytes) 
sum(memoryBytesSpilled) => 0 (0 Bytes) 
max(peakExecutionMemory) => 35651584 
sum(recordsRead) => 16972 
sum(bytesRead) => 22064469 (21.0 MB) 
sum(recordsWritten) => 0 
sum(bytesWritten) => 0 (0 Bytes) 
sum(shuffleTotalBytesRead) => 1431309 (1397.0 KB) 
sum(shuffleTotalBlocksFetched) => 4304 
sum(shuffleLocalBlocksFetched) => 4304 
sum(shuffleRemoteBlocksFetched) => 0 
sum(shuffleBytesWritten) => 1205067 (1176.0 KB) 
sum(shuffleRecordsWritten) => 8228 

集群模式

Scheduling mode = FIFO 
Spark Context default degree of parallelism = 16 
Aggregated Spark stage metrics: 
numStages => 1019 
sum(numTasks) => 3371 
elapsedTime => 119895 (2.0 min) 
sum(stageDuration) => 114501 (1.9 min) 
sum(executorRunTime) => 162486 (2.7 min) 
sum(executorCpuTime) => 29376 (29 s) 
sum(executorDeserializeTime) => 74342 (1.2 min) 
sum(executorDeserializeCpuTime) => 11919 (12 s) 
sum(resultSerializationTime) => 87 (87 ms) 
sum(jvmGCTime) => 7422 (7 s) 
sum(shuffleFetchWaitTime) => 20899 (21 s) 
sum(shuffleWriteTime) => 1144 (1 s) 
max(resultSize) => 61683 (60.0 KB) 
sum(numUpdatedBlockStatuses) => 6097 
sum(diskBytesSpilled) => 0 (0 Bytes) 
sum(memoryBytesSpilled) => 0 (0 Bytes) 
max(peakExecutionMemory) => 47185920 
sum(recordsRead) => 16962 
sum(bytesRead) => 22037962 (21.0 MB) 
sum(recordsWritten) => 0 
sum(bytesWritten) => 0 (0 Bytes) 
sum(shuffleTotalBytesRead) => 1480503 (1445.0 KB) 
sum(shuffleTotalBlocksFetched) => 4571 
sum(shuffleLocalBlocksFetched) => 2361 
sum(shuffleRemoteBlocksFetched) => 2210 
sum(shuffleBytesWritten) => 1211392 (1183.0 KB) 
sum(shuffleRecordsWritten) => 8228 

Answer 1

TL; DR您正在使用的工作错误的工具。

非常小的图（10-100个节点，每个图的10-100个边）完全不受益于分布式处理。

同时，像Spark这样的分布式系统中使用的高成本抽象增加了大量的开销，使得它占据了成本。

一般来说：

• 分布式图处理是困难的。
• 在大多数情况下，扩大规模都会扩大规模。