1
我已经配置我的Flume源是类型Spooldir。我有很多CSV文件,.xl3和.xls,我希望我的Flume代理将所有文件从spooldir加载到HDFS接收器。然而水槽代理回报例外水槽加载csv文件擅长hdfs水槽
这是我水槽源配置:
agent.sources.s1.type = spooldir
agent.sources.s1.spoolDir = /my-directory
agent.sources.s1.basenameHeader = true
agent.sources.batchSize = 10000
和我的HDFS下沉:
agent.sinks.sk1.type = hdfs
agent.sinks.sk1.hdfs.path = hdfs://...:8020/user/importflume/%Y/%m/%d/%H
agent.sinks.sk1.hdfs.filePrefix = %{basename}
agent.sinks.sk1.hdfs.rollSize = 0
agent.sinks.sk1.hdfs.rollCount = 0
agent.sinks.sk1.hdfs.useLocalTimeStamp = true
agent.sinks.sk1.hdfs.batchsize = 10000
agent.sinks.sk1.hdfs.fileType = DataStream
agent.sinks.sk1.serializer = avro_event
agent.sinks.sk1.serializer.compressionCodec = snappy