将批量数据加载到Elasticsearch时出错

我在python中使用Elasticsearch。我有熊猫框架（3列）中的数据，然后我添加了两列_index和_type，并使用pandas内置方法将数据转换为json。将批量数据加载到Elasticsearch时出错

data = data.to_json(orient='records')

这是我的数据，那么，

[{"op_key":99140046678,"employee_key":991400459,"Revenue Results":6625.76480192,"_index":"revenueindex","_type":"revenuetype"},  
{"op_key":99140045489,"employee_key":9914004258,"Revenue Results":6691.05435536,"_index":"revenueindex","_type":"revenuetype"}, 
...... 
}]

我的映射是：

Traceback (most recent call last): 
     File "/Users/adaggula/Documents/workspace/ElSearchPython/sample.py", line 59, in <module> 
     res = helpers.bulk(client,data) 
     File "/Users/adaggula/workspace/python/pve/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 188, in bulk 
     for ok, item in streaming_bulk(client, actions, **kwargs): 
     File "/Users/adaggula/workspace/python/pve/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 160, in streaming_bulk 
     for result in _process_bulk_chunk(client, bulk_actions, raise_on_exception, raise_on_error, **kwargs): 
     File "/Users/adaggula/workspace/python/pve/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 89, in _process_bulk_chunk 
     raise e 
    elasticsearch.exceptions.RequestError: TransportError(400, u'action_request_validation_exception', u'Validation Failed: 1: index is 
missing;2: type is missing;3: index is missing;4: type is missing;5: index is 
missing;6: ....... type is missing;999: index is missing;1000: type is missing;')

：

user_mapping = { 
     "settings" : { 
      "number_of_shards": 3, 
      "number_of_replicas": 2 
     }, 

     'mappings': { 
      'revenuetype': { 
       'properties': { 
        'op_key':{'type':'string'}, 
        'employee_key':{'type':'string'}, 
        'Revenue Results':{'type':'float','index':'not_analyzed'}, 
       } 
      } 
     } 
    }

同时使用helpers.bulk（ES，数据），然后对着这个错误

它看起来像每个json对象，索引和t ype's不见了。如何克服这一点？

来源

2016-07-04 Jack Daniel

熊猫数据帧到json转换是解决问题的技巧。

data = data.to_json(orient='records') 
data= json.loads(data)

来源

2016-07-04 10:10:49

我刚刚发表评论，这可能会缩短为'data = data.to_dict（orient ='records'）'。然后我对一个有1,000,000行和50列的数据框进行了一个简短的测试，发现你的版本执行速度明显更快......奇怪的是，'df.to_dict（）'的速度非常慢。 – Dirk

我有一个类似的错误，并通过在'obj.to_dict（include_meta = True）中添加'include_meta = True'来摆脱它' – Anupam

将批量数据加载到Elasticsearch时出错

回答

相关问题