格式MongoDB的查询中使用Pymongo

Question

我想插入我的查询对象在pymongo连接器创建大熊猫数据帧：

import pandas as pd 
from pymongo import MongoClient 


def _connect_mongo(host, port, username, password, db): 

    if username and password: 
     mongo_uri = 'mongodb://%s:%s@%s:%s/%s' % (username, password, host, port, db) 
     conn = MongoClient(mongo_uri) 
    else: 
     conn = MongoClient(host, port) 


    return conn[db] 


def read_mongo(db, collection, query={}, host='localhost', port=27017, username=None, password=None, no_id=True): 
    """ Read from Mongo and Store into DataFrame """ 

    # Connect to MongoDB 
    db = _connect_mongo(host=host, port=port, username=username, password=password, db=db) 

    # Make a query to the specific DB and Collection 
    cursor = db[collection].find(query) 
    # Expand the cursor and construct the DataFrame 
    df = pd.DataFrame(list(cursor)) 

    # Delete the _id 
    # if no_id: 
    # del df['_id'] 

    return df 

我的查询被定义为：

query_1 = "{ 
        "status" : {"$ne" : "deprecated"}, 
        "geoLocationData.date" : { $gte : new ISODate("2016-08-03") } 
       }, 
        { "geoLocationData.date": 1, 
         "geoLocationData.iso": 1, 
         "httpRequestData.ipAddress": 1, 
         "geoLocationData.city": 1, 
         "geoLocationData.latitude": 1, 
         "geoLocationData.longitude": 1 }" 

将其插入 - 获得一个数据帧大熊猫：

df = read_mongo(db, collection, query_1, host, port, username, password) 

我得到的错误：

TypeError: filter must be an instance of dict, bson.son.SON, or other type that inherits from collections.Mapping 

如果我只是省略子文档，查询工作得很好，我可以将其转换为数据框。

我想这是关于将我的查询转换成字典（与子文件）。 我该怎么做？

Answer 1

您的query_1变量是一个字符串，它似乎包含两个字典。方法find首先将字典作为过滤器参数，然后将投影作为第二个参数。你只是传递一个论点。

这应该工作：

def read_mongo(db, collection, filter={}, projection={}, host='localhost', port=27017, username=None, password=None, no_id=True): 
    """ Read from Mongo and Store into DataFrame """ 

    # Connect to MongoDB 
    db = _connect_mongo(host=host, port=port, username=username, password=password, db=db) 

    # Make a query to the specific DB and Collection 
    cursor = db[collection].find(filter, projection) 
    # Expand the cursor and construct the DataFrame 
    df = pd.DataFrame(list(cursor)) 

    # Delete the _id 
    # if no_id: 
    # del df['_id'] 

    return df 

query_filter = { 
        "status" : {"$ne" : "deprecated"}, 
        "geoLocationData.date" : { $gte : new ISODate("2016-08-03") } 
       } 
query_project = { "geoLocationData.date": 1, 
         "geoLocationData.iso": 1, 
         "httpRequestData.ipAddress": 1, 
         "geoLocationData.city": 1, 
         "geoLocationData.latitude": 1, 
         "geoLocationData.longitude": 1 } 

df = read_mongo(db, collection, query_filter, query_project, host, port, username, password) 

不过，我不认为这种方法可以联合你想要它做的相当的。这是因为find操作中的投影只能包含或排除它不以您想要的方式映射它们的字段。你可以遍历你的游标来处理数据，然后传入构造函数DataFrame（注意不要创建巨大的python列表）。更好的方法是不使用find在所有，但使用aggregate：

cursor = db[collection].aggregate([filter, projection]) 

现在你可以让你的过滤器match流水线阶段和投影一个project阶段。

query_project = { "geoDate":"$geoLocationData.date", 
        "geoLoc":"$geoLocationData.iso", 
        "ipAddress": "$httpRequestData.ipAddress", 
        "city": "$geoLocationData.city", 
         "lat": "$geoLocationData.latitude", 
         "long":"$geoLocationData.longitude"} 

df = read_mongo(db, collection, {"$match" : query_filter}, {"$project" : query_project}, host, port, username, password)