pandas在传递到kdb +时（使用qPython API）丢失索引

Question

我试图将时间序列数据从Python传递到q/kdb+。

一个解决方案是qPython module，提供从q表/字典到Pandas的无缝转换。

的问题是试图从熊猫传递到q，时间指数DataFrame（列Date）时并不完全使它的q侧。可重复码：

import pandas.io.data as web 
import datetime 
import numpy 
import qpython.qconnection as qconnection # requires installation of qPython module from https://github.com/exxeleron/qPython 

start = datetime.datetime(2010, 1, 1) 
end = datetime.datetime(2015, 2, 6) 
f=web.DataReader("F", 'yahoo', start, end) # download Ford stock data (ticker "F") from Yahoo Finance web service 
f.ix[:5] # explore first 5 rows of the DataFrame 
# Out: 
#    Open High Low Close Volume Adj Close 
# Date 
# 2010-01-04 10.17 10.28 10.05 10.28 60855800  9.43 
# 2010-01-05 10.45 11.24 10.40 10.96 215620200  10.05 
# 2010-01-06 11.21 11.46 11.13 11.37 200070600  10.43 
# 2010-01-07 11.46 11.69 11.32 11.66 130201700  10.69 
# 2010-01-08 11.67 11.74 11.46 11.69 130463000  10.72 

q = qconnection.QConnection(host = 'localhost', port = 5000, pandas = True) # define connection interface parameters. Assumes we have previously started q server on port 5000 with `q.exe -p 5000` command 
q.open() # open connection 
q('set', numpy.string_('yahoo'), f) # pass DataFrame to q table named `yahoo` 
q('5#yahoo') # display top 5 rows from newly created table on q server 
# Out: 
# Open High Low Close Volume Adj Close 
# 0 10.17 10.28 10.05 10.28 60855800  9.43 
# 1 10.45 11.24 10.40 10.96 215620200  10.05 
# 2 11.21 11.46 11.13 11.37 200070600  10.43 
# 3 11.46 11.69 11.32 11.66 130201700  10.69 
# 4 11.67 11.74 11.46 11.69 130463000  10.72 

正如你所看到的，问表不具备Date列存在于f数据帧的指数。

如何有效地（对于大数据）将日期时间索引传递给q？

Answer 1

在序列化DataFrame对象时，qPython检查是否存在meta属性。如果该属性不存在，则DataFrame被序列化为q表，并且索引列在该过程中被跳过。如果要保留索引列，则必须设置meta属性并提供类型提示以强制执行q键表的表示。

请看看修改后的样本：

import pandas.io.data as web 
import datetime 
import numpy 
import qpython.qconnection as qconnection # requires installation of qPython module from https://github.com/exxeleron/qPython 

from qpython import MetaData 
from qpython.qtype import QKEYED_TABLE 


start = datetime.datetime(2010, 1, 1) 
end = datetime.datetime(2015, 2, 6) 
f=web.DataReader("F", 'yahoo', start, end) # download Ford stock data (ticker "F") from Yahoo Finance web service 
f.ix[:5] # explore first 5 rows of the DataFrame 
# Out: 
#    Open High Low Close Volume Adj Close 
# Date 
# 2010-01-04 10.17 10.28 10.05 10.28 60855800  9.43 
# 2010-01-05 10.45 11.24 10.40 10.96 215620200  10.05 
# 2010-01-06 11.21 11.46 11.13 11.37 200070600  10.43 
# 2010-01-07 11.46 11.69 11.32 11.66 130201700  10.69 
# 2010-01-08 11.67 11.74 11.46 11.69 130463000  10.72 

q = qconnection.QConnection(host = 'localhost', port = 5000, pandas = True) # define connection interface parameters. Assumes we have previously started q server on port 5000 with `q.exe -p 5000` command 
q.open() # open connection 
f.meta = MetaData(**{'qtype': QKEYED_TABLE}) # enforce to serialize DataFrame as keyed table 
q('set', numpy.string_('yahoo'), f) # pass DataFrame to q table named `yahoo` 
q('5#yahoo') # display top 5 rows from newly created table on q server 
# Out: 
#    Open High Low Close  Volume Adj Close 
# Date               
# 2010-01-04 10.17 10.28 10.05 10.28 60855800  9.43 
# 2010-01-05 10.45 11.24 10.40 10.96 215620200  10.05 
# 2010-01-06 11.21 11.46 11.13 11.37 200070600  10.43 
# 2010-01-07 11.46 11.69 11.32 11.66 130201700  10.69 
# 2010-01-08 11.67 11.74 11.46 11.69 130463000  10.72 

Answer 2

您是否考虑尝试其他API之一？ http://www.timestored.com/kdb-guides/python-api 我也会在他们的github页面上报告问题。