为什么我不能在数据库中写入数据框？

我有32 GB RAM，我使用jupyter和熊猫。我的数据帧不是很大，但是当我想它写在北极的数据的基础上我有“的MemoryError”：为什么我不能在数据库中写入数据框？

而且我希望把它写：

from arctic import Arctic 
import arctic as arc 
store = Arctic('.....') 
lib = store['myLib'] 
lib.write('quotes', df_q)

MemoryError Traceback (most recent call last) in() 1 memory(df_q) ----> 2 lib.write('quotes', df_q)

/usr/local/lib/python2.7/dist-packages/arctic/decorators.pyc in f_retry(*args, **kwargs) 48 while True: 49 try: ---> 50 return f(*args, **kwargs) 51 except (DuplicateKeyError, ServerSelectionTimeoutError) as e: 52 # Re-raise errors that won't go away.

/usr/local/lib/python2.7/dist-packages/arctic/store/version_store.pyc in write(self, symbol, data, metadata, prune_previous_version, **kwargs) 561 562 handler = self._write_handler(version, symbol, data, **kwargs) --> 563 mongo_retry(handler.write)(self._arctic_lib, version, symbol, data, previous_version, **kwargs) 564 565 # Insert the new version into the version DB

/usr/local/lib/python2.7/dist-packages/arctic/decorators.pyc in f_retry(*args, **kwargs) 48 while True: 49 try: ---> 50 return f(*args, **kwargs) 51 except (DuplicateKeyError, ServerSelectionTimeoutError) as e: 52 # Re-raise errors that won't go away.

/usr/local/lib/python2.7/dist-packages/arctic/store/_pandas_ndarray_store.pyc in write(self, arctic_lib, version, symbol, item, previous_version) 301 def write(self, arctic_lib, version, symbol, item, previous_version): 302 item, md = self.to_records(item) --> 303 super(PandasDataFrameStore, self).write(arctic_lib, version, symbol, item, previous_version, dtype=md) 304 305 def append(self, arctic_lib, version, symbol, item, previous_version):

/usr/local/lib/python2.7/dist-packages/arctic/store/_ndarray_store.pyc in write(self, arctic_lib, version, symbol, item, previous_version, dtype) 385 version['type'] = self.TYPE 386 version['up_to'] = len(item) --> 387 version['sha'] = self.checksum(item) 388 389 if previous_version:

/usr/local/lib/python2.7/dist-packages/arctic/store/_ndarray_store.pyc in checksum(self, item) 370 def checksum(self, item): 371 sha = hashlib.sha1() --> 372 sha.update(item.tostring()) 373 return Binary(sha.digest()) 374

MemoryError:

WTF ？如果我使用df_q.to_csv（）我会等上好几年....

来源

2016-07-11 ooolllooo

你是否尝试在[chunks]（https://github.com/manahl/arctic/blob/master/arctic/chunkstore/chunkstore.py）中编写DF？ – MaxU

您的问题实际上不是内存问题。如果你读了你的错误，看起来你的程序库在访问你的数据时遇到了问题......

第一个错误：说你的服务器超时了。（ServerSelectionTimeoutError）

第二个错误：尝试更新MongoDB版本。

第三错误：重试访问您的服务器，失败（ServerSelectionTimeoutError）

等，所以基本上你的问题就出在北极包本身（见最后一个错误是一个校验和错误）。您也可以从df_q.to_csv()工作的事实推断出这一点，但由于它不像Artic那样进行优化，因此它非常缓慢。我建议尝试重新安装北极包

来源

2016-07-11 19:21:45

为什么我不能在数据库中写入数据框？

回答

相关问题