通过Python函数跟踪*最大*内存使用情况

Question

我想知道调用函数期间分配的最大内存量是多少（以Python为单位）。有上SO相关跟踪内存使用其他问题：

Which Python memory profiler is recommended?

How do I profile memory usage in Python?

但这些似乎让你更多的时间来跟踪内存使用量的heap()方法（在孔雀的情况下）叫做。然而，我想跟踪的是外部库中的一个函数，我无法修改它，并且增长到使用大量RAM，但是一旦函数执行完成后就会释放它。有什么方法可以找出函数调用期间使用的RAM的总量是多少？

Answer 1

这个问题似乎很有趣，它给了我一个理由去看望Guppy/Heapy，因为我感谢你。

我试了大约2个小时才让Heapy监视一个函数调用/进程而没有修改它的源代码零运气好。

我确实找到了一种方法来使用内置的Python库resource完成您的任务。请注意，该文档不指示RU_MAXRSS值返回的内容。另一个SO用户noted它是在kB。运行Mac OSX 7.3并观察我的系统资源在下面的测试代码中爬升，我相信返回的值是字节而不是千字节。

关于我如何使用resource库来监视库调用的10000ft视图是在单独的（可监视）线程中启动该函数，并在主线程中跟踪该进程的系统资源。下面我有两个文件，你需要运行来测试它。

图书馆资源监视器 - whatever_you_want.py

import resource 
import time 

from stoppable_thread import StoppableThread 


class MyLibrarySniffingClass(StoppableThread): 
    def __init__(self, target_lib_call, arg1, arg2): 
     super(MyLibrarySniffingClass, self).__init__() 
     self.target_function = target_lib_call 
     self.arg1 = arg1 
     self.arg2 = arg2 
     self.results = None 

    def startup(self): 
     # Overload the startup function 
     print "Calling the Target Library Function..." 

    def cleanup(self): 
     # Overload the cleanup function 
     print "Library Call Complete" 

    def mainloop(self): 
     # Start the library Call 
     self.results = self.target_function(self.arg1, self.arg2) 

     # Kill the thread when complete 
     self.stop() 

def SomeLongRunningLibraryCall(arg1, arg2): 
    max_dict_entries = 2500 
    delay_per_entry = .005 

    some_large_dictionary = {} 
    dict_entry_count = 0 

    while(1): 
     time.sleep(delay_per_entry) 
     dict_entry_count += 1 
     some_large_dictionary[dict_entry_count]=range(10000) 

     if len(some_large_dictionary) > max_dict_entries: 
      break 

    print arg1 + " " + arg2 
    return "Good Bye World" 

if __name__ == "__main__": 
    # Lib Testing Code 
    mythread = MyLibrarySniffingClass(SomeLongRunningLibraryCall, "Hello", "World") 
    mythread.start() 

    start_mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss 
    delta_mem = 0 
    max_memory = 0 
    memory_usage_refresh = .005 # Seconds 

    while(1): 
     time.sleep(memory_usage_refresh) 
     delta_mem = (resource.getrusage(resource.RUSAGE_SELF).ru_maxrss) - start_mem 
     if delta_mem > max_memory: 
      max_memory = delta_mem 

     # Uncomment this line to see the memory usuage during run-time 
     # print "Memory Usage During Call: %d MB" % (delta_mem/1000000.0) 

     # Check to see if the library call is complete 
     if mythread.isShutdown(): 
      print mythread.results 
      break; 

    print "\nMAX Memory Usage in MB: " + str(round(max_memory/1000.0, 3)) 

停止的线程 - stoppable_thread.py

import threading 
import time 

class StoppableThread(threading.Thread): 
    def __init__(self): 
     super(StoppableThread, self).__init__() 
     self.daemon = True 
     self.__monitor = threading.Event() 
     self.__monitor.set() 
     self.__has_shutdown = False 

    def run(self): 
     '''Overloads the threading.Thread.run''' 
     # Call the User's Startup functions 
     self.startup() 

     # Loop until the thread is stopped 
     while self.isRunning(): 
      self.mainloop() 

     # Clean up 
     self.cleanup() 

     # Flag to the outside world that the thread has exited 
     # AND that the cleanup is complete 
     self.__has_shutdown = True 

    def stop(self): 
     self.__monitor.clear() 

    def isRunning(self): 
     return self.__monitor.isSet() 

    def isShutdown(self): 
     return self.__has_shutdown 


    ############################### 
    ### User Defined Functions #### 
    ############################### 

    def mainloop(self): 
     ''' 
     Expected to be overwritten in a subclass!! 
     Note that Stoppable while(1) is handled in the built in "run". 
     ''' 
     pass 

    def startup(self): 
     '''Expected to be overwritten in a subclass!!''' 
     pass 

    def cleanup(self): 
     '''Expected to be overwritten in a subclass!!''' 
     pass 

Answer 2

这是可能的memory_profiler做到这一点。函数memory_usage返回一个值列表，这些值表示随时间变化的内存使用情况（默认情况下是0.1秒的块）。如果您需要最大限度的话，只需拿出该列表的最大值。小例子：

from memory_profiler import memory_usage 
from time import sleep 

def f(): 
    # a function that with growing 
    # memory consumption 
    a = [0] * 1000 
    sleep(.1) 
    b = a * 100 
    sleep(.1) 
    c = b * 100 
    return a 

mem_usage = memory_usage(f) 
print('Memory usage (in chunks of .1 seconds): %s' % mem_usage) 
print('Maximum memory usage: %s' % max(mem_usage)) 

在我的情况（memory_profiler 0.25），如果打印输出如下：

Memory usage (in chunks of .1 seconds): [45.65625, 45.734375, 46.41015625, 53.734375] 
Maximum memory usage: 53.734375 

Answer 3

这似乎在Windows下运行。不了解其他操作系统。

In [50]: import os 

In [51]: import psutil 

In [52]: process = psutil.Process(os.getpid()) 

In [53]: process.get_ext_memory_info().peak_wset 
Out[53]: 41934848 

Answer 4

一直在努力完成这项任务。在尝试了Adam的psutil和方法之后，我写了一个函数（学分给Adam Lewis）来测量特定函数使用的内存。人们可能会发现它更容易抓住和使用。

1）measure_memory_usage

2）test measure_memory_usage

我发现，大约线程和压倒一切的超材料是在理解亚当在他的剧本是做真正有用的。抱歉，由于我的“2链接”最大限制，我无法发布链接。

Answer 5

您可以使用python库资源来获取内存使用情况。

进口资源 resource.getrusage（resource.RUSAGE_SELF）.ru_maxrss

它会给内存使用量以KB为单位，由1000