1. 首页
  2. 问答
  3. net/protocol.rb分配的大量内存：153

net/protocol.rb分配的大量内存：153

2017-08-29 54 views
1

运行memory_profiler时，我注意到大量的内存被Ruby的net/protocol.rb组件分配。我在执行HTTP请求到服务器以下载文件时调用它。该文件大小为43.67MB，net/protocol.rb单独分配262,011,476字节来下载它。net/protocol.rb分配的大量内存：153

查看下面的分析器报告中的“按位置分配的内存”部分，我可以看到net/protocol.rb:172和http/response.rb:334每个分配50-60MB的内存，这大约与文件的大小有关，因此看起来合理。但是，最上面的条目（net/protocol.rb:153）让我担心：这是200MB的内存，至少是文件大小的4倍。

我有两个问题：

  1. 为什么网/协议需要分配5X文件的大小，以下载呢？
  2. 有什么我可以做的，以减少净/协议使用的内存量？

memory_profiler输出：

Total allocated: 314461424 bytes (82260 objects) 
Total retained: 0 bytes (0 objects) 

allocated memory by gem 
----------------------------------- 
314461304 ruby-2.1.2/lib 
     120 client/lib 

allocated memory by file 
----------------------------------- 
262011476 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb 
    52435727 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb 
     7971 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb 
     2178 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb 
     1663 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb 
     1260 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb 
     949 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb 
     120 /Users/andre.debrito/git/techserv-cache/client/lib/connections/cache_server_connection.rb 
     80 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/http.rb 

allocated memory by location 
----------------------------------- 
200483909 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153 
    60548199 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:172 
    52428839 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:334 
    978800 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:155 
     2537 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:61 
     2365 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:172 
     2190 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:54 
     1280 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:56 
     960 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:62 
     836 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:165 
     792 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:13 
     738 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:125 
     698 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:263 
     576 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:214 
     489 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:40 
     480 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:127 
     360 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:40 
     328 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:610 
     320 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:71 
     320 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:30 
     320 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:59 
     308 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:322 
     256 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:879 
     240 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1615 
     239 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:211 
     232 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:38 
     224 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:181 
     200 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:17 
     192 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:42 
     179 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:877 
     169 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1459 
     160 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1029 
     160 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:434 
     160 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:435 
     160 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:445 
     160 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1617 
     149 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1445 
     147 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1529 
     129 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:98 
     128 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1475 
     120 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:444 
     120 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:446 
     120 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:447 
     120 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:29 
     120 /Users/andre.debrito/git/techserv-cache/client/lib/connections/cache_server_connection.rb:45 
     96 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:899 
     80 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:39 
     80 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:45 
     80 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:46 
     80 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:145 

allocated memory by class 
----------------------------------- 
309678360 String 
    3445304 Thread::Backtrace 
    981096 Array 
    352376 IO::EAGAINWaitReadable 
     1960 MatchData 
     1024 Hash 
     328 Net::HTTP 
     256 TCPSocket 
     256 URI::HTTP 
     128 Time 
     120 Net::HTTP::Get 
     120 Net::HTTPOK 
     96 Net::BufferedIO 

allocated objects by gem 
----------------------------------- 
    82259 ruby-2.1.2/lib 
     1 client/lib 

allocated objects by file 
----------------------------------- 
    81908 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb 
     129 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb 
     127 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb 
     28 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb 
     23 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb 
     23 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb 
     19 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/http.rb 
     1 /Users/andre.debrito/git/techserv-cache/client/lib/connections/cache_server_connection.rb 

allocated objects by location 
----------------------------------- 
    36373 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153 
    24470 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:155 
    21057 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:172 
     48 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:61 
     38 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:54 
     32 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:56 
     31 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:172 
     24 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:62 
     12 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:127 
     9 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:40 
     8 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:71 
     8 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:165 
     8 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:30 
     8 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:59 
     6 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:214 
     6 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1615 
     5 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:263 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1029 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:322 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:17 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:434 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:435 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:445 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:42 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:125 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1617 
     3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1529 
     3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:444 
     3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:446 
     3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:447 
     3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:40 
     3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1445 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:877 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:39 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:45 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:46 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:13 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:145 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:31 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:111 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:144 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:98 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:179 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:181 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:213 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1640 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1642 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:343 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:530 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:557 

allocated objects by class 
----------------------------------- 
    47935 String 
    24519 Array 
     4894 IO::EAGAINWaitReadable 
     4894 Thread::Backtrace 
     7 MatchData 
     3 Hash 
     2 URI::HTTP 
     1 Net::BufferedIO 
     1 Net::HTTP 
     1 Net::HTTP::Get 
     1 Net::HTTPOK 
     1 TCPSocket 
     1 Time 

retained memory by gem 
----------------------------------- 
NO DATA 

retained memory by file 
----------------------------------- 
NO DATA 

retained memory by location 
----------------------------------- 
NO DATA 

retained memory by class 
----------------------------------- 
NO DATA 

retained objects by gem 
----------------------------------- 
NO DATA 

retained objects by file 
----------------------------------- 
NO DATA 

retained objects by location 
----------------------------------- 
NO DATA 

retained objects by class 
----------------------------------- 
NO DATA 


Allocated String Report 
----------------------------------- 
    11926 "" 
     7019 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:172 
     4894 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153 
     10 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:54 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:179 
     1 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:67 

     4894 "Resource temporarily unavailable - read would block" 
     4894 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153 

     4894 "UTF-8" 
     4894 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153 

     4894 "read would block" 
     4894 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153 
...

相关的代码：

report = MemoryProfiler.report do 
    begin 
     response = nil 
     Net::HTTP.new(uri.host, uri.port).start { |http| 
     request = Net::HTTP::Get.new uri.request_uri 
     response = http.request request 
     } 
    rescue Net::ReadTimeout => e 
     raise RequestTimeoutError.new(e.message) 
    rescue Exception => e 
     raise ServerConnectionError.new(e.message) 
    end 
    end 
    report.pretty_print
自Charles代理

网络流量数据：

  • 请求报头：168个字节
  • 响应头：288个字节
  • 请求： -
  • 响应：43.67 MB（45792735个字节）
  • 总计：43.67 MB（45793191个字节）

来源

2017-08-29 André de Brito

回答

1

几乎所有net/protocol.rb#L153分配这些字符串是短期住下来并在下一次GC运行中回收。这些分配的对象因此非常无害，并且不会导致显着更大的进程大小。

您会得到很多异常（用于此处的控制流以读取套接字）以及附加到缓冲区的实际读取数据。所有这些操作都会创建临时（内部使用）的对象。

因此，您可能正在测量错误的东西。可能更有意义的是：

  • 测量进程的最大RSS（即“已用”内存）;
  • 并测量读取后仍分配的额外内存量。

您会注意到（取决于计算机上的内存压力），RSS不会显着高于实际读取的数据量，并且读取后的参考内存与读取的大小大致相同大约没有内部中间对象的数据仍被引用。

来源

2017-08-29 16:47:12

+0

感谢您的回答。你是正确的：在代码运行后测量正在使用的内存，我看它稳定在77。5MB，所以Net /协议分配的所有内存都会非常快速地得到GC。 –

相关问题

 相关问题