Skip Menu |
 

This queue is for tickets about the ParallelUserAgent CPAN distribution.

Report information
The Basics
Id: 33137
Status: new
Priority: 0/
Queue: ParallelUserAgent

People
Owner: Nobody in particular
Requestors: david_v_wright [...] yahoo.com
Cc:
AdminCc:

Bug Information
Severity: Normal
Broken in: 2.57
Fixed in: (no value)



Subject: Out of memory! for Larger number of requests (4_000_000)
Download (untitled) / with headers
text/plain 5.7k
Apologies, here is the body to "Out of memory! for Larger number of requests (4_000_000)" bug report. Issue: Programs using ParallelUserAgent-2.57 for large numbers of requests, consume all available memory of the OS (linux in this case) and die with 'Out of memory!' ('large' threshold depends on your hardware/memory constraints of course) Environment: ParallelUserAgent-2.57 perl -v This is perl, v5.8.6 built for i686-linux-64int uname -a Linux myhost 2.6.9-34.0.2.EL #1 Fri Jun 30 10:23:19 EDT 2006 i686 free total used free shared buffers cached Mem: 4151308 796528 3354780 0 10328 150336 -/+ buffers/cache: 635864 3515444 Swap: 4096564 298668 3797896 To recreate 'Out of memory' message, modified one line of code in the t/http.t test: diff -Naur local/http.t local/http.t.outofmem --- local/http.t 2008-02-07 21:49:27.000000000 -0800 +++ local/http.t.outofmem 2008-02-07 22:02:21.000000000 -0800 @@ -5,7 +5,7 @@ $| = 1; # autoflush -$DEBUG = 0; +$DEBUG = 1; $NONBLOCK = 0; # set to 1 to try out non-blocking connects (new in 2.51) #use LWP::Debug qw(+debug +trace +conns); @@ -201,7 +201,7 @@ $ua->initialize; -for $i (0..40) { +for $i (0..4_000_000) { my $page = $i % 3; $req = new HTTP::Request GET => url("/page$page", $base); print STDERR "\tRegistering '".$req->url."'\n" if $DEBUG; command run: time perl local/http.t.outofmem truncated... Registering 'myhost:43998/page0' Registering 'myhost:43998/page1' Registering 'myhost:43998/page2' Registering 'myhost:43998/page0' Registering 'myhost:43998/page1' Out of memory! perl local/http.t.outofmem 770.83s user 22.72s system 41% cpu 31:33.20 total monitoring memory process: beginning OS state via top: top - 22:01:46 up 16 days, 6:36, 0 users, load average: 0.00, 0.06, 0.24 Tasks: 409 total, 1 running, 401 sleeping, 7 stopped, 0 zombie Cpu(s): 8.3% us, 0.4% sy, 0.0% ni, 90.5% id, 0.8% wa, 0.0% hi, 0.0% si Mem: 4151308k total, 1030128k used, 3121180k free, 45468k buffers Swap: 4096564k total, 298668k used, 3797896k free, 328572k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21924 dwright 15 0 2084 1076 664 R 3.8 0.0 0:00.03 top 1 root 16 0 3624 520 468 S 0.0 0.0 0:08.57 init during run: grep perl lwp_parallel_mem_usage |head 22495 dwright 15 0 11804 6984 2004 R 13.2 0.2 0:00.40 perl 22496 dwright 16 0 11172 5812 1860 S 4.6 0.1 0:00.14 perl 22495 dwright 16 0 14992 9.9m 2004 S 25.7 0.2 0:01.18 perl 22496 dwright 16 0 11172 5812 1860 S 0.0 0.1 0:00.14 perl 22495 dwright 25 0 24084 18m 2004 R 71.4 0.5 0:03.34 perl 22496 dwright 16 0 11172 5812 1860 S 0.0 0.1 0:00.14 perl 22495 dwright 25 0 36660 30m 2004 R 99.9 0.8 0:06.37 perl 22496 dwright 16 0 11172 5812 1860 S 0.0 0.1 0:00.14 perl 22495 dwright 25 0 48672 42m 2004 R 99.9 1.1 0:09.40 perl 22495 dwright 25 0 61972 55m 2004 R 99.8 1.4 0:12.42 perl grep perl lwp_parallel_mem_usage |less 22495 dwright 16 0 286m 279m 2004 S 28.7 6.9 1:08.93 perl 22495 dwright 15 0 289m 283m 2004 R 30.4 7.0 1:09.85 perl 22495 dwright 15 0 294m 287m 2004 S 36.0 7.1 1:10.94 perl 22495 dwright 16 0 297m 291m 2004 S 29.7 7.2 1:11.84 perl 22495 dwright 16 0 301m 294m 2004 R 29.4 7.3 1:12.73 perl 22495 dwright 16 0 304m 297m 2004 S 27.4 7.4 1:13.56 perl 22495 dwright 15 0 308m 302m 2004 R 35.7 7.5 1:14.64 perl 22495 dwright 16 0 313m 306m 2004 S 39.0 7.6 1:15.82 perl truncated... 22495 dwright 16 0 715m 706m 2004 S 48.9 17.4 2:58.01 perl 22495 dwright 15 0 720m 711m 2004 R 39.9 17.5 2:59.22 perl 22495 dwright 16 0 723m 714m 2004 S 29.1 17.6 3:00.10 perl 22495 dwright 16 0 727m 718m 2004 S 31.4 17.7 3:01.05 perl 22495 dwright 16 0 732m 723m 2004 S 44.9 17.9 3:02.41 perl 22495 dwright 16 0 735m 727m 2004 S 29.7 17.9 3:03.31 perl 22495 dwright 16 0 739m 731m 2004 S 34.4 18.0 3:04.35 perl 22495 dwright 16 0 743m 734m 2004 S 31.1 18.1 3:05.29 perl 22495 dwright 16 0 747m 738m 2004 S 32.7 18.2 3:06.28 perl truncated... 22495 dwright 16 0 1343m 1.3g 2004 S 36.3 32.8 5:36.54 perl 22495 dwright 16 0 1349m 1.3g 2004 S 45.3 33.0 5:37.91 perl 22495 dwright 15 0 1352m 1.3g 2004 R 30.1 33.1 5:38.82 perl 22495 dwright 16 0 1357m 1.3g 2004 S 41.2 33.2 5:40.07 perl 22495 dwright 15 0 1364m 1.3g 2004 R 54.5 33.3 5:41.72 perl 22495 dwright 16 0 1367m 1.3g 2004 S 31.7 33.4 5:42.68 perl 22495 dwright 16 0 1373m 1.3g 2004 S 46.3 33.6 5:44.08 perl 22495 dwright 15 0 1378m 1.3g 2004 R 45.3 33.7 5:45.45 perl 22495 dwright 16 0 1382m 1.3g 2004 S 38.6 33.8 5:46.62 perl 22495 dwright 16 0 1386m 1.3g 2004 S 30.1 33.9 5:47.53 perl grep perl lwp_parallel_mem_usage |tail 22495 dwright 21 0 3030m 2.9g 2004 R 99.9 74.2 12:44.21 perl 22495 dwright 25 0 3030m 2.9g 2004 R 99.8 74.2 12:47.23 perl 22495 dwright 25 0 3030m 2.9g 2004 R 99.9 74.2 12:50.26 perl 22495 dwright 25 0 3030m 2.9g 2004 R 99.9 74.2 12:53.29 perl 22495 dwright 25 0 3030m 2.9g 2004 R 99.9 74.2 12:56.32 perl 22495 dwright 25 0 3030m 2.9g 2004 R 99.8 74.2 12:59.34 perl 22495 dwright 25 0 3030m 2.9g 2004 R 99.9 74.2 13:02.37 perl 22495 dwright 25 0 3030m 2.9g 2004 R 99.8 74.2 13:05.39 perl 22495 dwright 25 0 3030m 2.9g 2004 R 99.9 74.2 13:08.42 perl 22495 dwright 25 0 3030m 2.9g 2004 R 99.9 74.2 13:11.45 perl a steady consistent consumption of memory until all consumed. Desired Result: regardless of number of requests, ParallelUserAgent will not consume all the memory of the box Thanks, David


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.