Skip Menu |
 

This queue is for tickets about the libwww-perl CPAN distribution.

Report information
The Basics
Id: 88287
Status: resolved
Priority: 0/
Queue: libwww-perl

People
Owner: Nobody in particular
Requestors: calvillo.net [...] me.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: Memory leak when calling certain HTTPS addresses
Date: Fri, 30 Aug 2013 09:27:50 -0500
To: bug-libwww-perl [...] rt.cpan.org
From: "Roberto G. Calvillo" <calvillo.net [...] me.com>
Download (untitled) / with headers
text/plain 304b
Hello, I think I found a memory leak when connecting to certain addresses over HTTPS. I'm using Perl 5.12.4 / LWP 6.05 / Mac OS X 10.8.4. I talked about the bug on Stack Overflow: http://stackoverflow.com/questions/12681857/memory-leak-with-perls-lwp-using-https Thank you. Your library is awesome!
Download (untitled) / with headers
text/plain 551b
On 2013-08-30 07:28:03, calvillo.net@me.com wrote: Show quoted text
> Hello, > > I think I found a memory leak when connecting to certain addresses > over HTTPS. I'm using Perl 5.12.4 / LWP 6.05 / Mac OS X 10.8.4. > > I talked about the bug on Stack Overflow: > > http://stackoverflow.com/questions/12681857/memory-leak-with-perls- > lwp-using-https > > Thank you. Your library is awesome!
Are you able to try this out with newer perls (5.16 or 5.18), and new LWP and SSL libraries? The answer in that thread suggests that the problem has already been solved.
Subject: Re: [rt.cpan.org #88287] Memory leak when calling certain HTTPS addresses
Date: Fri, 30 Aug 2013 12:09:29 -0500
To: bug-libwww-perl [...] rt.cpan.org
From: "Roberto G. Calvillo" <calvillo.net [...] me.com>
Download (untitled) / with headers
text/plain 1.5k
Hello Karen, Right now I'm in the middle of a project and it would be very risky for me to upgrade my Perl version. However, I did upgrade all of the LWP's dependencies, and the bug is still there. The bug is: If you call any URL via HTTP, there's no memory leak. If you call most URL's via HTTPS, there's no memory leak either. If you call some URL's via HTTPS, such as https://www.google.com, there's a very notorious memory leak. For testing, I create a simple script with this code: use LWP::UserAgent; $ua = LWP::UserAgent->new(); $request = HTTP::Request->new(GET=>'https://www.google.com/'); while (1) { $response = $ua->request($request); sleep(1); } Then I run it on the Terminal, open the Activity Monitor, and see the memory grow and grow. After a day or two memory usage goes up to 500 MB. On 30/08/2013, at 10:46, Karen Etheridge via RT <bug-libwww-perl@rt.cpan.org> wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=88287 > > > On 2013-08-30 07:28:03, calvillo.net@me.com wrote:
>> Hello, >> >> I think I found a memory leak when connecting to certain addresses >> over HTTPS. I'm using Perl 5.12.4 / LWP 6.05 / Mac OS X 10.8.4. >> >> I talked about the bug on Stack Overflow: >> >> http://stackoverflow.com/questions/12681857/memory-leak-with-perls- >> lwp-using-https >> >> Thank you. Your library is awesome!
> > > Are you able to try this out with newer perls (5.16 or 5.18), and new LWP and SSL libraries? The answer in that thread suggests that the problem has already been solved.
Download (untitled) / with headers
text/plain 1.7k
yep, seems it grows for google.com perl 5.12.2 Ubuntu, LWP 6.03. for other sites it does not grow. On Fri Aug 30 21:09:52 2013, calvillo.net@me.com wrote: Show quoted text
> Hello Karen, > > Right now I'm in the middle of a project and it would be very risky > for me to upgrade my Perl version. However, I did upgrade all of the > LWP's dependencies, and the bug is still there. > > The bug is: > > If you call any URL via HTTP, there's no memory leak. > If you call most URL's via HTTPS, there's no memory leak either. > If you call some URL's via HTTPS, such as https://www.google.com, > there's a very notorious memory leak. > > For testing, I create a simple script with this code: > > use LWP::UserAgent; > > $ua = LWP::UserAgent->new(); > $request = HTTP::Request->new(GET=>'https://www.google.com/'); > while (1) { > $response = $ua->request($request); > sleep(1); > } > Then I run it on the Terminal, open the Activity Monitor, and see the > memory grow and grow. After a day or two memory usage goes up to 500 > MB. > > > On 30/08/2013, at 10:46, Karen Etheridge via RT <bug-libwww- > perl@rt.cpan.org> wrote: >
> > <URL: https://rt.cpan.org/Ticket/Display.html?id=88287 > > > > > On 2013-08-30 07:28:03, calvillo.net@me.com wrote:
> >> Hello, > >> > >> I think I found a memory leak when connecting to certain addresses > >> over HTTPS. I'm using Perl 5.12.4 / LWP 6.05 / Mac OS X 10.8.4. > >> > >> I talked about the bug on Stack Overflow: > >> > >> http://stackoverflow.com/questions/12681857/memory-leak-with-perls- > >> lwp-using-https > >> > >> Thank you. Your library is awesome!
> > > > > > Are you able to try this out with newer perls (5.16 or 5.18), and new > > LWP and SSL libraries? The answer in that thread suggests that the > > problem has already been solved.
Download (untitled) / with headers
text/plain 440b
I'm seeing this issue with api.fastly.com and www.fastly.com. I see it on linux amd64 machines. One runs perl 5.14.2 with LWP/LWP::UserAgent 6.03 and HTTP::Request/HTTP::Request::Common 6.00 The other runs perl 5.18.2 with LWP/LWP::UserAgent 6.05 and HTTP::Request 6.00 and HTTP::Request::Common 6.04 Will try it in a chroot with fresh installs of the latest version of the entire suite to see if it still exists in the latest versions
Download (untitled) / with headers
text/plain 4.8k
I have verified that the issue exists with linux/amd64, perl 5.20.2, LWP/LWP::UserAgent 6.13 and HTTP::Request/HTTP::Request::Common 6.11. It only affects some sites, and it would appear google is unaffected now, but it does affect various other sites I care about. (ubuntu_15_10)root@chroot:/home/ski# for h in www.fastly.com www.thinkgeek.com www.google.com www.facebook.com www.cnn.com ; do perl -e 'use strict; use warnings; $|++; use LWP::UserAgent; use HTTP::Request::Common; for(0..499) { my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new( GET => "https://'$h'/foo" ); $ua->request( $req ); }' & sleep 2 ; ps auxww | grep LWP::UserAgent | grep -v grep | awk '{ print "before, host: $h, VSZ: " $5 ", RSS: " $6 }' ; sleep 15 ; ps auxww | grep LWP::UserAgent | grep -v grep | awk '{ print "after, host: $h, VSZ: " $5 ", RSS: " $6 }' ; kill %1 ; sleep 1 ; done [1] 5116 before, host: $h, VSZ: 74852, RSS: 25712 after, host: $h, VSZ: 75504, RSS: 26400 [1]+ Terminated perl -e 'use strict; use warnings; $|++; use LWP::UserAgent; use HTTP::Request::Common; for(0..499) { my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new( GET => "https://'$h'/foo" ); $ua->request( $req ); }' [1] 5129 before, host: $h, VSZ: 70724, RSS: 23504 after, host: $h, VSZ: 70728, RSS: 23888 [1]+ Terminated perl -e 'use strict; use warnings; $|++; use LWP::UserAgent; use HTTP::Request::Common; for(0..499) { my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new( GET => "https://'$h'/foo" ); $ua->request( $req ); }' [1] 5141 before, host: $h, VSZ: 70708, RSS: 23752 after, host: $h, VSZ: 70708, RSS: 23752 [1]+ Terminated perl -e 'use strict; use warnings; $|++; use LWP::UserAgent; use HTTP::Request::Common; for(0..499) { my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new( GET => "https://'$h'/foo" ); $ua->request( $req ); }' [1] 5154 before, host: $h, VSZ: 74820, RSS: 25796 after, host: $h, VSZ: 74820, RSS: 25852 [1]+ Terminated perl -e 'use strict; use warnings; $|++; use LWP::UserAgent; use HTTP::Request::Common; for(0..499) { my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new( GET => "https://'$h'/foo" ); $ua->request( $req ); }' [1] 5166 before, host: $h, VSZ: 59484, RSS: 20500 after, host: $h, VSZ: 60008, RSS: 21140 [1]+ Terminated perl -e 'use strict; use warnings; $|++; use LWP::UserAgent; use HTTP::Request::Common; for(0..499) { my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new( GET => "https://'$h'/foo" ); $ua->request( $req ); }' With perl 5.18.2, LWP/LWP::UserAgent 6.05, HTTP::Request 6.00 and HTTP::Request::Common 6.04: ski@ubuntu14.04:~$ for h in www.fastly.com www.thinkgeek.com www.google.com www.facebook.com www.cnn.com ; do perl -e 'use strict; use warnings; $|++; use LWP::UserAgent; use HTTP::Request::Common; for(0..499) { my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new( GET => "https://'$h'/foo" ); $ua->request( $req ); }' & sleep 2 ; ps auxww | grep LWP::UserAgent | grep -v grep | awk '{ print "before, host: $h, VSZ: " $5 ", RSS: " $6 }' ; sleep 15 ; ps auxww | grep LWP::UserAgent | grep -v grep | awk '{ print "after, host: $h, VSZ: " $5 ", RSS: " $6 }' ; kill %1 ; sleep 1 ; done [1] 5187 before, host: $h, VSZ: 79544, RSS: 21908 after, host: $h, VSZ: 79868, RSS: 22256 [1]+ Terminated perl -e 'use strict; use warnings; $|++; use LWP::UserAgent; use HTTP::Request::Common; for(0..499) { my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new( GET => "https://'$h'/foo" ); $ua->request( $req ); }' [1] 5199 before, host: $h, VSZ: 79528, RSS: 21780 after, host: $h, VSZ: 79544, RSS: 21820 [1]+ Terminated perl -e 'use strict; use warnings; $|++; use LWP::UserAgent; use HTTP::Request::Common; for(0..499) { my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new( GET => "https://'$h'/foo" ); $ua->request( $req ); }' [1] 5218 before, host: $h, VSZ: 79488, RSS: 21808 after, host: $h, VSZ: 79488, RSS: 21816 [1]+ Terminated perl -e 'use strict; use warnings; $|++; use LWP::UserAgent; use HTTP::Request::Common; for(0..499) { my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new( GET => "https://'$h'/foo" ); $ua->request( $req ); }' [1] 5231 before, host: $h, VSZ: 79480, RSS: 21720 after, host: $h, VSZ: 79560, RSS: 21772 [1]+ Terminated perl -e 'use strict; use warnings; $|++; use LWP::UserAgent; use HTTP::Request::Common; for(0..499) { my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new( GET => "https://'$h'/foo" ); $ua->request( $req ); }' [1] 5244 before, host: $h, VSZ: 70452, RSS: 18868 after, host: $h, VSZ: 71368, RSS: 19732 [1]+ Terminated perl -e 'use strict; use warnings; $|++; use LWP::UserAgent; use HTTP::Request::Common; for(0..499) { my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new( GET => "https://'$h'/foo" ); $ua->request( $req ); }'
Subject: [rt.cpan.org #88287]
Date: Mon, 22 Feb 2016 15:50:58 +0100
To: bug-libwww-perl [...] rt.cpan.org
From: Marc Ballarin <marc.ballarin [...] 1und1.de>
Download (untitled) / with headers
text/plain 2.1k
Hi, I can reproduce this reliably using internal servers using something simple like: #!/usr/bin/perl -w use strict; use LWP::UserAgent; my $ua = LWP::UserAgent->new(); my $request = HTTP::Request->new(GET => 'https://...'); for my $count (1..100) { my $response = $ua->request($request); } Reproducible on Debian Jessie (Perl 5.20, ) and Stretch (Perl 5.22). NOT reproducible on Wheezy (Perl 5.14) or an all three versions when using Net::SSLeay's "get_https" (without validation). Memory::Usage gives the following output: time vsz ( diff) rss ( diff) shared ( diff) code ( diff) data ( diff) 0 33376 ( 33376) 9564 ( 9564) 3992 ( 3992) 8 ( 8) 6004 ( 6004) before requests 0 71664 ( 38288) 23240 ( 13676) 6924 ( 2932) 8 ( 0) 16372 ( 10368) after 1 requests 0 71664 ( 0) 23284 ( 44) 6924 ( 0) 8 ( 0) 16372 ( 0) after 2 requests 0 71664 ( 0) 23368 ( 84) 6924 ( 0) 8 ( 0) 16372 ( 0) after 3 requests 0 71776 ( 112) 23420 ( 52) 6924 ( 0) 8 ( 0) 16484 ( 112) after 4 requests 1 71776 ( 0) 23420 ( 0) 6924 ( 0) 8 ( 0) 16484 ( 0) after 5 requests 1 71776 ( 0) 23452 ( 32) 6924 ( 0) 8 ( 0) 16484 ( 0) after 6 requests 1 71776 ( 0) 23480 ( 28) 6924 ( 0) 8 ( 0) 16484 ( 0) after 7 requests 1 71888 ( 112) 23512 ( 32) 6924 ( 0) 8 ( 0) 16596 ( 112) after 8 requests 1 71888 ( 0) 23532 ( 20) 6924 ( 0) 8 ( 0) 16596 ( 0) after 9 requests 1 71888 ( 0) 23564 ( 32) 6924 ( 0) 8 ( 0) 16596 ( 0) after 10 requests 1 71888 ( 0) 23592 ( 28) 6924 ( 0) 8 ( 0) 16596 ( 0) after 11 requests 1 72000 ( 112) 23644 ( 52) 6924 ( 0) 8 ( 0) 16708 ( 112) after 12 requests ... This pattern continues. Using keepalive can somewhat mitigate the issue: Memory is only leaked when the connection expires and a new one is reestablished.
Download (untitled) / with headers
text/plain 264b
Hi Everyone, This seems like it's related to the open issue in IO::Socket::SSL that has to do with OCSP stapling: https://rt.cpan.org/Public/Bug/Display.html?id=120643 I'd say we can close this out here as it's outside of the scope of libwww-perl. Thanks, Chase
Download (untitled) / with headers
text/plain 335b
On Fri Mar 31 13:28:23 2017, CAPOEIRAB wrote: Show quoted text
> Hi Everyone, > > This seems like it's related to the open issue in IO::Socket::SSL that > has to do with OCSP stapling: > https://rt.cpan.org/Public/Bug/Display.html?id=120643 > > I'd say we can close this out here as it's outside of the scope of > libwww-perl. > > Thanks, > Chase


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.