Skip Menu |
 

This queue is for tickets about the File-Slurp CPAN distribution.

Report information
The Basics
Id: 38640
Status: resolved
Priority: 0/
Queue: File-Slurp

People
Owner: Nobody in particular
Requestors: cpan [...] pjedwards.co.uk
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: 9999.13
Fixed in: (no value)



Subject: Note: memory not returned to OS.
Download (untitled) / with headers
text/plain 702b
Hello and thanks for File::Slurp. I was using File::Slurp with some large files and I noticed memory wasn't being returned to the OS when scalars fell out of scope, I thought it worth mentioning, and was thinking it would be useful in the docs. perl -e "use strict;use File::Slurp;sub foo{print system(q{ps aux | grep -E 'BIG|MEM'}).qq{\n}};foo();my \$t=File::Slurp::read_file ('BIG_file.txt');foo();undef(\$t);foo();" %MEM 0.0 %MEM 5.8 %MEM 2.9 Compared to: perl -e "use strict;sub foo{print system(q{ps aux | grep - E 'BIG|MEM'}).qq{\n}};foo();my \$t;{local \$/;open(my \$fh,'BIG_file.txt');\$t=<\$fh>;};foo();undef(\$t);foo();" %MEM 0.0 %MEM 2.9 %MEM 0.0 Cheers, Peter (Stig) Edwards
CORRECTION, it's not when scalars fell out of scope, but when I call undef() on them.
Download (untitled) / with headers
text/plain 615b
On Thu Aug 21 11:22:33 2008, cpan@pjedwards.co.uk wrote: Show quoted text
> Hello and thanks for File::Slurp. > > I was using File::Slurp with some large files and I noticed memory > wasn't being returned to the OS when scalars fell out of scope, I > thought it worth mentioning, and was thinking it would be useful in the > docs.
This is just how Perl works. It never returns memory that it allocates to the OS. This can make slurping problematic, but it's not specific to this module. Anything that loads a lot of data into memory could be a problem. I suppose a note in the docs wouldn't hurt, but it's not module-specific.
Download (untitled) / with headers
text/plain 1.4k
On Sun Oct 26 14:43:34 2008, DROLSKY wrote: Show quoted text
> On Thu Aug 21 11:22:33 2008, cpan@pjedwards.co.uk wrote:
> > Hello and thanks for File::Slurp. > > > > I was using File::Slurp with some large files and I noticed memory > > wasn't being returned to the OS when scalars fell out of scope, I > > thought it worth mentioning, and was thinking it would be useful in
the Show quoted text
> > docs.
> > This is just how Perl works. It never returns memory that it allocates > to the OS. This can make slurping problematic, but it's not specific
to Show quoted text
> this module. Anything that loads a lot of data into memory could be a > problem. > > I suppose a note in the docs wouldn't hurt, but it's not module-
specific. Thanks for the reply, here are some more tests I performed. I thought they might be useful. Using buf_ref avoids the $buf (line 80 of Slurp.pm in File-Slurp- 9999.13) being used, so for example: perl -e "use strict;use File::Slurp;sub foo{print system(q{ps aux | grep -E 'BIG|MEM'}).qq{\n}};foo();my \$t;File::Slurp::read_file ('BIG_file.txt',buf_ref=>\\\$t);foo();undef(\$t);foo();" %MEM 0.0 %MEM 2.9 %MEM 0.0 uses half the memory (as reported by ps %MEM) of $t=File::Slurp::read_file('BIG_file.txt') Also using scalar_ref=>1, uses less memory: perl -e "use strict;use File::Slurp;sub foo{print system(q{ps aux | grep -E 'BIG|MEM'}).qq{\n}};foo();my \$t=File::Slurp::read_file ('BIG_file.txt',scalar_ref=>1);foo();undef(\$t);foo();" %MEM 0.0 %MEM 2.9 %MEM 0.0 Cheers.
CC: cpan [...] pjedwards.co.uk
Subject: Re: [rt.cpan.org #38640] Note: memory not returned to OS.
Date: Thu, 22 Apr 2010 01:52:33 -0400
To: bug-File-Slurp [...] rt.cpan.org
From: "Uri Guttman" <uri [...] StemSystems.com>
Download (untitled) / with headers
text/plain 1.2k
Show quoted text
>>>>> "DvR" == DROLSKY via RT <bug-File-Slurp@rt.cpan.org> writes:
Show quoted text
DvR> Queue: File-Slurp DvR> Ticket <URL: http://rt.cpan.org/Ticket/Display.html?id=38640 >
Show quoted text
DvR> On Thu Aug 21 11:22:33 2008, cpan@pjedwards.co.uk wrote:
Show quoted text
>> Hello and thanks for File::Slurp. >> >> I was using File::Slurp with some large files and I noticed memory >> wasn't being returned to the OS when scalars fell out of scope, I >> thought it worth mentioning, and was thinking it would be useful in the >> docs.
Show quoted text
DvR> This is just how Perl works. It never returns memory that it allocates DvR> to the OS. This can make slurping problematic, but it's not specific to DvR> this module. Anything that loads a lot of data into memory could be a DvR> problem.
Show quoted text
DvR> I suppose a note in the docs wouldn't hurt, but it's not DvR> module-specific.
i have added notes to the pod for scalar_ref and buf_ref that using them will be faster and save ram usage by not doing an extra copy. i am working (finally) on a new release and catching up on many requests. thanx, uri -- Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.