Skip Menu |
 

This queue is for tickets about the File-Slurp CPAN distribution.

Report information
The Basics
Id: 38640
Status: resolved
Priority: 0/
Queue: File-Slurp

People
Owner: Nobody in particular
Requestors: cpan [...] pjedwards.co.uk
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: 9999.13
Fixed in: (no value)



Subject: Note: memory not returned to OS.
MIME-Version: 1.0
X-Mailer: MIME-tools 5.426 (Entity 5.426)
Content-Type: text/plain
Charset: utf8
Content-Disposition: inline
Content-Transfer-Encoding: binary
X-RT-Original-Encoding: utf-8
Content-Length: 702
Download (untitled) / with headers
text/plain 702b
Hello and thanks for File::Slurp. I was using File::Slurp with some large files and I noticed memory wasn't being returned to the OS when scalars fell out of scope, I thought it worth mentioning, and was thinking it would be useful in the docs. perl -e "use strict;use File::Slurp;sub foo{print system(q{ps aux | grep -E 'BIG|MEM'}).qq{\n}};foo();my \$t=File::Slurp::read_file ('BIG_file.txt');foo();undef(\$t);foo();" %MEM 0.0 %MEM 5.8 %MEM 2.9 Compared to: perl -e "use strict;sub foo{print system(q{ps aux | grep - E 'BIG|MEM'}).qq{\n}};foo();my \$t;{local \$/;open(my \$fh,'BIG_file.txt');\$t=<\$fh>;};foo();undef(\$t);foo();" %MEM 0.0 %MEM 2.9 %MEM 0.0 Cheers, Peter (Stig) Edwards
MIME-Version: 1.0
X-Mailer: MIME-tools 5.426 (Entity 5.426)
Content-Disposition: inline
Charset: utf8
Message-Id: <rt-3.6.HEAD-28921-1219332904-1703.38640-0-0 [...] rt.cpan.org>
Content-Type: text/plain
Content-Transfer-Encoding: binary
X-RT-Original-Encoding: utf-8
Content-Length: 86
CORRECTION, it's not when scalars fell out of scope, but when I call undef() on them.
MIME-Version: 1.0
X-Mailer: MIME-tools 5.426 (Entity 5.426)
Content-Disposition: inline
Charset: utf8
Message-Id: <rt-3.6.HEAD-10654-1225046614-1388.38640-0-0 [...] rt.cpan.org>
Content-Type: text/plain
Content-Transfer-Encoding: binary
X-RT-Original-Encoding: utf-8
Content-Length: 615
Download (untitled) / with headers
text/plain 615b
On Thu Aug 21 11:22:33 2008, cpan@pjedwards.co.uk wrote: Show quoted text
> Hello and thanks for File::Slurp. > > I was using File::Slurp with some large files and I noticed memory > wasn't being returned to the OS when scalars fell out of scope, I > thought it worth mentioning, and was thinking it would be useful in the > docs.
This is just how Perl works. It never returns memory that it allocates to the OS. This can make slurping problematic, but it's not specific to this module. Anything that loads a lot of data into memory could be a problem. I suppose a note in the docs wouldn't hurt, but it's not module-specific.
MIME-Version: 1.0
In-Reply-To: <rt-3.6.HEAD-10654-1225046614-1388.38640-0-0 [...] rt.cpan.org>
X-Mailer: MIME-tools 5.426 (Entity 5.426)
Content-Disposition: inline
Charset: utf8
References: <rt-3.6.HEAD-10654-1225046614-1388.38640-0-0 [...] rt.cpan.org>
Message-Id: <rt-3.6.HEAD-22741-1225735526-303.38640-0-0 [...] rt.cpan.org>
Content-Type: text/plain
Content-Transfer-Encoding: binary
X-RT-Original-Encoding: utf-8
Content-Length: 1479
Download (untitled) / with headers
text/plain 1.4k
On Sun Oct 26 14:43:34 2008, DROLSKY wrote: Show quoted text
> On Thu Aug 21 11:22:33 2008, cpan@pjedwards.co.uk wrote:
> > Hello and thanks for File::Slurp. > > > > I was using File::Slurp with some large files and I noticed memory > > wasn't being returned to the OS when scalars fell out of scope, I > > thought it worth mentioning, and was thinking it would be useful in
the Show quoted text
> > docs.
> > This is just how Perl works. It never returns memory that it allocates > to the OS. This can make slurping problematic, but it's not specific
to Show quoted text
> this module. Anything that loads a lot of data into memory could be a > problem. > > I suppose a note in the docs wouldn't hurt, but it's not module-
specific. Thanks for the reply, here are some more tests I performed. I thought they might be useful. Using buf_ref avoids the $buf (line 80 of Slurp.pm in File-Slurp- 9999.13) being used, so for example: perl -e "use strict;use File::Slurp;sub foo{print system(q{ps aux | grep -E 'BIG|MEM'}).qq{\n}};foo();my \$t;File::Slurp::read_file ('BIG_file.txt',buf_ref=>\\\$t);foo();undef(\$t);foo();" %MEM 0.0 %MEM 2.9 %MEM 0.0 uses half the memory (as reported by ps %MEM) of $t=File::Slurp::read_file('BIG_file.txt') Also using scalar_ref=>1, uses less memory: perl -e "use strict;use File::Slurp;sub foo{print system(q{ps aux | grep -E 'BIG|MEM'}).qq{\n}};foo();my \$t=File::Slurp::read_file ('BIG_file.txt',scalar_ref=>1);foo();undef(\$t);foo();" %MEM 0.0 %MEM 2.9 %MEM 0.0 Cheers.
From uri [...] sysarch.com Thu Apr 22 01: 53:02 2010
CC: cpan [...] pjedwards.co.uk
MIME-Version: 1.0
X-Spam-Status: No, score=-6.599 tagged_above=-99.9 required=10 tests=[AWL=4.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] autolearn=ham
In-Reply-To: <rt-3.6.HEAD-10654-1225046614-1388.38640-5-0 [...] rt.cpan.org> (DROLSKY via RT's message of "Sun\, 26 Oct 2008 14\:43\:40 -0400")
X-Spam-Flag: NO
References: <RT-Ticket-38640 [...] rt.cpan.org> <rt-3.6.HEAD-10654-1225046614-1388.38640-5-0 [...] rt.cpan.org>
X-Virus-Scanned: Debian amavisd-new at bestpractical.com
Message-ID: <87mxwwnie6.fsf [...] quad.sysarch.com>
Content-Type: text/plain; charset="utf-8"
X-RT-Original-Encoding: utf-8
X-Spam-Score: -6.599
Received: from localhost (localhost [127.0.0.1]) by hipster.bestpractical.com (Postfix) with ESMTP id C0D322405C6 for <cpan-bug+File-Slurp [...] hipster.bestpractical.com>; Thu, 22 Apr 2010 01:53:02 -0400 (EDT)
Received: from hipster.bestpractical.com ([127.0.0.1]) by localhost (hipster.bestpractical.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kh8Nn57ktM-A for <cpan-bug+File-Slurp [...] hipster.bestpractical.com>; Thu, 22 Apr 2010 01:52:59 -0400 (EDT)
Received: from la.mx.develooper.com (x1.develooper.com [207.171.7.70]) by hipster.bestpractical.com (Postfix) with SMTP id 85B372405BB for <bug-File-Slurp [...] rt.cpan.org>; Thu, 22 Apr 2010 01:52:59 -0400 (EDT)
Received: (qmail 1062 invoked by uid 103); 22 Apr 2010 05:53:00 -0000
Received: from x16.dev (10.0.100.26) by x1.dev with QMQP; 22 Apr 2010 05:53:00 -0000
Received: from vms173005pub.verizon.net (HELO vms173005pub.verizon.net) (206.46.173.5) by 16.mx.develooper.com (qpsmtpd/0.80) with ESMTP; Wed, 21 Apr 2010 22:52:55 -0700
Received: from sysarch.com ([unknown] [173.48.104.108]) by vms173005.mailsrvcs.net (Sun Java(tm) System Messaging Server 7u2-7.02 32bit (built Apr 16 2009)) with ESMTPA id <0L19006NEKBLO463 [...] vms173005.mailsrvcs.net> for bug-File-Slurp [...] rt.cpan.org; Thu, 22 Apr 2010 00:52:38 -0500 (CDT)
Received: (qmail 2782 invoked by uid 1000); Thu, 22 Apr 2010 05:52:33 +0000
Delivered-To: cpan-bug+File-Slurp [...] hipster.bestpractical.com
Subject: Re: [rt.cpan.org #38640] Note: memory not returned to OS.
User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/22.2 (gnu/linux)
Return-Path: <uri [...] sysarch.com>
X-Spam-Check-BY: 16.mx.develooper.com
X-Original-To: cpan-bug+File-Slurp [...] hipster.bestpractical.com
X-RT-Mail-Extension: file-slurp
Date: Thu, 22 Apr 2010 01:52:33 -0400
X-Spam-Level:
To: bug-File-Slurp [...] rt.cpan.org
From: "Uri Guttman" <uri [...] StemSystems.com>
RT-Message-ID: <rt-3.8.HEAD-26540-1271915585-1127.38640-0-0 [...] rt.cpan.org>
Content-Length: 1318
Download (untitled) / with headers
text/plain 1.2k
Show quoted text
>>>>> "DvR" == DROLSKY via RT <bug-File-Slurp@rt.cpan.org> writes:
Show quoted text
DvR> Queue: File-Slurp DvR> Ticket <URL: http://rt.cpan.org/Ticket/Display.html?id=38640 >
Show quoted text
DvR> On Thu Aug 21 11:22:33 2008, cpan@pjedwards.co.uk wrote:
Show quoted text
>> Hello and thanks for File::Slurp. >> >> I was using File::Slurp with some large files and I noticed memory >> wasn't being returned to the OS when scalars fell out of scope, I >> thought it worth mentioning, and was thinking it would be useful in the >> docs.
Show quoted text
DvR> This is just how Perl works. It never returns memory that it allocates DvR> to the OS. This can make slurping problematic, but it's not specific to DvR> this module. Anything that loads a lot of data into memory could be a DvR> problem.
Show quoted text
DvR> I suppose a note in the docs wouldn't hurt, but it's not DvR> module-specific.
i have added notes to the pod for scalar_ref and buf_ref that using them will be faster and save ram usage by not doing an extra copy. i am working (finally) on a new release and catching up on many requests. thanx, uri -- Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.