Skip Menu |
 

This queue is for tickets about the Tie-File CPAN distribution.

Report information
The Basics
Id: 3222
Status: open
Worked: 45 min
Priority: 0/
Queue: Tie-File

People
Owner: Nobody in particular
Requestors: pprasadb [...] planet-save.com
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in: 0.93
Fixed in: (no value)



Date: Mon, 28 Jul 2003 12:24:10 -0700 (PDT)
From: Prasad Poruporuthan <pprasadb [...] planet-save.com>
To: mjd-perl-tiefile+ [...] plover.com
Subject: using Tie::File
Download (untitled) / with headers
text/plain 936b
Hi mjd, I have written a Parser module and I am trying to use your Tie::File to handle large files... I was trying out some code like this, ======================================================= use Tie::File; use Devel::Size qw(total_size); my @array; tie @array, 'Tie::File', $ARGV[0], memory => 20000000, recsep => '~'; $size = total_size (\@array); print "diagnostics : total_size(\@array) = $size\n"; ======================================================= However, I found that even though I defined max cache use to 20,000,000, the Tie::File module was using a lot more memory. If I specify files as large 50 MB, it overflows my 256MB RAM, and even my 512 MB swap. Could u let me know, if I am doing something wrong. Your help would be appreciated. Thanks, Prasad Show quoted text
_____________________________________________________________ Save rainforest for free with a Planet-Save.com e-mail account: http://www.planet-save.com
To: bug-Tie-File [...] rt.cpan.org
Subject: Re: [cpan #3222] using Tie::File
Date: Tue, 12 Aug 2003 15:57:41 -0400
From: Mark Jason Dominus <mjd [...] plover.com>
RT-Send-Cc:
Download (untitled) / with headers
text/plain 1.1k
Show quoted text
> However, I found that even though I defined max cache use to 20,000,000, > the Tie::File module was using a lot more memory. > > If I specify files as large 50 MB, it overflows my 256MB RAM, > and even my 512 MB swap. > > Could u let me know, if I am doing something wrong.
I'm not sure, but my guess is that it is a problem with the module, not with your code. The manual says: The "memory" value is not an absolute or exact limit on the memory used. "Tie::File" objects contains some structures besides the read cache and the deferred write buffer, whose sizes are not charged against "memory". If your file has many lines, it's possible that this overhead is very large. Can you please tell me how many lines are in your 50MB file? I should probably add that whatever you're trying to use Devel::Size for here, it probably won't work. Using Devel::Size on a tied array only tells you how much space the array is actually using. Since none of the file data or anything else is stored in the array, Devel::Size will report a very small size. If you want to know how much space Tie::File is using, you probably should do total_size(tied @array); instead.
To: bug-Tie-File [...] rt.cpan.org
Subject: Re: [cpan #3222] using Tie::File
Date: Tue, 12 Aug 2003 17:15:21 -0400
From: Mark Jason Dominus <mjd [...] plover.com>
RT-Send-Cc:
Download (untitled) / with headers
text/plain 927b
Show quoted text
> ======================================================= > use Tie::File; > use Devel::Size qw(total_size); > > my @array; > tie @array, 'Tie::File', $ARGV[0], memory => 20000000, recsep => '~'; > > $size = total_size (\@array); > print "diagnostics : total_size(\@array) = $size\n"; > ======================================================= > > However, I found that even though I defined max cache use to 20,000,000, > the Tie::File module was using a lot more memory. > > If I specify files as large 50 MB, it overflows my 256MB RAM, > and even my 512 MB swap.
It turns out that in your program above, Devel::Size wil load the entire file into memory. It doesn't surprise me that this program runs out of memory, but this is not a problem with Tie::File; it's a problem with the way Devel::Size is designed, or with the way you are using it. Do you also have problems when you use Tie::File without Devel::Size?
From: Prasad
Download (untitled) / with headers
text/plain 899b
Mark, I still have problems even if don't use the Devel::Size module. I am using a file with Size 200 MB approx Lines 10,000,000 approx each line length varies between 10 to 100 characters. code executed is use Tie::File; tie @array, 'Tie::File', $ARGV[0], memory => 20000000; $size = @array; Memory usage is about 230 MB. (130 MB Main memory and 100 MB swap). From what you pointed out, even though I have defined memory => 20MB, the Tie::File would use more memory for maintaining its internal structures. But here it is using 250 MB, which seems to be very large.So specifying the memory => 20MB is not really limiting the memory that the Tie::File is using. Is the additional memory that Tie::File dependant on the File size ? Is this behaviour expected ? If I use files > 500MB, the memory usage will still be more. What file sizes can Tie::File use efficiently ? Thanks Prasad


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.