Skip Menu |
 

This queue is for tickets about the Tie-Cache CPAN distribution.

Report information
The Basics
Id: 97508
Status: resolved
Priority: 0/
Queue: Tie-Cache

People
Owner: Nobody in particular
Requestors: Ralf.Neubauer [...] wido.bv.aok.de
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: 0.21



Subject: Tie::Cache together with threads crashes perl
Date: Fri, 25 Jul 2014 18:15:06 +0000
To: "bug-Tie-Cache [...] rt.cpan.org" <bug-Tie-Cache [...] rt.cpan.org>
From: "Neubauer, Ralf" <ralf.neubauer [...] wido.bv.aok.de>
Download (untitled) / with headers
text/plain 1.6k
Hi. This sometimes crashes perl under Linux (debian 7.6 i386, packages perl 5.14.2-21+deb7u1 and libtie-cache-perl 0.17-4): perl -e "use threads; use Tie::Cache; tie %c, 'Tie::Cache', 100000; $c{$_} = 1 for 1..21825; (async { print 'Hallo' })->join;" Speicherzugriffsfehler ("memory access error", may be the translation of "segmentation fault"). And this always crashes: perl -e "use threads; use Tie::Cache; tie %c, 'Tie::Cache', 100000; $c{$_} = 1 for 1..30000; (async { print 'Hallo' })->join;" This crashes perl under Windows 7 64bit (Tie::Cache 0.21 with the stable strawberry-perl-5.18.2.2-64bit.msi and Tie::Cache 0.19 with ActivePerl 32 bit v5.14.2 build 1402): perl -e "use threads; use Tie::Cache; tie %c, 'Tie::Cache', 100000; $c{$_} = 1 for 1..4784; (async { print 'Hallo' })->join;" Problemsignatur: Problemereignisname: APPCRASH Anwendungsname: perl.exe Anwendungsversion: 5.18.2.2 Anwendungszeitstempel: 534d2a2b Fehlermodulname: ntdll.dll Fehlermodulversion: 6.1.7601.18247 Fehlermodulzeitstempel: 521eaf24 Ausnahmecode: c00000fd Ausnahmeoffset: 0000000000026062 Betriebsystemversion: 6.1.7601.2.1.0.256.48 Gebietsschema-ID: 1031 Zusatzinformation 1: 87c3 Zusatzinformation 2: 87c3633f67ee1f0fe9a9859875d91123 Zusatzinformation 3: e9ed Zusatzinformation 4: e9ed7338a06dde530f696acafd450751 But with one hash entry less it works: perl -e "use threads; use Tie::Cache; tie %c, 'Tie::Cache', 100000; $c{$_} = 1 for 1..4783; (async { print 'Hallo' })->join;" As Tie::Cache is pure perl, this may be a bug in perl's memory manager, but I only encountered it with Tie::Cache so far. Ralf
Download (untitled) / with headers
text/plain 275b
On 2014-07-25 14:15:21, Ralf.Neubauer@wido.bv.aok.de wrote: Show quoted text
> As Tie::Cache is pure perl, this may be a bug in perl's memory > manager, but I only encountered it with Tie::Cache so far.
The issue is that Tie::Cache is not threads safe. See 'perldoc threads' for more info.
RT-Send-CC: Ralf.Neubauer [...] wido.bv.aok.de
I agree there is a bug here. Sadly I suspect you are right that it is mostly likely in perl's memory management with threads, which I suspect since the segfault gets clear as the range in this line increases: $c{$_} = 1 for 1..30000 30000 seems to be the number that drives this. At 10000 and 20000 I was able to run this fine. Internally the memory use and structures of the Tie::Cache grow as this 30000 range increases and more of the Tie::Cache gets used. Interestingly enough if the size of the Tie::Cache is constrained to smaller size, this code runs: perl -e "use threads; use Tie::Cache; tie %c, 'Tie::Cache', 10000; for(1..30000) { \$c{\$_} = 1 }; (async { print 'Hallo' })->join;" So the Tie::Cache object in this case cannot grow past 10000 elements, so 30000 writes to it does not increase it to 30000 elements. This points further to some thread size memory management issue. I suspect this has to do with a thread stack size running out of space. However I could not prove this, as when I tried to increase thread stack size the problem was not fixed. If its the case, then Tie::Cache might be using upwards of 1K per entry in threaded mode so it seems this is not a good solution under perl threads. On Fri Jul 25 14:15:21 2014, Ralf.Neubauer@wido.bv.aok.de wrote: Show quoted text
> Hi. > > This sometimes crashes perl under Linux (debian 7.6 i386, packages > perl 5.14.2-21+deb7u1 and libtie-cache-perl 0.17-4): > perl -e "use threads; use Tie::Cache; tie %c, 'Tie::Cache', 100000; > $c{$_} = 1 for 1..21825; (async { print 'Hallo' })->join;" > Speicherzugriffsfehler ("memory access error", may be the translation > of "segmentation fault"). > > And this always crashes: > perl -e "use threads; use Tie::Cache; tie %c, 'Tie::Cache', 100000; > $c{$_} = 1 for 1..30000; (async { print 'Hallo' })->join;" > > > This crashes perl under Windows 7 64bit (Tie::Cache 0.21 with the > stable strawberry-perl-5.18.2.2-64bit.msi and Tie::Cache 0.19 with > ActivePerl 32 bit v5.14.2 build 1402): > perl -e "use threads; use Tie::Cache; tie %c, 'Tie::Cache', 100000; > $c{$_} = 1 for 1..4784; (async { print 'Hallo' })->join;" > > Problemsignatur: > Problemereignisname: APPCRASH > Anwendungsname: perl.exe > Anwendungsversion: 5.18.2.2 > Anwendungszeitstempel: 534d2a2b > Fehlermodulname: ntdll.dll > Fehlermodulversion: 6.1.7601.18247 > Fehlermodulzeitstempel: 521eaf24 > Ausnahmecode: c00000fd > Ausnahmeoffset: 0000000000026062 > Betriebsystemversion: 6.1.7601.2.1.0.256.48 > Gebietsschema-ID: 1031 > Zusatzinformation 1: 87c3 > Zusatzinformation 2: 87c3633f67ee1f0fe9a9859875d91123 > Zusatzinformation 3: e9ed > Zusatzinformation 4: e9ed7338a06dde530f696acafd450751 > > But with one hash entry less it works: > > perl -e "use threads; use Tie::Cache; tie %c, 'Tie::Cache', 100000; > $c{$_} = 1 for 1..4783; (async { print 'Hallo' })->join;" > > > As Tie::Cache is pure perl, this may be a bug in perl's memory > manager, but I only encountered it with Tie::Cache so far. > > > Ralf
Download (untitled) / with headers
text/plain 400b
On 2014-07-25 15:15:25, JDHEDDEN wrote: Show quoted text
> On 2014-07-25 14:15:21, Ralf.Neubauer@wido.bv.aok.de wrote: >
> > As Tie::Cache is pure perl, this may be a bug in perl's memory > > manager, but I only encountered it with Tie::Cache so far.
> > The issue is that Tie::Cache is not threads safe. See 'perldoc > threads' for more info.
My error. This is an out-of-memory issue; not a threads safe issue.
Download (untitled) / with headers
text/plain 853b
On Fri Jul 25 16:56:49 2014, JDHEDDEN wrote: Show quoted text
> On 2014-07-25 15:15:25, JDHEDDEN wrote:
> > On 2014-07-25 14:15:21, Ralf.Neubauer@wido.bv.aok.de wrote: > >
> > > As Tie::Cache is pure perl, this may be a bug in perl's memory > > > manager, but I only encountered it with Tie::Cache so far.
> > > > The issue is that Tie::Cache is not threads safe. See 'perldoc > > threads' for more info.
> > My error. This is an out-of-memory issue; not a threads safe issue.
I believe this is a thread + memory problem... This program runs fine for me outside of a thread context with a larger cache: perl -e "use Tie::Cache; tie %c, 'Tie::Cache', 200000; for(1..1000000) { \$c{\$_} = 1 }; " But there is no thread stack size issue in this case. Do note that the memory overhead here is significant and the program used around 100M memory for 200K has size.
Subject: [rt.cpan.org #97508] trying to simplify...
Date: Sat, 26 Jul 2014 10:36:10 +0200
To: bug-Tie-Cache [...] rt.cpan.org
From: Ralf Neubauer <ralf [...] strcmp.de>
Download (untitled) / with headers
text/plain 1.7k
I tried to simplify the test case. I started with: perl -e 'use threads; use Tie::Cache; tie %c, "Tie::Cache", 200000; for(1..30000) { $c{$_} = 1 }; (async { print qq{Hello\n} })->join' Segmentation fault But with a my (note %c is in scope in the async block): perl -e 'use threads; use Tie::Cache; my %c; tie %c, "Tie::Cache", 200000; for(1..30000) { $c{$_} = 1 }; (async { print qq{Hello\n} })->join' Hello Extracting just the Tie::Cache variable and "unblessing": perl -e 'use Storable; use Tie::Cache; $t; { tie my %c, "Tie::Cache", 200000; for(1..30000) { $c{$_} = 1 }; $t = { %{tied %c}, class => undef }; } print $t; store $t, "/tmp/tiecache.dump" and print qq{Ok\n}' HASH(0xa3b42d0)Ok perl -e 'use Storable; my $t = retrieve "/tmp/tiecache.dump";' Segmentation fault But: perl -e 'use Storable; use Tie::Cache; $t; { tie my %c, "Tie::Cache", 200000; for(1..20000) { $c{$_} = 1 }; $t = { %{tied %c}, class => undef }; } print $t; store $t, "/tmp/tiecache.dump" and print qq{Ok\n}' HASH(0x90a9a78)Ok perl -e 'use Storable; my $t = retrieve "/tmp/tiecache.dump"; print qq{Ok\n}' Ok I tried to use Data::Dump or Data::Dumper instead of Storable, but with element counts as big as 20000 i only get out of memory errors. With 20 the data structure looks fine, though. Note there is no reference to threads in the last test. Tie::Cache created the structure, but there is no reference to Tie::Cache in the dump file -- it is just a boring perl data structure. Of course this could simply be a bug in Storable. Why does threads crash with the same data size as Storable? Does threads use Storable internally? The next step would be to create the hash+doubly linked list in a simple loop without using any modules and test starting threads or retrieving this simplified structure. Ralf
Subject: [rt.cpan.org #97508] simplified even more
Date: Sat, 26 Jul 2014 10:57:20 +0200
To: bug-Tie-Cache [...] rt.cpan.org
From: Ralf Neubauer <ralf [...] strcmp.de>
Download (untitled) / with headers
text/plain 2.3k
perl -e 'my @a = map [ $_, 0, 1, undef, undef ], 0..30000; for (0..$#a-1) { $a[$_+1][3] = $a[$_]; $a[$_][4] = $a[$_+1] }; my %h = map +($_->[0] => $_), @a; %c = (node => \%h, head => $a[0], tail => $a[$#a]); use threads; (async { print qq{Hello\n} })->join' Segmentation fault perl -e 'my @a = map [ $_, 0, 1, undef, undef ], 0..20000; for (0..$#a-1) { $a[$_+1][3] = $a[$_]; $a[$_][4] = $a[$_+1] }; my %h = map +($_->[0] => $_), @a; %c = (node => \%h, head => $a[0], tail => $a[$#a]); use threads; (async { print qq{Hello\n} })->join' Hello perl -e 'my @a = map [ $_, 0, 1, undef, undef ], 0..30000; for (0..$#a-1) { $a[$_+1][3] = $a[$_]; $a[$_][4] = $a[$_+1] }; my %h = map +($_->[0] => $_), @a; %c = (node => \%h, head => $a[0], tail => $a[$#a]); use Storable qw(dclone); dclone \%c; print qq{Hello\n}' Segmentation fault perl -e 'my @a = map [ $_, 0, 1, undef, undef ], 0..20000; for (0..$#a-1) { $a[$_+1][3] = $a[$_]; $a[$_][4] = $a[$_+1] }; my %h = map +($_->[0] => $_), @a; %c = (node => \%h, head => $a[0], tail => $a[$#a]); use Storable qw(dclone); dclone \%c; print qq{Hello\n}' Hello This is with the rather ancient perl 5.14.2-21+deb7u1 i386 on debian. With perl 5.18.2-2ubuntu1 amd64 on ubuntu I get: perl -e 'my @a = map [ $_, 0, 1, undef, undef ], 0..20000; for (0..$#a-1) { $a[$_+1][3] = $a[$_]; $a[$_][4] = $a[$_+1] }; my %h = map +($_->[0] => $_), @a; %c = (node => \%h, head => $a[0], tail => $a[$#a]); use Storable qw(dclone); dclone \%c; print qq{Hello\n}' Hello perl -e 'my @a = map [ $_, 0, 1, undef, undef ], 0..30000; for (0..$#a-1) { $a[$_+1][3] = $a[$_]; $a[$_][4] = $a[$_+1] }; my %h = map +($_->[0] => $_), @a; %c = (node => \%h, head => $a[0], tail => $a[$#a]); use Storable qw(dclone); dclone \%c; print qq{Hello\n}' Segmentation fault (core dumped) perl -e 'my @a = map [ $_, 0, 1, undef, undef ], 0..30000; for (0..$#a-1) { $a[$_+1][3] = $a[$_]; $a[$_][4] = $a[$_+1] }; my %h = map +($_->[0] => $_), @a; %c = (node => \%h, head => $a[0], tail => $a[$#a]); use threads; (async { print qq{Hello\n} })->join' Hello perl -e 'my @a = map [ $_, 0, 1, undef, undef ], 0..40000; for (0..$#a-1) { $a[$_+1][3] = $a[$_]; $a[$_][4] = $a[$_+1] }; my %h = map +($_->[0] => $_), @a; %c = (node => \%h, head => $a[0], tail => $a[$#a]); use threads; (async { print qq{Hello\n} })->join' Segmentation fault (core dumped) Note I had to raise the number of elements a bit for the last test.
Subject: [rt.cpan.org #97508] simplified even more...
Date: Sat, 26 Jul 2014 11:16:13 +0200
To: bug-Tie-Cache [...] rt.cpan.org
From: Ralf Neubauer <ralf [...] strcmp.de>
perl 5.18.2-2ubuntu1 amd64: perl -e 'my @a = map [ undef, undef ], 0..40000; for (0..$#a-1) { $a[$_+1][0] = $a[$_]; $a[$_][1] = $a[$_+1] }; use Storable qw(dclone); dclone $a[0]; print qq{Hello\n}' Segmentation fault (core dumped) perl -e 'my @a = map [ undef, undef ], 0..20000; for (0..$#a-1) { $a[$_+1][0] = $a[$_]; $a[$_][1] = $a[$_+1] }; use Storable qw(dclone); dclone $a[0]; print qq{Hello\n}' Hello perl -e 'use threads; my @a = map [ undef, undef ], 0..40000; for (0..$#a-1) { $a[$_+1][0] = $a[$_]; $a[$_][1] = $a[$_+1] }; (async { print qq{Hello\n} })->join' Segmentation fault (core dumped) perl -e 'use threads; my @a = map [ undef, undef ], 0..20000; for (0..$#a-1) { $a[$_+1][0] = $a[$_]; $a[$_][1] = $a[$_+1] }; (async { print qq{Hello\n} })->join' Hello So it's not a genuine bug in Tie::Cache, but how and where can this bug be reassigned? Is it a memory management bug or just a bug in the copying routines? The data structure is not _that_ uncommon -- if it's not a linear list but a more complicated graph, you can't emulate it with arrays.
Subject: [rt.cpan.org #97508] shorter...
Date: Sat, 26 Jul 2014 12:43:44 +0200
To: bug-Tie-Cache [...] rt.cpan.org
From: Ralf Neubauer <ralf [...] strcmp.de>
$ perl -e 'use threads; $t = $t->[1] = [$t] for 1..40000; (async { print qq{Hello\n} })->join' Segmentation fault (core dumped) $ perl -e 'use Storable qw(dclone); $t = $t->[1] = [$t] for 1..40000; dclone $t' Segmentation fault (core dumped) $ perl -e 'use Clone qw(clone); $t = $t->[1] = [$t] for 1..40000; clone $t' Segmentation fault (core dumped) $ ulimit -s 8192 $ ulimit -s 16384 $ perl -e 'use Clone qw(clone); $t = $t->[1] = [$t] for 1..40000; clone $t' $ perl -e 'use Clone qw(clone); $t = $t->[1] = [$t] for 1..80000; clone $t' Segmentation fault (core dumped) If for some reason no non-recursive algorithm is possible, an error message 'clone: out of memory' or 'recursion limit reached' would be much nicer than a segfault, which gives no hint about which part of the program crashed (took a while to find out the async {} in a big program crashed because of some object I allocated and filled somewhere completely else). Ok, this is completely off topic for Tie::Cache... Just wanted to write this down for whoever is interested in this bug.
Subject: [rt.cpan.org #97508] even shorter...
Date: Sat, 26 Jul 2014 15:16:41 +0200
To: bug-Tie-Cache [...] rt.cpan.org
From: Ralf Neubauer <ralf [...] strcmp.de>
Download (untitled) / with headers
text/plain 356b
$ perl -e 'use Clone qw(clone); $t = [$t] for 1..40000; clone $t' Segmentation fault (core dumped) $ perl -e 'use Storable qw(dclone); $t = [$t] for 1..30000; dclone $t' Segmentation fault (core dumped) $ perl -e 'use threads; $t = [$t] for 1..30000; (async {})->join' Segmentation fault (core dumped) I only wonder why I never encountered this before...
Download (untitled) / with headers
text/plain 172b
This old bug report seems to be resolved with more recent versions of perl! No longer and issue, and not a bug with Tie::Cache in any case as was documented in bug report.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.