Skip Menu |
 

This queue is for tickets about the Memoize CPAN distribution.

Report information
The Basics
Id: 21707
Status: open
Priority: 0/
Queue: Memoize

People
Owner: Nobody in particular
Requestors: Josh-Engelman [...] deshaw.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



CC: <bug-Memoize [...] rt.cpan.org>
Subject: Memoize causes fatal errors in subsequent threads
Date: Mon, 25 Sep 2006 14:01:08 -0400
To: <mjd-perl-memoize [...] plover.com>
From: "Engelman, Josh" <Josh-Engelman [...] deshaw.com>
Hi Mark Jason. My name is Josh Engelman; I'm a software developer at D.E. Shaw & Co. We've noticed a problem with Memoize.pm when used with multiple threads in perl 5.8.6. I'm not sure if you're still maintaining this module; if there's someone else you think we should contact, please let me know. The problem seems to be that function addresses change in the new thread, but the %memotable and %revmemotable keys still refer to the old function addresses. This means that any attempt to use a memoized function from a second thread raises an exception that kills that thread. Here's a very simple demonstration: $ perl -e ' use threads; use Memoize; Memoize::memoize("foo"); sub foo { $_[0] } my $t1 = threads->new(\&foo, 1); printf "%d\n", $t1->join(); ' thread failed to start: Anonymous function called in forbidden scalar context; faulting at -e line 6 0 This is unfortunate, because it means that any use of a memoized function in *any* perl module makes that module unsafe to use in multi-threaded programs. Since Memoize.pm is part of the core perl distribution, this has far-reaching ramifications. I haven't been able to come up with a good way to make Memoize generically thread-safe -- doing so requires re-engineering of the way memotables work, or rebuilding them when the new thread is created. However, a very small patch will allow memoized functions to continue working in subsequent threads. Basically, the idea is to pass the serialized version of $cref to the _memoizer(), rather than passing the reference itself. The reference itself will have a different serialized form in subsequent threads, but we want to continue using the same %memotable key. Here's the patch diff: diff -c /usr/local/lib/perl5/site_perl/5.8.6/Memoize.pm Memoize.pm *** /usr/local/lib/perl5/site_perl/5.8.6/Memoize.pm Fri Jul 12 13:16:00 2002 --- Memoize.pm Mon Sep 25 13:21:24 2006 *************** *** 72,78 **** # goto version everywhere. my $wrapper = $Config{usethreads} ! ? eval "sub $proto { &_memoizer(\$cref, \@_); }" : eval "sub $proto { unshift \@_, \$cref; goto &_memoizer; }"; my $normalizer = $options{NORMALIZER}; --- 72,78 ---- # goto version everywhere. my $wrapper = $Config{usethreads} ! ? eval "sub $proto { &_memoizer('$cref', \@_); }" : eval "sub $proto { unshift \@_, \$cref; goto &_memoizer; }"; my $normalizer = $options{NORMALIZER}; Note that this solution will not fix unmemoize() in subsequent threads, nor address any complications caused by tied caches. But I believe it fixes the most common case. I've attached the full patched module along with a slightly more extensive test script. Note that I've tested both the problem and the solution with perl 5.8.6 on Solaris 10 x86, Solaris 10 sparc, Windows XP, <<Memoize.pm>> a <<break_memoize.pl>> nd MacOS 10.4. Please let me know if I can provide any further information or assistance. Thx. -josh === Josh Engelman Vice President, D.E. Shaw & Co. Ph. 212.478.0614 Fax 212.845.1614
Download Memoize.pm
text/x-perl 34.4k

Message body is not shown because sender requested not to inline it.

Download break_memoize.pl
text/x-perl 1.1k

Message body is not shown because sender requested not to inline it.

Subject: RE: [rt.cpan.org #21707] Memoize causes fatal errors in subsequent threads
Date: Mon, 25 Sep 2006 14:07:41 -0400
To: <bug-Memoize [...] rt.cpan.org>
From: "Engelman, Josh" <Josh-Engelman [...] deshaw.com>
Download (untitled) / with headers
text/plain 239b
An easier-to-read form of the diff: diff /usr/local/lib/perl5/site_perl/5.8.6/Memoize.pm Memoize.pm 75c75 < ? eval "sub $proto { &_memoizer(\$cref, \@_); }" --- Show quoted text
> ? eval "sub $proto { &_memoizer('$cref', \@_); }"
-josh
From: ted [...] tedcarnahan.com
Download (untitled) / with headers
text/plain 408b
On Mon Sep 25 14:07:53 2006, Josh-Engelman@deshaw.com wrote: Show quoted text
> An easier-to-read form of the diff: > > diff /usr/local/lib/perl5/site_perl/5.8.6/Memoize.pm Memoize.pm > 75c75 > < ? eval "sub $proto { &_memoizer(\$cref, \@_); }" > ---
> > ? eval "sub $proto { &_memoizer('$cref', \@_); }"
> > -josh
Fixes the problem for me on Debian testing under Apache 2.2 and mod_perl. Perl 5.8.8.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.