Skip Menu |
 

This queue is for tickets about the File-Path CPAN distribution.

Report information
The Basics
Id: 94209
Status: resolved
Priority: 0/
Queue: File-Path

People
Owner: RICHE [...] cpan.org
Requestors: ppisar [...] redhat.com
Cc: jkeenan [...] cpan.org
AdminCc:

Bug Information
Severity: (no value)
Broken in: 2.09
Fixed in: (no value)



Subject: remove_tree() is not thread-safe due to chdir
Download (untitled) / with headers
text/plain 780b
I have a multi-threaded program which calls make_path() and remove_tree(). I observed some occasional failure from subprocesses about missing working directory. Now I got a croak from remove_tree() that removing directory is not empty. It happened that one thread started remove_tree() which made chdir(), then the other thread called and finished make_path(), and finally, the former thread continued in remove_tree() which failed because concurrent make_path() created relative directory tree into working directory changed by remove_tree(). All this is tread-unsafety is due to using chdir(). Working directory is a process attribute shared between threads. It would be nice to reimplement File::Path without changing working directory. Or at least document this deficiency.
Subject: Re: [rt.cpan.org #94209] AutoReply: remove_tree() is not thread-safe due to chdir
Date: Wed, 26 Mar 2014 15:51:08 +0100
To: Bugs in File-Path via RT <bug-File-Path [...] rt.cpan.org>
From: Petr Pisar <ppisar [...] redhat.com>
Download (untitled) / with headers
text/plain 135b
This issue has already been reported into perl RT without any reply. (https://rt.perl.org/Public/Bug/Display.html?id=112008). -- Petr
Download (untitled)
application/pgp-signature 230b

Message body not shown because it is not plain text.

Subject: Re: [rt.cpan.org #94209] AutoReply: remove_tree() is not thread-safe due to chdir
Date: Thu, 27 Mar 2014 07:46:53 +0100
To: Bugs in File-Path via RT <bug-File-Path [...] rt.cpan.org>
From: Petr Pisar <ppisar [...] redhat.com>
File::Path::Tiny does not fiddle with working directory. -- Petr
Download (untitled)
application/pgp-signature 230b

Message body not shown because it is not plain text.

Download (untitled) / with headers
text/plain 2.1k
On Wed Mar 26 09:41:20 2014, ppisar wrote: Show quoted text
> I have a multi-threaded program which calls make_path() and > remove_tree(). I observed some occasional failure from subprocesses > about missing working directory. Now I got a croak from remove_tree() > that removing directory is not empty. > > It happened that one thread started remove_tree() which made chdir(), > then the other thread called and finished make_path(), and finally, > the former thread continued in remove_tree() which failed because > concurrent make_path() created relative directory tree into working > directory changed by remove_tree(). > > All this is tread-unsafety is due to using chdir(). Working directory > is a process attribute shared between threads. > > It would be nice to reimplement File::Path without changing working > directory. Or at least document this deficiency.
Richard Elberger and I have become co-maintainers of File-Path on CPAN, so the library is now under active maintenance. This problem is definitely on our radar. Having given some study to this issues (which is also the subject of https://rt.cpan.org/Ticket/Display.html?id=99230), I think the *best* solution is to re-implement sub remove_tree() to remove its internal reliance on 'chdir'. But to do that, we -- and by that I mean not just the maintainers but all who have contributed to these RTs -- have to understand why 'chdir' was used as part of the implementation of remove_tree in the first place. As best I can tell, this was the commit to Perl 5 source code which first introduced this implementation: ##### commit 0b3d36bd61fec90809936fcf1a90441d970d875e Author: David Landgren <david@landgren.net> Date: Sat Sep 8 12:46:15 2007 +0200 sync blead with File-Path 2.00_11 Message-ID: <46E26157.4050307@landgren.net> p4raw-id: //depot/perl@31819 ##### David Landgren presumably had good reason to take this approach. So we have to understand what benefits that brought us in order to retain them as best as we can while we address the costs it also brought us. Since all of the above, is non-trivial, I recommend that in the short-run we adopt a *second best* solution of simply documenting the lack of thread safety. Thank you very much. Jim Keenan
Download (untitled) / with headers
text/plain 218b
On Wed Mar 26 10:51:23 2014, ppisar wrote: Show quoted text
> This issue has already been reported into perl RT without any reply. > (https://rt.perl.org/Public/Bug/Display.html?id=112008). > > -- Petr
Attaching text of that report.
Subject: rt_perl_org_112008.txt
This is a bug report for perl from steve@rhythm.com, generated with the help of perlbug 1.39 running under perl 5.14.2. ----------------------------------------------------------------- [Please describe your issue here] On Linux, threads are typically created with pthread_create which eventually ends up being a clone() call with CLONE_FS set which means all threads share one /proc/self/cwd. The code within File::Path does a chdir(). If a another thread does a chdir() files might be deleted that are not suppose to be deleted because the rmtree will not be in the directory that it expects to be in. I wrote a simple test case which shows the wrong files being deleted: Deleting /tmp/test/junk Thread 1 terminated abnormally: previous directory .. changed before entering /tmp/test/junk/1/2, expected dev=2051 inode=4105812, actual dev=2051 ino=4105807, aborting. at ./rmtree.pl line 56 thread 1 CWD: /tmp/test/precious/1 found 971 files in /tmp/test/junk found 29 files in /tmp/test/precious I have replicated this problem on linux with perl v5.10.0 and v5.14.2. OS X perl v5.10.0 also exhibits this issue. Here is the test program: #!/usr/bin/perl use File::Path qw( rmtree mkpath); use File::Find; use Cwd 'getcwd'; use threads; mkpath("/tmp/test/precious/1/2",0755); mkpath("/tmp/test/junk/1/2",0755); for( $x = 0 ;$x < 1000 ; $x++){ open(F,">/tmp/test/junk/1/2/file$x"); open(F,">/tmp/test/precious/1/2/file$x"); } chdir("/tmp"); threads->new(\&sub1); threads->new(\&sub2); $_->join() for threads->list(); print "CWD: " . getcwd() . "\n"; sub wanted { if( -f $_){ $count++; } } foreach my $dir ( qw(/tmp/test/junk /tmp/test/precious)){ $count = 0 ; find(\&wanted, $dir); print "found $count files in $dir\n"; } sub sub2(){ while($z < 1000 ){ chdir("/tmp/test/precious/1/2"); $z++; } } sub sub1(){ _delDir("/tmp/test/junk"); print "THREAD CWD 2: " . getcwd() . "\n"; $isdone = 1; } sub _delDir { my ( $dir ) = @_; my $rmTreeErrors; if ( $dir and ( -d $dir ) ) { print ( "Deleting $dir\n" ); rmtree( $dir, { verbose => 0, error => \$rmTreeErrors } ); foreach my $rmTreeErr ( @{ $rmTreeErrors } ) { my ( $file, $message ) = each %{ $rmTreeErr }; if ( $file eq '' ) { print ( "Problem with rmtree($dir): $message\n" ); } else { print ( "Error in rmtree($dir): Problem deleting $file -- $message\n" ); } } } return $rmTreeErrors; } [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=library severity=medium module=File::Path --- Site configuration information for perl 5.14.2: Configured by steve at Mon Mar 26 13:11:01 PDT 2012. Summary of my perl5 (revision 5 version 14 subversion 2) configuration: Platform: osname=linux, osvers=2.6.27.23-10rh, archname=linux-linux-thread uname='linux lid9 2.6.27.23-10rh #4 smp wed dec 28 16:40:10 pst 2011 x86_64 x86_64 x86_64 gnulinux ' config_args='-ds -e' hint=previous, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=undef useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2', cppflags='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64' ccversion='', gccversion='4.3.2 [gcc-4_3-branch revision 141291]', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='cc', ldflags =' -fstack-protector -lpthread' libpth=/usr/local/lib /lib/../lib64 /usr/lib/../lib64 /lib /usr/lib /lib64 /usr/lib64 /usr/local/lib64 libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc libc=/lib/libc-2.9.so, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='2.9' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -fstack-protector -lpthread' Locally applied patches: --- @INC for perl 5.14.2: /usr/lsd/lib/perl /usr/local/prod/perl /usr/local/rnh/perl /opt/perl-5.14.2/lib/site_perl/5.14.2/linux-linux /opt/perl-5.14.2/lib/site_perl/5.14.2 /opt/perl-5.14.2/lib/5.14.2/linux-linux /opt/perl-5.14.2/lib/5.14.2 . --- Environment for perl 5.14.2: HOME=/home/steve LANG=C LANGUAGE (unset) LD_LIBRARY_PATH=/opt/randh/mesa-7.4.3/lib64:/opt/randh/mesa-7.4.3/lib:/usr/lib64/mpi/gcc/openmpi/lib64 LOGDIR (unset) PATH=/opt/perl-5.14.2/bin:/home/steve/bin.Linux_i686:/usr/apps/ffmpeg/ffmpeg-git20111012/bin:/muse/bin:/usr/apps/bin:/usr/local/prod/bin:/usr/local/rnh/bin:/usr/site/bin:/usr/local/bin:/opt/kde3/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11:/usr/lsd/bin:/opt/gnome/bin:/usr/games:. PERL5LIB=/usr/lsd/lib/perl:/usr/local/prod/perl:/usr/local/rnh/perl PERL_BADLANG (unset) SHELL=/bin/tcsh
Download (untitled) / with headers
text/plain 611b
On Sat Jul 11 09:56:20 2015, JKEENAN wrote: Show quoted text
> On Wed Mar 26 10:51:23 2014, ppisar wrote:
> > This issue has already been reported into perl RT without any reply. > > (https://rt.perl.org/Public/Bug/Display.html?id=112008). > > > > -- Petr
> > Attaching text of that report.
For quite some time this limitation has been documented under: BUGS AND LIMITATIONS > MULTITHREAD APPLICATIONS I think we need to change the following text: The implementation that surfaces this limitation may change in a future release. to: The implementation that surfaces this limitation will not be changed.
Subject: Re: [rt.cpan.org #94209] remove_tree() is not thread-safe due to chdir
Date: Mon, 12 Oct 2015 10:15:33 +0200
To: Richard Elberger via RT <bug-File-Path [...] rt.cpan.org>
From: Petr Pisar <ppisar [...] redhat.com>
Download (untitled) / with headers
text/plain 255b
On Sun, Oct 11, 2015 at 07:57:51AM -0400, Richard Elberger via RT wrote: Show quoted text
> For quite some time this limitation has been documented under: > > BUGS AND LIMITATIONS > MULTITHREAD APPLICATIONS >
Thanks. The text is there since File-Path-2.10_005. -- Petr
Download signature.asc
application/pgp-signature 213b

Message body not shown because it is not plain text.



This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.