Skip Menu |
 

This queue is for tickets about the Encode CPAN distribution.

Report information
The Basics
Id: 113164
Status: resolved
Priority: 0/
Queue: Encode

People
Owner: Nobody in particular
Requestors: nigelgregoire [...] gmail.com
Cc: pali [...] cpan.org
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: 2.87



Subject: encode() returns Encode::XS object, dies
Date: Fri, 18 Mar 2016 18:13:57 -0700
To: bug-Encode [...] rt.cpan.org
From: Nigel Gregoire <nigelgregoire [...] gmail.com>
Download (untitled) / with headers
text/plain 1.1k
Hello! On my system encode() sometimes returns an Encode::XS object instead of the resulting string. Here's some sample code: nfg@bill-ubuntu-ng:/home/main/SMS on master *$ cat encode2.pl #!/usr/bin/env perl use strict; use warnings; use Encode; use Data::Dumper; my $str = "You" . chr(8217) . "re doomed!"; binmode STDOUT, ":encoding(utf8)"; binmode STDERR, ":encoding(utf8)"; print $str, "\n"; my $data; my $cb = sub { $data = [ ('?') x 12_500 ]; return ";"; }; my $octets = encode('iso-8859-1', $str, $cb); print Dumper($octets), "\n"; print "And again: ", "\n"; $octets = encode('iso-8859-1', $str, $cb); print Dumper($octets), "\n"; OUTPUT: nfg@bill-ubuntu-ng:/home/main/SMS on master *$ perl encode2.pl You’re doomed! $VAR1 = bless( do{\(my $o = '140472427705184')}, 'Encode::XS' ); And again: $VAR1 = 'You;re doomed!'; Segmentation fault (core dumped) I ran into this with some code that uses Text::Unidecode in the CHECK callback. Text::Unidecode can dynamically require packages while running. This is with Encode 2.82. I've reproduced it on Perl 5.8.9, 5.14.2, 5.16.0, 5.20.3, and 5.22.1 , as long as I allocate enough memory. FWIW, Cheers! -- Nigel
Download (untitled) / with headers
text/plain 1.5k
I could not reproduce your symptom on OS X v10.11.4, Ubuntu 14.04 LTS, and FreeBSD 10.2-RELEASE. Would you tell me more about your environment in details? the output of "perl -V" is pretty helpful. Dan the Maintainer Thereof On Fri Mar 18 21:14:16 2016, nigelgregoire@gmail.com wrote: Show quoted text
> Hello! On my system encode() sometimes returns an Encode::XS object > instead of the resulting string. Here's some sample code: > > nfg@bill-ubuntu-ng:/home/main/SMS on master *$ cat encode2.pl > #!/usr/bin/env perl > use strict; > use warnings; > > use Encode; > use Data::Dumper; > > my $str = "You" . chr(8217) . "re doomed!"; > binmode STDOUT, ":encoding(utf8)"; > binmode STDERR, ":encoding(utf8)"; > print $str, "\n"; > > my $data; > > my $cb = sub { > $data = [ ('?') x 12_500 ]; > return ";"; > }; > > my $octets = encode('iso-8859-1', $str, $cb); > print Dumper($octets), "\n"; > > print "And again: ", "\n"; > $octets = encode('iso-8859-1', $str, $cb); > print Dumper($octets), "\n"; > > OUTPUT: > > nfg@bill-ubuntu-ng:/home/main/SMS on master *$ perl encode2.pl > You’re doomed! > $VAR1 = bless( do{\(my $o = '140472427705184')}, 'Encode::XS' ); > > And again: > $VAR1 = 'You;re doomed!'; > > Segmentation fault (core dumped) > > > > I ran into this with some code that uses Text::Unidecode in the CHECK > callback. Text::Unidecode can dynamically require packages while > running. > > This is with Encode 2.82. I've reproduced it on Perl 5.8.9, 5.14.2, > 5.16.0, 5.20.3, and 5.22.1 , as long as I allocate enough memory. > FWIW, > > Cheers! >
Subject: Re: [rt.cpan.org #113164] encode() returns Encode::XS object, dies
Date: Tue, 22 Mar 2016 10:28:50 -0700
To: bug-Encode [...] rt.cpan.org
From: Nigel Gregoire <nigelgregoire [...] gmail.com>
Download (untitled) / with headers
text/plain 5.9k
Here you go. FYI, all my testing was done via perlbrew. nfg@bill-ubuntu-ng:/home/main/SMS on master *$ lsb_release -a LSB Version: core-2.0-amd64:core-2.0-noarch:core-3.0-amd64:core-3.0-noarch:core-3.1-amd64:core-3.1-noarch:core-3.2-amd64:core-3.2-noarch:core-4.0-amd64:core-4.0-noarch:core-4.1-amd64:core-4.1-noarch:security-4.0-amd64:security-4.0-noarch:security-4.1-amd64:security-4.1-noarch Distributor ID: Ubuntu Description: Ubuntu 15.10 Release: 15.10 Codename: wily nfg@bill-ubuntu-ng:/home/main/SMS on master *$ perl -V Summary of my perl5 (revision 5 version 20 subversion 3) configuration: Platform: osname=linux, osvers=4.2.0-27-generic, archname=x86_64-linux uname='linux bill-ubuntu-ng 4.2.0-27-generic #32-ubuntu smp fri jan 22 04:49:08 utc 2016 x86_64 x86_64 x86_64 gnulinux ' config_args='-de -Dprefix=/home/ad/nigelg/perl5/perlbrew/perls/perl-5.20.3' hint=recommended, useposix=true, d_sigaction=define useithreads=undef, usemultiplicity=undef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2', cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include' ccversion='', gccversion='5.2.1 20151010', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='cc', ldflags =' -fstack-protector -L/usr/local/lib' libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/5/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc libc=libc-2.21.so, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='2.21' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector' Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV USE_64_BIT_ALL USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF Locally applied patches: Devel::PatchPerl 1.30 Built under linux Compiled at Mar 18 2016 13:31:29 %ENV: PERL5LIB="/home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev/lib/perl5/x86_64-linux:/home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev/lib/perl5" PERLBREW_BASHRC_VERSION="0.43" PERLBREW_HOME="/home/ad/nigelg/.perlbrew" PERLBREW_LIB="dev" PERLBREW_MANPATH="/home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev/man:/home/ad/nigelg/perl5/perlbrew/perls/perl-5.20.3/man" PERLBREW_PATH="/home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev/bin:/home/ad/nigelg/perl5/perlbrew/bin:/home/ad/nigelg/perl5/perlbrew/perls/perl-5.20.3/bin" PERLBREW_PERL="perl-5.20.3" PERLBREW_ROOT="/home/ad/nigelg/perl5/perlbrew" PERLBREW_VERSION="0.43" PERL_LOCAL_LIB_ROOT="/home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev" PERL_MB_OPT="--install_base /home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev" PERL_MM_OPT="INSTALL_BASE=/home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev" @INC: /home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev/lib/perl5/x86_64-linux /home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev/lib/perl5/x86_64-linux /home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev/lib/perl5 /home/ad/nigelg/perl5/perlbrew/perls/perl-5.20.3/lib/site_perl/5.20.3/x86_64-linux /home/ad/nigelg/perl5/perlbrew/perls/perl-5.20.3/lib/site_perl/5.20.3 /home/ad/nigelg/perl5/perlbrew/perls/perl-5.20.3/lib/5.20.3/x86_64-linux /home/ad/nigelg/perl5/perlbrew/perls/perl-5.20.3/lib/5.20.3 . On 22 March 2016 at 01:39, Dan Kogai via RT <bug-Encode@rt.cpan.org> wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=113164 > > > I could not reproduce your symptom on OS X v10.11.4, Ubuntu 14.04 LTS, and FreeBSD 10.2-RELEASE. Would you tell me more about your environment in details? > the output of "perl -V" is pretty helpful. > > Dan the Maintainer Thereof > > On Fri Mar 18 21:14:16 2016, nigelgregoire@gmail.com wrote:
>> Hello! On my system encode() sometimes returns an Encode::XS object >> instead of the resulting string. Here's some sample code: >> >> nfg@bill-ubuntu-ng:/home/main/SMS on master *$ cat encode2.pl >> #!/usr/bin/env perl >> use strict; >> use warnings; >> >> use Encode; >> use Data::Dumper; >> >> my $str = "You" . chr(8217) . "re doomed!"; >> binmode STDOUT, ":encoding(utf8)"; >> binmode STDERR, ":encoding(utf8)"; >> print $str, "\n"; >> >> my $data; >> >> my $cb = sub { >> $data = [ ('?') x 12_500 ]; >> return ";"; >> }; >> >> my $octets = encode('iso-8859-1', $str, $cb); >> print Dumper($octets), "\n"; >> >> print "And again: ", "\n"; >> $octets = encode('iso-8859-1', $str, $cb); >> print Dumper($octets), "\n"; >> >> OUTPUT: >> >> nfg@bill-ubuntu-ng:/home/main/SMS on master *$ perl encode2.pl >> You’re doomed! >> $VAR1 = bless( do{\(my $o = '140472427705184')}, 'Encode::XS' ); >> >> And again: >> $VAR1 = 'You;re doomed!'; >> >> Segmentation fault (core dumped) >> >> >> >> I ran into this with some code that uses Text::Unidecode in the CHECK >> callback. Text::Unidecode can dynamically require packages while >> running. >> >> This is with Encode 2.82. I've reproduced it on Perl 5.8.9, 5.14.2, >> 5.16.0, 5.20.3, and 5.22.1 , as long as I allocate enough memory. >> FWIW, >> >> Cheers! >>
> > >
-- Nigel http://www.forgreatjustice.ca/
Download (untitled) / with headers
text/plain 3.3k
I don't see a segfault either, but valgrind says: ==13517== Invalid write of size 8 ==13517== at 0x609A2B6: XS_Encode__XS_encode (in /bbbike/perl-5.20.1/lib/site_perl/5.20.1/x86_64-linux/auto/Encode/Encode.so) ==13517== by 0x4ABEC6: Perl_pp_entersub (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x4A4E32: Perl_runops_standard (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x43FFD3: perl_run (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x422284: main (in /bbbike/perl-5.20.1/bin/perl) ==13517== Address 0x5c98d18 is 8 bytes inside a block of size 1,024 free'd ==13517== at 0x4C28CCE: realloc (vg_replace_malloc.c:632) ==13517== by 0x48AC72: Perl_safesysrealloc (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x4A356E: Perl_av_extend_guts (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x4D1C19: Perl_stack_grow (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x4C4C05: Perl_pp_repeat (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x4A4E32: Perl_runops_standard (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x4399E2: Perl_call_sv (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x60989BB: do_fallback_cb (in /bbbike/perl-5.20.1/lib/site_perl/5.20.1/x86_64-linux/auto/Encode/Encode.so) ==13517== by 0x609A05A: encode_method (in /bbbike/perl-5.20.1/lib/site_perl/5.20.1/x86_64-linux/auto/Encode/Encode.so) ==13517== by 0x609A2B5: XS_Encode__XS_encode (in /bbbike/perl-5.20.1/lib/site_perl/5.20.1/x86_64-linux/auto/Encode/Encode.so) ==13517== by 0x4ABEC6: Perl_pp_entersub (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x4A4E32: Perl_runops_standard (in /bbbike/perl-5.20.1/bin/perl) ==13517== The "Perl_stack_grow" call looks suspicious... Regards, Slaven On 2016-03-22 04:39:48, DANKOGAI wrote: Show quoted text
> I could not reproduce your symptom on OS X v10.11.4, Ubuntu 14.04 LTS, > and FreeBSD 10.2-RELEASE. Would you tell me more about your > environment in details? > the output of "perl -V" is pretty helpful. > > Dan the Maintainer Thereof > > On Fri Mar 18 21:14:16 2016, nigelgregoire@gmail.com wrote:
> > Hello! On my system encode() sometimes returns an Encode::XS object > > instead of the resulting string. Here's some sample code: > > > > nfg@bill-ubuntu-ng:/home/main/SMS on master *$ cat encode2.pl > > #!/usr/bin/env perl > > use strict; > > use warnings; > > > > use Encode; > > use Data::Dumper; > > > > my $str = "You" . chr(8217) . "re doomed!"; > > binmode STDOUT, ":encoding(utf8)"; > > binmode STDERR, ":encoding(utf8)"; > > print $str, "\n"; > > > > my $data; > > > > my $cb = sub { > > $data = [ ('?') x 12_500 ]; > > return ";"; > > }; > > > > my $octets = encode('iso-8859-1', $str, $cb); > > print Dumper($octets), "\n"; > > > > print "And again: ", "\n"; > > $octets = encode('iso-8859-1', $str, $cb); > > print Dumper($octets), "\n"; > > > > OUTPUT: > > > > nfg@bill-ubuntu-ng:/home/main/SMS on master *$ perl encode2.pl > > You’re doomed! > > $VAR1 = bless( do{\(my $o = '140472427705184')}, 'Encode::XS' ); > > > > And again: > > $VAR1 = 'You;re doomed!'; > > > > Segmentation fault (core dumped) > > > > > > > > I ran into this with some code that uses Text::Unidecode in the CHECK > > callback. Text::Unidecode can dynamically require packages while > > running. > > > > This is with Encode 2.82. I've reproduced it on Perl 5.8.9, 5.14.2, > > 5.16.0, 5.20.3, and 5.22.1 , as long as I allocate enough memory. > > FWIW, > > > > Cheers! > >
Subject: Re: [rt.cpan.org #113164] encode() returns Encode::XS object, dies
Date: Fri, 22 Apr 2016 16:50:27 -0700
To: bug-Encode [...] rt.cpan.org
From: Nigel Gregoire <nigelgregoire [...] gmail.com>
Download (untitled) / with headers
text/plain 2.9k
Hello! I switched to plenv and got a coredump: plenvtest@bill-ubuntu-ng:/home/main/SMS/t/encoding$ gdb ~/.plenv/versions/5.22.1-debug/bin/perl ~/unpacked/CoreDump GNU gdb (Ubuntu 7.10-1ubuntu2) 7.10 Copyright (C) 2015 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /home/plenvtest/.plenv/versions/5.22.1-debug/bin/perl...done. warning: core file may not match specified executable file. [New LWP 16676] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `perl broken_encode.t'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f4eb03d0267 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 55 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) backtrace #0 0x00007f4eb03d0267 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 #1 0x00007f4eb03d1eca in __GI_abort () at abort.c:89 #2 0x00007f4eb0413bf3 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7f4eb052c168 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175 #3 0x00007f4eb0419767 in malloc_printerr (action=<optimized out>, str=0x7f4eb052829d "corrupted double-linked list", ptr=<optimized Show quoted text
out>) at malloc.c:4965
#4 0x00007f4eb041dc52 in _int_malloc (av=av@entry=0x7f4eb075fc00 <main_arena>, bytes=bytes@entry=1024) at malloc.c:3589 #5 0x00007f4eb0420055 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3213 #6 0x0000000000421db4 in S_new_slab (sz=128) at op.c:181 #7 Perl_Slab_Alloc (sz=9) at op.c:281 #8 0x00000000004266d7 in Perl_newLISTOP (type=192, flags=0, first=0x238eef0, last=0x238ef50) at op.c:4694 #9 0x00000000004674a4 in Perl_yyparse (gramtype=gramtype@entry=258) at perly.y:238 #10 0x00000000004e2e4d in S_doeval (gimme=gimme@entry=2, outside=outside@entry=0x22a77a8, seq=<optimized out>, hh=hh@entry=0x0) at pp_ctl.c:3491 #11 0x00000000004eda02 in Perl_pp_entereval () at pp_ctl.c:4302 #12 0x00000000004ab3b3 in Perl_runops_standard () at run.c:41 #13 0x0000000000442c76 in S_run_body (oldscope=1) at perl.c:2448 #14 perl_run (my_perl=<optimized out>) at perl.c:2371 #15 0x000000000041fcb5 in main (argc=2, argv=0x7ffd725f28b8, env=0x7ffd725f28d0) at perlmain.c:116 Does this help at all? My gdb-fu is weak, but I'll do whatever I can to debug this thing. Cheers! -- Nigel
Download (untitled) / with headers
text/plain 336b
On Pia mar 18 21:14:16 2016, nigelgregoire@gmail.com wrote: Show quoted text
> Hello! On my system encode() sometimes returns an Encode::XS object > instead of the resulting string.
Bug in Encode.xs. It modifies perl stack arguments more times between two sequence points. This is undefined behaviour in C. I'm preparing patch for fixing this problem.
Download (untitled) / with headers
text/plain 488b
On Pia Sep 30 06:12:35 2016, PALI wrote: Show quoted text
> On Pia mar 18 21:14:16 2016, nigelgregoire@gmail.com wrote:
> > Hello! On my system encode() sometimes returns an Encode::XS object > > instead of the resulting string.
> > Bug in Encode.xs. It modifies perl stack arguments more times between > two sequence points. This is undefined behaviour in C. > > I'm preparing patch for fixing this problem.
Pull request with patch: https://github.com/dankogai/p5-encode/pull/72 Fixed in Encode 2.87.
CC: pali [...] cpan.org
Subject: Re: [rt.cpan.org #113164] encode() returns Encode::XS object, dies
Date: Fri, 28 Oct 2016 10:54:37 -0700
To: bug-Encode [...] rt.cpan.org
From: Nigel Gregoire <nigelgregoire [...] gmail.com>
Download (untitled) / with headers
text/plain 794b
Hey! Thank you so much for looking into this. I just ran my tests on the new Encode and everything passes. :D On 28 October 2016 at 05:30, Pali via RT <bug-Encode@rt.cpan.org> wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=113164 > > > On Pia Sep 30 06:12:35 2016, PALI wrote:
>> On Pia mar 18 21:14:16 2016, nigelgregoire@gmail.com wrote:
>> > Hello! On my system encode() sometimes returns an Encode::XS object >> > instead of the resulting string.
>> >> Bug in Encode.xs. It modifies perl stack arguments more times between >> two sequence points. This is undefined behaviour in C. >> >> I'm preparing patch for fixing this problem.
> > Pull request with patch: https://github.com/dankogai/p5-encode/pull/72 > > Fixed in Encode 2.87.
-- Nigel http://www.forgreatjustice.ca/


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.