Skip Menu |
 

This queue is for tickets about the Term-ReadLine-Gnu CPAN distribution.

Maintainer(s)' notes

When you report a bug, please provide the following information;

- output of
	perl -V
	perl Makefile.PL verbose
	make test TEST_VERBOSE=1
	perl -Mblib t/00checkver.t
	echo $TERM
- terminal emulator which you are using
- compiler which is used to compile the GNU Readline Library (libreadline.a) if you can know.
Read INSTALL in the distribution for more details.

Report information
The Basics
Id: 59832
Status: resolved
Priority: 0/
Queue: Term-ReadLine-Gnu

People
Owner: HAYASHI [...] cpan.org
Requestors: CARNIL [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: 1.24



Subject: clobbers binmode layers on filehandles
Date: Thu, 29 Jul 2010 20:28:35 +0200
To: bug-Term-ReadLine-Gnu [...] rt.cpan.org
From: Salvatore Bonaccorso <carnil [...] cpan.org>
Download (untitled) / with headers
text/plain 2.1k
Hi In Debian we recieved the following bugreport. Could you have a look at it? Many thanks in advance Bests Salvatore ------------------------------------------------------------------------ From: Andrew Pimlott <andrew@pimlott.net> To: Debian Bug Tracking System <submit@bugs.debian.org> Subject: libterm-readline-gnu-perl: clobbers binmode layers on filehandles Date: Thu, 17 Sep 2009 13:31:24 -0700 Package: libterm-readline-gnu-perl Version: 1.19-1 Severity: normal If I call Term::ReadLine->new('test', \*IN, \*OUT); any encoding layers I have set on OUT seem to be removed. I tracked it down to the line in the perl source that sets $Attribs{outstream}. I looked briefly at the .xs source for _rl_store_iostream, but I can't see what's causing this. I hope someone who knows perl internals can figure it out. Here's a complete test program. I run it on a utf-8 terminal. The first print works correctly (the correct utf-8 byte sequence is output), and the second shows a unicode unknown character (U+fffd), because perl outputs the byte \xf3. use Term::ReadLine; binmode(STDOUT, ':encoding(utf-8)'); print STDOUT ">", chr(0xf3), "<\n"; $Term::ReadLine::Gnu::Attribs{outstream} = \*STDOUT; print STDOUT ">", chr(0xf3), "<\n"; Andrew -- System Information: Debian Release: squeeze/sid APT prefers unstable APT policy: (500, 'unstable') Architecture: i386 (i686) Kernel: Linux 2.6.30-1-686 (SMP w/1 CPU core) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/bash Versions of packages libterm-readline-gnu-perl depends on: ii libc6 2.9-26 GNU C Library: Shared libraries ii libncurses5 5.7+20090803-2 shared libraries for terminal hand ii libreadline5 5.2-6 GNU readline and history libraries ii perl 5.10.0-25 Larry Wall's Practical Extraction ii perl-base [perlapi-5.10.0 5.10.0-25 minimal Perl system libterm-readline-gnu-perl recommends no packages. libterm-readline-gnu-perl suggests no packages. -- no debconf information ------------------------------------------------------------------------
Download signature.asc
application/pgp-signature 835b

Message body not shown because it is not plain text.

RT-Send-CC: carnil [...] cpan.org
Download (untitled) / with headers
text/plain 1.3k
Hi, Show quoted text
> If I call > > Term::ReadLine->new('test', \*IN, \*OUT); > > any encoding layers I have set on OUT seem to be removed. I tracked
Term::Gnu::ReadLine "is an implementation of Term::ReadLine using the GNU Readline/History Library." as described in the document. The GNU Readline Library may call low level APIs directly. So this is not a bug. BTW if the reporter wants to handle multibyte (ex. UTF-8) characters, the GNU Readline Library support them and, as the result, Term::Gnu::ReadLine also support them. Refer the documents for the GNU Readline Library for details. Show quoted text
> it > down to the line in the perl source that sets $Attribs{outstream}. I > looked briefly at the .xs source for _rl_store_iostream, but I can't > see > what's causing this. I hope someone who knows perl internals can > figure > it out.
Here is a comment in Gnu.pm; --------------------------------------- # Each variable in the GNU Readline Library is tied to an entry of # this hash (%Attribs). By accessing the hash entry, you can read # and/or write the variable in the GNU Readline Library. See the # package definition of Term::ReadLine::Gnu::Var and following code # for more details. --------------------------------------- Refer the chapter 'Tied Variables' in Perl manual, or the following URL; http://perldoc.perl.org/perltie.html I hope this helps. Thanks.
Thanks! Bests Salvatore
CC: Hiroo_HAYASHI via RT <bug-Term-ReadLine-Gnu [...] rt.cpan.org>, CARNIL [...] cpan.org
Subject: Re: Bug#547231 closed by Salvatore Bonaccorso <salvatore.bonaccorso@gmail.com> ([rt.cpan.org #59832] clobbers binmode layers on filehandles)
Date: Wed, 18 Aug 2010 13:35:56 -0700
To: 547231 <547231 [...] bugs.debian.org>
From: Andrew Pimlott <andrew [...] pimlott.net>
Download (untitled) / with headers
text/plain 2.9k
Salvatore, thanks for following up on this report. I have some questions below. Excerpts from owner's message of Wed Aug 18 04:30:08 -0700 2010: Show quoted text
> This is an automatic notification regarding your Bug report
... Show quoted text
> Hi Andrew > > I'm forwarding you the reply from upstream according to your report. > > From: Hiroo_HAYASHI via RT <bug-Term-ReadLine-Gnu@rt.cpan.org> >
> > If I call > > > > Term::ReadLine->new('test', \*IN, \*OUT); > > > > any encoding layers I have set on OUT seem to be removed. I tracked
> > Term::Gnu::ReadLine "is an implementation of Term::ReadLine using the > GNU Readline/History Library." as described in the document. > The GNU Readline Library may call low level APIs directly. So this is > not a bug.
I realize that Readline has to do some low-level things. But I don't understand why it has to clobber the encoding layers on the filehandle. Is it possible to preserve them? Do you not agree that the test program I sent is terribly confusing? If Term::ReadLine::Gnu changes the behavior of the filehandles like this, it should at least be documented. Show quoted text
> BTW if the reporter wants to handle multibyte (ex. UTF-8) characters, > the GNU Readline Library support them and, as the result, > Term::Gnu::ReadLine also support them. Refer the documents for the GNU > Readline Library for details.
I couldn't find the documentation you are refering to, or how I'm supposed to handle multibyte characters with Term::ReadLine::GNU in Perl. I notice that if I set the encoding layer back to UTF-8, it seems to work. In this test program, the first and third print statements output the expected character, and only the second is broken: use Term::ReadLine; binmode(STDOUT, ':encoding(UTF-8)'); print STDOUT ">", chr(0xf3), "<\n"; $Term::ReadLine::Gnu::Attribs{outstream} = \*STDOUT; print STDOUT ">", chr(0xf3), "<\n"; binmode(STDOUT, ':encoding(UTF-8)'); print STDOUT ">", chr(0xf3), "<\n"; Will I have problems if I do it this way? If this works, I don't see why Term::ReadLine::GNU had to remove the encoding layer in the first place. Show quoted text
> > it > > down to the line in the perl source that sets $Attribs{outstream}. I > > looked briefly at the .xs source for _rl_store_iostream, but I can't > > see > > what's causing this. I hope someone who knows perl internals can > > figure > > it out.
> > Here is a comment in Gnu.pm; > --------------------------------------- > # Each variable in the GNU Readline Library is tied to an entry of > # this hash (%Attribs). By accessing the hash entry, you can read > # and/or write the variable in the GNU Readline Library. See the > # package definition of Term::ReadLine::Gnu::Var and following code > # for more details.
I understand that part. I traced the setting of $Attribs{outstream} through the tie into _rl_store_iostream in Gnu.xs. The code there looks like rl_outstream = PerlIO_findFILE(stream); Somehow this breaks my encoding layers, but I don't understand why. Andrew
Subject: Re: Bug#547231 closed by Salvatore Bonaccorso <salvatore.bonaccorso@gmail.com> ([rt.cpan.org #59832] clobbers binmode layers on filehandles)
Date: Thu, 19 Aug 2010 22:23:58 +0900
To: bug-Term-ReadLine-Gnu [...] rt.cpan.org
From: 林宏雄 <hiroo.hayashi [...] computer.org>
Download (untitled) / with headers
text/plain 431b
Hi, Show quoted text
> rl_outstream = PerlIO_findFILE(stream);
Show quoted text
> Somehow this breaks my encoding layers, but I don't understand why.
I don't, either. As you see, Term::ReadLine::Gnu simply passes the filehandle to the GNU Readline Library directly. Show quoted text
> If Term::ReadLine::Gnu changes the > behavior of the filehandles like this, it should at least be documented.
This may be an undocumented behavior of Perl XS API. -- Hiroo Hayashi
Download (untitled) / with headers
text/plain 2.1k
Hi, Finally I've found a fix for this issue. I found PerlIO::get_layers() function in http://perldoc.perl.org/PerlIO.html. (If I remember correctly, I could not find good documentation of PerlIO when this ticket was opened.) ------------ use Term::ReadLine; my @layers; binmode(STDOUT, ':encoding(utf-8)'); @layers = PerlIO::get_layers(STDOUT); print join(':', @layers), "\n"; print STDOUT ">", chr(0xf3), "<\n"; $Term::ReadLine::Gnu::Attribs{outstream} = \*STDOUT; @layers = PerlIO::get_layers(STDOUT); print join(':', @layers), "\n"; print STDOUT ">", chr(0xf3), "<\n"; ------------ The code above outputs the following lines. ------------ unix:perlio:encoding(utf-8-strict):utf8 Show quoted text
>ó<
unix:perlio:encoding(utf-8-strict):utf8:stdio Show quoted text
>�<
------------ As you see; rl_outstream = PerlIO_findFILE(stream); adds the stdio layer. By popping the layer, ------------ use Term::ReadLine; my @layers; binmode(STDOUT, ':encoding(utf-8)'); @layers = PerlIO::get_layers(STDOUT); print join(':', @layers), "\n"; print STDOUT ">", chr(0xf3), "<\n"; $Term::ReadLine::Gnu::Attribs{outstream} = \*STDOUT; @layers = PerlIO::get_layers(STDOUT); print join(':', @layers), "\n"; print STDOUT ">", chr(0xf3), "<\n"; binmode(STDOUT, ":pop"); @layers = PerlIO::get_layers(STDOUT); print join(':', @layers), "\n"; print STDOUT ">", chr(0xf3), "<\n"; ------------ The code above outputs the following lines. ------------ unix:perlio:encoding(utf-8-strict):utf8 Show quoted text
>ó<
unix:perlio:encoding(utf-8-strict):utf8:stdio Show quoted text
>�<
unix:perlio:encoding(utf-8-strict):utf8 Show quoted text
>ó<
------------ Here is the patch. It will be included in Term-ReadLine-Gnu-1.23 soon. --- Gnu.pm (revision 458) +++ Gnu.pm (working copy) @@ -697,7 +698,12 @@ } elsif ($type eq 'F') { return _rl_store_function($value, $id); } elsif ($type eq 'IO') { - return _rl_store_iostream($value, $id); + #my @layers; + my $FH = _rl_store_iostream($value, $id); + #@layers = PerlIO::get_layers($FH); warn "<", join(':', @layers), ">\n"; + binmode($FH, ":pop"); + #@layers = PerlIO::get_layers($FH); warn "<", join(':', @layers), ">\n"; + return $FH; } elsif ($type eq 'K' || $type eq 'LF') { carp "Term::ReadLine::Gnu::Var::STORE: read only variable `$name'\n"; return undef;
The fix is in 1.23.
The fix in 1.23 did not work with perl 5.8.9 and older. The fix is in 1.24.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.