Skip Menu |
 

This queue is for tickets about the HTML-Parser CPAN distribution.

Report information
The Basics
Id: 99456
Status: rejected
Priority: 0/
Queue: HTML-Parser

People
Owner: Nobody in particular
Requestors: YKAR [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: HTML::TokeParser utf8 flag is not always preserved
Please look at the test case.
Subject: preserve_utf8.t
Download preserve_utf8.t
text/x-perl 271b
use strict; use warnings; use Test::More; use HTML::TokeParser; my $s = "Hello World"; utf8::upgrade($s); my $p = HTML::TokeParser->new(\$s); ok(utf8::is_utf8($s), 'input is utf8'); my $t = $p->get_text; ok(utf8::is_utf8($t), 'output is utf8'); done_testing();
Subject: Re: [rt.cpan.org #99456] HTML::TokeParser utf8 flag is not always preserved
Date: Mon, 13 Oct 2014 08:50:08 -0700
To: Yuri Karaban via RT <bug-HTML-Parser [...] rt.cpan.org>
From: Karen Etheridge <ether [...] cpan.org>
Download (untitled) / with headers
text/plain 178b
Why is this important? If the string does not actually contain any non-ascii characters, the utf8 flag should not be relied upon to contain anything meaningful to the end user.
RT-Send-CC: ether [...] cpan.org
Download (untitled) / with headers
text/plain 627b
On Mon Oct 13 11:50:26 2014, ETHER wrote: Show quoted text
> Why is this important? If the string does not actually contain any > non-ascii characters, the utf8 flag should not be relied upon to contain > anything meaningful to the end user.
HTML::TokeParser decodes HTML entity &nbsp; to non ASCII character 0xa0. When string is Unicode it means the U+00A0 code point (non-breaking space), but for raw octets 0xa0 does not have any particular meaning. It's not even a question of preserving utf8 flag. HTML::TokeParser should set utf8 flag if it has decoded HTML entities which does not map to ASCII (even if input document was pure ASCII).
Subject: Re: [rt.cpan.org #99456] HTML::TokeParser utf8 flag is not always preserved
Date: Mon, 13 Oct 2014 11:01:29 -0700
To: Yuri Karaban via RT <bug-HTML-Parser [...] rt.cpan.org>
From: Karen Etheridge <ether [...] cpan.org>
Download (untitled) / with headers
text/plain 1.1k
On Mon, Oct 13, 2014 at 01:47:22PM -0400, Yuri Karaban via RT wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=99456 > > > On Mon Oct 13 11:50:26 2014, ETHER wrote: >
> > Why is this important? If the string does not actually contain any > > non-ascii characters, the utf8 flag should not be relied upon to contain > > anything meaningful to the end user.
> > HTML::TokeParser decodes HTML entity &nbsp; to non ASCII character 0xa0. When string is Unicode it means the U+00A0 code point (non-breaking space), but for raw octets 0xa0 does not have any particular meaning.
HTML::TokeParser shouldn't be using the is_utf8 flag to make any decisions. It should treat every incoming string equally - either 0xa0 always means non-breaking space, or it doesn't. Show quoted text
> It's not even a question of preserving utf8 flag. HTML::TokeParser should set utf8 flag if it has decoded HTML entities which does not map to ASCII (even if input document was pure ASCII).
No, if it ever sets the utf8 flag, it should *always* set it, even if all the characters fall in the ascii range. It's not correct to only set it if some values are not ascii.
RT-Send-CC: ether [...] cpan.org
Download (untitled) / with headers
text/plain 1.1k
On Mon Oct 13 14:01:46 2014, ETHER wrote: Show quoted text
> HTML::TokeParser shouldn't be using the is_utf8 flag to make any > decisions. > It should treat every incoming string equally - either 0xa0 always > means > non-breaking space, or it doesn't.
I'm not asking to treat input differently depending utf8 flag. I'm asking to set utf8 flag on output. Show quoted text
> > It's not even a question of preserving utf8 flag. HTML::TokeParser > > should set utf8 flag if it has decoded HTML entities which does not > > map to ASCII (even if input document was pure ASCII).
> > No, if it ever sets the utf8 flag, it should *always* set it, even if > all > the characters fall in the ascii range. It's not correct to only set > it if > some values are not ascii.
I'm all for unconditionally setting utf8 on output. Unfortunately HTML::TokeParser is setting utf8 flag only if output contains characters with code points greater than 255. Example: my $p = HTML::TokeParser->new(\'&trade;'); # U+2122 say '&trade; results in utf8' if utf8::is_utf8($p->get_text); $p = HTML::TokeParser->new(\'&copy;'); # U+00A9 say "&copy; dosn't result in utf8" unless utf8::is_utf8($p->get_text);
This is done on purpose.  The parser selects the most compact way to represent the strings internally.  You should not put any semantic meaning of the internal UTF8-flag.
RT-Send-CC: ether [...] cpan.org
Download (untitled) / with headers
text/plain 1.1k
On Tue Oct 14 16:10:33 2014, GAAS wrote: Show quoted text
> This is done on purpose. The parser selects the most compact > way to represent the strings internally. You should not put > any semantic meaning of the internal UTF8-flag.
You both right. I'm sorry for taking your time. I wrongly identified the problem, offending module is DBD::mysql. mysql_enable_utf8 has effect just in one way, it decodes data on the way from mysql->client with sv_utf8_decode, but does not have an effect on data coming from client->mysql. It's just using SvPV when binding parameters, so if string is internally encoded in utf8 it's sent correctly, but if string is octet string it's sent as raw octets. Marc Lehmann has opened ticked regarding this problem: https://rt.cpan.org/Ticket/Display.html?id=87428 Unfortunately this problem does not have simple solution without breaking existing code. Automatic upgrade of octet strings to utf8 strings would break code which passes already encoded utf8 strings. A maintainer has suggested a new API which won't break other code, which would encode/decode data in both ways. But it's still not implemented. For now, safe but ugly way is to always call utf8::encode when binding parameters.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.