This queue is for tickets about the Net-OpenID-Consumer CPAN distribution.

Report information
The Basics
Id:
106930
Status:
open
Priority:
Low/Low

People
Owner:
Nobody in particular
Requestors:
oholecek [...] suse.com
Cc:
AdminCc:

BugTracker
Severity:
(no value)
Broken in:
(no value)
Fixed in:
(no value)



Subject: Verification fail when in some fields contain unicode characters
Date: Mon, 7 Sep 2015 15:15:48 +0200
To: <bug-Net-OpenID-Consumer@rt.cpan.org>
From: Ondrej Holecek <oholecek@suse.com>
Hi, when some fields returned by openid server contain unicode characters subsequent HTTP verification fail (server returns is_valid: false). But when "charset=UTF-8" is added to request, verification succeed. Thanks, OH Module: perl-Net-OpenID-Consumer-1.16-2.1.noarch Distri: openSUSE Tumbleweed (20150903) Perl: v5.22.0 Example diff of my solution: diff --git a/lib/Net/OpenID/Consumer.pm b/lib/Net/OpenID/Consumer.pm index a2261cb..9a9953b 100644 --- a/lib/Net/OpenID/Consumer.pm +++ b/lib/Net/OpenID/Consumer.pm @@ -976,7 +976,7 @@ sub verified_identity { $post{"openid.mode"} = "check_authentication"; my $req = HTTP::Request->new(POST => $server); - $req->header("Content-Type" => "application/x-www-form-urlencoded"); + $req->header("Content-Type" => "application/x-www-form-urlencoded; charset=UTF-8"); $req->content(join("&", map { "$_=" . uri_escape_utf8($post{$_}) } keys %post)); my $ua = $self->ua;

Message body not shown because it is not plain text.

according to the spec, application/x-www-form-urlencoded does not actually take a charset parameter and according to https://bugzilla.mozilla.org/show_bug.cgi?id=289060#c8 it's a bad idea to add one because various servers in the wild will get confused even if the one you're corresponding with happens to accept this. I'm also inclined to think this is a server failure since I'm not sure why the server *wouldn't* be decoding parameter values in OpenID messages as UTF-8 anyway, but without an actual example I'm not clear on what's actually happening in your case.
Subject: Re: [rt.cpan.org #106930] Verification fail when in some fields contain unicode characters
Date: Fri, 15 Jan 2016 14:38:47 +0100
To: bug-Net-OpenID-Consumer@rt.cpan.org
From: Ondrej Holecek <oholecek@suse.com>
Show quoted text
> according to the spec, > application/x-www-form-urlencoded does not actually take a charset parameter > and according to https://bugzilla.mozilla.org/show_bug.cgi?id=289060#c8 > it's a bad idea to add one because various servers in the wild will get > confused even if the one you're corresponding with happens to accept this.
Make sense. Do you have some other suggestions how to explicitly inform server it's in utf-8 encoding?
Show quoted text
> > I'm also inclined to think this is a server failure since I'm not sure why > the server *wouldn't* be decoding parameter values in OpenID messages as > UTF-8 anyway, but without an actual example I'm not clear on what's > actually happening in your case.
I don't have access to server configuration. OpenID provider in question was https://www.opensuse.org/openid/user/ if you want to check. The actual problem was that some peoples names contain non ascii characters and when they tried to log in then request verification failed.

Message body not shown because it is not plain text.

Two possibilities: (1) the HTML5 solution to this is evidently to add a hidden field named '_charset_' to the form; browser then fills this in when POSTing and server is supposed to check for its existence in a POST request and decode accordingly. I'm guessing legacy OpenID servers are unlikely to know about this but it may be worth a try. (if it works it's probably the simplest solution). (2) Use multipart/form-data instead of application/x-www-form-urlencoded. This allows individual Content-Type: headers on each of the fields and seems to be the Right Thing but getting LWP::UserAgent to actually do it looks cumbersome.
On Mon Sep 07 09:16:14 2015, oholecek@suse.com wrote:
Show quoted text
> Hi, > > when some fields returned by openid server contain unicode characters > subsequent HTTP verification fail (server returns is_valid: false). > But when > "charset=UTF-8" is added to request, verification succeed.
I don't think this analysis is correct. I have done some research, and the issue seems to be that the server returns UTF-8 data in a URL-encoded query. Since you're running a CGI script, this data is passed through an environment variable (QUERY_STRING). The CGI module does not treat that data as UTF-8. It cannot do that, because a query argument may in fact be an arbitrary byte stream. Net::OpenID does not that data either, so the UTF-8 encoded byte stream is treated as a Latin1 string, and that's why it fails. I can make it work by reverting commit c392be3a2180b742e940f8ccd42bedcb8e8e6cd6, but it seems Evgeniy had a different setup that was fixed by that commit. See also RT 80329.
On Thu Nov 21 15:40:29 2019, petr@tesarici.cz wrote:
Show quoted text
> On Mon Sep 07 09:16:14 2015, oholecek@suse.com wrote:
> > Hi, > > > > when some fields returned by openid server contain unicode characters > > subsequent HTTP verification fail (server returns is_valid: false). > > But when > > "charset=UTF-8" is added to request, verification succeed.
> > I don't think this analysis is correct. I have done some research, and > the issue seems to be that the server returns UTF-8 data in a URL- > encoded query. Since you're running a CGI script, this data is passed > through an environment variable (QUERY_STRING). The CGI module does > not treat that data as UTF-8. It cannot do that, because a query > argument may in fact be an arbitrary byte stream. Net::OpenID does not > that data either, so the UTF-8 encoded byte stream is treated as a > Latin1 string, and that's why it fails. > > I can make it work by reverting commit > c392be3a2180b742e940f8ccd42bedcb8e8e6cd6, but it seems Evgeniy had a > different setup that was fixed by that commit. See also RT 80329.
Um, actually, no, to make things work I need both: reverting commit c392be3a _AND_ adding charset=UTF-8. At this point, I think the situation with www.opensuse.org is hopeless.


This service runs on Request Tracker, is sponsored by The Perl Foundation, and maintained by Best Practical Solutions.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.