Skip Menu |
 

This queue is for tickets about the Net-SSLeay CPAN distribution.

Report information
The Basics
Id: 102640
Status: rejected
Priority: 0/
Queue: Net-SSLeay

People
Owner: MIKEM [...] cpan.org
Requestors: KENTNL [...] cpan.org
Cc: ether [...] cpan.org
AdminCc:

Bug Information
Severity: (no value)
Broken in: 1.68
Fixed in: (no value)



Subject: HTTP POST requests over 90kb fail with OpenSSL 1.0.2

I've had a hard time nailing this issue down as to where its actually going wrong.

All the facts I know are, that this issue was not present on 1.0.1l, and is present on 1.0.2

Under 1.0.2 , https://metacpan.org/source/ETHER/libwww-perl-6.13/lib/LWP/Protocol/http.pm#L282 <-- this line fails  when the socket is an IO::Socket::SSL socket.

Using an MITM to unwrap the SSL and provide plain HTTP to the same requests results in no error.

Downgrading openssl to 1.0.1l also results in no error.

The error dies with $! being "write failed: Bad address" which is a C Level error to do with bad pointers ( usually ).

And I've found similar issues with similar upload thresholds triggering the same Bad address error on ruby projects ( https://github.com/excon/excon/issues/467 )

But there's no obvious smoking gun, there's no related articles I can see at OpenSSL's official forums, so at best I might assume there's some kind of ABI missmatch 1.0.2 introduces that needs to be adjusted for.

Unfortunately I don't know what other data might be helpful in replicating this issue, other than to say that uploading the following files to PAUSE using cpan-uploader ( LWP -> IO::Socket::SSL -> Net::SSLeay )  reliably replicated the failure, and circumvention was required to get uploads working:

https://cpan.metacpan.org/authors/id/K/KE/KENTNL/Color-Library-Dictionary-XKCD-1.000000.tar.gz

https://cpan.metacpan.org/authors/id/K/KE/KENTNL/Dist-Zilla-PluginBundle-Author-KENTNL-2.024001.tar.gz
https://cpan.metacpan.org/authors/id/K/KE/KENTNL/Dist-Zilla-PluginBundle-Author-KENTNL-2.024000.tar.gz





 

Subject: Re: [rt.cpan.org #102640] HTTP POST requests over 90kb fail with OpenSSL 1.0.2
Date: Mon, 09 Mar 2015 15:34:05 +1000
To: bug-Net-SSLeay [...] rt.cpan.org
From: Mike McCauley <mikem [...] airspayce.com>
Download (untitled) / with headers
text/plain 2.4k
Hello Kent, I havent been able to reproduce this with cpan-upload If you can provide some sample code or command line to demonstrate the problem I will do my best with it. Cheers. On Sunday, March 08, 2015 11:46:00 AM you wrote: Show quoted text
> Sun Mar 08 11:45:59 2015: Request 102640 was acted upon. > Transaction: Ticket created by KENTNL > Queue: Net-SSLeay > Subject: HTTP POST requests over 90kb fail with OpenSSL 1.0.2 > Broken in: 1.68 > Severity: (no value) > Owner: Nobody > Requestors: KENTNL@cpan.org > Status: new > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=102640 > > > > I've had a hard time nailing this issue down as to where its actually going > wrong. > > All the facts I know are, that this issue was not present on 1.0.1l, and is > present on 1.0.2 > > Under 1.0.2 , > https://metacpan.org/source/ETHER/libwww-perl-6.13/lib/LWP/Protocol/http.pm# > L282 <-- this line fails when the socket is an IO::Socket::SSL socket. > > Using an MITM to unwrap the SSL and provide plain HTTP to the same requests > results in no error. > > Downgrading openssl to 1.0.1l also results in no error. > > The error dies with $! being "write failed: Bad address" which is a C Level > error to do with bad pointers ( usually ). > > And I've found similar issues with similar upload thresholds triggering the > same Bad address error on ruby projects ( > https://github.com/excon/excon/issues/467 ) > > But there's no obvious smoking gun, there's no related articles I can see at > OpenSSL's official forums, so at best I might assume there's some kind of > ABI missmatch 1.0.2 introduces that needs to be adjusted for. > > Unfortunately I don't know what other data might be helpful in replicating > this issue, other than to say that uploading the following files to PAUSE > using cpan-uploader ( LWP -> IO::Socket::SSL -> Net::SSLeay ) reliably > replicated the failure, and circumvention was required to get uploads > working: > > https://cpan.metacpan.org/authors/id/K/KE/KENTNL/Color-Library-Dictionary-XK > CD-1.000000.tar.gz > > https://cpan.metacpan.org/authors/id/K/KE/KENTNL/Dist-Zilla-PluginBundle-Aut > hor-KENTNL-2.024001.tar.gz > https://cpan.metacpan.org/authors/id/K/KE/KENTNL/Dist-Zilla-PluginBundle-Au > thor-KENTNL-2.024000.tar.gz
-- Mike McCauley VK4AMM mikem@airspayce.com Airspayce Pty Ltd 9 Bulbul Place Currumbin Waters QLD 4223 Australia http://www.airspayce.com Phone +61 7 5598-7474

Earlier today RJBS was seen on #toolchain having the same issue. But it appeared semi-transient for him.

 

I've been trying to make a reduced case, but the code I'm trying to simplify and retain the failure condition is way way beyond my mental capacity to process and I've not been able to get anyone to assist me.

 

I need to find a way to reduce the code to its absolute minimum retaining the PAUSE failure, and then when I'm sure the code can be simplified no more, I have to make a non-pause server that assists in replicating the failure.

 

And then I have to work out what series of OpenSSL compilation options I happen to have that built my system OpenSSL install that cause the issue.

 

But I haven't even done 1/3 :(

 

Download (untitled) / with headers
text/plain 869b
I have the feeling that this problem might be related to CVE-2015-0290 "OpenSSL Multiblock Corrupted Pointer Issue". The diff at https://git.openssl.org/?p=openssl.git;a=commitdiff;h=77c77f0a1b9f15b869ca3342186dfbedd1119d0e;hp=8b84495380098592ef7bb2fa9209ccb87803bf1d indicates for me that this situation might happen sometimes with a combination of a multiblock-capable cipher (AES-NI) and non-blocking I/O when the server does not read data fast enough so that the socket buffer will fill up and BIO_write will not be able to write the full SSL fragment to the socket. This would explain also if it is happening with larger data only, because it takes some amount of data to fill up the socket buffer. This OpenSSL problem was fixed in 1.02a which was only released after this bug report. So it would be good to know if this problem still happens with OpenSSL 1.02a.

>
> This OpenSSL problem was fixed in 1.02a which was only released after
> this bug report. So it would be good to know if this problem still
> happens with OpenSSL 1.02a.

Indeed. I haven't personally seen it myself in a long time now, ( having upgraded to 1.0.2a around March 21 ) which is good because it happened repeatedly before.

 

The ruby instance I referenced with the same error also appears to be solved in the same way.
 

So I'm relatively comfortable in putting this one to bed and just having some kind of "Your OpenSSL library version is in a list of ones known to be critically broken, plz upgrade" advice.

Download (untitled) / with headers
text/plain 141b
Just to reiterate: apparently due to a bug in OPenSSL that is fixed in OpenSSL 1.0.2a. Thanks to Steffen Ulrich for shedding light on this.
Download (untitled) / with headers
text/plain 233b
On 2015-05-26 01:51:15, MIKEM wrote: Show quoted text
> Just to reiterate: > apparently due to a bug in OPenSSL that is fixed in OpenSSL 1.0.2a.
Maybe this issue could be mentioned in Net::SSLeay's Pod page, in the "KNOWN BUGS AND CAVEATS" section?
Subject: Re: [rt.cpan.org #102640] HTTP POST requests over 90kb fail with OpenSSL 1.0.2
Date: Tue, 26 May 2015 16:22:49 +1000
To: bug-Net-SSLeay [...] rt.cpan.org
From: Mike McCauley <mikem [...] airspayce.com>
Download (untitled) / with headers
text/plain 601b
Done. Cheers. On Tuesday, May 26, 2015 02:17:25 AM you wrote: Show quoted text
> Queue: Net-SSLeay > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=102640 > > > On 2015-05-26 01:51:15, MIKEM wrote:
> > Just to reiterate: > > apparently due to a bug in OPenSSL that is fixed in OpenSSL 1.0.2a.
> > Maybe this issue could be mentioned in Net::SSLeay's Pod page, in the "KNOWN > BUGS AND CAVEATS" section?
-- Mike McCauley VK4AMM mikem@airspayce.com Airspayce Pty Ltd 9 Bulbul Place Currumbin Waters QLD 4223 Australia http://www.airspayce.com Phone +61 7 5598-7474


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.