Skip Menu |
 

This queue is for tickets about the libwww-perl CPAN distribution.

Report information
The Basics
Id: 1894
Status: open
Priority: 0/
Queue: libwww-perl

People
Owner: Nobody in particular
Requestors: chs [...] newmail.ru
Cc: DDUMONT [...] cpan.org
JGMYERS [...] cpan.org
AdminCc:

Bug Information
Severity: Normal
Broken in: 5.66
Fixed in: (no value)



Subject: LWP::UserAgent can't reach https sites via proxy
Download (untitled) / with headers
text/plain 558b
Win98 + ActivePerl libwwwperl 5.66 Crypt::SSLeay; If now proxy connect to site successful, if proxy - no connection ti site. #!/usr/bin/perl use strict; use LWP::UserAgent; use HTTP::Request; my $url = 'https://www.mts.ru/'; my $ua = new LWP::UserAgent; $ua->proxy(['http','https'],'http://youproxy.com/'); my $req = new HTTP::Request('GET'=>$url); my $res = $ua->request($req); if ($res->is_success) { open(OUT,"> page.html"); print OUT $res->content; close OUT; } else { print "Error: " . $res->status_line . "\n"; }
From: sam [...] att.com
Download (untitled) / with headers
text/plain 806b
I get a negotiation failure on Solaris. Have you found any work-around or fix? res.status_line=500 SSL negotiation failed: -- thanks, sam berman [guest - Fri Dec 27 17:06:18 2002]: Show quoted text
> Win98 + ActivePerl > libwwwperl 5.66 > Crypt::SSLeay; > > If now proxy connect to site successful, if proxy - no connection ti site. > > #!/usr/bin/perl > > use strict; > > use LWP::UserAgent; > use HTTP::Request; > > my $url = 'https://www.mts.ru/'; > my $ua = new LWP::UserAgent; > > $ua->proxy(['http','https'],'http://youproxy.com/'); > > my $req = new HTTP::Request('GET'=>$url); > > my $res = $ua->request($req); > > > if ($res->is_success) { > open(OUT,"> page.html"); > print OUT $res->content; > close OUT; > } else { > print "Error: " . $res->status_line . "\n"; > }
Download (untitled) / with headers
text/plain 156b
[guest - Tue Dec 28 11:30:27 2004]: Show quoted text
> I get a negotiation failure on Solaris. Have you found any
work-around Show quoted text
> or fix?
No, may be your will more lucky.:)
From: bulb [...] ucw.cz
Download (untitled) / with headers
text/plain 1.4k
I have found a workaround for this problem. The problem is, that when libwww sees proxy settings for https, it tries to handle it like it handles http proxy, which is bogus. I mean, when setting is: https => https://proxy:port it attempts a SSL connection to proxy:port and passes GET <URI> request there. The proxy usualy does not accept SSL connections. When the setting is: https => http://proxy.port it attempts a normal conenction to proxy:port, but it still passes GET <URI> request instead of a CONNECT request. Proxies generaly refuse to answer GET https://<anything> The workaround is, that the Crypt::SSLeay can do proxy connections on it's own. When it sees $ENV{https_proxy}, it issues the right CONNECT request to the proxy. Thus the workaround is to have https_proxy set, but for LWP it must be explicitely turned on. Ie: $ua->proxy(https => undef); Then it works. It's not a good solution obviously. A good solution should be possible along the lines of: 1) LWP::Protocol::https shall override the request method. It should put the $proxy somewhere safe and dispatch to the parent with $proxy = undef 2) LWP::Protocol::https shall override _new_socket method. If there is no saved $proxy, it should dispatch to the parent. If there is, it should create a CONNECT request, dispatch it and have the socket upgraded to SSL. I have not, however, succeeded in actualy implementing it in the first take and, upon finding the workaround, gave up on further attempts.
From: Jason Smith
Download (untitled) / with headers
text/plain 635b
On Thu Jan 13 03:14:38 2005, BULB wrote: Show quoted text
> I have found a workaround for this problem. > > The workaround is, that the Crypt::SSLeay can do proxy connections on > it's own. When it sees $ENV{https_proxy}, it issues the right CONNECT > request to the proxy. Thus the workaround is to have https_proxy set, > but for LWP it must be explicitely turned on. Ie: > $ua->proxy(https => undef); > Then it works.
Can this workaround at least be added to the pod documentation until a correct fix is found? It took a lot of debugging and searching for me to figure out why it was not working and find this bug report. Thanks, ~Jason
From: lars.eggert [...] gmx.net
Download (untitled) / with headers
text/plain 294b
On Wed May 24 10:26:38 2006, guest wrote: Show quoted text
> Can this workaround at least be added to the pod documentation until a > correct fix is found? It took a lot of debugging and searching for me > to figure out why it was not working and find this bug report.
FYI, the workaround doesn't work for me.
Download (untitled) / with headers
text/plain 300b
Hi, This patch at least localizes the $ENV{HTTPS_PROXY} variable by stripping and storing the proxy parameter on the way into LWP::Protocol::http::request(), which is where the aforementioned bogusness happens. So now you can have two user agents running through different proxies. Thanks, Eric
Subject: lwp-ssl.patch
Download lwp-ssl.patch
text/x-diff 564b
diff --git a/lib/LWP/Protocol/https.pm b/lib/LWP/Protocol/https.pm index 367c8f7..62cee8f 100644 --- a/lib/LWP/Protocol/https.pm +++ b/lib/LWP/Protocol/https.pm @@ -41,6 +41,15 @@ sub _get_sock_info } } +sub request +{ + my $self = shift; + # if proxy option is set, remove it and temporarily stuff in the env + local $ENV{HTTPS_PROXY} = defined($_[1]) ? $_[1] : $ENV{HTTPS_PROXY}; + $_[1]=undef; + return $self->SUPER::request(@_); +} + #----------------------------------------------------------- package LWP::Protocol::https::Socket;
Download (untitled) / with headers
text/plain 533b
Птн Авг 27 21:38:47 2010, EWILHELM писал: Show quoted text
> Hi, > > This patch at least localizes the $ENV{HTTPS_PROXY} variable by > stripping and storing the proxy parameter on the way into > LWP::Protocol::http::request(), which is where the aforementioned > bogusness happens. > > So now you can have two user agents running through different proxies. > > Thanks, > Eric
This patch works for me. I think this is the best way to get it worked for now without much modifications. Please include it to the mainstream or comment.
Subject: [rt.cpan.org #1894] this is debian bug #503440
Date: Mon, 20 Dec 2010 15:39:26 -0500
To: bug-libwww-perl [...] rt.cpan.org
From: Daniel Kahn Gillmor <dkg [...] fifthhorseman.net>
Download (untitled) / with headers
text/plain 405b
rt.cpan.org's #1894 is the root cause of http://bugs.debian.org/503440 which is causing problems for gpgkeys_hkpms (from msva-perl) when the user has --keyserver-option http-proxy set: https://labs.riseup.net/code/issues/2677 It's also referenced here: http://www.annocpan.org/~GAAS/libwww-perl-5.834/lib/LWP/UserAgent.pm#note_751 any suggestions for how/when it might be resolved upstream? --dkg
Download signature.asc
application/pgp-signature 900b

Message body not shown because it is not plain text.

Download (untitled) / with headers
text/plain 108b
Proposed fix in the attached three patches. The fix affects this module, LWP-Protocol-https, and Net-HTTP.
Subject: libwww-perl-6.03-httpsproxy.patch
Only in .: blib diff -ru ../libwww-perl-6.03-0orig/lib/LWP/Protocol/http.pm ./lib/LWP/Protocol/http.pm --- ../libwww-perl-6.03-0orig/lib/LWP/Protocol/http.pm 2011-04-17 04:45:47.000000000 -0700 +++ ./lib/LWP/Protocol/http.pm 2011-10-18 15:08:58.000000000 -0700 @@ -15,7 +15,7 @@ sub _new_socket { - my($self, $host, $port, $timeout) = @_; + my($self, $host, $port, $timeout, $connectproxy) = @_; my $conn_cache = $self->{ua}{conn_cache}; if ($conn_cache) { if (my $sock = $conn_cache->withdraw($self->socket_type, "$host:$port")) { @@ -35,6 +35,7 @@ Timeout => $timeout, KeepAlive => !!$conn_cache, SendTE => 1, + ConnectProxy => $connectproxy, $self->_extra_sock_opts($host, $port), ); @@ -88,18 +89,26 @@ sub _fixup_header { - my($self, $h, $url, $proxy) = @_; + my($self, $h, $url, $proxy, $method) = @_; # Extract 'Host' header my $hhost = $url->authority; if ($hhost =~ s/^([^\@]*)\@//) { # get rid of potential "user:pass@" - # add authorization header if we need them. HTTP URLs do - # not really support specification of user and password, but - # we allow it. - if (defined($1) && not $h->header('Authorization')) { - require URI::Escape; - $h->authorization_basic(map URI::Escape::uri_unescape($_), - split(":", $1, 2)); + if ($method eq "CONNECT") { + if (defined($1)) { + require URI::Escape; + $h->proxy_authorization_basic(map URI::Escape::uri_unescape($_), + split(":", $1, 2)); + } + } else { + # add authorization header if we need them. HTTP URLs do + # not really support specification of user and password, but + # we allow it. + if (defined($1) && not $h->header('Authorization')) { + require URI::Escape; + $h->authorization_basic(map URI::Escape::uri_unescape($_), + split(":", $1, 2)); + } } } $h->init_header('Host' => $hhost); @@ -140,9 +149,13 @@ } my $url = $request->uri; - my($host, $port, $fullpath); + my($host, $port, $fullpath, $connectproxy); # Check if we're proxy'ing + if (defined $proxy && $url->scheme() eq 'https') { + $connectproxy = $proxy; + undef $proxy; + } if (defined $proxy) { # $proxy is an URL to an HTTP server which will proxy this request $host = $proxy->host; @@ -156,10 +169,11 @@ $port = $url->port; $fullpath = $url->path_query; $fullpath = "/$fullpath" unless $fullpath =~ m,^/,; - } + $fullpath =~ s,^/,, if $method eq "CONNECT"; + } # connect to remote site - my $socket = $self->_new_socket($host, $port, $timeout); + my $socket = $self->_new_socket($host, $port, $timeout, $connectproxy); my $http_version = ""; if (my $proto = $request->protocol) { @@ -174,7 +188,7 @@ my @h; my $request_headers = $request->headers->clone; - $self->_fixup_header($request_headers, $url, $proxy); + $self->_fixup_header($request_headers, $url, $proxy, $method); $request_headers->scan(sub { my($k, $v) = @_; Only in ./lib/LWP/Protocol: http.pm~ Only in ./lib/LWP/Protocol: http.pm.rej diff -ru ../libwww-perl-6.03-0orig/lib/LWP/UserAgent.pm ./lib/LWP/UserAgent.pm --- ../libwww-perl-6.03-0orig/lib/LWP/UserAgent.pm 2011-10-15 06:30:33.000000000 -0700 +++ ./lib/LWP/UserAgent.pm 2011-10-18 15:06:59.000000000 -0700 @@ -166,7 +166,7 @@ # Locate protocol to use my $proxy = $request->{proxy}; - if ($proxy) { + if ($proxy && $scheme ne 'https') { $scheme = $proxy->scheme; } Only in ./lib/LWP: UserAgent.pm~ Only in .: Makefile Only in .: pm_to_blib Only in ./t: CAN_TALK_TO_OURSELF
Subject: Net-HTTP-6.01-httpsproxy.patch
Only in .: blib diff -ru ../Net-HTTP-6.01-0orig/lib/Net/HTTP/Methods.pm ./lib/Net/HTTP/Methods.pm --- ../Net-HTTP-6.01-0orig/lib/Net/HTTP/Methods.pm 2011-03-20 04:36:18.000000000 -0700 +++ ./lib/Net/HTTP/Methods.pm 2011-10-18 12:24:35.000000000 -0700 @@ -135,6 +135,7 @@ push(@{${*$self}{'http_request_method'}}, $method); my $ver = ${*$self}{'http_version'}; my $peer_ver = ${*$self}{'http_peer_http_version'} || "1.0"; + $ver = '1.0' if $method eq 'CONNECT'; my @h; my @connection; @@ -162,7 +163,7 @@ if ($given{te}) { push(@connection, "TE") unless grep lc($_) eq "te", @connection; } - elsif ($self->send_te && gunzip_ok()) { + elsif ($self->send_te && gunzip_ok() && $method ne 'CONNECT') { # gzip is less wanted since the IO::Uncompress::Gunzip interface for # it does not really allow chunked decoding to take place easily. push(@h2, "TE: deflate,gzip;q=0.3"); Only in ./lib/Net/HTTP: Methods.pm~ Only in .: Makefile Only in .: pm_to_blib
Subject: LWP-Protocol-https-6.02-httpsproxy.patch
Only in .: blib diff -ru ../LWP-Protocol-https-6.02-0orig/lib/LWP/Protocol/https.pm ./lib/LWP/Protocol/https.pm --- ../LWP-Protocol-https-6.02-0orig/lib/LWP/Protocol/https.pm 2011-03-27 04:54:01.000000000 -0700 +++ ./lib/LWP/Protocol/https.pm 2011-10-18 15:03:21.000000000 -0700 @@ -43,7 +43,7 @@ } } $self->{ssl_opts} = \%ssl_opts; - return (%ssl_opts, $self->SUPER::_extra_sock_opts); + return (%ssl_opts, UA => $self->{ua}, $self->SUPER::_extra_sock_opts); } sub _check_sock @@ -86,6 +86,27 @@ require Net::HTTPS; our @ISA = qw(Net::HTTPS LWP::Protocol::http::SocketMethods); +sub new +{ + my $self = shift(); + my $arg_hash = { @_ }; + + my $proxy = delete $arg_hash->{ConnectProxy}; + my $ua = delete $arg_hash->{UA}; + + if ($proxy) { + my $uri = $proxy->clone; + $uri->path($arg_hash->{PeerAddr}.':'.$arg_hash->{PeerPort}); + my $response = $ua->request(HTTP::Request->new('CONNECT', $uri)); + + return $self->error($response->status_line) if $response->is_error; + eval { $response->{client_socket}->blocking(1); }; + return $self->start_SSL($response->{client_socket}, $arg_hash); + } + + return $self->SUPER::new(%$arg_hash); +} + 1; __END__ Only in ./lib/LWP/Protocol: https.pm~ Only in .: Makefile Only in ../LWP-Protocol-https-6.02-0orig: Makefile.old Only in .: pm_to_blib
Download (untitled) / with headers
text/plain 272b
Arguably when the method is "CONNECT", Net::HTTP::Methods::format_request should filter out more of the passed in headers, at least ignoring Connection and TE. We do want it to use any passed-in User-Agent. If requested by the maintainer, I will update the proposed fix.
From: yusufnulwala [...] indiatimes.com
Download (untitled) / with headers
text/plain 3.4k
On Tue Oct 18 18:24:00 2011, JGMYERS wrote: Show quoted text
> Proposed fix in the attached three patches. The fix affects this module, > LWP-Protocol-https, and Net-HTTP. >
After going thru your solution I did the necessary changes in the perl modules but the https requests are failing via proxy. My http request runs fine with proxy even when the direct line is connected the https code works fine it is only via proxy that it starts giving problems. This is what comes to https::new function DEBUGing: https.pm called:LWP::Protocol::https::Socket|PeerAddr|secure.lme.com|PeerPort|443|LocalAddr||Proto|tcp|Timeout|30|KeepAlive||SendTE|1|ConnectProxy|http://196.1.1.14:8080|SSL_verifycn_scheme|www|SSL_ca_file|/usr/lib/perl5/site_perl/5.14.2/Mozilla/CA/cacert.pem|SSL_verify_mode|1|UA|LWP::UserAgent=HASH(0x786360) The request object dumped out is : $VAR1 = bless( { '_content' => '', '_uri' => bless( do{\(my $o = 'http://196.1.1.14:8080/secure.lme.com:443')}, 'URI:: http' ), '_headers' => bless( {}, 'HTTP::Headers' ), '_method' => 'CONNECT' }, 'HTTP::Request' ); Prepare request http://196.1.1.14:8080/secure.lme.com:443 LWP::UserAgent=HASH(0x786360)|request_preprepare|HTTP::Request=HASH(0xf64b40) LWP::UserAgent=HASH(0x786360)|request_prepare|HTTP::Request=HASH(0xf64b40) LWP::UserAgent=HASH(0x786360)|request_send|HTTP::Request=HASH(0xf64b40) http request LWP::Protocol::http=HASH(0xdd50d8)|HTTP::Request=HASH(0xf64b40)|http://196.1.1.14:8080|||30 debugging: http.pm called:LWP::Protocol::http=HASH(0xdd50d8)|196.1.1.14|8080|30| LWP::UserAgent=HASH(0x786360)|response_done|HTTP::Response=HASH(0x11c76b0) Response object dumped : $VAR1 = bless( { '_protocol' => 'HTTP/1.0', '_content' => '', 'client_socket' => bless( \*Symbol::GEN0, 'LWP::Protocol::http::Socket' ), '_rc' => '403', '_headers' => bless( { 'proxy-connection' => 'close', 'date' => 'Mon, 19 Dec 2011 10:41:28 GMT', 'x-squid-error' => 'ERR_ACCESS_DENIED 0', 'client-peer' => '196.1.1.14:8080', 'content-length' => '445', 'client-date' => 'Mon, 19 Dec 2011 10:41:28 GMT', 'via' => '1.0 none.local:3128 (squid)', 'content-type' => 'text/html', 'server' => 'squid', 'x-cache' => 'MISS from none.local', 'x-cache-lookup' => 'NONE from none.local:3128', 'expires' => 'Mon, 19 Dec 2011 10:41:28 GMT' }, 'HTTP::Headers' ), '_msg' => 'Forbidden', '_request' => bless( { '_content' => '', '_uri' => bless( do{\(my $o = 'http://196.1.1.14:8080/secure.lme.com:443')}, 'URI::http' ), '_headers' => bless( { 'user-agent' => 'MSIE/6.0' }, 'HTTP::Headers' ), '_method' => 'CONNECT', '_uri_canonical' => $VAR1->{'_request'}{'_uri'}, 'proxy' => bless( do{\(my $o = 'http://196.1.1.14:8080')}, ' URI::http' ) }, 'HTTP::Request' ) }, 'HTTP::Response' ); Any further inputs in this regards shall be helpfull
Download (untitled) / with headers
text/plain 321b
On Mon Dec 19 06:59:36 2011, Yusuf wrote: Show quoted text
> After going thru your solution I did the necessary changes in the perl > modules but the https requests are failing via proxy.
I don't see the "ConnectProxy" and "UA" parameters added by the patch to LWP-Protocol-https, so it doesn't look like you applied all of the patches.
From: yusufnulwala [...] indiatimes.com
Download (untitled) / with headers
text/plain 9.2k
On Mon Dec 19 13:34:42 2011, JGMYERS wrote: Show quoted text
> On Mon Dec 19 06:59:36 2011, Yusuf wrote:
> > After going thru your solution I did the necessary changes in the perl > > modules but the https requests are failing via proxy.
> > I don't see the "ConnectProxy" and "UA" parameters added by the patch to > LWP-Protocol-https, so it doesn't look like you applied all of the
patches. Thanks for your reply. As suggested I have applied the following patches in the order given below. 1. lwp-ssl.patch 2. Net-HTTP-6.01-httpsproxy.patch 3. LWP-Protocol-https-6.02-httpsproxy.patch 4. libwww-perl-6.03-httpsproxy.patch But still I am not able to connect via proxy. Thanks for your valued help. Regards, My test program is: #! /opt/bin/perl5 -w use LWP::UserAgent; $ua = LWP::UserAgent->new(); $ua->proxy(['https', 'http', 'ftp'] => "http://196.1.1.14:8080"); $ua->protocols_allowed([ 'https', 'http', 'ftp' ]); $req = new HTTP::Request 'GET' => 'https://www.cmie.biz/updates/af/'; $req->header('Accept' => '*/*'); # send request $res = $ua->request($req); # check the outcome if ($res->is_success) { print $res->content; } else { print "Error: " . $res->status_line . "\n"; } Debug outputs: Case 1: Comment subroutine "_request" in LWP/Protocol/https.pm with the following patched "new" in LWP/Protocol/https.pm (patch is to enable debugging). Note: Squid Proxy Server reports the following in its access_log "1324363103.407 0 196.1.1.28 TCP_DENIED/403 778 CONNECT 196.1.1.14:8080 - NONE/- text/html" sub new { my $self = shift(); my $arg_hash = { @_ }; print STDERR "x" x 80, "\n"; print STDERR "LWP::Protocol::https new -> value of arg_hash before assigning=", Dumper($arg_hash), "\n"; print STDERR "x" x 80, "\n"; my $proxy = delete $arg_hash->{ConnectProxy}; my $ua = delete $arg_hash->{UA}; if ($proxy) { my $uri = $proxy->clone; print STDERR "uri=", Dumper($uri), "\n"; print STDERR "x" x 80, "\n"; $uri->path($arg_hash->{PeerAddr}.':'.$arg_hash->{PeerPort}); my $response = $ua->request(HTTP::Request->new('CONNECT', $uri)); return $self->error($response->status_line) if $response->is_error; eval { $response->{client_socket}->blocking(1); }; return $self->start_SSL($response->{client_socket}, $arg_hash); } return $self->SUPER::new(%$arg_hash); } Output of Case 1: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx LWP::Protocol::https new -> value of arg_hash before assigning=$VAR1 = { 'Proto' => 'tcp', 'SSL_verifycn_scheme' => 'www', 'PeerPort' => 443, 'SSL_ca_path' => '/etc/ssl/certs', 'SSL_verify_mode' => 1, 'LocalAddr' => undef, 'KeepAlive' => '', 'PeerAddr' => 'www.cmie.biz', 'SendTE' => 1, 'UA' => bless( { 'max_redirect' => 7, 'ssl_opts' => { 'verify_hostname' => 1 }, 'protocols_forbidden' => undef, 'show_progress' => undef, 'handlers' => { 'response_header' => bless( [ { 'owner' => 'LWP::UserAgent::parse_head', 'callback' => sub { "DUMMY" }, 'm_media_type' => 'html', 'line' => 'LWP/UserAgent.pm:658' } ], 'HTTP::Config' ), 'request_preprepare' => bless( [ { 'owner' => 'LWP::UserAgent::proxy', 'callback' => sub { "DUMMY" }, 'line' => 'LWP/UserAgent.pm:966' } ], 'HTTP::Config' ) }, 'no_proxy' => [], 'protocols_allowed' => [ 'https', 'http', 'ftp' ], 'local_address' => undef, 'use_eval' => 1, 'requests_redirectable' => [ 'GET', 'HEAD' ], 'timeout' => 180, 'def_headers' => bless( { 'user-agent' => 'libwww-perl/6.03' }, 'HTTP::Headers' ), 'proxy' => { 'ftp' => 'http://196.1.1.14:8080', 'http' => 'http://196.1.1.14:8080', 'https' => 'http://196.1.1.14:8080' }, 'max_size' => undef }, 'LWP::UserAgent' ), 'ConnectProxy' => bless( do{\(my $o = 'http://196.1.1.14:8080')}, 'URI::http' ), 'Timeout' => 180 }; xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx uri=$VAR1 = bless( do{\(my $o = 'http://196.1.1.14:8080')}, 'URI::http' ); xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Error: 500 Can't connect to www.cmie.biz:443 Case 2: With "request" in LWP/Protocol/https.pm Output of Case 2: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx LWP::Protocol::https new -> value of arg_hash before assigning=$VAR1 = { 'Proto' => 'tcp', 'SSL_verifycn_scheme' => 'www', 'PeerPort' => 443, 'SSL_ca_path' => '/etc/ssl/certs', 'SSL_verify_mode' => 1, 'LocalAddr' => undef, 'KeepAlive' => '', 'PeerAddr' => 'www.cmie.biz', 'SendTE' => 1, 'UA' => bless( { 'max_redirect' => 7, 'ssl_opts' => { 'verify_hostname' => 1 }, 'protocols_forbidden' => undef, 'show_progress' => undef, 'handlers' => { 'response_header' => bless( [ { 'owner' => 'LWP::UserAgent::parse_head', 'callback' => sub { "DUMMY" }, 'm_media_type' => 'html', 'line' => 'LWP/UserAgent.pm:658' } ], 'HTTP::Config' ), 'request_preprepare' => bless( [ { 'owner' => 'LWP::UserAgent::proxy', 'callback' => sub { "DUMMY" }, 'line' => 'LWP/UserAgent.pm:966' } ], 'HTTP::Config' ) }, 'no_proxy' => [], 'protocols_allowed' => [ 'https', 'http', 'ftp' ], 'local_address' => undef, 'use_eval' => 1, 'requests_redirectable' => [ 'GET', 'HEAD' ], 'timeout' => 180, 'def_headers' => bless( { 'user-agent' => 'libwww-perl/6.03' }, 'HTTP::Headers' ), 'proxy' => { 'ftp' => 'http://196.1.1.14:8080', 'http' => 'http://196.1.1.14:8080', 'https' => 'http://196.1.1.14:8080' }, 'max_size' => undef }, 'LWP::UserAgent' ), 'ConnectProxy' => undef, 'Timeout' => 180 }; xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Error: 500 Can't connect to www.cmie.biz:443 (Connection timed out)
From: yusufnulwala [...] indiatimes.com
Download (untitled) / with headers
text/plain 802b
hi In continuation of the above i.e. after applying the given patches and some minor change I am able to get the page. This code works fine. use Net::SSL; $ua = new LWP::UserAgent(ssl_opts => { verify_hostname => 0 }); New Program : #! /opt/bin/perl5 -w use Net::SSL; use LWP::UserAgent; $ua = new LWP::UserAgent(ssl_opts => { verify_hostname => 0 }); $ua->proxy(['https', 'http', 'ftp'] => "http://196.1.1.14:8080"); $ua->protocols_allowed([ 'https', 'http', 'ftp' ]); $req = new HTTP::Request 'GET' => 'https://www.cmie.biz/updates/af/'; $req->header('Accept' => '*/*'); # send request $res = $ua->request($req); # check the outcome if ($res->is_success) { print $res->content; } else { print "Error: " . $res->status_line . "\n"; } The only drawback being verify_hostname is off.
Download (untitled) / with headers
text/plain 365b
On Tue Dec 20 01:50:40 2011, Yusuf wrote: Show quoted text
> As suggested I have applied the following patches in the order given > below. > > 1. lwp-ssl.patch > 2. Net-HTTP-6.01-httpsproxy.patch > 3. LWP-Protocol-https-6.02-httpsproxy.patch > 4. libwww-perl-6.03-httpsproxy.patch
You should not have applied lwp-ssl.patch. That is not from me and is not part of the fix.
From: yusufnulwala [...] indiatimes.com
Download (untitled) / with headers
text/plain 1.4k
On Tue Dec 20 11:11:59 2011, JGMYERS wrote: Show quoted text
> On Tue Dec 20 01:50:40 2011, Yusuf wrote:
> > As suggested I have applied the following patches in the order given > > below. > > > > 1. lwp-ssl.patch > > 2. Net-HTTP-6.01-httpsproxy.patch > > 3. LWP-Protocol-https-6.02-httpsproxy.patch > > 4. libwww-perl-6.03-httpsproxy.patch
> > You should not have applied lwp-ssl.patch. That is not from me and is > not part of the fix.
After applying your patch i.e. Net-HTTP-6.01-httpsproxy.patch, LWP-Protocol-https-6.02-httpsproxy.patch, libwww-perl-6.03-httpsproxy.patch In my program if I write the following then things work use Net::SSL; $ua = new LWP::UserAgent(ssl_opts => { verify_hostname => 0 }); i.e. I have to switch of verify_hostname => 0 otherwise it gives an error. Is this the correct way because hostname verification is lost. Please advice. An a millions thanks for your prompt feedback and patch they really have helped. New Program : #! /opt/bin/perl5 -w use Net::SSL; use LWP::UserAgent; $ua = new LWP::UserAgent(ssl_opts => { verify_hostname => 0 }); $ua->proxy(['https', 'http', 'ftp'] => "http://196.1.1.14:8080"); $ua->protocols_allowed([ 'https', 'http', 'ftp' ]); $req = new HTTP::Request 'GET' => 'https://www.cmie.biz/updates/a f/'; $req->header('Accept' => '*/*'); # send request $res = $ua->request($req); # check the outcome if ($res->is_success) { print $res->content; } else { print "Error: " . $res->status_line . "\n"; }
Download (untitled) / with headers
text/plain 190b
Hostname verification has worked in the tests I have run. Your latest comment is not a proper bug report, as it is missing detailed reproduction steps, expected results, and actual results.
From: yusufnulwala [...] indiatimes.com
On Wed Dec 21 14:42:36 2011, JGMYERS wrote: Show quoted text
> Hostname verification has worked in the tests I have run. Your latest > comment is not a proper bug report, as it is missing detailed > reproduction steps, expected results, and actual results.
The program is : #! /opt/bin/perl5 -w use Net::SSL; use LWP::UserAgent; $ua = new LWP::UserAgent(ssl_opts => { verify_hostname => 1 }); $ua->proxy(['https', 'http', 'ftp'] => "http://196.1.1.14:8080"); $ua->protocols_allowed([ 'https', 'http', 'ftp' ]); $req = new HTTP::Request 'GET' => 'https://www.cmie.biz/updates/'; $req->header('Accept' => '*/*'); # send request $res = $ua->request($req); # check the outcome if ($res->is_success) { print $res->content; } else { print "Error: " . $res->status_line . "\n"; }; Error statement : Error: 500 Can't connect to www.cmie.biz:443 (Crypt-SSLeay can't verify hostnames) But when I change the object creation statement to $ua = new LWP::UserAgent(ssl_opts => { verify_hostname => 0 }); It successfully downloads the page.
From: yusufnulwala [...] indiatimes.com
Download (untitled) / with headers
text/plain 715b
On Fri Dec 30 02:47:53 2011, Yusuf wrote: Show quoted text
> On Wed Dec 21 14:42:36 2011, JGMYERS wrote:
> > Hostname verification has worked in the tests I have run. Your latest > > comment is not a proper bug report, as it is missing detailed > > reproduction steps, expected results, and actual results.
>
I also tried it adding the following use Mozilla::CA; $ua = new LWP::UserAgent( ssl_opts => { verify_hostname => 1, SSL_ca_file => Mozilla::CA::SSL_ca_file() } ); But it still give the same error. [Error: 500 Can't connect to www.cmie.biz:443 (Crypt-SSLeay can't verify hostnames)
Download (untitled) / with headers
text/plain 370b
On Fri Dec 30 02:47:53 2011, Yusuf wrote: Show quoted text
> Error: 500 Can't connect to www.cmie.biz:443 (Crypt-SSLeay can't verify > hostnames)
Did you try without using a proxy, to determine whether the proxy support is a contributing factor. If it doesn't work without proxy support, then it's an unrelated problem. Your problem might be that Crypt-SSLeay can't verify hostnames.
From: victor [...] vsespb.ru
Download (untitled) / with headers
text/plain 769b
Seems the following fix works for me (found here http://www.perlmonks.org/?node_id=994683) #!/usr/bin/perl use MIME::Base64; use Net::SSL (); # From Crypt-SSLeay use LWP::UserAgent; $Net::HTTPS::SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL for my $p (qw!http://127.0.0.1:3128 http://127.0.0.1:3333!) {# http://127.0.0.1:3333 http://127.0.0.1:3128 $ENV{HTTPS_PROXY} = $p; $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 1; $ENV{"HTTPS_PROXY_USERNAME"} = 'puser'; $ENV{"HTTPS_PROXY_PASSWORD"} = 'puser'; my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new('GET','https://bing.com/'); my $res = $ua->request($req); print $res->dump(); print "\n"; } (it supports HTTPS proxy, password protected proxies, different proxies for different requests)
Download (untitled) / with headers
text/plain 1.3k
Also, for one who testing this or who want to make sure their code actually use the proxy (in different environments): you can verify that $response->header('Client-Peer') is actually points to the proxy. $response->header('Client-Peer') is actual IP address of the peer. Here is the code from LWP: Show quoted text
> if (defined(my $peerhost = $sock->peerhost)) { > $res->header("Client-Peer" => "$peerhost:" . $sock->peerport); > }
(note that Clietn-Peer is only IP, not DNS name) On Mon Jan 07 02:14:10 2013, vsespb wrote: Show quoted text
> > Seems the following fix works for me (found here > http://www.perlmonks.org/?node_id=994683) > > > #!/usr/bin/perl > > use MIME::Base64; > use Net::SSL (); # From Crypt-SSLeay > use LWP::UserAgent; > $Net::HTTPS::SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL > > for my $p (qw!http://127.0.0.1:3128 http://127.0.0.1:3333!) {# > http://127.0.0.1:3333 http://127.0.0.1:3128 > $ENV{HTTPS_PROXY} = $p; > $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 1; > $ENV{"HTTPS_PROXY_USERNAME"} = 'puser'; > $ENV{"HTTPS_PROXY_PASSWORD"} = 'puser'; > > > my $ua = LWP::UserAgent->new(); > my $req = HTTP::Request->new('GET','https://bing.com/'); > my $res = $ua->request($req); > print $res->dump(); > print "\n"; > } > > (it supports HTTPS proxy, password protected proxies, different proxies > for different requests)
RT-Send-CC: dkg [...] fifthhorseman.net
Download (untitled) / with headers
text/plain 1.6k
ok. I'm giving this a try. I've applied the 3 patches from JGMYERS on my Debian/sid packages (which are up-to-date compared to CPAN). I'm trying to connect to https servers using tinyproxy. This may not be the best proxy, but it's one I can install easily on my laptop so I can check the logs. First of all, https request coming from Firefox ares accepted and generate the following tinyproxy logs: CONNECT Oct 11 19:47:50 [8443]: Connect (file descriptor 6): localhost [127.0.0.1] CONNECT Oct 11 19:47:50 [8443]: Request (file descriptor 6): CONNECT rt.cpan.org:443 HTTP/1.1 INFO Oct 11 19:47:50 [8443]: No upstream proxy for rt.cpan.org But the https connection coming from patched LWP::UserAgent fails: CONNECT Oct 11 19:38:10 [16212]: Connect (file descriptor 6): localhost [127.0.0.1] CONNECT Oct 11 19:38:10 [16212]: Request (file descriptor 6): CONNECT 127.0.0.1:8888 HTTP/1.0 INFO Oct 11 19:38:10 [16212]: Refused CONNECT method on port 8888 I believe that doing CONNECT on the proxy itself (127.0.0.1) is wrong. I don't know how to fix that. Turns out that tinyproxy rejection triggers error handling in LWP::Protocol::https::Socket::new. I get the following message: 500 Usage: IO::Handle::error(handle) Looks like this line from Net-HTTP-6.01-httpsproxy.patch is not correct: return $self->error($response->status_line) if $response->is_error; But that another (minor) issue. Feel free to get back to me to perform further test. I can test behind tinyproxy and behind a corporate firewall. Once I have a working solution, I can upload patched packages in Debian/experimental to have more user feedback. All the best
Subject: LWP::UserAgent with JGMYERS patches *can* reach https sites via proxy
Download (untitled) / with headers
text/plain 546b
ok, I had a case of PEBKAC. The following program is working fine with JGMYERS patches and a local tinyproxy setup : #!/usr/bin/perl use LWP::UserAgent; for my $p (qw!http://127.0.0.1:8888!) { $ENV{HTTPS_PROXY} = $p; $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 1; my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new('GET','https://bing.com/'); my $res = $ua->request($req); print $res->dump(); print "\n"; } Using this stuff cannot be simpler :-) JGMYERS, thanks a bunch for your patches All the best
Download (untitled) / with headers
text/plain 957b
Sorry about the previous message. I was wrong: the program I posted bypasses the proxy. First of all, the URL suggested in the example above is a bad idea: https://bing.com sends back a 301 and redirects toward an http url. This is really confusing when testing proxies. I'm now doing tests with a repo url from github. (no redirection) I've tried again JGMYERS patches, but they work only when setting env_proxy to 0. Which does not bring obvious benefits compared to the solution suggested by Crypt::SSLeay man page ("PROXY SUPPORT" section). To get a LWP::UserAgent that work through a proxy with both http and https request (with CONNECT), a simple solution is to modify LWP::UserAgent to avoid storing https_proxy. (See https://github.com/libwww-perl/libwww-perl/pull/51) This should solve most of the use cases. On the other hand, calling $ua->proxy('https',...) will not use SSLeay so the proxy connection will not work with most proxies.
Download (untitled) / with headers
text/plain 623b
On Sun Oct 27 11:08:43 2013, DDUMONT wrote: Show quoted text
> I've tried again JGMYERS patches, but they work only when setting > env_proxy to 0.
My patches are not intended to be used with env_proxy. Show quoted text
> Which does not bring obvious benefits compared to the > solution suggested by Crypt::SSLeay man page ("PROXY SUPPORT" > section).
The Crypt::SSLeay solution is not thread safe, as the environment cannot be changed in a thread safe manner. Two different threads cannot have two different proxy configurations. The only safe approach with that solution is to either have all connections use the same proxy or none of them use a proxy.
Download (untitled) / with headers
text/plain 581b
I've added another way to patch the problem and added it as pull requests to libwww-perl and lwp-protocol-https repositories on github. My patches work with Net::SSL and IO::Socket::SSL backends, work with keep-alive, proxy-authorization etc. With IO::Socket::SSL backend one can even switch to SSL inside an already established socket, e.g. after some plain HTTP requests. And, the change contains an extensiv test to make sure that all features work as intended. See https://github.com/libwww-perl/libwww-perl/pull/52, https://github.com/libwww-perl/lwp-protocol-https/pull/7
Download (untitled) / with headers
text/plain 803b
On Sat Nov 09 15:05:03 2013, SULLR wrote: Show quoted text
> > I've added another way to patch the problem and added it as pull > requests to libwww-perl and lwp-protocol-https repositories on github. > My patches work with Net::SSL and IO::Socket::SSL backends, work with > keep-alive, proxy-authorization etc. With IO::Socket::SSL backend one > can even switch to SSL inside an already established socket, e.g. > after some plain HTTP requests. > And, the change contains an extensiv test to make sure that all > features work as intended. > > See https://github.com/libwww-perl/libwww-perl/pull/52, > https://github.com/libwww-perl/lwp-protocol-https/pull/7
That looked like a lot of work. Thank you. I hope it makes it in. (You don’t want to see the crazy workarounds my scripts have used up till now. :-)
Download (untitled) / with headers
text/plain 197b
Le Sam 09 Nov 2013 22:28:11, SPROUT a écrit : Show quoted text
> That looked like a lot of work. Thank you. I hope it makes it in.
Stephen patches are merged upstream. I hope a new release will be done soon.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.