Skip Menu |
 

This queue is for tickets about the Crypt-SSLeay CPAN distribution.

Report information
The Basics
Id: 41007
Status: open
Priority: 0/
Queue: Crypt-SSLeay

People
Owner: nanis [...] runu.moc.invalid
Requestors: t-bitcard-perlbug [...] snowelm.com
Cc: kmx [...] cpan.org
cpan [...] barely3am.com
AdminCc:

Bug Information
Severity: Normal
Broken in: 0.57_01
Fixed in: (no value)



Subject: [PATCH} ithreads support
Download (untitled) / with headers
text/plain 813b
Using perl ithreads, we can set up multiple LWP::UserAgents to fetch many pages simultaneously. However, when more than two threads are accessing https:// URLs, perl occasionally reports corrupted memory and dies. This is because Crypt::SSLeay lacks mutex setup that is required by OpenSSL library. The attached patch is a trial to fix this problem. The patch is for Crypt::SSLeay 0.57_01. Unfortunately the patch is pthread-specific, so more work is necessary to support other thread libraries. I tested the patch on Linux 2.6 (Debian etch) and it seems working. Some code in the patch is copied from http://www.nabble.com/When-to-use-CRYPTO_set_locking_callback()-and- CRYPTO_set_id_callback()--td5849882.html I hope (more complete) thread support is included in the future version of Crypt::SSLeay.
Subject: crypt.diff
Download crypt.diff
text/x-diff 4k
diff -r -c Crypt-SSLeay-0.57_01/META.yml Crypt-SSLeay-0.57_01+ithreads/META.yml *** Crypt-SSLeay-0.57_01/META.yml 2008-02-18 23:43:14.000000000 +0900 --- Crypt-SSLeay-0.57_01+ithreads/META.yml 2008-11-13 09:53:09.000000000 +0900 *************** *** 1,6 **** --- #YAML:1.0 name: Crypt-SSLeay ! version: 0.57_01 abstract: OpenSSL support for LWP license: perl author: --- 1,6 ---- --- #YAML:1.0 name: Crypt-SSLeay ! version: 0.57_01+ithreads abstract: OpenSSL support for LWP license: perl author: diff -r -c Crypt-SSLeay-0.57_01/SSLeay.pm Crypt-SSLeay-0.57_01+ithreads/SSLeay.pm *** Crypt-SSLeay-0.57_01/SSLeay.pm 2008-02-18 23:33:14.000000000 +0900 --- Crypt-SSLeay-0.57_01+ithreads/SSLeay.pm 2008-11-13 09:51:28.000000000 +0900 *************** *** 2,8 **** use strict; use vars '$VERSION'; ! $VERSION = '0.57_01'; eval { require XSLoader; --- 2,8 ---- use strict; use vars '$VERSION'; ! $VERSION = '0.57_01+ithreads'; eval { require XSLoader; diff -r -c Crypt-SSLeay-0.57_01/SSLeay.xs Crypt-SSLeay-0.57_01+ithreads/SSLeay.xs *** Crypt-SSLeay-0.57_01/SSLeay.xs 2008-02-18 23:11:15.000000000 +0900 --- Crypt-SSLeay-0.57_01+ithreads/SSLeay.xs 2008-11-13 11:37:55.000000000 +0900 *************** *** 18,23 **** --- 18,24 ---- #define PERL5 1 #endif + #define OPENSSL_THREAD_DEFINES /* ssl.h or openssl/ssl.h is included from the crypt_ssleay_version * file which is written when building with perl Makefile.PL * #include "ssl.h" *************** *** 25,34 **** --- 26,73 ---- #include "crypt_ssleay_version.h" #undef Free /* undo namespace pollution from crypto.h */ + + #if defined(OPENSSL_THREADS) + #include <pthread.h> + #endif #ifdef __cplusplus } #endif + #if defined(OPENSSL_THREADS) + static pthread_mutex_t *mutex_buf = NULL; + static int mutex_use = 0; + static int mutex_free = 0; + static pthread_mutex_t mutex_lib; + + /** + * OpenSSL locking function. + * + * @param mode lock mode + * @param n lock number + * @param file source file name + * @param line source file line number + * @return none + */ + static void locking_function(int mode, int n, const char *file, int line) + { + if (mode & CRYPTO_LOCK) { + pthread_mutex_lock(&mutex_buf[n]); + } else { + pthread_mutex_unlock(&mutex_buf[n]); + } + } + + /** + * OpenSSL uniq id function. + * + * @return thread id + */ + static unsigned long id_function(void) + { + return ((unsigned long) pthread_self()); + } + #endif // OPENSSL_THREADS /* moved this out to Makefile.PL so user can * see value being used printed during build *************** *** 103,109 **** --- 142,171 ---- static int bNotFirstTime; char buf[1024]; int rand_bytes_read; + int i; + #if defined(OPENSSL_THREADS) + if( mutex_use == 0 ) + { + mutex_use++; + pthread_mutex_init( &mutex_lib, NULL ); + pthread_mutex_lock( &mutex_lib ); + /* static locks area */ + mutex_buf = malloc(CRYPTO_num_locks() * sizeof(pthread_mutex_t)); + //if (mutex_buf == NULL) { + // return (-1); + //} + for (i = 0; i < CRYPTO_num_locks(); i++) { + pthread_mutex_init(&mutex_buf[i], NULL); + } + /* static locks callbacks */ + CRYPTO_set_locking_callback(locking_function); + CRYPTO_set_id_callback(id_function); + } else { + pthread_mutex_lock(&mutex_lib); + mutex_use++; + } + #endif // OPENSSL_THREADS if(!bNotFirstTime) { SSLeay_add_all_algorithms(); SSL_load_error_strings(); *************** *** 136,147 **** --- 198,214 ---- SSL_CTX_set_default_verify_paths(ctx); SSL_CTX_set_verify(ctx, SSL_VERIFY_NONE, NULL); RETVAL = ctx; + #if defined(OPENSSL_THREADS) + pthread_mutex_unlock(&mutex_lib); + #endif OUTPUT: RETVAL void SSL_CTX_free(ctx) SSL_CTX* ctx + CODE: + mutex_free++; int SSL_CTX_set_cipher_list(ctx, ciphers)
Download (untitled) / with headers
text/plain 1.5k
The comments at http://www.nabble.com/When-to-use-CRYPTO_set_locking_callback()-and-CRYPTO_set_id_callback()--td5849882.html also give Windows related information as well as point out: #> static unsigned long id_function(void) #> { #> return ((unsigned long) pthread_self()); #> } # # This is not technically legal. The results of casting a # 'pthread_t' to an 'unsigned long' are undefined and there is no # guarantee that this will always return the same value for the # same thread. (That's why 'pthread_equal' exists.) # To do this correctly under pthreads, you have to use TLS, storing # a unique 'unsigned long' for each thread and assigning them yourself. On Mon Nov 17 20:11:31 2008, tmakino wrote: Show quoted text
> Using perl ithreads, we can set up multiple LWP::UserAgents to fetch > many pages simultaneously. However, when more than two threads are > accessing https:// URLs, perl occasionally reports corrupted memory and > dies. > This is because Crypt::SSLeay lacks mutex setup that is required by > OpenSSL library. > The attached patch is a trial to fix this problem. The patch is for > Crypt::SSLeay 0.57_01. > Unfortunately the patch is pthread-specific, so more work is necessary > to support other thread libraries. I tested the patch on Linux 2.6 > (Debian etch) and it seems working. > > Some code in the patch is copied from > http://www.nabble.com/When-to-use-CRYPTO_set_locking_callback()-and- > CRYPTO_set_id_callback()--td5849882.html > > I hope (more complete) thread support is included in the future version > of Crypt::SSLeay.
Download (untitled) / with headers
text/plain 307b
i started a partial fix to 0.59_03 https://github.com/collectiveintel/Crypt- SSLeay/commit/72c592a15ea97354830ece418e5a92ed2b0cd168 if others want to test/contribute to this, will test and help contrib back.. i haven't done much testing yet (add other threading support, fixed the 'unsigned' issue, etc).
RT-Send-CC: nanis [...] cpan.org
Download (untitled) / with headers
text/plain 688b
Thank you very much for your contribution. I am going to do some work on Crypt-SSLeay soon and this comes at a very good time. RT seems to have broken up the GitHub URL below, so I am going try including it again: <https://github.com/collectiveintel/Crypt-SSLeay/commit/72c592a15ea97354830ece418e5a92ed2b0cd168> On Fri Jul 27 12:38:43 2012, SAXJAZMAN wrote: Show quoted text
> i started a partial fix to 0.59_03 > > https://github.com/collectiveintel/Crypt- > SSLeay/commit/72c592a15ea97354830ece418e5a92ed2b0cd168 > > if others want to test/contribute to this, will test and help contrib > back.. i haven't done much > testing yet (add other threading support, fixed the 'unsigned' issue, > etc).
Download (untitled) / with headers
text/plain 592b
thanks to t-bitcard-perlbug@snowelm.com, i just swiped their patch and git-hub'd it... feel free to just swipe that commit and i'll kill the fork, wasn't sure how active the author(s) were and needed to fix it. turns out there's a race condition issue with either this or lwp, where if you get an https:// ... 301 redirect, depending how long the redirect takes, you'll get a seg fault... could be related to this bug, could be something else (i'm doing this in multiple threads and it's hard to pin down). let me know when you get crack'n on this and i be happy to help out.. cheers,
My plan is to get a release out without the thread support first, and work on thread support next. 0.60 should will feature a better Makefile.PL that should make it easier to add optional support for threads. I would really appreciate a test case, if you have written a short, generic script using threads to explore the issue, that would be wonderful. I'll post updates on this ticket. Thank you, -- Sinan On Sat Jul 28 07:51:12 2012, SAXJAZMAN wrote: Show quoted text
> thanks to t-bitcard-perlbug@snowelm.com, i just swiped their patch and > git-hub'd it... > > feel free to just swipe that commit and i'll kill the fork, wasn't > sure how active the author(s) were > and needed to fix it. > > turns out there's a race condition issue with either this or lwp, > where if you get an https:// ... > 301 redirect, depending how long the redirect takes, you'll get a seg > fault... > > could be related to this bug, could be something else (i'm doing this > in multiple threads and it's > hard to pin down). > > let me know when you get crack'n on this and i be happy to help out.. > > cheers,
Please keep in mind that the proposed patch will not work on MS Windows as it is "pthreads-only".

You might also find inspiring how Net::SSLeay implements thread-safety

--
kmx

Download (untitled) / with headers
text/plain 408b
On Tue Jul 31 15:56:28 2012, KMX wrote: Show quoted text
> Please keep in mind that the proposed patch will not work on MS > Windows as it is "pthreads-only".
I know. I would like to put together something with 1) as few #ifdefs as possible; and 2) has a reasonable chance of working on both Windows and Linux initially. Show quoted text
> You might also find inspiring how Net::SSLeay implements thread-safety
You read my mind. -- Sinan
On Tue Jul 31 16:09:23 2012, NANIS wrote:
Show quoted text
> On Tue Jul 31 15:56:28 2012, KMX wrote:
>
> > Please keep in mind that the proposed patch will not work on MS
> > Windows as it is "pthreads-only".
>
> I know. I would like to put together something with 1) as few #ifdefs as
> possible; and 2) has a reasonable chance of working on both Windows and
> Linux initially.
>
> > You might also find inspiring how Net::SSLeay implements thread-safety
>
> You read my mind.

Consider using perl's functions:
MUTEX_INIT
MUTEX_LOCK
MUTEX_UNLOCK
you do not need to care about whether you are in pthreads or win32threads environment (perl handles it well enough)

--
kmx

Download (untitled) / with headers
text/plain 435b
On Tue Jul 31 16:17:09 2012, KMX wrote: Show quoted text
> Consider using perl's functions: > MUTEX_INIT > MUTEX_LOCK > MUTEX_UNLOCK > you do not need to care about whether you are > in pthreads or win32threads environment > (perl handles it well enough)
Thank you. That is indeed the plan, but I have been puzzled by something else since yesterday. Regardless, I appreciate your support and feedback, and I am going to turn to this next. -- Sinan


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.