Skip Menu |
 

This queue is for tickets about the Spreadsheet-XLSX CPAN distribution.

Report information
The Basics
Id: 67954
Status: open
Priority: 0/
Queue: Spreadsheet-XLSX

People
Owner: Nobody in particular
Requestors: leon.panokarren [...] farmersinsurance.com
Cc:
AdminCc:

Bug Information
Severity: Critical
Broken in: 0.13
Fixed in: (no value)



Subject: Out of Memory Failure
Download (untitled) / with headers
text/plain 2.7k
Hello Dmitry, We are using the Spreadsheet XLSX module in our Production environment for reading Excel files on our AIX box and transform to CSV files. We were running fine until we received XLSX files greater than 40MB. We are now failing with an 'Out of memory' - I know this is because we are reading the whole file at once - but dont know if there is a way to reduce the memory usage (like in Spreadsheet::ParseExcel, I remember you had provided the cell handler function that reduced the size of the cell object). Is there something similar available for Spreadsheet::XLSX? Your help is greatly appreciated! Also, here is our perl environment information to aid your review - Summary of my perl5 (revision 5 version 10 subversion 1) configuration: Platform: osname=aix, osvers=5.3.0.0, archname=aix uname='aix laatdm62 3 5 00c396bc4c00 ' config_args='-de -Dcc=cc_r' hint=recommended, useposix=true, d_sigaction=define useithreads=undef, usemultiplicity=undef useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=undef, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc_r -q32', ccflags ='-D_ALL_SOURCE -D_ANSI_C_SOURCE - D_POSIX_SOURCE -qmaxmem=-1 -qnoansialias -DUSE_NATIVE_DLOPEN -q32 - D_LARGE_FILES -qlonglong', optimize='-O', cppflags='-D_ALL_SOURCE -D_ANSI_C_SOURCE -D_POSIX_SOURCE -qmaxmem=- 1 -qnoansialias -DUSE_NATIVE_DLOPEN' ccversion='', gccversion='', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='ld', ldflags =' -brtl -bdynamic -b32 -bmaxdata:0x80000000' libpth=/lib /usr/lib /usr/ccs/lib libs=-lbind -lnsl -ldbm -ldl -lld -lm -lcrypt -lc perllibs=-lbind -lnsl -ldl -lld -lm -lcrypt -lc libc=/lib/libc.a, so=a, useshrplib=false, libperl=libperl.a gnulibc_version='' Dynamic Linking: dlsrc=dl_aix.xs, dlext=so, d_dlsymun=undef, ccdlflags=' - bE:/usr/local/lib/perl5/5.10.1/aix/CORE/perl.exp' cccdlflags=' ', lddlflags=' -bhalt:4 -G -bI:$(PERL_INC)/perl.exp - bE:$(BASEEXT).exp -bnoentry -lc -lm' Characteristics of this binary (from libperl): Compile-time options: PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP USE_LARGE_FILES USE_PERLIO Built under aix Compiled at Feb 26 2011 02:13:00 @INC: /usr/local/lib/perl5/5.10.1/aix /usr/local/lib/perl5/5.10.1 /usr/local/lib/perl5/site_perl/5.10.1/aix /usr/local/lib/perl5/site_perl/5.10.1 Also, here is our OS information - AIX laatdm62 3 5 00C396BC4C00 Thanks for the help!
Subject: Re: [rt.cpan.org #67954] AutoReply: Out of Memory Failure
Date: Thu, 5 May 2011 13:44:29 -0700
To: bug-Spreadsheet-XLSX [...] rt.cpan.org, do_ [...] pochta.ru
From: leon.panokarren [...] farmersinsurance.com
Download (untitled) / with headers
text/plain 4.8k
Hello Dmitry, I was hoping this will help your review in providing a fix. I noticed the module fails trying to read the entire XML worksheet content at the biggest worksheet in my spreadsheet with 1 million rows in it - foreach ($member_sheet -> contents =~ /(\<.*?\/?\>|.*?(?=\<))/g) { Can we change this go line-by-line? or in chunks? not sure I totally understand how this works. Thanks! Leon Panokarren Work: 714 - 385 - 3793 Cell: 732 - 692 - 4110 "Bugs in Spreadsheet-XLSX via RT" <bug-Spreadsheet-XLSX@rt.cpan.org> 05/04/2011 05:01 PM Please respond to bug-Spreadsheet-XLSX@rt.cpan.org To leon.panokarren@farmersinsurance.com cc Subject [rt.cpan.org #67954] AutoReply: Out of Memory Failure Greetings, This message has been automatically generated in response to the creation of a trouble ticket regarding: "Out of Memory Failure", a summary of which appears below. There is no need to reply to this message right now. Your ticket has been assigned an ID of [rt.cpan.org #67954]. Your ticket is accessible on the web at: https://rt.cpan.org/Ticket/Display.html?id=67954 Please include the string: [rt.cpan.org #67954] in the subject line of all future correspondence about this issue. To do so, you may reply to this message. Thank you, bug-Spreadsheet-XLSX@rt.cpan.org ------------------------------------------------------------------------- Hello Dmitry, We are using the Spreadsheet XLSX module in our Production environment for reading Excel files on our AIX box and transform to CSV files. We were running fine until we received XLSX files greater than 40MB. We are now failing with an 'Out of memory' - I know this is because we are reading the whole file at once - but dont know if there is a way to reduce the memory usage (like in Spreadsheet::ParseExcel, I remember you had provided the cell handler function that reduced the size of the cell object). Is there something similar available for Spreadsheet::XLSX? Your help is greatly appreciated! Also, here is our perl environment information to aid your review - Summary of my perl5 (revision 5 version 10 subversion 1) configuration: Platform: osname=aix, osvers=5.3.0.0, archname=aix uname='aix laatdm62 3 5 00c396bc4c00 ' config_args='-de -Dcc=cc_r' hint=recommended, useposix=true, d_sigaction=define useithreads=undef, usemultiplicity=undef useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=undef, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc_r -q32', ccflags ='-D_ALL_SOURCE -D_ANSI_C_SOURCE - D_POSIX_SOURCE -qmaxmem=-1 -qnoansialias -DUSE_NATIVE_DLOPEN -q32 - D_LARGE_FILES -qlonglong', optimize='-O', cppflags='-D_ALL_SOURCE -D_ANSI_C_SOURCE -D_POSIX_SOURCE -qmaxmem=- 1 -qnoansialias -DUSE_NATIVE_DLOPEN' ccversion='', gccversion='', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='ld', ldflags =' -brtl -bdynamic -b32 -bmaxdata:0x80000000' libpth=/lib /usr/lib /usr/ccs/lib libs=-lbind -lnsl -ldbm -ldl -lld -lm -lcrypt -lc perllibs=-lbind -lnsl -ldl -lld -lm -lcrypt -lc libc=/lib/libc.a, so=a, useshrplib=false, libperl=libperl.a gnulibc_version='' Dynamic Linking: dlsrc=dl_aix.xs, dlext=so, d_dlsymun=undef, ccdlflags=' - bE:/usr/local/lib/perl5/5.10.1/aix/CORE/perl.exp' cccdlflags=' ', lddlflags=' -bhalt:4 -G -bI:$(PERL_INC)/perl.exp - bE:$(BASEEXT).exp -bnoentry -lc -lm' Characteristics of this binary (from libperl): Compile-time options: PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP USE_LARGE_FILES USE_PERLIO Built under aix Compiled at Feb 26 2011 02:13:00 @INC: /usr/local/lib/perl5/5.10.1/aix /usr/local/lib/perl5/5.10.1 /usr/local/lib/perl5/site_perl/5.10.1/aix /usr/local/lib/perl5/site_perl/5.10.1 Also, here is our OS information - AIX laatdm62 3 5 00C396BC4C00 Thanks for the help! ***** PLEASE NOTE ***** This E-Mail/telefax message and any documents accompanying this transmission may contain privileged and/or confidential information and is intended solely for the addressee(s) named above. If you are not the intended addressee/recipient, you are hereby notified that any use of, disclosure, copying, distribution, or reliance on the contents of this E-Mail/telefax information is strictly prohibited and may result in legal action against you. Please reply to the sender advising of the error in transmission and immediately delete/destroy the message and any accompanying documents. Thank you.*****
On Thu May 05 16:44:50 2011, ETLTCHFIG wrote: See http://rt.cpan.org/Ticket/Display.html?id=66516


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.