Skip Menu |
 

This queue is for tickets about the DBD-Pg CPAN distribution.

Report information
The Basics
Id: 44225
Status: resolved
Priority: 0/
Queue: DBD-Pg

People
Owner: greg [...] turnstep.com
Requestors: szbalint [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in: 2.11.8
Fixed in: 2.12.0



Subject: array retrieval leaks memory
Download (untitled) / with headers
text/plain 447b
Hi, DBD::Pg leaks memory when it's accessing PostgreSQL array values. Creating the included table and running leak.t produces a constant and sizeable memory leak for me. When the placeholder is changed from 'foo' to 'bar', so that for the given row the array is NULL, the leak stops and the program has constant memory usage over time. I've used 5.8.8 and the latest DBD::Pg version: $ perl5.8.8 -MDBD::Pg -le 'print $DBD::Pg::VERSION' 2.11.8
Subject: test_leak.sql
Download test_leak.sql
text/x-sql 289b
drop table if exists test_leak cascade; create table test_leak ( id serial NOT NULL, numbers integer[], string text NOT NULL, CONSTRAINT test_leak_pkey PRIMARY KEY (id) ); INSERT INTO test_leak (string,numbers) VALUES ('foo', '{}'); INSERT INTO test_leak (string) VALUES ('bar');
Subject: leak.t
Download leak.t
text/x-perl 437b
use strict; use warnings; use DBI; my $dbh = DBI->connect('dbi:Pg:host=localhost;dbname=test;port=5432;',"postgres","postgres") or die($DBI::errstr); my $count = 0; my $sth = $dbh->prepare("SELECT * FROM test_leak WHERE string = ?"); while (1) { { $sth->execute('foo'); my $res = $sth->fetchall_arrayref({}); if ($res && @{$res}) { $count += @{$res}; } else { exit; } print "$count\n" if ($count % 1000 == 0); } }
Download (untitled) / with headers
text/plain 445b
Thanks for the report (and test case!). I think this is fixed by making pg_destringify_array return a non-incremented version of the arrayref. This is committed in r12613. Please confirm that it fixes the problem you are seeing and does not cause any others. It's simple enough to patch if you don't want to download the whole of HEAD: just change the last line of pg_destringify_array from return newRV((SV*)av) to return newRV_noinc((SV*)av);
Download (untitled) / with headers
text/plain 568b
On Wed Mar 18 15:28:13 2009, greg@turnstep.com wrote: Show quoted text
> Thanks for the report (and test case!). I think this is fixed by making > pg_destringify_array return a non-incremented version of the arrayref. > This is committed in r12613. Please confirm that it fixes the problem > you are seeing and does not cause any others. It's simple enough to > patch if you don't want to download the whole of HEAD: just change the > last line of pg_destringify_array from return newRV((SV*)av) to return > newRV_noinc((SV*)av);
It seems that this fixes the problem indeed. Cheers.
Download (untitled) / with headers
text/plain 309b
On Fri Mar 20 01:41:00 2009, SZBALINT wrote: Show quoted text
> It seems that this fixes the problem indeed.
Actually, scratch that. I was accidentally testing with the row that had a NULL array, not the one that contained the data. The noinc fix doesn't seem to fix the problem unfortunately, something else is causing it.
Download (untitled) / with headers
text/plain 196b
Show quoted text
> The noinc fix doesn't seem to fix the problem unfortunately, something > else is causing it.
That's unfortunate. It fixed it on my system, so I guess at this point we need a better test case?
Download (untitled) / with headers
text/plain 649b
On Wed Mar 25 11:12:18 2009, greg@turnstep.com wrote: Show quoted text
> > The noinc fix doesn't seem to fix the problem unfortunately, something > > else is causing it.
> > That's unfortunate. It fixed it on my system, so I guess at this point > we need a better test case?
You're correct. Testing is hard :) This is what I think happens at the moment: The noinc change appears to fix the problem that is presented by the testcase, which is a non-NULL postgres array value, that happens to contain no elements. ('{}'). I still see a memory leak if the array does contain one or more elements though. I've attached a test case where the array contains elements.
Download test_leak2.sql
text/x-sql 351b
drop table if exists test_leak cascade; create table test_leak ( id serial NOT NULL, numbers integer[], string text NOT NULL, CONSTRAINT test_leak_pkey PRIMARY KEY (id) ); INSERT INTO test_leak (string,numbers) VALUES ('foo', '{}'); INSERT INTO test_leak (string) VALUES ('bar'); INSERT INTO test_leak (string,numbers) VALUES ('baz', '{1}');
Download leak2.t
text/x-perl 437b
use strict; use warnings; use DBI; my $dbh = DBI->connect('dbi:Pg:host=localhost;dbname=test;port=5432;',"postgres","postgres") or die($DBI::errstr); my $count = 0; my $sth = $dbh->prepare("SELECT * FROM test_leak WHERE string = ?"); while (1) { { $sth->execute('baz'); my $res = $sth->fetchall_arrayref({}); if ($res && @{$res}) { $count += @{$res}; } else { exit; } print "$count\n" if ($count % 1000 == 0); } }
Download (untitled) / with headers
text/plain 237b
Please see if the changes in r12635 have fixed the problem. I mortalized the temporary strings we use to populate the sub-parts of the array: http://git.postgresql.org/?p=dbdpg.git;a=commitdiff;h=3508ab706546d898a217ab95815a76d56129541a
Download (untitled) / with headers
text/plain 438b
On Thu Mar 26 12:29:53 2009, greg@turnstep.com wrote: Show quoted text
> Please see if the changes in r12635 have fixed the problem. I > mortalized > the temporary strings we use to populate the sub-parts of the array: > >
http://git.postgresql.org/?p=dbdpg.git;a=commitdiff;h=3508ab706546d898a217ab95815a76d56129541a Looks like this commit indeed fixes the memory leak shown by the updated test case. Thank you for the fix and the quick response time!


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.