Skip Menu |
 

This queue is for tickets about the DBIx-Class CPAN distribution.

Report information
The Basics
Id: 105376
Status: open
Priority: 0/
Queue: DBIx-Class

People
Owner: Nobody in particular
Requestors: mca [...] sanger.ac.uk
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: Conflict with autodie < 2.26 ?
Date: Fri, 19 Jun 2015 13:34:42 +0100
To: bug-DBIx-Class [...] rt.cpan.org
From: Matthew Astley <mca [...] sanger.ac.uk>
Download (untitled) / with headers
text/plain 1.7k
Thanks for DBIx::Class, and (specifically in this instance) the explanation about what's going on with Text::Balanced. I found that autodie 2.22 has the exception comparison problem, DBIx::Class::_Util::is_exception(): External exception class autodie::exception implements partial (broken) overloading preventing its instances from being used in simple ($x eq $y) comparisons. Given Perl's "globally cooperative" exception handling this type of brokenness is extremely dangerous on exception objects, as it may (and often does) result in silent "exception substitution". DBIx::Class tries to work around this as much as possible, but other parts of your software stack may not be even aware of this. Please submit a bugreport against the distribution containing autodie::exception and in the meantime apply a fix similar to the one shown at http://v.gd/DBIC_overload_tempfix/, in order to ensure your exception handling is saner application-wide. What follows is the actual error text as generated by Perl itself: Operation "ne": no method found, left argument in overloaded package autodie::exception, right argument has no overloaded magic at [...]/dbix-class/blib/lib/DBIx/Class/_Util.pm line 133. and this is fixed in autodie 2.26 due to https://github.com/pjf/autodie/pull/57 Therefore, would it be appropriate to declare in DBIx::Class a conflict against the earlier versions? Possibly Module::Install::CheckConflicts or Dist::CheckConflicts ? It might save someone else these couple of hours' digging... -- Matthew -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
Subject: Re: [rt.cpan.org #105376] Conflict with autodie < 2.26 ?
Date: Fri, 19 Jun 2015 14:41:41 +0200
To: bug-DBIx-Class [...] rt.cpan.org
From: Peter Rabbitson <ribasushi [...] cpan.org>
Download (untitled) / with headers
text/plain 2.5k
On 06/19/2015 02:34 PM, Matthew Astley via RT wrote: Show quoted text
> Fri Jun 19 08:34:57 2015: Request 105376 was acted upon. > Transaction: Ticket created by mca@sanger.ac.uk > Queue: DBIx-Class > Subject: Conflict with autodie < 2.26 ? > Broken in: (no value) > Severity: (no value) > Owner: Nobody > Requestors: mca@sanger.ac.uk > Status: new > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=105376 > > > > Thanks for DBIx::Class, and (specifically in this instance) the > explanation about what's going on with Text::Balanced. > > I found that autodie 2.22 has the exception comparison problem, > > DBIx::Class::_Util::is_exception(): External exception class > autodie::exception implements partial (broken) overloading preventing its > instances from being used in simple ($x eq $y) comparisons. Given Perl's > "globally cooperative" exception handling this type of brokenness is > extremely dangerous on exception objects, as it may (and often does) result > in silent "exception substitution". DBIx::Class tries to work around this as > much as possible, but other parts of your software stack may not be even > aware of this. Please submit a bugreport against the distribution containing > autodie::exception and in the meantime apply a fix similar to the one shown > at http://v.gd/DBIC_overload_tempfix/, in order to ensure your exception > handling is saner application-wide. What follows is the actual error text as > generated by Perl itself: > > Operation "ne": no method found, > left argument in overloaded package autodie::exception, > right argument has no overloaded magic at [...]/dbix-class/blib/lib/DBIx/Class/_Util.pm line 133. > > and this is fixed in autodie 2.26 due to > https://github.com/pjf/autodie/pull/57 > > > Therefore, would it be appropriate to declare in DBIx::Class a conflict > against the earlier versions? Possibly Module::Install::CheckConflicts > or Dist::CheckConflicts ?
It wouldn't for the simple reason that DBIx::Class itself does not use autodie for anything. All you see is a "Public Service Announcement" of sorts (there are several of those) - DBIC keeps on trucking after it encounters (and recovers) from such a condition. Therefor a "conflicts" would be a lie, as everything continues working. Show quoted text
> It might save someone else these couple of hours' digging...
Can you expand on this? What was the exact failure mode and why did it end up costing you hours? The very goal of this PSA was to save people time... Cheers!
Subject: Re: [rt.cpan.org #105376] Conflict with autodie < 2.26 ?
Date: Fri, 19 Jun 2015 15:27:07 +0100
To: Peter Rabbitson via RT <bug-DBIx-Class [...] rt.cpan.org>
From: Matthew Astley <mca [...] sanger.ac.uk>
On Fri, Jun 19, 2015 at 08:42:08AM -0400, Peter Rabbitson via RT wrote: Show quoted text
> On 06/19/2015 02:34 PM, Matthew Astley via RT wrote:
Show quoted text
> > I found that autodie 2.22 has the exception comparison problem, > > > > DBIx::Class::_Util::is_exception(): External exception class > > autodie::exception implements partial (broken) overloading [...] > > > > Operation "ne": no method found, > > left argument in overloaded package autodie::exception, > > right argument has no overloaded magic at [...]/dbix-class/blib/lib/DBIx/Class/_Util.pm line 133.
(This PSA text was from a cloned dbix-class on current/blead) Show quoted text
> > [...] would it be appropriate to declare in DBIx::Class a conflict > > against the earlier versions?
> > It wouldn't for the simple reason that DBIx::Class itself does not use > autodie for anything. All you see is a "Public Service Announcement" of > sorts [...]
OK. It does seem that, in every case, the transaction was rolled back. The fallout was afterwards. Show quoted text
> > It might save someone else these couple of hours' digging...
> > Can you expand on this? What was the exact failure mode
In short, an old DBIx::Class gave a strange and less helpful error message and then interacted badly with DESTROY of other objects. When I first met it, I was using DBIx::Class 0.08250 autodie 2.22 and my code, which did this... mca@cgpfoo:~/gitwk-cgp/cgpDataOut$ perl script/cgpout --db canpg init -d /foo (in cleanup) (in cleanup) Operation "ne": no method found, left argument in overloaded package autodie::exception, right argument has no overloaded magic at /software/perl-5.16.2/lib/site_perl/5.16.2/DBIx/Class/Storage/TxnScopeGuard.pm line 60. Not an ARRAY reference at /nfs/users/nfs_m/mca/gitwk-cgp/cgpDataOut/lib/Sanger/CGP/DataOut/TxnMinder.pm line 96. 13@cgpfoo:~/gitwk-cgp/cgpDataOut$ (the 13 is $? being non-zero, and I'm not sure where it came from) ...for the (execution-ordered) pseudocode use warnings FATAL=>'all'; # not relevant? use autodie ':all'; # a local convention try { # Try::Tiny my $g = $schema->txn_minder; # my wrapper object # in $g constructor $g->{_dbtxn} = $schema->txn_scope_guard; open my $fh, '<', '/does/not/exist'; # boom # in $g->DESTROY my ($self) = @_; # that $g again my $R = delete $self->{_rollbacks}; # ARRAY of CODE delete $self->{_dbtxn}; # database ROLLBACK, if necessary # I deduce that it went boom again here, in $_dbtxn->DESTROY if ($R && @$R) { # the error that stops the program? # then $g->DESTROY happens again later, because we made a new # reference to it and then dropped it } catch { # didn't happen }; # not reached Actually it looks like the entire try{} block is happening again, because when I put a print at the top it happens twice..?! Presumably this is due to some deep failure with DESTROYing things while propagating exceptions. Then, more weirdly, that @$R changes to the printed text (?!) when I'm printing at the top of the try. try { my $g = $schema->txn_minder; print "init = $self<\n"; ... and code similar to above, with more prints ... produces mca@cgpfoo:~/gitwk-cgp/cgpDataOut$ perl script/cgpout --db canpg init -d /foo init = Sanger::CGP::DataOut::Command::init=HASH(0x1e9f268)< DESTROY! deleted _rollbacks: r=ARRAY(0x22c4d30)=r deleted _dbtxn g=DBIx::Class::Storage::TxnScopeGuard=HASH(0x1fdd608)=g (in cleanup) (in cleanup) Operation "ne": no method found, left argument in overloaded package autodie::exception, right argument has no overloaded magic at /software/perl-5.16.2/lib/site_perl/5.16.2/DBIx/Class/Storage/TxnScopeGuard.pm line 60. init = Sanger::CGP::DataOut::Command::init=HASH(0x1e9f268)< Can't use string ("init = Sanger::CGP::DataOut::Com"...) as an ARRAY ref while "strict refs" in use at /nfs/users/nfs_m/mca/gitwk-cgp/cgpDataOut/lib/Sanger/CGP/DataOut/TxnMinder.pm line 101. 20160DESTROY! 13@cgpfoo:~/gitwk-cgp/cgpDataOut$ The only place 20160 appears is as 14 days' worth of minutes, which will be the value of a field in the row INSERTed & rolled back. I don't know why it was printed. "DESTROY!\n" is my $g going away. I haven't done anything fancy with STDOUT. We weren't planning on publishing this because it depends strongly on internal systems, but I can put it up somewhere if you want to look..? Above was with branch weirdness commitid 86e4455, with a set of dependencies I can trace from commitid 909e9053 in ./CPAN/ on top of a potentially variable background install of perl-5.16.2 plus site-wide libraries. The TxnMinder itself might interest you. Its purpose is to run a LIFO list of CODE blocks, after the database rollbacks are completed, for things like removing files which were being built off the uncommitted rows. However, I'm now suspecting that doing this much work in DESTROY is inadvisable. Show quoted text
> and why did it end up costing you hours? The very goal of this PSA > was to save people time...
It was obvious that autodie's exception class was disagreeing with TxnScopeGuard, but I had not yet seen the PSA and started digging. I concluded fairly quickly that there was a bug, so I set about writing a test: clone dbix-class build enough of it to `prove -bv t/storage/txn_scope_guard.t` start trying to provoke the problem, and see how the outcome shows up. Possibly this was premature, since I didn't know how the failure in my $g->DESTROY or $_dbtxn->DESTROY was managing to change $R to something true-but-not-ARRAYref in time for the @$R. At this point I saw the PSA... Why am I using an old Perl with old libraries installed? It is our production environment. You could say we chose stability and old bugs over instability, fixes and new bugs. Thanks for reading. Does this answer your question? -- Matthew -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
Download (untitled) / with headers
text/plain 1.8k
Hi, sorry for the delayed reply. On Fri Jun 19 16:27:29 2015, mca@sanger.ac.uk wrote: Show quoted text
> > Actually it looks like the entire try{} block is happening again, > because when I put a print at the top it happens twice..?! > Presumably this is due to some deep failure with DESTROYing things > while propagating exceptions. >
This is most likely due to the behavior of txn_do (it tries to run things twice on a disconnect) There have been many (not yet released) improvements in the last couple months in this area. Set DBIC_STORAGE_RETRY_DEBUG=1 for extra info on if this is happening or not. Show quoted text
> > The TxnMinder itself might interest you. Its purpose is to run a LIFO > list of CODE blocks, after the database rollbacks are completed, for > things like removing files which were being built off the uncommitted > rows. >
I am very interested in what is the API that you are using for this (the implementation isn't that important). I am going to implement something very similar next week to fix another problem with the DBIC deferred constraint support. Show quoted text
> However, I'm now suspecting that doing this much work in DESTROY is > inadvisable. >
Generally yes, but it may be moot if I can fit your needs into the thing I mentioned above. So please do get back to me with what your calling conventions look like (in private if need be). Show quoted text
> > Why am I using an old Perl with old libraries installed? It is our > production environment. You could say we chose stability and old bugs > over instability, fixes and new bugs.
I wouldn't say that. Instead I'd say you are being a reasonable user of software, in a world where this is not the norm: https://gist.github.com/ribasushi/d3769fa75822b7b080e9#file-tilt_description-L178-L201 Show quoted text
> > Thanks for reading. Does this answer your question?
Yes it did, in full! Thank you very much for taking the time to explain this. I will keep the ticket open to remind myself about the TxnMinder thing.
Subject: Re: [rt.cpan.org #105376] Conflict with autodie < 2.26 ?
Date: Fri, 1 Apr 2016 14:52:28 +0100
To: Peter Rabbitson via RT <bug-DBIx-Class [...] rt.cpan.org>
From: Matthew Astley <mca [...] sanger.ac.uk>
Download (untitled) / with headers
text/plain 4.4k
On Wed, Mar 30, 2016 at 11:18:46AM -0400, Peter Rabbitson via RT wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=105376 > > > Hi, sorry for the delayed reply.
No worries. It does mean I have semi-garbage-collected most of the context, so I don't remember what was important for me. I think my original request, that others be warned of autodie<2.26, is addressed by your explanation of the PSA messages. And maybe also the fact that search engines can find this thread. Show quoted text
> On Fri Jun 19 16:27:29 2015, mca@sanger.ac.uk wrote:
[ execution-event-annotated pseudocode was here ] Show quoted text
> > Actually it looks like the entire try{} block is happening again, > > because when I put a print at the top it happens twice..?! > > Presumably this is due to some deep failure with DESTROYing things > > while propagating exceptions.
> > This is most likely due to the behavior of txn_do (it tries to run > things twice on a disconnect)
OK. I guess a 'cluck' might show that up, but I didn't think of that at the time. I had previously concluded that DESTROY blocks may happen more than once per object, even without the re-blessing mentioned in L<perlobj/Destructors>, so I try to remember to keep them idempotent. Show quoted text
> There have been many (not yet released) improvements in the last > couple months in this area. Set DBIC_STORAGE_RETRY_DEBUG=1 for extra > info on if this is happening or not.
Thanks for the tip. I made a note on my cgpDataOut project to try this, (internal : https://gitlab.internal.sanger.ac.uk/CancerIT/cgpDataOut/issues/1 ) but it may be a while before I'm back on the DBIC side of this. Show quoted text
> > The TxnMinder itself might interest you. Its purpose is to run a LIFO > > list of CODE blocks, after the database rollbacks are completed, for > > things like removing files which were being built off the uncommitted > > rows.
> > I am very interested in what is the API that you are using for this > (the implementation isn't that important). I am going to implement > something very similar next week to fix another problem with the > DBIC deferred constraint support.
(I'm not sure I understood your question.) I was building a command line tool and underlying API to supply temporary copies of large data files to a compute cluster, and track those files in the database via DBIC in order to clean them up later. What I wanted was a way to keep the disk and database in sync, for commit or rollback, and to report on the progress of this. Assuming that the Perl process doesn't suddenly disappear. It seemed that a D:C:S:TxnScopeGuard-like object was the answer, because then I can ask it to (e.g.) mkdir and it will correctly rmdir if the database transaction is rolled back. Therefore I create my object (to BEGIN transaction), add to it some CODE to undo each non-db piece of work as I do the work, and then tell it when to COMMIT. The last line of POD says | This "has a" DBIx::Class::Storage::TxnScopeGuard rather than | inheriting, to avoid needing to mess with the privates of the | superclass. Possibly I would have had less trouble if subclassing TxnScopeGuard was explicitly OK? In this case I preferred to avoid poking undocumented magic. Show quoted text
> > However, I'm now suspecting that doing this much work in DESTROY > > is inadvisable.
(But I haven't found a more convenient way. Scope::Guard-like behaviour really simplifies code and that removes places to hide bugs.) Show quoted text
> Generally yes, but it may be moot if I can fit your needs into the > thing I mentioned above. So please do get back to me with what your > calling conventions look like (in private if need be).
If the description above isn't enough, I can probably just publish the project. It would probably be GPL if it could possibly be interesting to anyone outside the organisation; but I should do the licensing thing if I do. Show quoted text
> > Why am I using an old Perl with old libraries installed? It is > > our production environment. You could say we chose stability and > > old bugs over instability, fixes and new bugs.
> > I wouldn't say that. Instead I'd say you are being a reasonable user > of software, in a world where this is not the norm: > https://gist.github.com/ribasushi/d3769fa75822b7b080e9#file-tilt_description-L178-L201
Thanks. 8-) -- Matthew -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
Subject: Re: [rt.cpan.org #105376] Conflict with autodie < 2.26 ?
Date: Fri, 1 Apr 2016 18:41:14 +0200
To: bug-DBIx-Class [...] rt.cpan.org
From: Peter Rabbitson <ribasushi [...] cpan.org>
On 04/01/2016 03:52 PM, Matthew Astley via RT wrote: Show quoted text
> I think my original request, that others be warned of autodie<2.26, is > addressed by your explanation of the PSA messages. And maybe also the > fact that search engines can find this thread.
This is correct. The ticket is alive due to secondary concerns now, see below. Show quoted text
> I had previously concluded that DESTROY blocks may happen more than > once per object, even without the re-blessing mentioned in > L<perlobj/Destructors>, so I try to remember to keep them idempotent.
This is half-incorrect (funny word that ;). In general DESTROY can not happen twice on the same object by definition, as DESTROY is called right before the refcount goes to 0 and a free() cascade is triggered. However there is a loophole in how DESTROY works itself that allows to re-reference the object, thus preventing the entire process (and of course someone could just do $foo->DESTROY). Sometimes this is used for good[1] which allows [2] to work Sometimes it can result in evil[3] My recommendation would be to use a guard like [4] (employed as [5]) instead of trying to make your destructors idempotent. Show quoted text
> (I'm not sure I understood your question.)
See last paragraph of this email. Show quoted text
> The last line of POD says > > | This "has a" DBIx::Class::Storage::TxnScopeGuard rather than > | inheriting, to avoid needing to mess with the privates of the > | superclass. > > Possibly I would have had less trouble if subclassing TxnScopeGuard > was explicitly OK? In this case I preferred to avoid poking > undocumented magic.
It is besides the point: the scope guard is the wrong spot to implement any of this (i.e. what about txn_do()? ) Show quoted text
>> Generally yes, but it may be moot if I can fit your needs into the >> thing I mentioned above. So please do get back to me with what your >> calling conventions look like (in private if need be).
> If the description above isn't enough, I can probably just publish the > project. It would probably be GPL if it could possibly be interesting > to anyone outside the organisation; but I should do the licensing > thing if I do.
Basically what I am trying to get a good grasp on is *how exactly* does your implementation get used (note - not how it works internally). Going all the way to publish just for that is way more effort than needed, especially given I won't be able to reuse any of it (both on conceptual and possibly on licensing grounds). It would be best if you can show me (in private) snippets of code where you define the "run these tasks on commit/rollback", so I can confirm the new system in question will be able to replicate your needs in full. Does this make more sense? [1] https://metacpan.org/source/RIBASUSHI/DBIx-Class-0.082821/lib/DBIx/Class/Schema.pm#L1387-1417 [2] https://metacpan.org/source/RIBASUSHI/DBIx-Class-0.082821/t/52leaks.t#L400-434 [3] https://github.com/dbsrgits/dbix-class/pull/63 [4] https://metacpan.org/source/RIBASUSHI/DBIx-Class-0.082821/lib/DBIx/Class/_Util.pm#L170-216 [5] https://metacpan.org/source/RIBASUSHI/DBIx-Class-0.082821/lib/DBIx/Class/Storage/TxnScopeGuard.pm#L52-53


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.