Skip Menu |
 

This queue is for tickets about the Parse-RecDescent CPAN distribution.

Report information
The Basics
Id: 74593
Status: open
Priority: 0/
Queue: Parse-RecDescent

People
Owner: Nobody in particular
Requestors: mjgardner [...] cpan.org
Cc: domm [...] zsi.at
PLOBBES [...] cpan.org
ribasushi [...] leporine.io
AdminCc:

Bug Information
Severity: Important
Broken in: 1.966_002
Fixed in: (no value)



CC: domm [...] zsi.at
Subject: Recent changes break Module::ExtractUse and its dependents
Download (untitled) / with headers
text/plain 339b
Most of Module::ExtractUse's tests fail with Parse::RecDescent Show quoted text
>=1.966_002. M::E's dependents include Module::CPANTS::Analyse, so a
lot of CPAN and Kwalitee tests now fail too. I understand if this is more an issue with Module::ExtractUse than Parse::RecDescent, but the former hasn't been updated since 2008 so I thought I'd start here.
Thank you for the report. Module::ExtractUse ships with a pre-compiled grammar. Unfortunately, I did change some of the internal functions of Parse::Descent, and those are used by the pre-compiled grammar. I've confirmed that Module::ExtractUse does pass its own tests if the grammar is re-compiled. Long term, the right solution is probably to have pre-compiled grammars not take any dependency on Parse::RecDescent. Short term, users should probably either downgrade Parse::RecDescent, or should recompile Module::ExtractUse's grammar before running make. Not an ideal situation, I know. Let me think about how to best solve this and get back to you. Jeremy On Wed Feb 01 13:00:21 2012, MJGARDNER wrote: Show quoted text
> Most of Module::ExtractUse's tests fail with Parse::RecDescent
> >=1.966_002. M::E's dependents include Module::CPANTS::Analyse, so a
> lot of CPAN and Kwalitee tests now fail too. > > I understand if this is more an issue with Module::ExtractUse than > Parse::RecDescent, but the former hasn't been updated since 2008 so I > thought I'd start here.
Download (untitled) / with headers
text/plain 515b
I've posted a patch for Module::ExtractUse that will Precompile the parser during the build process, avoiding this issue: https://rt.cpan.org/Ticket/Display.html?id=74688 The issue can always reoccur if Parse::RecDescent is upgraded after installing a precompiled parser. I've prepared some changes to the Precompilation step that will remove the dependency on Parse::RecDescent, preventing the issue from happening with Precompiled parsers in the future. Those changes should be present in a future release.
Download (untitled) / with headers
text/plain 531b
In case anyone else is interested, here is a list of more related tickets on how the unfortunately non-backwards compatible changes have broken other modules: https://rt.cpan.org/Ticket/Display.html?id=74733 https://rt.cpan.org/Ticket/Display.html?id=74618 https://rt.cpan.org/Ticket/Display.html?id=74637 https://rt.cpan.org/Ticket/Display.html?id=74640 Any thoughts about keeping (or adding back in) backwards compatibility for a release or two instead of just pulling the rug out of others that use this Parse::RecDescent?
Download (untitled) / with headers
text/plain 1.2k
Yes, I'm interested. Let me think about it over the weekend. I'm certainly open to it, though it will be challenging to maintain backwards compatibility for P:RD's internals. Adding backwards-compatibility back in will either hurt performance, require two copies of the internals to be maintained, or re-introduce other parsing bugs that were fixed. None of which are good options. Is there a better way to reach out to the users of Parse::RecDescent to get them to update their modules to use the standalone parsing feature(s)? Did you just search rt.cpan.org for "Parse::RecDescent" and fails, or am I not on a critical mailing list where these are getting discussed? Jeremy On Thu Mar 01 23:16:19 2012, PLOBBES wrote: Show quoted text
> In case anyone else is interested, here is a list of more related tickets > on how the unfortunately non-backwards compatible changes have broken > other modules: > > https://rt.cpan.org/Ticket/Display.html?id=74733 > https://rt.cpan.org/Ticket/Display.html?id=74618 > https://rt.cpan.org/Ticket/Display.html?id=74637 > https://rt.cpan.org/Ticket/Display.html?id=74640 > > Any thoughts about keeping (or adding back in) backwards compatibility
for Show quoted text
> a release or two instead of just pulling the rug out of others that use > this Parse::RecDescent?
Subject: Re: [rt.cpan.org #74593] Recent changes break Module::ExtractUse and its dependents
Date: Fri, 02 Mar 2012 16:58:25 -0500
To: bug-Parse-RecDescent [...] rt.cpan.org
From: Phil Lobbes <phil [...] perkpartners.com>
Download (untitled) / with headers
text/plain 864b
Show quoted text
> Is there a better way to reach out to the users of Parse::RecDescent > to get them to update their modules to use the standalone parsing > feature(s)?
No idea. Show quoted text
> Did you just search rt.cpan.org for "Parse::RecDescent" and fails, or > am I not on a critical mailing list where these are getting discussed?
I just did a quick google search and looked at similar new/open bugs. Then did a quick search on rt.cpan.org with the search term "recdescent" and you'll find these recent bugs: 74637 Fails with Parse::RecDescent >= 1.966_002 Cindy 74640 Fails with Parse::RecDescent >= 1.966_002 CSS 74733 Fails with Parse::RecDescent >= 1.966_002 Mail-IMAPClient 74879 Failures due to Parse::RecDescent changes Module-ExtractUse 75130 Minimum requirement for Parse::RecDescent seems to be 1.966002 75334 Fails with Parse::RecDescent >= 1.967003 Lingua-Phonology Phil
RT-Send-CC: domm [...] zsi.at, phil [...] perkpartners.com
Download (untitled) / with headers
text/plain 1.2k
On Fri Mar 02 16:59:35 2012, phil@perkpartners.com wrote: Show quoted text
> > Is there a better way to reach out to the users of Parse::RecDescent > > to get them to update their modules to use the standalone parsing > > feature(s)?
> > No idea. >
> > Did you just search rt.cpan.org for "Parse::RecDescent" and fails, or > > am I not on a critical mailing list where these are getting discussed?
> > I just did a quick google search and looked at similar new/open bugs. > Then did a quick search on rt.cpan.org with the search term "recdescent" > and you'll find these recent bugs: > > 74637 Fails with Parse::RecDescent >= 1.966_002 Cindy > 74640 Fails with Parse::RecDescent >= 1.966_002 CSS > 74733 Fails with Parse::RecDescent >= 1.966_002 Mail-IMAPClient > 74879 Failures due to Parse::RecDescent changes Module-ExtractUse > 75130 Minimum requirement for Parse::RecDescent seems to be 1.966002 > 75334 Fails with Parse::RecDescent >= 1.967003 Lingua-Phonology > > Phil
You might want to look at this list: http://deps.cpantesters.org/depended-on-by.pl?module=Parse%3A%3ARecDescent It includes all distributions that depend on Parse::RecDescent. They would have to be tested individually to see whether they are broken by the new release.
Download (untitled) / with headers
text/plain 108b
I looked into this some tonight, and I can't exactly tell what's going on yet. Thank you for your patience.
On Tue Mar 13 00:22:57 2012, jtbraun wrote: Show quoted text
> I looked into this some tonight, and I can't exactly tell what's going > on yet. Thank you for your patience.
I spoke too soon. Two things happened: 1.966_002 introduced some extra arguments to internal rules/functions (this was probably the first break). I think that this is easily fixed by swapping the argument ordering around. Recent namespace cleanup improvements cause instances of precompiled parsers to delete their own namespaces (preventing future parse calls). This is typically guarded against by precompiled parsers setting their $self->{_precompiled} flag. For some reason Module::ExtractUse doesn't have this flag set, likely M::EU pre-dates this change to P::RD. I can add an extra guard around that, and more of the test cases begin to work. I'll work through the rest of the failures this week and see if I can come up with a comprehensive strategy for fixing these errors, as well as providing some sort of warning to users of modules dependent on Parse::RecDescent that older precompiled modules will break in the future. Jeremy
Download (untitled) / with headers
text/plain 405b
Thank you so much for the bug report. Two prior bug fixes contributed to the problem. I've checked that Module::ExtractUse and Mail::IMAPClient now pass 'make test' against Parse::RecDescent 1.967_008. Once this passes muster w/ CPAN Testers, I'll update a non-dev release to CPAN. You can find the fix here: https://github.com/jtbraun/Parse-RecDescent/commit/8b4f6daab2e55f8169afe0bd64f7a6b31a58b12c
Download (untitled) / with headers
text/plain 1.7k
On Wed Mar 14 01:43:24 2012, jtbraun wrote: Show quoted text
> Thank you so much for the bug report. Two prior bug fixes contributed > to the problem. I've checked that Module::ExtractUse and > Mail::IMAPClient now pass 'make test' against Parse::RecDescent > 1.967_008. > > Once this passes muster w/ CPAN Testers, I'll update a non-dev release > to CPAN.
Hi! I would like to clarify if this backwards compatibility will be retained in the future or this is a temporary "being nice" kind of thing. In particular will the following scenario work: t1: User installs SQL::Translator with grammars precompiled at dist-build time using the *author*'s P::RD v1.967009. At the same time user has a P::RD v1.965001 on their machine, which happens to satisfy the SQL::Translator requirements - this scenario currently works - will it work in the future? t2: At a later stage a user install some module X, which pulls in a (at the time available) P::RD v1.989000. Will the grammars compiled with v1.967009 continue to work? Or will there be imexplicabale breakage? If any of the above is in fact a problem - is there a way to tell *what* generated a grammar, so that if the grammar version and the currently installed P::RD version do not match, things will revert to compile-on-demand? The reason for this question is that SQL::Translator *almost* started shipping precompiled grammars, but then backed out the changes given the turbulence of Parse::RecDescent. Currently the state is as follows: https://github.com/dbsrgits/sql-translator/commit/0eb3b94a5 . What would your recommendation be, if the priorities are to not break the users code by a drive-by upgrade? Thanks! P.S. Sorry for reopening this ticket, it seemed the best place to continue this discussion.
Download (untitled) / with headers
text/plain 4.5k
Hi Peter, thank you for your patience while I got back to you. First off, let me assure you that I have no interest in breaking current uses of P::RD. I have every intention of preserving the current behavior of the module, while improving the interface and performance. I would recommend against using the current version (v1.967009) of standalone parsers with SQL::Translator. The standalone parsers generated with this version will end up including most of P::RD as the package Parse::RecDescent::_Runtime. This works to isolate your module from future changes to P::RD, and against past implementations of P::RD. But if your user 'use's your precompiled parser with someone else's precompiled parser, they will end up with numerous redefinitions of functions in the "Parse::RecDescent::_Runtime" namespace. Annoying. I've pushed some modifications to the standalone precompiled parser generator (https://github.com/jtbraun/Parse-RecDescent/commit/21f9ad82aba94ee05b806b4e99f21e6d1dc47e4b) to github. It allows you to do two things: 1. Declare the package name where P::RD's internals are copied to, so that it avoids the redefinition problem above. 2. Change the name of the Parse::RecDescent dependencies to the package from (1), without including P::RD. This will be of particular interest to you, as you'll want to include a single copy of the P::RD run-time as SQL::Translator::ParseRD::_Runtime (or equivalent), as well as reference that package from each of your parsers. I'd appreciate your assistance in vetting this feature, as SQL::Translator is the only module I know of that would be shipping multiple pre-compiled parsers. t2: My intention is that precompiled _standalone_ parsers should be 100% independent of the P::RD installed on the system. Standalone parsers should not 'use' or 'require' P::RD. The only dependencies will be on: 1. The P::RD installed on the module maintainer's machine when they compile the standalone parser. 2. The P::RD installed on the user's machine at module build/install time, when that module's build process creates the standalone parsers. I do not believe that there is currently a way to determine which version of P::RD precompiled a given parser. I've filed RT#77001 to track adding this. No worries about re-opening the ticket. I'm excited to see P::RD improve, and I'm rather embarrassed that I broke so many modules by changing the calling convention of &_parserepeat. I _REALLY_ appreciate any testing help or suggestions you can provide to ensure that my and your users are not adversely impacted by the changes I'm making to P::RD. Please take a look at the most recent changes to P::RD on github, and let me know if the interface works for you. Once we agree on an implementation, I'll push out a release on PAUSE. Jeremy On Sat Apr 28 11:26:43 2012, RIBASUSHI wrote: Show quoted text
> On Wed Mar 14 01:43:24 2012, jtbraun wrote:
> > Thank you so much for the bug report. Two prior bug fixes contributed > > to the problem. I've checked that Module::ExtractUse and > > Mail::IMAPClient now pass 'make test' against Parse::RecDescent > > 1.967_008. > > > > Once this passes muster w/ CPAN Testers, I'll update a non-dev release > > to CPAN.
> > Hi! > > I would like to clarify if this backwards compatibility will be retained > in the future or this is a temporary "being nice" kind of thing. In > particular will the following scenario work: > > t1: User installs SQL::Translator with grammars precompiled at > dist-build time using the *author*'s P::RD v1.967009. At the same time > user has a P::RD v1.965001 on their machine, which happens to satisfy > the SQL::Translator requirements - this scenario currently works - will > it work in the future? > > t2: At a later stage a user install some module X, which pulls in a (at > the time available) P::RD v1.989000. Will the grammars compiled with > v1.967009 continue to work? Or will there be imexplicabale breakage? > > If any of the above is in fact a problem - is there a way to tell *what* > generated a grammar, so that if the grammar version and the currently > installed P::RD version do not match, things will revert to > compile-on-demand? > > The reason for this question is that SQL::Translator *almost* started > shipping precompiled grammars, but then backed out the changes given the > turbulence of Parse::RecDescent. Currently the state is as follows: > https://github.com/dbsrgits/sql-translator/commit/0eb3b94a5 . What would > your recommendation be, if the priorities are to not break the users > code by a drive-by upgrade? > > Thanks! > > P.S. Sorry for reopening this ticket, it seemed the best place to > continue this discussion.
Download (untitled) / with headers
text/plain 4.1k
On Sat May 05 01:18:36 2012, jtbraun wrote: Show quoted text
> Hi Peter, thank you for your patience while I got back to you. > > First off, let me assure you that I have no interest in breaking > current > uses of P::RD. I have every intention of preserving the current > behavior of the module, while improving the interface and performance.
That's great to hear! Show quoted text
> I would recommend against using the current version (v1.967009) of > standalone parsers with SQL::Translator. The standalone parsers > generated with this version will end up including most of P::RD as the > package Parse::RecDescent::_Runtime. This works to isolate your > module > from future changes to P::RD, and against past implementations of > P::RD. > But if your user 'use's your precompiled parser with someone else's > precompiled parser, they will end up with numerous redefinitions of > functions in the "Parse::RecDescent::_Runtime" namespace. Annoying.
Right. Show quoted text
> I've pushed some modifications to the standalone precompiled parser > generator > (https://github.com/jtbraun/Parse- > RecDescent/commit/21f9ad82aba94ee05b806b4e99f21e6d1dc47e4b) > to github. It allows you to do two things: > 1. Declare the package name where P::RD's internals are copied to, so > that it avoids the redefinition problem above. > 2. Change the name of the Parse::RecDescent dependencies to the > package > from (1), without including P::RD. This will be of particular > interest > to you, as you'll want to include a single copy of the P::RD run-time > as > SQL::Translator::ParseRD::_Runtime (or equivalent), as well as > reference > that package from each of your parsers.
Excellent. Show quoted text
> I'd appreciate your assistance in vetting this feature, as > SQL::Translator is the only module I know of that would be shipping > multiple pre-compiled parsers.
Unfortunately I am very short of time lately, and won't be able to play with this for at least 2 weeks. SQLT has already been long overdue, so we will ship a release with the precompiled parsers disabled *for the time being*. In the meantime it is extremely easy for you to test your changes yourself: all you have to do is back-out this commit: https://github.com/dbsrgits/sql-translator/commit/0eb3b94a54140cedfc37172290f3fc8c1e15ddc9 As you can see all the code is already there, it is just commented. Running `perl Makefile.PL` from a git checkout detects you are in "author" mode, and generates the correct grammars. Then the core of SQLT knows how to find them/use them properly as well, so a mere prove -lr will suffice to diagnose what you messed up. Ignore the "FIXME FIXME FIXME" line - you already explained how to deal with this in https://rt.cpan.org/Ticket/Display.html?id=74088, I just didn't have time to integrate it. Show quoted text
> t2: My intention is that precompiled _standalone_ parsers should be > 100% > independent of the P::RD installed on the system. Standalone parsers > should not 'use' or 'require' P::RD. The only dependencies will be > on: > 1. The P::RD installed on the module maintainer's machine when they > compile the standalone parser. > 2. The P::RD installed on the user's machine at module build/install > time, when that module's build process creates the standalone parsers.
This is an excellent idea, though in this case 2. would not even be necessary. We just ship the precompiled parsers directly to the user, so they have one less dependency to install. I.e. P::RD becomes an author-only dependency similar to https://github.com/dbsrgits/sql-translator/blob/master/Makefile.PL#L91 Show quoted text
> I do not believe that there is currently a way to determine which > version of P::RD precompiled a given parser. I've filed RT#77001 to > track adding this.
If you implement the above, the version that made the parser becomes irrelevant, as it becomes truly standalone. Show quoted text
> No worries about re-opening the ticket. I'm excited to see P::RD > improve, and I'm rather embarrassed that I broke so many modules by > changing the calling convention of &_parserepeat. I _REALLY_ > appreciate > any testing help or suggestions you can provide to ensure that my and > your users are not adversely impacted by the changes I'm making to > P::RD.
Thanks for working on this! Much appreciated.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.