Skip Menu |
 

This queue is for tickets about the SQL-Tokenizer CPAN distribution.

Report information
The Basics
Id: 44918
Status: open
Priority: 0/
Queue: SQL-Tokenizer

People
Owner: Nobody in particular
Requestors: JKRAMER [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in: 0.18
Fixed in: (no value)

Attachments
0001-Name-quote-and-separator-fix.patch



Subject: Dot gets lost when combined with field quotations
MIME-Version: 1.0
X-Mailer: MIME-tools 5.427 (Entity 5.427)
Content-Type: text/plain
Charset: utf8
Content-Disposition: inline
Content-Transfer-Encoding: binary
X-RT-Original-Encoding: utf-8
Content-Length: 640
Download (untitled) / with headers
text/plain 640b
I'm creating a new bug report for this because the other dot-bug is slightly different from this one and seems to be already fixed. When quoting table/column names, the seperating dot seems to get lost. For example, in SELECT "foo"."bar" FROM "foo" "foo"."bar" becomes [ '"foo"', '"bar"' ] instead of [ '"foo"', '.', '"bar"']. When using MySQL quotation, the quotes get lost too: SELECT `foo`.`bar` FROM `foo`. Becomes: [ ... 'foo', 'bar, ... ] instead of [ ..., '`foo`', '.', '`bar`', ...] The bug appears in 0.18, but older versions might be affected too (didn't test them). Best regards and thanks for the great module, Jonas
MIME-Version: 1.0
X-Mailer: MIME-tools 5.427 (Entity 5.427)
Charset: utf8
Content-Type: multipart/mixed; boundary="----------=_1239358936-2538-183"
Message-ID: <rt-3.6.HEAD-2538-1239358936-1361.44918-0-0 [...] rt.cpan.org>
X-RT-Original-Encoding: utf-8
Content-Length: 0
Content-Disposition: inline
Content-Type: text/plain
Content-Transfer-Encoding: binary
X-RT-Original-Encoding: utf-8
Content-Length: 267
Download (untitled) / with headers
text/plain 267b
Hi again, I made a quick patch which seems to fix the problem. I didn't test it too much, but the SQL-Tokenizer tests run perfectly. What I did: 1. Allow `` quoted strings (similar to "" and '') 2. Add dot to punctuation characters Hope it doesn't break anything.
MIME-Version: 1.0
X-Mailer: MIME-tools 5.427 (Entity 5.427)
Content-Type: multipart/mixed; boundary="----------=_1239358936-2538-182"
Charset: utf8
Content-Length: 0
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: binary
X-RT-Original-Encoding: iso-8859-1
Content-Length: 0
Content-Type: application/octet-stream; name="0001-Name-quote-and-separator-fix.patch"
Content-Disposition: inline; filename="0001-Name-quote-and-separator-fix.patch"
Content-Transfer-Encoding: base64
Content-Length: 2324
From dc9639b640fe536e75e372d05a7af35b86cb87f2 Mon Sep 17 00:00:00 2001 From: Jonas Kramer <jkramer@nex.scrapping.cc> Date: Fri, 10 Apr 2009 12:11:06 +0200 Subject: [PATCH] Name quote and separator fix. --- lib/SQL/Tokenizer.pm | 5 ++++- t/11-name-sep.t | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+), 1 deletions(-) create mode 100644 t/11-name-sep.t diff --git a/lib/SQL/Tokenizer.pm b/lib/SQL/Tokenizer.pm index 7d5b1c8..6e8b158 100644 --- a/lib/SQL/Tokenizer.pm +++ b/lib/SQL/Tokenizer.pm @@ -20,7 +20,7 @@ my $re= qr{ (?:<>|<=>|>=|<=|==|=|!=|!|<<|>>|<|>|\|\||\||&&|&|-|\+|\*(?!/)|/(?!\*)|\%|~|\^|\?) # operators and tests | - [\[\]\(\),;] # punctuation (parenthesis, comma) + [\[\]\(\),;.] # punctuation (parenthesis, comma) | \'\'(?!\') # empty single quoted string | @@ -28,6 +28,9 @@ my $re= qr{ | ".*?(?:(?:""){1,}"|(?<!["\\])"(?!")|\\"{2}) # anything inside double quotes, ungreedy + | + `.*?(?:(?:``){1,}`|(?<![`\\])`(?!`)|\\`{2}) + # anything inside backticks quotes, ungreedy | '.*?(?:(?:''){1,}'|(?<!['\\])'(?!')|\\'{2}) # anything inside single quotes, ungreedy. diff --git a/t/11-name-sep.t b/t/11-name-sep.t new file mode 100644 index 0000000..9964f8f --- /dev/null +++ b/t/11-name-sep.t @@ -0,0 +1,37 @@ +use strict; +use warnings; + +use Test::More; + +use SQL::Tokenizer; + +use constant SPACE => ' '; +use constant COMMA => ','; +use constant NL => "\n"; + +my $query; +my @query; +my @tokenized; + +my @tests = ( + { + description => q{PostgreSQL style}, + query => qq{SELECT "foo"."bar"}, + wanted => [ 'SELECT', SPACE, '"foo"', '.', '"bar"' ], + }, { + description => q{MySQL style}, + query => qq{SELECT `foo`.`bar`}, + wanted => [ 'SELECT', SPACE, '`foo`', '.', '`bar`' ], + } + +); + +plan tests => scalar @tests; + +foreach my $test (@tests) { + my @tokenized= SQL::Tokenizer->tokenize( $test->{query} ); + is_deeply( \@tokenized, $test->{wanted}, $test->{description} ); +} + +__END__ + -- 1.6.2.1
MIME-Version: 1.0
X-Mailer: MIME-tools 5.427 (Entity 5.427)
Content-Disposition: inline
Charset: utf8
Content-Type: text/plain
Message-ID: <rt-3.6.HEAD-2538-1239375047-1267.44918-0-0 [...] rt.cpan.org>
Content-Transfer-Encoding: binary
X-RT-Original-Encoding: utf-8
Content-Length: 83
Hi Jonas, Your patch was applied and I'm uploading 0.19 right now. Thanks! -Igor


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.