Preferred bug tracker

Please visit the preferred bug tracker to report your issue.

This queue is for tickets about the DBD-SQLite CPAN distribution.

Report information
The Basics
Id:
72418
Status:
resolved
Priority:
Low/Low
Queue:

People
Owner:
Nobody in particular
Requestors:
bohica [...] ntlworld.com
Cc:
AdminCc:

BugTracker
Severity:
Normal
Broken in:
  • 1.29
  • 1.33
Fixed in:
(no value)



Subject: unicode table names not decoded in table_info_all call
Hi, Attached a script I wrote for DBI and testing unicode in various DBDs. It shows that when you create a unicode table name and call table_info_all you cannot find the table by name comparison and yet it does exist. If the names returned by table_info_all are decoded as UTF-8 you can find the table. So that suggests DBD::SQLite is not decoding table names. The same probably applies to unicode column names but although there is a test to check you can use them and find them there is no additional code to test if they are decoded as UTF-8 you can find them. BTW, some tests DBD::SQLite fails no one really expects it to pass like unicode parameter name markers so please ignore that test - the script was written for discovery not to demonstrate any particular bugs. Output when run on Windows (hence the unicode data out is displayed wrong so ignore that) is: martin@bragi:~/svn/dbi/trunk/ex$ perl unicode_test.pl dbi:SQLite:db=test.db # Driver DBD::SQLite-1.29 # Using DBMS_NAME 'SQLite' # Using DBMS_VER '3.6.22' # Using DRIVER_NAME undef # Using DRIVER_VER undef # SQL_IDENTIFIER_CASE undef # LANG = en_GB.UTF-8 # create table "fred" ( "a" varchar(20)) ok 1 - table for unicode data ok 2 - insert unicode data into table ok 3 - unicode data out = unicode data in, no where ok 4 - length of output data the same ok 5 - db length of unicode data correct ok 6 - select unicode data via parameterised where ok 7 - select unicode data via inline where # create table "fred" ( "a" varchar(20),"b" blob) ok 8 - table for unicode data ok 9 - insert unicode data and blob into table ok 10 - unicode data out = unicode data in, no where with blob ok 11 - utf8 flag not set on blob data ok 12 - retrieved blob = inserted blob # create table "fred" ( "a" int) ok 13 - test table for unicode parameter markers DBD::SQLite::st bind_param failed: Unknown named parameter: fredÔé¼ at unicode_t est.pl line 429. not ok 14 - bind parameter with unicode parameter marker # Failed test 'bind parameter with unicode parameter marker' # at unicode_test.pl line 431. # died: DBD::SQLite::st bind_param failed: Unknown named parameter: fred├ó┬é┬¼ a t unicode_test.pl line 429. # create table "fred" ( "dave─Ç" int) ok 15 - unicode column name supported not ok 16 - unicode column found in unqualified column_info # Failed test 'unicode column found in unqualified column_info' # at unicode_test.pl line 306. not ok 17 - unicode column found by qualified column_info # Failed test 'unicode column found by qualified column_info' # at unicode_test.pl line 311. # create table "fred Ç" ( "a" int) ok 18 - unicode table name supported not ok 19 - unicode table found in unqualified table_info # Failed test 'unicode table found in unqualified table_info' # at unicode_test.pl line 269. not ok 20 - Table not found initially but when table name decoded it was found as fred Ç THAT TEST ABOVE^ # Failed test 'Table not found initially but when table name decoded it was found as fred Ç' # at unicode_test.pl line 275. # Did not find tables with utf8 on not ok 21 - unicode table found by qualified table_info # Failed test 'unicode table found by qualified table_info' # at unicode_test.pl line 287. ok 22 # skip table not found 1..22 # Looks like you failed 6 tests of 22. martin@bragi:~/svn/dbi/trunk/ex$ I tried it with 1.33 too - same result. Martin -- Martin J. Evans Wetherby, UK
The test code can be found in DBI subversion trunk in ex/unicode_test.pl but attached here in case it changes. Martin -- Martin J. Evans Wetherby, UK
Subject: unicode_test.pl
# # Copyright Martin J. Evans # # Test unicode in a DBD - written for DBD::ODBC but should work for other # DBDs if you change the column types at the start of this script. # # Usage: # unicode_test.pl DSN USERNAME PASSWORD # # NOTE: will attempt to create tables called fred and # fredĀ (LATIN CAPITAL LETTER A WITH MACRON) # # NOTE: there are multiple ways of doing named parameter markers in DBDs. # some do: # insert into sometable (a_column) values(:fred); # bind_param(':fred', x); # some do: # insert into sometable (a_column) values(:fred); # bind_param('fred', x); # This script does the latter - see unicode_param_markers # # DBD::ODBC currently fails: # not ok 3 - unicode table found by qualified table_info # not ok 6 - unicode column found by qualified column_info # not ok 18 - bind parameter with unicode parameter marker # All of which is documented in the DBD::ODBC pod. The first 2 are because # table_info/column_info XS code uses char * instead of Perl scalars and # the latter is because DBD::ODBC parses the SQL looking for placeholders # and it does this as bytes not UTF-8 encoded strings. # use DBI qw(:sql_types data_diff); use strict; use warnings; use Data::Dumper; use utf8; use Test::More; use Test::More::UTF8; # set utf8 mode on failure,out and todo handles use Test::Exception; use List::Util qw(first); use Encode; # unicode to use in tests for insert/select # the simley ("\x{263A}") is useful because it always has a multibyte encoding my $unicode_sample = "\x{263A}"; # short binary string that is invalid utf8 and includes nul bytes my $binary_sample = "\xFF\x01\x00" x 20; # This script tries to guess the types for unicode columns and binary columns # using type_info_all - it may fail (e.g., if you don't support type_info_all # or if your type_info_all does not return column types this script can # identify as char/binary columns. If it does set the types below or change # the possible SQL types in the calls to find_types below. # my $unicode_column_type; # 'nvarchar for MS SQL Server' my $blob_column_type; # = 'image' for MS SQL Server my $blob_bind_type; # type to pass to bind_param for blobs # may be different in different SQL support # if your DBD/db needs a different function to return the length in # characters of a column redefine $length_fn in a DBD specific section later # in this script my $length_fn = 'length'; my $h = do_connect(); # output a load of data my $driver = $h->{Driver}->{Name}; #note("Driver being used is $driver"); my $dbd="DBD::$h->{Driver}{Name}"; note("Driver " . $dbd,"-",$dbd->VERSION); my $dbms_name = $h->get_info(17); my $dbms_ver = $h->get_info(18); my $driver_name = $h->get_info(6); my $driver_ver = $h->get_info(7); my $identifier_case = $h->get_info(28); note("Using DBMS_NAME " . DBI::neat($dbms_name)); note("Using DBMS_VER " . DBI::neat($dbms_ver)); note("Using DRIVER_NAME " . DBI::neat($driver_name)); note("Using DRIVER_VER " . DBI::neat($driver_ver)); # annoyingly some databases take lowercase table names but create # them uppercase (if unquoted) and so when you ask for a list # of table they come back uppercase. Problem is pattern matching # with unicode and /i is dodgy unless you've got a really recent Perl. note("SQL_IDENTIFIER_CASE " . DBI::neat($identifier_case)); # dump entire env - some people might end up wanting to remove some of this # so changed to specific env vars #note("Environment:\n" . Dumper(\%ENV)); foreach my $env (qw(LANG LC_ NLS_)) { note(map {"$_ = $ENV{$_}\n"} grep(/$env/, keys %ENV)); } # the following sets the "magic" unicode/utf8 flag for each DBD # and sets the column types for DBDs which do not support type_info_all # which is pretty much all of them if ($driver eq 'SQLite') { # does not support type_info_all $blob_column_type = 'blob'; $blob_bind_type = SQL_BLOB; $unicode_column_type = 'varchar'; $h->{sqlite_unicode} = 1; } elsif ($driver eq 'CSV') { # does not support column_info #####$blob_column_type = 'blob'; #####$blob_bind_type = SQL_BLOB; #####$unicode_column_type = 'varchar'; $h->{f_encoding} = 'UTF-8'; $h->{f_ext} = '.csv/r'; $length_fn = 'char_length'; } elsif ($driver eq 'Pg') { $unicode_column_type = 'varchar'; } elsif ($driver eq 'mysql') { # does not support type_info_all $h->{mysql_enable_utf8} = 1; #####$blob_column_type = 'blob'; #####$blob_bind_type = SQL_BLOB; #####$unicode_column_type = 'varchar'; $length_fn = 'char_length'; } elsif ($driver eq 'ODBC') { # DBD::ODBC has type_info_all and column_info support $length_fn = 'len'; } elsif ($driver eq 'Unify') { # Unify does not have varchar $h->{ChopBlanks} = 1; $blob_column_type = 'binary'; $unicode_column_type = 'char'; # or text $h->{uni_unicode} = 1; # Available in the upcoming 0.81 $length_fn = 'undefined'; # I don't think Unify has a function like this } if (!defined($blob_column_type)) { ($blob_column_type, $blob_bind_type) = # -98 for DB2 which gets true blob column type find_type($h, [30, -98, SQL_LONGVARBINARY, SQL_BINARY, SQL_VARBINARY], length($binary_sample)); } BAIL_OUT("Could not find an image/blob type in type_info_all - you will need to change this script to specify the type") if !defined($blob_column_type); if (!defined($unicode_column_type)) { ($unicode_column_type) = find_type($h, [SQL_WVARCHAR, SQL_VARCHAR]); } BAIL_OUT("Could not find a unicode type in type_info_all - you will need to change this script to specify the type") if !defined($unicode_column_type); unicode_data($h); mixed_lob_unicode_data($h); # Without disconnecting after the above test DBD::CSV gets upset # refusing to create fred.csv as it already exists when it certainly # does not exist. # disconnect($h); $h = do_connect(); unicode_param_markers($h); unicode_in_column_name($h); unicode_in_table_name($h); $h->disconnect; unlink 'unitest_8.db' if $driver eq "SQLite"; done_testing; exit 0; # ====== sub do_connect { # eg unicode_test.pl "dbi:Pg(AutoCommit=0):host=example.com;port=6000;db=name" user pass my ($dsn, $user, $pass, %attr) = @ARGV; $user //= $ENV{DBI_USER} // undef; $pass //= $ENV{DBI_PASS} // undef; # A (semi)sane set of defaults my %dsn = ( csv => [ "dbi:CSV:", $user, $pass ], mysql => [ "dbi:mysql:database=test", $user, $pass ], odbc => [ "dbi:ODBC:DSN=asus2", $user, $pass ], oracle => [ "dbi:Oracle:host=betoracle.easysoft.local;sid=devel", 'bet', 'b3t' ], pg => [ "dbi:Pg:dbname=test", $user, $pass ], sqlite => [ "dbi:SQLite:dbname=unitest_8.db", "", "" ], unify => [ "dbi:Unify:", $ENV{USCHEMA}, undef ], ); # Either pass a fully qualified DSN or use the default shortcuts # eg unicode_test.pl CSV $dsn //= "SQLite"; $dsn =~ m/:/ or ($dsn, $user, $pass) = @{$dsn{lc $dsn} // die "No connect info\n"}; my $h = DBI->connect($dsn, $user, $pass, { RaiseError => 1, %attr }); return $h; } sub disconnect { my $h = shift; $h->disconnect; } sub drop_table { my ($h, $table) = @_; eval { local $h->{PrintError} = 0; $table = $h->quote_identifier ($table); my $s = $h->prepare(qq/drop table $table/); $s->execute; }; $h->commit if $driver eq 'Unify'; # DBD::CSV seems to get upset by the mixed_lob_unicode_data test # and fails to drop the table with: # Execution ERROR: utf8 "\x89" does not map to Unicode at /usr/lib/perl/5.10/IO/Handle.pm line 167. unlink 'fred.csv' if $driver eq 'CSV'; #diag($@) if $@; } # create the named table with columns specified in $columns which is # an arrayref with each element a hash of name and type sub create_table { my ($h, $testmsg, $table, $columns) = @_; $table = $h->quote_identifier ($table); my $sql = qq/create table $table ( / . join(",", map {join " " => $h->quote_identifier ($_->{name}), $_->{type}} @$columns) . ')'; return lives_ok { diag ($sql); my $s = $h->prepare($sql); $s->execute; $dbd eq "DBD::Unify" and $h->commit; } $testmsg; } sub unicode_in_table_name { my $h = shift; my $table = "fred\x{0100}"; drop_table($h, $table); my $created = create_table($h, 'unicode table name supported', $table, [{name => 'a', type => 'int'}]); SKIP: { skip "Failed to create unicode table name", 2 unless $created; find_table($h, $table); drop_table($h, $table); } } # NOTE: some DBs may uppercase table names sub find_table { my ($h, $table) = @_; # won't find a match if the returned data is not utf8 decoded my $s = $h->table_info(undef, undef, undef, 'TABLE'); my $r = $s->fetchall_arrayref; my $found = first { $_->[2] =~ /$table/i} @$r; ok($found, 'unicode table found in unqualified table_info'); SKIP: { skip "table found via table_info", 1 if $found; $found = first { Encode::decode_utf8($_->[2]) =~ /$table/i} @$r; ok(!$found, "Table not found initially but when table name decoded it was found as $table"); }; my $found_some_utf8_tables; foreach ($r) { $found_some_utf8_tables++ if Encode::is_utf8($_->[2]); } note(($found_some_utf8_tables ? 'Found' : 'Did not find') , ' tables with utf8 on'); $s = $h->table_info(undef, undef, $table, 'TABLE'); $r = $s->fetchall_arrayref; $found = first {$_->[2] =~ /$table/i} @$r; ok($found, 'unicode table found by qualified table_info'); SKIP: { skip "table not found", 1 if !$found; ok(Encode::is_utf8($found), 'utf8 flag set of unicode table name'); } } sub find_column { my ($h, $table, $column) = @_; my $s = $h->column_info(undef, undef, $table, undef); if (!$s) { note("This driver does not seem to support column_info"); note("Skipping this test"); return; } my $r = $s->fetchall_arrayref; my $found = first {$_->[3] =~ /$column/i} @$r; ok($found, 'unicode column found in unqualified column_info'); $s = $h->column_info(undef, undef, $table, $column); $r = $s->fetchall_arrayref; $found = first {$_->[3] =~ /$column/i} @$r; ok($found, 'unicode column found by qualified column_info'); } sub unicode_in_column_name { my $h = shift; my $table = 'fred'; my $column = "dave\x{0100}"; drop_table($h, $table); my $created = create_table($h, 'unicode column name supported', $table, [{name => $column, type => 'int'}]); SKIP: { skip "table with unicode column not created", 2 unless $created; find_column($h, $table, $column); drop_table($h, $table); }; } sub unicode_data { my $h = shift; my $table = 'fred'; my $column = 'a'; drop_table($h, $table); create_table($h, 'table for unicode data', $table, [{name => $column, type => $unicode_column_type . "(20)"}]); lives_ok { my $s = $h->prepare(qq/insert into $table ($column) values (?)/); $s->execute($unicode_sample); } 'insert unicode data into table'; my $s = $h->prepare(qq/select $column from $table/); $s->execute; my $r = $s->fetchall_arrayref; is($r->[0][0], $unicode_sample, 'unicode data out = unicode data in, no where') or diag(data_diff($r->[0][0]), $unicode_sample); # probably redundant but does not hurt: is(length($r->[0][0]), length($unicode_sample), 'length of output data the same') or diag(data_diff($r->[0][0], $unicode_sample)); # check db thinks the chr is 1 chr eval { # we might not have the correct length fn $s = $h->prepare(qq/select $length_fn($column) from $table/); $s->execute; }; if ($@) { note "!!db probably does not have length function!! - $@"; } else { $r = $s->fetchall_arrayref; is($r->[0][0], length($unicode_sample), 'db length of unicode data correct'); } $s = $h->prepare(qq/select $column from $table where $column = ?/); $s->execute($unicode_sample); $r = $s->fetchall_arrayref; is(scalar(@$r), 1, 'select unicode data via parameterised where'); $s = $h->prepare(qq/select $column from $table where $column = / . $h->quote($unicode_sample)); $s->execute; $r = $s->fetchall_arrayref; is(scalar(@$r), 1, 'select unicode data via inline where'); drop_table($h, $table); } sub mixed_lob_unicode_data { my $h = shift; my $table = 'fred'; my $column1 = 'a'; my $column2 = 'b'; drop_table($h, $table); create_table($h, 'table for unicode data', $table, [{name => $column1, type => $unicode_column_type . "(20)"}, {name => $column2, type => $blob_column_type}]); lives_ok { my $s = $h->prepare(qq/insert into $table ($column1, $column2) values (?,?)/); $s->bind_param(1, $unicode_sample); $s->bind_param(2, $binary_sample, {TYPE => $blob_bind_type}); #$s->execute($unicode_sample, $binary_sample); $s->execute; } 'insert unicode data and blob into table'; # argh - have to set LongReadLen before doing a prepare in DBD::Oracle # because it picks a LongReadLen value when it describes the result-set $h->{LongReadLen} = length($binary_sample) * 2; my $s = $h->prepare(qq/select $column1, $column2 from $table/, {ora_pers_lob => 1}); $s->execute; my $r = $s->fetchall_arrayref; is($r->[0][0], $unicode_sample, 'unicode data out = unicode data in, no where with blob'); ok(!Encode::is_utf8($r->[0][1]), 'utf8 flag not set on blob data'); ok($binary_sample eq $r->[0][1], 'retrieved blob = inserted blob'); drop_table($h, $table); } sub unicode_param_markers { my $h = shift; my $table = 'fred'; drop_table($h, $table); create_table($h, 'test table for unicode parameter markers', $table, [{name => 'a', type => 'int'}]); my $param_marker = "fred\x{20ac}"; lives_ok { my $s = $h->prepare(qq/insert into $table (a) values (:$param_marker)/); $s->bind_param($param_marker, 1); $s->execute; } 'bind parameter with unicode parameter marker'; drop_table($h, $table); } sub find_type { my ($h, $types, $minsize) = @_; my $r = $h->type_info_all; #print Dumper($r); my $indexes = shift @$r; my $sql_type_idx = $indexes->{SQL_DATA_TYPE}; my $type_name_idx = $indexes->{TYPE_NAME}; my $column_size_idx = $indexes->{COLUMN_SIZE}; if (!defined($sql_type_idx)) { note("type_info_all has no key for SQL_DATA_TYPE - falling back on DATA_TYPE"); $sql_type_idx = $indexes->{DATA_TYPE}; } if (!$column_size_idx) { note("type_info_all has no key for COLUMN_SIZE so not performing size checks"); } BAIL_OUT("DBD does not seem to support type_info_all - you will need to edit this script to specify column types") if !$r || (scalar(@$r) == 0); foreach my $type (@$types) { foreach (@$r) { note("Found type $_->[$sql_type_idx] ($_->[$type_name_idx]) size=" . ($column_size_idx ? $_->[$column_size_idx] // 'undef' : 'undef')); if ($_->[$sql_type_idx] eq $type) { if ((!defined($minsize)) || (!defined($column_size_idx)) || ($minsize && ($_->[$column_size_idx] > $minsize))) { note("Found $type type which is $_->[$type_name_idx] and max size of " . ($column_size_idx ? $_->[$column_size_idx] : 'undef')); return ($_->[$type_name_idx], $_->[$sql_type_idx]); } else { note("$type type ($_->[$type_name_idx]) but the max length of $_->[$column_size_idx] is less than the required length $minsize"); } } } } }
On Mon Nov 14 14:11:18 2011, MJEVANS wrote:
Show quoted text
> The test code can be found in DBI subversion trunk in ex/unicode_test.pl > but attached here in case it changes. > > Martin
The test code has changed to fix a bug - best to get the latest version from DBI subversion tree in ex/unicode_test.pl. Martin -- Martin J. Evans Wetherby, UK
Hi. There was 2 resons why 6 tests fail. 1. sqltite_unicode option is only affects on connect method. DBI->connect($dsn, $user, $pass, { sqlite_unicode => 1}); # ok $dbh->{sqltite_unicode} = 1; # not ok 2. A test case is wrong 'bind parameter' test case seems to be wrong. There is no unicode bug. my $s = $h->prepare(qq/insert into $table (a) values (:$param_marker)/); - $s->bind_param($param_marker, 1); + $s->bind_param(":$param_marker", 1); Fixing two of them makes all unicode_test.pl tests passed. To fix No.1, I guess sqlite_db_login6 should some how detect $dbh->{sqlite_unicode} as well as sqlite_db_STORE_attrib. I'm just looking for how. In addition to No.1 issue there is still a column name bug due to lack of SvUTF8_on. The topic below describes the detail. https://rt.cpan.org/Public/Bug/Display.html?id=78833
On Fri Aug 10 11:41:54 2012, JAMADAM wrote:
Show quoted text
> Hi. > > There was 2 resons why 6 tests fail. > > 1. sqltite_unicode option is only affects on connect method. > > DBI->connect($dsn, $user, $pass, { sqlite_unicode => 1}); # ok > $dbh->{sqltite_unicode} = 1; # not ok
Because of a typo? ;) (sql"t"ite_unicode)
Show quoted text
> > 2. A test case is wrong > > 'bind parameter' test case seems to be wrong. There is no unicode bug. > > my $s = $h->prepare(qq/insert into $table (a) values (:
$param_marker)/);
Show quoted text
> - $s->bind_param($param_marker, 1); > + $s->bind_param(":$param_marker", 1);
This is noted at the top of unicode_test.pl, but thanks for the heads- up. I'll remember to tweak this when I test.
Show quoted text
> Fixing two of them makes all unicode_test.pl tests passed.
I'm not sure exactly what you did to pass the tests, but didn't you simply add sqlite_unicode => 1 in the "do_connect" function? Also, note that "*_info" methods reside in lib/DBD/SQLite.pm.
Show quoted text
> > To fix No.1, I guess sqlite_db_login6 should some how detect > $dbh->{sqlite_unicode} as well as sqlite_db_STORE_attrib. > I'm just looking for how. > > In addition to No.1 issue there is still a column name bug due to lack > of SvUTF8_on. > The topic below describes the detail. > > https://rt.cpan.org/Public/Bug/Display.html?id=78833
On 2012-8月-09 木 23:37:32, ISHIGAKI wrote:
Show quoted text
> On Fri Aug 10 11:41:54 2012, JAMADAM wrote:
> > Hi. > > > > There was 2 resons why 6 tests fail. > > > > 1. sqltite_unicode option is only affects on connect method. > > > > DBI->connect($dsn, $user, $pass, { sqlite_unicode => 1}); # ok > > $dbh->{sqltite_unicode} = 1; # not ok
> > Because of a typo? ;) (sql"t"ite_unicode) >
Sorry, the report is wrong. I tested without typo.
Show quoted text
> > > > 2. A test case is wrong > > > > 'bind parameter' test case seems to be wrong. There is no unicode bug. > > > > my $s = $h->prepare(qq/insert into $table (a) values (:
> $param_marker)/);
> > - $s->bind_param($param_marker, 1); > > + $s->bind_param(":$param_marker", 1);
> > This is noted at the top of unicode_test.pl, but thanks for the heads- > up. I'll remember to tweak this when I test.
I overlooked that.
Show quoted text
>
> > Fixing two of them makes all unicode_test.pl tests passed.
> > I'm not sure exactly what you did to pass the tests, but didn't you > simply add sqlite_unicode => 1 in the "do_connect" function?
Yes, I exactly did it to identify the issue is on sqlite_unicode detection. The driver still needs fix for passing original unicode_test.pl.
Show quoted text
> > Also, note that "*_info" methods reside in lib/DBD/SQLite.pm. >
With the test modification, attached unicode_test.pl doesn't pass 'unicode table name' test but the newest one on svn passes.
On Fri Aug 10 13:29:24 2012, JAMADAM wrote:
Show quoted text
> On 2012-8月-09 木 23:37:32, ISHIGAKI wrote:
> > On Fri Aug 10 11:41:54 2012, JAMADAM wrote:
> > > Hi. > > > > > > There was 2 resons why 6 tests fail. > > > > > > 1. sqltite_unicode option is only affects on connect method. > > > > > > DBI->connect($dsn, $user, $pass, { sqlite_unicode => 1}); # ok > > > $dbh->{sqltite_unicode} = 1; # not ok
> > > > Because of a typo? ;) (sql"t"ite_unicode) > >
> > Sorry, the report is wrong. I tested without typo. >
> > > > > > 2. A test case is wrong > > > > > > 'bind parameter' test case seems to be wrong. There is no unicode
bug.
Show quoted text
> > > > > > my $s = $h->prepare(qq/insert into $table (a) values (:
> > $param_marker)/);
> > > - $s->bind_param($param_marker, 1); > > > + $s->bind_param(":$param_marker", 1);
> > > > This is noted at the top of unicode_test.pl, but thanks for the
heads-
Show quoted text
> > up. I'll remember to tweak this when I test.
> > I overlooked that. >
> >
> > > Fixing two of them makes all unicode_test.pl tests passed.
> > > > I'm not sure exactly what you did to pass the tests, but didn't you > > simply add sqlite_unicode => 1 in the "do_connect" function?
> > Yes, I exactly did it to identify the issue is on sqlite_unicode
detection.
Show quoted text
> The driver still needs fix for passing original unicode_test.pl.
Then try adding $h->{sqlite_unicode} = 1; after $h = do_connect(); in the middle of the test. The test will pass, but that's not the point of this test.
Show quoted text
> > > > Also, note that "*_info" methods reside in lib/DBD/SQLite.pm. > >
> > With the test modification, attached unicode_test.pl doesn't pass > 'unicode table name' test but the newest one on svn passes. >
You're right. I'd misunderstood the test. There is no bug on sqlite_unicode detection. I will retry to investigate. Thanks. On 2012-8月-10 金 00:40:57, ISHIGAKI wrote:
Show quoted text
> On Fri Aug 10 13:29:24 2012, JAMADAM wrote:
> > On 2012-8月-09 木 23:37:32, ISHIGAKI wrote:
> > > On Fri Aug 10 11:41:54 2012, JAMADAM wrote:
> > > > Hi. > > > > > > > > There was 2 resons why 6 tests fail. > > > > > > > > 1. sqltite_unicode option is only affects on connect method. > > > > > > > > DBI->connect($dsn, $user, $pass, { sqlite_unicode => 1}); # ok > > > > $dbh->{sqltite_unicode} = 1; # not ok
> > > > > > Because of a typo? ;) (sql"t"ite_unicode) > > >
> > > > Sorry, the report is wrong. I tested without typo. > >
> > > > > > > > 2. A test case is wrong > > > > > > > > 'bind parameter' test case seems to be wrong. There is no unicode
> bug.
> > > > > > > > my $s = $h->prepare(qq/insert into $table (a) values (:
> > > $param_marker)/);
> > > > - $s->bind_param($param_marker, 1); > > > > + $s->bind_param(":$param_marker", 1);
> > > > > > This is noted at the top of unicode_test.pl, but thanks for the
> heads-
> > > up. I'll remember to tweak this when I test.
> > > > I overlooked that. > >
> > >
> > > > Fixing two of them makes all unicode_test.pl tests passed.
> > > > > > I'm not sure exactly what you did to pass the tests, but didn't you > > > simply add sqlite_unicode => 1 in the "do_connect" function?
> > > > Yes, I exactly did it to identify the issue is on sqlite_unicode
> detection.
> > The driver still needs fix for passing original unicode_test.pl.
> > Then try adding $h->{sqlite_unicode} = 1; after $h = do_connect(); in > the middle of the test. The test will pass, but that's not the point of > this test. >
> > > > > > Also, note that "*_info" methods reside in lib/DBD/SQLite.pm. > > >
> > > > With the test modification, attached unicode_test.pl doesn't pass > > 'unicode table name' test but the newest one on svn passes. > >
> >
Hi. DBD::SQLite 1.38_01 should fix the issue, though the script in the DBI repo still needs some fix. Please refer to t/ rt_78833_utf8_flag_for_column_names.t as well. Closed this ticket as resolved for now. Thanks. On Tue Nov 15 04:07:22 2011, MJEVANS wrote:
Show quoted text
> Hi, > > Attached a script I wrote for DBI and testing unicode in various DBDs. > It shows that when you create a unicode table name and call > table_info_all you cannot find the table by name comparison and yet it > does exist. If the names returned by table_info_all are decoded as
UTF-8
Show quoted text
> you can find the table. So that suggests DBD::SQLite is not decoding > table names. The same probably applies to unicode column names but > although there is a test to check you can use them and find them there > is no additional code to test if they are decoded as UTF-8 you can
find
Show quoted text
> them. > > BTW, some tests DBD::SQLite fails no one really expects it to pass
like
Show quoted text
> unicode parameter name markers so please ignore that test - the script > was written for discovery not to demonstrate any particular bugs. > > Output when run on Windows (hence the unicode data out is displayed > wrong so ignore that) is: > > martin@bragi:~/svn/dbi/trunk/ex$ perl unicode_test.pl
dbi:SQLite:db=test.db
Show quoted text
> # Driver DBD::SQLite-1.29 > # Using DBMS_NAME 'SQLite' > # Using DBMS_VER '3.6.22' > # Using DRIVER_NAME undef > # Using DRIVER_VER undef > # SQL_IDENTIFIER_CASE undef > # LANG = en_GB.UTF-8 > # create table "fred" ( "a" varchar(20)) > ok 1 - table for unicode data > ok 2 - insert unicode data into table > ok 3 - unicode data out = unicode data in, no where > ok 4 - length of output data the same > ok 5 - db length of unicode data correct > ok 6 - select unicode data via parameterised where > ok 7 - select unicode data via inline where > # create table "fred" ( "a" varchar(20),"b" blob) > ok 8 - table for unicode data > ok 9 - insert unicode data and blob into table > ok 10 - unicode data out = unicode data in, no where with blob > ok 11 - utf8 flag not set on blob data > ok 12 - retrieved blob = inserted blob > # create table "fred" ( "a" int) > ok 13 - test table for unicode parameter markers > DBD::SQLite::st bind_param failed: Unknown named parameter: fredÔé¼ at > unicode_t > est.pl line 429. > not ok 14 - bind parameter with unicode parameter marker > # Failed test 'bind parameter with unicode parameter marker' > # at unicode_test.pl line 431. > # died: DBD::SQLite::st bind_param failed: Unknown named parameter: > fred├ó┬é┬¼ a > t unicode_test.pl line 429. > # create table "fred" ( "dave─Ç" int) > ok 15 - unicode column name supported > not ok 16 - unicode column found in unqualified column_info > # Failed test 'unicode column found in unqualified column_info' > # at unicode_test.pl line 306. > not ok 17 - unicode column found by qualified column_info > # Failed test 'unicode column found by qualified column_info' > # at unicode_test.pl line 311. > # create table "fred Ç" ( "a" int) > ok 18 - unicode table name supported > not ok 19 - unicode table found in unqualified table_info > # Failed test 'unicode table found in unqualified table_info' > # at unicode_test.pl line 269. > not ok 20 - Table not found initially but when table name decoded it
was
Show quoted text
> found as fred Ç > > THAT TEST ABOVE^ > > # Failed test 'Table not found initially but when table name decoded > it was found as fred Ç' > # at unicode_test.pl line 275. > # Did not find tables with utf8 on > not ok 21 - unicode table found by qualified table_info > # Failed test 'unicode table found by qualified table_info' > # at unicode_test.pl line 287. > ok 22 # skip table not found > 1..22 > # Looks like you failed 6 tests of 22. > martin@bragi:~/svn/dbi/trunk/ex$ > > I tried it with 1.33 too - same result. > > Martin


This service runs on Request Tracker, is sponsored by The Perl Foundation, and maintained by Best Practical Solutions.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.