#24835: Net::Cmd 2.27 (libnet 1.20) incorrectly upgrades everything to UTF-8

Hello Graham,

I'm afraid libnet 1.20 has introduced a quite important bug for people
using characters outside ASCII: unconditionally calling utf8::encode()
on any data passed to datasend() has the side effect to convert
everything to UTF-8, even when it's not expected to be. As a result,
all the accented characters appear as the usual Unicode junk: "é"
becomes "é", "ê" becomes "ê" and the like.

Therefore, perfectly valid programs that were sending correct mails
will send rubbish as soon as libnet is upgraded to version 1.20.

As suggested in ticket#18589, there may be a need to pass additional
parameter to libnet modules in order to indicate the encoding
(although I'm not sure if one can assume that encoding stays the
same through an entire mail or if it can change from one part to
another).

In the mean time, I'd suggest to remove the call to utf8::encode()
from Net::Cmd:

--- lib/Net/Cmd.pm.orig 2006-10-27 13:08:07.000000000 +0200
+++ lib/Net/Cmd.pm 2007-02-07 19:27:54.328532000 +0100
@@ -21,8 +21,6 @@
}
}

-my $doUTF8 = eval { require utf8 };
-
$VERSION = "2.27";
@ISA = qw(Exporter);
@EXPORT = qw(CMD_INFO CMD_OK CMD_MORE CMD_REJECT CMD_ERROR CMD_PENDING);
@@ -395,8 +393,6 @@
my $arr = @_ == 1 && ref($_[0]) ? $_[0] : \@_;
my $line = join("" ,@$arr);

- utf8::encode($line) if $doUTF8;
-
return 0 unless defined(fileno($cmd));

my $last_ch = ${*$cmd}{'net_cmd_last_ch'};



Best Regards,

--
Close the world, txEn eht nepO.

Hello,

Confirming the bug, it is affecting us. I also e-mailed Graham about this.

On Wed Feb 07 13:30:48 2007, SAPER wrote:

> As suggested in ticket#18589, there may be a need to pass additional
> parameter to libnet modules in order to indicate the encoding
> (although I'm not sure if one can assume that encoding stays the
> same through an entire mail or if it can change from one part to
> another).

This is not a good idea, a multi-part MIME message can have each part in
a different character set. People shouldn't be passing on strings to
Net::CMD anyway, they should be passing octets (variables with the
internal utf8 flag off).

And as there is the internal perl flag for utf8ness I don't see any
reason to pass on encoding as a parameter.

Net::CMD could just die if the flag is on if you want to be strict.
If you don't then the line

utf8::encode($line) if $doUTF8;

should/could be replaced with:

if ($doUTF8) {
# encode to individual utf8 bytes if
# $line is a string (in internal UTF-8)
utf8::encode($line) if utf8::is_utf8($line);
}

to fix the bug with latin-1 and to do what people probably expect if
they would feed UTF8 lines to it.



For reference:
http://perldoc.perl.org/utf8.html
http://www.perlmonks.org/?displaytype=print;node_id=551676

Fixed in libnet 1.21

Graham.