|Subject:||JSON::Syck cannot handle utf-8/unicode correctly.|
JSON is defined to be an octet stream (by default encoded in UTF-8), while strings within JSON are defined to be unicode. There seems to be no way to handle this correctly with JSON::Syck, which makes it impossible to properly exchange data using non-ascii characters. The module offers $JSON::Syck::ImplicitUnicode, but neither 0 or 1 result in correct behaviour. If it is 0, then JSON::Syck does not corretcly encode perl strings into json objects. If it is 1, it sometimes returns json objects with "bytes" Show quoted text
>255.The problem is likely the wrong mental model. E.g. the documentation incorrectly says: Perl (UTF-8 bytes) => JSON (Unicode flagged) Perl has no such thing as a "unicode flag". Perl has a utf-8 flag, but that doesn't flag a scalar as unicode, its jsut a different encoding of the scalar data. Perl only knows about binary strings (octet strings) and text strings (unicode character strings), both of which can have the UTF-8 flag set or cleared. The correct handling is to always encode the resulting json object correctly (preferably in UTF-8), and always create perl text strings (in either UTF-8 or latin1 encoding), as json strings are defined to be text.