Subject: Re: Bug#649729: uniq: merges obscure Cyrillic characters

From: Bob Proulx <[email-masked]>
Date: Thu, 2 Feb 2012 23:47:45 -0700

forcemerge 139861 649729
thanks

Alex Shinn wrote:
> As the link states, it's pretty clearly a bug - the correct behavior
> would be to sort the unknown characters after all known characters
> and consider them distinct.  As a workaround, adding values for
> all characters to every locale file in /usr/share/i18n/locales/ should
> work.

See also this issue:

  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=139861

It is a known deficiency in coreutils that the utilities are not
multibyte aware.  The following can be found in the upstream source
package TODO file.

  Adapt tools like wc, tr, fmt, etc. (most of the textutils) to be
    multibyte aware.  The problem is that I want to avoid duplicating
    significant blocks of logic, yet I also want to incur only minimal
    (preferably `no') cost when operating in single-byte mode.

Some vendors have hacked in patches to make the utilities multibyte
aware but none of those patches have been considered clean enough to
incorporate into the upstream source yet.  Debian's maintainer has
stated that he does not want to diverge from upstream this radically.
The patches are very messy and incomplete.  The best course of action
would be to get this resolved upstream with the functionally properly
integrated.  Until then this remains a known deficiency.

Bob