Subject: Bug#431231: tr fails with UTF-8

From: Juhapekka Tolvanen <[email-masked]>
Date: Sat, 30 Jun 2007 23:29:25 +0300

Package: coreutils
Version: 5.97-5.3
Severity: important

juhtolv@juhtolv:/home/juhtolv % echo 'huuhaa öljy äiti über' | tr '[:lower:]' '[:upper:]'
HUUHAA öLJY äITI üBER
juhtolv@juhtolv:/home/juhtolv % echo 'huuhaa öljy äiti über' | tr '[:lower:]' '[:upper:]' | tr 'åäöü' 'ÅÄÖÜ'
HUUHAA ÖLJY ÄITI ÜBER
juhtolv@juhtolv:/home/juhtolv %

See locale below.

-- System Information:
Debian Release: 4.0
  APT prefers stable
  APT policy: (990, 'stable'), (500, 'testing-proposed-updates'), (500, 'proposed-updates'), (101, 'testing'), (99, 'unstable')
Architecture: i386 (i686)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.18-5-686
Locale: LANG=fi_FI.utf8, LC_CTYPE=fi_FI.utf8 (charmap=UTF-8)

Versions of packages coreutils depends on:
ii  libacl1                       2.2.41-1   Access control list shared library
ii  libc6                         2.5-9+b1   GNU C Library: Shared libraries
ii  libselinux1                   2.0.15-2   SELinux shared libraries

coreutils recommends no packages.

-- no debconf information

-- 
Juhapekka "naula" Tolvanen * http colon slash slash iki dot fi slash juhtolv
"Denn du bist was du isst und ihr wisst was es ist. Es ist mein Teil – nein.
Mein Teil – nein. Da das ist mein Teil – nein. Mein Teil – nein."  Rammstein

---

From: [email-masked] (Bob Proulx)
Date: Sat, 30 Jun 2007 15:33:12 -0600

merge 431231 139861
thanks

Juhapekka Tolvanen wrote:
> ...report of a locale problem in tr deleted...

See also these related issues.

  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=139861
  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=388689

It is a known deficiency in coreutils in general that the utilities
are not multibyte aware.  The following can be found in the upstream
source package TODO file.

  Adapt tools like wc, tr, fmt, etc. (most of the textutils) to be
    multibyte aware.  The problem is that I want to avoid duplicating
    significant blocks of logic, yet I also want to incur only minimal
    (preferably `no') cost when operating in single-byte mode.

Some vendors have hacked in patches to make the utilities multibyte
aware but none of those patches have been considered clean enough to
incorporate into the upstream source yet.  Debian's maintainer has
stated that he does not want to diverge from upstream this radically.
The patches are very messy and incomplete.  The best course of action
would be to get this resolved upstream with the functionally properly
integrated.

Bob

---

From: Juhapekka Tolvanen <[email-masked]>
Date: Tue, 25 Dec 2007 06:48:59 +0200

On Sun, 01 Jul 2007, +01:24:14 EEST (UTC +0300),

> source package TODO file.

Look at this:

% echo 'huuhaa öljy äiti über' | tr '[:lower:]' '[:upper:]'
HUUHAA öLJY äITI üBER
% echo 'huuhaa öljy äiti über' | /opt/heirloom/5bin/tr '[:lower:]' '[:upper:]'
HUUHAA ÖLJY ÄITI ÜBER

That Heirloom Toolchest is available here:

http://heirloom.sourceforge.net/

IMNSHO Debian project should package that toolchest, too. Maybe GNU
tools should be replaced with them, but that would be very radical.

Also other software from Heirloom -project should be Debian-packaged,
because they have superior UTF-8 -support when compared to
GNU-counterparts. For example their roff -implementation can handle
UTF-8 and OpenType-fonts.

-- 
Juhapekka "naula" Tolvanen * http colon slash slash iki dot fi slash juhtolv
"eiga wo miyou kimi no yakusoku dohri te wo tsunai de. yoru ni wa owakare
desu ringo to ichigo ga kusaru mae ni. yume wa hirogaru kimi no yakusoku
dohri kisu wo shi nagara."                                       Dir en grey

---

From: Colin Watson <[email-masked]>
Date: Mon, 31 Dec 2007 17:14:08 +0000

> UTF-8 and OpenType-fonts.

Somebody else is welcome to try to do so, but I don't have the time, I'm
afraid.

Cheers,

-- 
Colin Watson                                       [[email-masked]]

---

From: Lucas Nussbaum <[email-masked]>
Date: Tue, 22 Jan 2008 20:56:51 +0100

forcemerge 139861 388689 431231
tags 139861 + upstream confirmed wontfix
found 139861 6.10~20071127-1
thanks

Hi,

I'm merging these bugs (all about tr not supporting UTF-8), that still
affects the current coreutils in experimental. "wontfix" indicates that
this is not going to be fixed by a debian-specific patch, but that the
problem should be fixed upstream first.
-- 
| Lucas Nussbaum
| [email-masked]   http://www.lucas-nussbaum.net/ |
| jabber: [email-masked]             GPG: 1024D/023B3F4F |

---

From: Jonathan Nieder <[email-masked]>
Date: Fri, 11 Feb 2011 04:04:13 -0600

tags 139861 - wontfix
quit

> problem should be fixed upstream first.

That isn't what wontfix usually means, is it?

Thanks; I do agree that it seems best for anyone working on this to
just communicate directly with upstream.