Subject: Re: Bug#493465: libncurses5: allocates memory that is never freed

From: Lars Doelle <[email-masked]>
Date: Fri, 8 May 2009 23:37:29 +0200

Hi,

i stumbled over this bug, too, and write to ask what the state of resolution is.
The libncurses5-dbg is not distributed with --disable-leaks, at time of writing.

If no semantic issues are know i would suggest not only to build the debugging
library with --disable-leaks, but the production libraries, as well, to allow to build
memory balanced applications and up-stream libraries.

Thanks,

  Lars

---

From: Thomas Dickey <[email-masked]>
Date: Fri, 8 May 2009 18:02:02 -0400 (EDT)

> The libncurses5-dbg is not distributed with --disable-leaks, at time of writing.

--disable-leaks won't (by itself) free "all" leaks.
For that, you'd need the non-public function which frees all memory.

But doing that, some features of ncurses (like any curses implementation)
would not work.  In short the _nc_freeall, etc., are only useful for
debugging, not for a production library.

-- 
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

---

From: Lars Doelle <[email-masked]>
Date: Fri, 15 May 2009 06:36:32 +0200

Thomas,

(thank you for maintaining ncurses for so long.)

> debugging, not for a production library.

But this precisely is the point i do not understand. If i call '_nc_freeall' under
the obligation not to call any ncurses routine again, how on earth could this
change the semantics?

Likely, keeping track of the allocated items, as needed for '_nc_freeall' to do
its work, is also something that should be absolutely transparent for the items
themselves. So no semantics should be affected either by the 'debugging'
mode as far as storage is concerned.

So '_nc_freeall' (which is a storage policy thingy, IMO) and semantic changing
debugging features got mixed, or what is it?

Kind regards,
  Lars

---

From: Thomas Dickey <[email-masked]>
Date: Fri, 15 May 2009 19:26:40 -0400

> the obligation not to call any ncurses routine again, how on earth could this

man endwin gives a hint:

       A  program  should  always  call endwin before exiting or escaping from
       curses mode temporarily.  This routine restores tty  modes,  moves  the
       cursor  to the lower left-hand corner of the screen and resets the ter-
       minal into the proper non-visual mode.  Calling refresh or doupdate af-
       ter a temporary escape causes the program to resume visual mode.

(note the "temporarily")

-- 
Thomas E. Dickey <[email-masked]>
http://invisible-island.net
ftp://invisible-island.net

---

From: Lars Doelle <[email-masked]>
Date: Sat, 16 May 2009 23:14:05 +0200

Thomas

>        ter a temporary escape causes the program to resume visual mode.

If this means that the semantics is /not/ affected under the obligation /not/ to
issue any other ncurses calls /after/ calling '_nc_freeall', i do not see why this
call cannot be distributed in a production library.

(It is not even distributed in the debugging library, btw. right now, and that alone
would not help much, since everything upstream is properly infected by the
storage policy. E.g.
$ ldd /usr/lib/debug/libreadline.so.5
        libncurses.so.5 => /lib/libncurses.so.5)

Personally, i do not think, this is an issue aside. Right now it is not possible to write
/any/ program providing a non-trivial UI with a balanced memory. This does not
only break the local storage policy (which could demand that all memory is returned
/before/ exit), but also development tools and the development methods. I for one,
prefer to run all my programs with valgrind /while/ developing to detect not only
memory leaks but a lot of other hard to diagnose issues in the moment i introduce
them. Not possible to do that as soon as libncurses get involved. The policy imposed
creeps upstream. See, i add a single call (readline) and get 4 or 5 pages of error
messages in valgrind, covering everything that could of /real/ interest. Ok, i could do
an #ifdef and use getline to test /my/ stuff, but i cannot longer use 'readline' while
developing which only makes life harder for no reason.

Though off-topic, the same issue at precisely the same place under Xlib (mostly
configuration database stuff), successfully preventing any decision whether one
of the graphical programs truly leaks memory or not. Hmm, they almost certainly
all do, and the problem starts with Xlib's memory policy.

It is not that bad with character based UIs, but only because the upstream chain
is shorter. Repeating the summary, it is not possible right now to write /any/
memory balanced program with a decent UI (graphical or character) for a pure
policy issue.

Thomas, i'm well aware that preserving semantics with a thing like curses is a
delicate issue to say the least, but to impose a particular policy (i.e. not to allow
to balance memory before exit) is another to the very best of my understanding.
Thus my asking for any technical reason. If there are non, i vote to include
'_nc_freeall' for production.

As said, though the '_nc_freeall' feature exists (thank you, Thomas), right now it
is not distributed on debian at all, which leaves only to compile a local version.
Thus, technically, it appears to be solved for ncurses, but the solution is simply
not distributed.

Kind regards,
   Lars

---

From: Thomas Dickey <[email-masked]>
Date: Sun, 17 May 2009 20:53:13 -0400 (EDT)

On Sat, 16 May 2009, Lars Doelle wrote:

> If this means that the semantics is /not/ affected under the obligation 
> /not/ to issue any other ncurses calls /after/ calling '_nc_freeall', i 
> do not see why this call cannot be distributed in a production library.

Well, there are several issues - first, being able to discard
most of the memory is necessarily an extension to X/Open.
The description of endwin makes that plain enough (to me, anyway).

But so far, the only use that I'm aware of for this is for analysis -
no real program would be using it.  (memory gets allocated, but doesn't
grow, unless there is a bug ;-)

Then, curses libraries have a variety of higher/lower-level interfaces
(tgetent, setupterm, newterm, initscr), and a lot of legacy applications
that assume things about their behavior.

For debugging, it's sufficient to be able to audit the memory and discard
everything (though not necessarily leave memory in a state that would
permit restarting the library).

Making a cleanup routine a regular part of the API would lead very
quickly to demands that it do-the-right-thing and permit restart.

Not all of the memory can be freed - except in special cases, and then the 
application's dead (such as setbuf).  The check in _nc_free_and_exit 
depends on a special case (a single screen).

> developing which only makes life harder for no reason.

looking at the suppress-files for glibc shows that life is hard to
start with.

(a recent update from Debian got me about a thousand lines of valgrind
warnings in a "clean" program - I suppose it'll get fixed in those
".supp" files ;-).

> Though off-topic, the same issue at precisely the same place under Xlib 
> (mostly configuration database stuff), successfully preventing any 
> decision whether one of the graphical programs truly leaks memory or 
> not. Hmm, they almost certainly all do, and the problem starts with 
> Xlib's memory policy.

I normally get about 4000 lines of listing from running xterm -
with its --disable-leaks option turned on.  Without, that doubles.

>
> It is not that bad with character based UIs, but only because the 
> upstream chain is shorter. Repeating the summary, it is not possible 
> right now to write /any/ memory balanced program with a decent UI 
> (graphical or character) for a pure policy issue.
>
> Thomas, i'm well aware that preserving semantics with a thing like 
> curses is a delicate issue to say the least, but to impose a particular 
> policy (i.e. not to allow to balance memory before exit) is another to 
> the very best of my understanding. Thus my asking for any technical 
> reason. If there are non, i vote to include '_nc_freeall' for 
> production.