Debian Bug report logs - #835904
tail --retry doesn't

Reply or subscribe to this bug.

Display info messages

Message #5 received at [email protected] (full text, mbox, reply):

From: Harald Dunkel <[email protected]>

To: [email protected]

Subject: tail --retry doesn't

Date: Mon, 29 Aug 2016 09:55:13 +0200

Package: coreutils
Version: 8.23-4

I am running tail in a dedicated window for some days now:

% ps -ef | grep tai[l]
hdunkel   2928  2915  0 Aug15 pts/1    00:00:00 tail --retry --max-unchanged-stats=5 -f /var/log/messages

Problem: It's frozen. /var/log/messages is changing all the
time, but tail doesn't recognize.

lsof -p shows:

% lsof -p 2928
lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
      Output information may be incomplete.
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME
tail    2928 hdunkel  cwd    DIR   0,40    20480 189573782 /home/hdunkel (nfs-data:/space/home)
tail    2928 hdunkel  rtd    DIR  259,0     4096         2 /
tail    2928 hdunkel  txt    REG  259,0    64232    657851 /usr/bin/tail
tail    2928 hdunkel  mem    REG  259,0  1738176   1724334 /lib/x86_64-linux-gnu/libc-2.19.so
tail    2928 hdunkel  mem    REG  259,0   140928   1724331 /lib/x86_64-linux-gnu/ld-2.19.so
tail    2928 hdunkel    0u   CHR  136,1      0t0         4 /dev/pts/1
tail    2928 hdunkel    1u   CHR  136,1      0t0         4 /dev/pts/1
tail    2928 hdunkel    2u   CHR  136,1      0t0         4 /dev/pts/1
tail    2928 hdunkel    3r   REG  259,0   100176    263640 /var/log/messages.1 (deleted)
tail    2928 hdunkel    4r  0000   0,11        0      8294 anon_inode


AFAICT this is not supposed to happen. According to the man
page "tail --retry --max-unchanged-stats=5" should reopen
/var/log/messages.


Regards
Harri

Message #10 received at [email protected] (full text, mbox, reply):

From: Pádraig Brady <[email protected]>

To: Harald Dunkel <[email protected]>, [email protected], [email protected]

Subject: Re: Bug#835904: tail --retry doesn't

Date: Mon, 29 Aug 2016 11:57:42 +0100

On 29/08/16 08:55, Harald Dunkel wrote:
> Package: coreutils
> Version: 8.23-4
> 
> I am running tail in a dedicated window for some days now:
> 
> % ps -ef | grep tai[l]
> hdunkel   2928  2915  0 Aug15 pts/1    00:00:00 tail --retry --max-unchanged-stats=5 -f /var/log/messages
> 
> Problem: It's frozen. /var/log/messages is changing all the
> time, but tail doesn't recognize.
> 
> lsof -p shows:
> 
> % lsof -p 2928
> lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
>       Output information may be incomplete.
> COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME
> tail    2928 hdunkel  cwd    DIR   0,40    20480 189573782 /home/hdunkel (nfs-data:/space/home)
> tail    2928 hdunkel  rtd    DIR  259,0     4096         2 /
> tail    2928 hdunkel  txt    REG  259,0    64232    657851 /usr/bin/tail
> tail    2928 hdunkel  mem    REG  259,0  1738176   1724334 /lib/x86_64-linux-gnu/libc-2.19.so
> tail    2928 hdunkel  mem    REG  259,0   140928   1724331 /lib/x86_64-linux-gnu/ld-2.19.so
> tail    2928 hdunkel    0u   CHR  136,1      0t0         4 /dev/pts/1
> tail    2928 hdunkel    1u   CHR  136,1      0t0         4 /dev/pts/1
> tail    2928 hdunkel    2u   CHR  136,1      0t0         4 /dev/pts/1
> tail    2928 hdunkel    3r   REG  259,0   100176    263640 /var/log/messages.1 (deleted)
> tail    2928 hdunkel    4r  0000   0,11        0      8294 anon_inode
> 
> 
> AFAICT this is not supposed to happen. According to the man
> page "tail --retry --max-unchanged-stats=5" should reopen
> /var/log/messages.

There were tail fixes in this area in v8.24.
Previously renames may have been missed when inotify was used

Message #20 received at [email protected] (full text, mbox, reply):

From: Harald Dunkel <[email protected]>

To: [email protected]

Subject: Re: tail --retry doesn't

Date: Mon, 5 Sep 2016 08:07:44 +0200

coreutils 8.25-2 seems to have the same problem. Sample:


% ps -ef | grep tail
harri     3815  3783  0 Aug30 pts/3    00:00:00 tail --retry --max-unchanged-stats=5 -f /var/log/messages
harri    11756 11708  0 08:00 pts/4    00:00:00 grep tail

% lsof -p 3815
COMMAND  PID  USER   FD      TYPE DEVICE SIZE/OFF    NODE NAME
tail    3815 harri  cwd       DIR    8,4    20480 4456449 /home/harri
tail    3815 harri  rtd       DIR    8,3     4096       2 /
tail    3815 harri  txt       REG    8,3    64232   15185 /usr/bin/tail
tail    3815 harri  mem       REG    8,3  1697504 1054034 /lib/x86_64-linux-gnu/libc-2.23.so
tail    3815 harri  mem       REG    8,3   153288 1054030 /lib/x86_64-linux-gnu/ld-2.23.so
tail    3815 harri    0u      CHR  136,3      0t0       6 /dev/pts/3
tail    3815 harri    1u      CHR  136,3      0t0       6 /dev/pts/3
tail    3815 harri    2u      CHR  136,3      0t0       6 /dev/pts/3
tail    3815 harri    3r      REG    8,3   159279  140486 /var/log/messages.1
tail    3815 harri    4r  a_inode   0,11        0    4381 inotify

% ls -alrt /var/log/messages*
-rw-r----- 1 root adm   2057 Aug 15 07:54 /var/log/messages.4.gz
-rw-r----- 1 root adm   1762 Aug 21 08:03 /var/log/messages.3.gz
-rw-r----- 1 root adm  29000 Aug 29 07:53 /var/log/messages.2.gz
-rw-r----- 1 root adm 159279 Sep  4 08:00 /var/log/messages.1
-rw-r----- 1 root adm   2881 Sep  5 07:52 /var/log/messages

% dpkg -l coreutils
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                Version                Architecture           Description
+++-===================================-======================-======================-============================================================================
ii  coreutils                           8.25-2                 amd64                  GNU core utilities


Regards
Harri

Message #25 received at [email protected] (full text, mbox, reply):

From: Pádraig Brady <[email protected]>

To: Harald Dunkel <[email protected]>, [email protected]

Subject: Re: Bug#835904: tail --retry doesn't

Date: Mon, 5 Sep 2016 10:15:47 +0100

On 05/09/16 07:07, Harald Dunkel wrote:
> coreutils 8.25-2 seems to have the same problem. Sample:
> 
> 
> % ps -ef | grep tail
> harri     3815  3783  0 Aug30 pts/3    00:00:00 tail --retry --max-unchanged-stats=5 -f /var/log/messages
> harri    11756 11708  0 08:00 pts/4    00:00:00 grep tail
> 
> % lsof -p 3815
> COMMAND  PID  USER   FD      TYPE DEVICE SIZE/OFF    NODE NAME
> tail    3815 harri  cwd       DIR    8,4    20480 4456449 /home/harri
> tail    3815 harri  rtd       DIR    8,3     4096       2 /
> tail    3815 harri  txt       REG    8,3    64232   15185 /usr/bin/tail
> tail    3815 harri  mem       REG    8,3  1697504 1054034 /lib/x86_64-linux-gnu/libc-2.23.so
> tail    3815 harri  mem       REG    8,3   153288 1054030 /lib/x86_64-linux-gnu/ld-2.23.so
> tail    3815 harri    0u      CHR  136,3      0t0       6 /dev/pts/3
> tail    3815 harri    1u      CHR  136,3      0t0       6 /dev/pts/3
> tail    3815 harri    2u      CHR  136,3      0t0       6 /dev/pts/3
> tail    3815 harri    3r      REG    8,3   159279  140486 /var/log/messages.1
> tail    3815 harri    4r  a_inode   0,11        0    4381 inotify
> 
> % ls -alrt /var/log/messages*
> -rw-r----- 1 root adm   2057 Aug 15 07:54 /var/log/messages.4.gz
> -rw-r----- 1 root adm   1762 Aug 21 08:03 /var/log/messages.3.gz
> -rw-r----- 1 root adm  29000 Aug 29 07:53 /var/log/messages.2.gz
> -rw-r----- 1 root adm 159279 Sep  4 08:00 /var/log/messages.1
> -rw-r----- 1 root adm   2881 Sep  5 07:52 /var/log/messages
> 
> % dpkg -l coreutils
> Desired=Unknown/Install/Remove/Purge/Hold
> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> ||/ Name                                Version                Architecture           Description
> +++-===================================-======================-======================-============================================================================
> ii  coreutils                           8.25-2                 amd64                  GNU core utilities

Oh the file is renamed.
You probably need `tail -F` rather than `tail -f --retry`

Message #30 received at [email protected] (full text, mbox, reply):

From: Harald Dunkel <[email protected]>

To: Pádraig Brady <[email protected]>, [email protected]

Subject: Re: Bug#835904: tail --retry doesn't

Date: Tue, 6 Sep 2016 09:19:33 +0200

On 09/05/2016 11:15 AM, Pádraig Brady wrote:
> 
> Oh the file is renamed.
> You probably need `tail -F` rather than `tail -f --retry`
> 

Please check the man page.

Message #33 received at [email protected] (full text, mbox, reply):

From: Bob Proulx <[email protected]>

To: Harald Dunkel <[email protected]>, [email protected]

Subject: Re: Bug#835904: tail --retry doesn't

Date: Tue, 6 Sep 2016 11:05:24 -0600

Harald Dunkel wrote:
> Pádraig Brady wrote:
> > > tail --retry doesn't
> > Oh the file is renamed.
> > You probably need `tail -F` rather than `tail -f --retry`
> 
> Please check the man page.

The man page says:

       -f, --follow[={name|descriptor}]
              output appended data as the file grows;

              an absent option argument means 'descriptor'

The behavior of 'tail -f /var/log/messages' is to open the file once
and use file descriptor.  Which means that if the file is renamed that
tail follows the renamed file.  When the file is renamed to messages.1
then tail -f will be following the messages.1 file.

It also includes this note specific to this situation.

       With  --follow  (-f),  tail  defaults to following the file descriptor,
       which means that even if a tail'ed file is renamed, tail will  continue
       to  track  its  end.   This  default behavior is not desirable when you
       really want to track the actual name of the file, not the file descrip‐
       tor (e.g., log rotation).  Use --follow=name in that case.  That causes
       tail to track the named file  in  a  way  that  accommodates  renaming,
       removal and creation.

When used with syslog and logrotate file rotation /var/log/syslog and
/var/log/messages are periodically rotated.  A program that opens the
file before rotation will have the open file descriptor pointing to
the rotated file after rotation.

Before rotation (pseudocode):

  tail -f /var/log/messages
    ...internally tail will have fd = open(/var/log/messages) == 3
  mv /var/log/messages /var/log/messages.1
    ...syslog is signaled to open a new /var/log/messages file
    ...the internal fd remains at 3 and pointing to the messages.1
    ...this is the tail follows the file descriptor mode

After rotation:

  ...tail is now following messages.1 file...

After rotation tail will internally still have fd == 3 but that fd
will still be pointing to the original file which has now been renamed
to /var/log/messages.1.  But syslog is not writing to the messages.1
file.  Syslog is now writing to the newly created messages file which
it was signaled to change output to.  messages.1 never gets any new
updates and tail -f never emits any new data.

       --retry
              keep trying to open a file if it is inaccessible

Using --retry is not appropriate in this situation since tail is
accessing the messages.1 file okay.  The file is accessible.  But the
renamed file is no longer being written to by syslog.

Strictly speaking the man page says that this is the appropriate
command for following the new file after a renaming of files and
creation of a new file.

  tail --follow=name /var/log/messages

However -F is shorter and the --retry makes it robust in the face of
file rotation.

       -F     same as --follow=name --retry

Hope this helps to explain what is happening.

Bob

Here are the longer details from the full tail documentation.

‘-f’
‘--follow[=HOW]’
     Loop forever trying to read more characters at the end of the file,
     presumably because the file is growing.  If more than one file is
     given, ‘tail’ prints a header whenever it gets output from a
     different file, to indicate which file that output is from.

     There are two ways to specify how you’d like to track files with
     this option, but that difference is noticeable only when a followed
     file is removed or renamed.  If you’d like to continue to track the
     end of a growing file even after it has been unlinked, use
     ‘--follow=descriptor’.  This is the default behavior, but it is not
     useful if you’re tracking a log file that may be rotated (removed
     or renamed, then reopened).  In that case, use ‘--follow=name’ to
     track the named file, perhaps by reopening it periodically to see
     if it has been removed and recreated by some other program.  Note
     that the inotify-based implementation handles this case without the
     need for any periodic reopening.

     No matter which method you use, if the tracked file is determined
     to have shrunk, ‘tail’ prints a message saying the file has been
     truncated and resumes tracking the end of the file from the
     newly-determined endpoint.

     When a file is removed, ‘tail’’s behavior depends on whether it is
     following the name or the descriptor.  When following by name, tail
     can detect that a file has been removed and gives a message to that
     effect, and if ‘--retry’ has been specified it will continue
     checking periodically to see if the file reappears.  When following
     a descriptor, tail does not detect that the file has been unlinked
     or renamed and issues no message; even though the file may no
     longer be accessible via its original name, it may still be
     growing.

     The option values ‘descriptor’ and ‘name’ may be specified only
     with the long form of the option, not with ‘-f’.

     The ‘-f’ option is ignored if no FILE operand is specified and
     standard input is a FIFO or a pipe.  Likewise, the ‘-f’ option has
     no effect for any operand specified as ‘-’, when standard input is
     a FIFO or a pipe.

     With kernel inotify support, output is triggered by file changes
     and is generally very prompt.  Otherwise, ‘tail’ sleeps for one
     second between checks— use ‘--sleep-interval=N’ to change that
     default—which can make the output appear slightly less responsive
     or bursty.  When using tail without inotify support, you can make
     it more responsive by using a sub-second sleep interval, e.g., via
     an alias like this:

          alias tail='tail -s.1'

‘-F’
     This option is the same as ‘--follow=name --retry’.  That is, tail
     will attempt to reopen a file when it is removed.  Should this
     fail, tail will keep trying until it becomes accessible again.

‘--retry’
     Indefinitely try to open the specified file.  This option is useful
     mainly when following (and otherwise issues a warning).

     When following by file descriptor (i.e., with
     ‘--follow=descriptor’), this option only affects the initial open
     of the file, as after a successful open, ‘tail’ will start
     following the file descriptor.

     When following by name (i.e., with ‘--follow=name’), ‘tail’
     infinitely retries to re-open the given files until killed.

     Without this option, when ‘tail’ encounters a file that doesn’t
     exist or is otherwise inaccessible, it reports that fact and never
     checks it again.

Message #38 received at [email protected] (full text, mbox, reply):

From: Harald Dunkel <[email protected]>

To: Bob Proulx <[email protected]>, [email protected]

Subject: Re: Bug#835904: tail --retry doesn't

Date: Tue, 13 Sep 2016 13:53:41 +0200

Hi Bob,

On 09/06/2016 07:05 PM, Bob Proulx wrote:
> 
> The man page says:
> 
>        -f, --follow[={name|descriptor}]
>               output appended data as the file grows;
> 
>               an absent option argument means 'descriptor'
> 

I see. "name" is not a file name, but a literal. My failure.
New test with -F is running.

Wouldn't you admit that this is misleading? It is pretty common
that a command line option as in

	command --flag arg

does the same as

	command --flag=arg


Thanx very much
Harri

Send a report that this bug log contains spam.