Subject: amavisd-new using sophos-av via savi hangs after virus sig updates

From: "M. Techter" <[email-masked]>
Date: Wed, 07 May 2014 20:45:07 +0200

Package: amavisd-new
Version: 1:2.7.1-2
Severity: important

Hi,

since Mon Apr 28 2014 we have a _new_ kind of problem with amavis
using sophos-av (sophos anti virus) via the so called savi interface
(from ./SAVI-Perl-0.30.tar.gz).

We got a mail server on Debian Squeeze with some Lenny components on
hold to keep amavis, savi, and sophos-av in its original Lenny
setup. savd --the control, logging and email alerting daemon from
sophos-- running in the background takes care of updating virus
signatures.

This server has been in business for years, virtually untouched for
the last months, since 28. April 2014 we encountered the following so
far unknown problem:

After updates of the signatures amavis processes taking care
of the next content scan log this:

        Apr 30 01:17:19 mgw amavis[11740]: (11740-06) Requesting process rundown due to stale Sophos virus data

afterwards a look in the process table shows:

        ps -f  ax  | grep 'amavi[s]'
        =>
        amavis   11738     1  0 Apr30 ?        Ss     0:00 /usr/sbin/amavisd-new (master)
        amavis   11739 11738  0 Apr30 ?        S      0:01 /usr/sbin/amavisd-new (ch6-finish)
        amavis   11740 11738  0 Apr30 ?        S      0:01 /usr/sbin/amavisd-new (ch6-finish)

before it was:

        amavis   11738     1  0 Apr30 ?        Ss     0:00 /usr/sbin/amavisd-new (master)
        amavis   11739 11738  0 Apr30 ?        S      0:01 /usr/sbin/amavisd-new (ch6-avail)
        amavis   11740 11738  0 Apr30 ?        S      0:01 /usr/sbin/amavisd-new (ch6-avail)

initially it was:

        amavis   11738     1  0 Apr30 ?        Ss     0:00 /usr/sbin/amavisd-new (master)
        amavis   11739 11738  0 Apr30 ?        S      0:01 /usr/sbin/amavisd-new (virgin child)
        amavis   11740 11738  0 Apr30 ?        S      0:01 /usr/sbin/amavisd-new (virgin child)

after some time the mailq commands informs about a none responsive content scanner service.

E.g.

    mgw # mailq
    =>
    -Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
    6D58C64        1522 We 30 Apr 1 01:11:07  [email-masked]
    (conversation with 127.0.0.1[127.0.0.1] timed out while receiving the initial server greeting)
                                               [email-masked]

strace gives the following information

        strace -p 11738
        =>
        ...
        kill(11739, SIG_0)                      = 0
        kill(11740, SIG_0)                      = 0
        select(0, NULL, NULL, NULL, {10, 0})    = 0 (Timeout)
        time(NULL)                              = 1398954023
        select(0, NULL, NULL, NULL, {10, 0})    = 0 (Timeout)
        time(NULL)                              = 1398954033
        select(0, NULL, NULL, NULL, {10, 0})    = 0 (Timeout)
        time(NULL)                              = 1398954043
        select(0, NULL, NULL, NULL, {10, 0})    = 0 (Timeout)
        time(NULL)                              = 1398954053
        kill(11739, SIG_0)                      = 0
        kill(11740, SIG_0)                      = 0
        select(0, NULL, NULL, NULL, {10, 0})    = 0 (Timeout)
        ...
        ...

        strace -p 11739
        =>
        Process 11739 attached - interrupt to quit
        futex(0xb509f24, FUTEX_WAIT_PRIVATE, 1, NULL

        strace -p 11740
        =>
        Process 11740 attached - interrupt to quit
        futex(0xb509f24, FUTEX_WAIT_PRIVATE, 1, NULL

To get the mail gateway going again I have to restart amavis.  The
session leader is killed, a new session leader is started, and two new
virgin childs are created, but the old processes in the -finish state
keep hanging around:

        ps -f  ax  | grep 'amavi[s]'
        =>
        amavis   11739 11738  0 Apr30 ?        S      0:01 /usr/sbin/amavisd-new (ch6-finish)
        amavis   11740 11738  0 Apr30 ?        S      0:01 /usr/sbin/amavisd-new (ch6-finish)
        amavis   12733     1  0 14:48 ?        Ss     0:00 /usr/sbin/amavisd-new (master)
        amavis   12734 12733  0 14:48 ?        S      0:01 /usr/sbin/amavisd-new (virgin child)
        amavis   12735 12733  0 14:48 ?        S      0:01 /usr/sbin/amavisd-new (virgin child)

One possibility to get things going again is to kill the amavis
processes in the -finish state, and to call a

        postfix flush

the system then works normally until the virus signatures are updated.
And again amavis hangs as described above.

The problem was motivation to speed up a migration to a new wheezy
setup with a complete new installation on far more powerful machine.
Savi and sophos-av where newly installed too.

We got the same problem on the new wheezy host now.

regards
max.

PS
        we inform the sophos support about the issue with a pointer to this bug report

-- System Information:
Debian Release: 7.5
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: i386 (i686)

Kernel: Linux 3.2.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/bash

Versions of packages amavisd-new depends on:
ii  adduser                             3.113+nmu3
ii  debconf [debconf-2.0]               1.5.49
ii  file                                5.11-2+deb7u3
ii  libarchive-zip-perl                 1.30-6
ii  libberkeleydb-perl                  0.51-1
ii  libconvert-tnef-perl                0.17-11
ii  libconvert-uulib-perl               1:1.4~dfsg-1+b1
pn  libdigest-md5-perl                  <none>
ii  libio-stringy-perl                  2.110-5
ii  libmail-dkim-perl                   0.39-1
ii  libmailtools-perl                   2.09-1
pn  libmime-base64-perl                 <none>
ii  libmime-tools-perl                  5.503-1
ii  libnet-server-perl                  2.006-1+deb7u1
ii  libunix-syslog-perl                 1.1-2+b2
ii  pax                                 1:20120606-2
ii  perl [libtime-hires-perl]           5.14.2-21+deb7u1
ii  perl-modules [libarchive-tar-perl]  5.14.2-21+deb7u1

Versions of packages amavisd-new recommends:
ii  altermime              0.3.10-7
pn  libnet-patricial-perl  <none>
ii  ripole                 0.2.0+20081101.0215-1

Versions of packages amavisd-new suggests:
pn  apt-listchanges      <none>
pn  arj                  <none>
pn  cabextract           <none>
pn  clamav               <none>
pn  clamav-daemon        <none>
ii  cpio                 2.11+dfsg-0.1
pn  dspam                <none>
pn  lha                  <none>
pn  lhasa                <none>
pn  libauthen-sasl-perl  <none>
pn  libdbi-perl          <none>
ii  libmail-dkim-perl    0.39-1
pn  libnet-ldap-perl     <none>
pn  libsnmp-perl         <none>
pn  lzop                 <none>
pn  nomarch              <none>
pn  p7zip                <none>
pn  rpm                  <none>
pn  spamassassin         <none>
pn  unrar                <none>
pn  unrar-free           <none>
pn  zoo                  <none>

-- Configuration Files:
/etc/amavis/conf.d/05-node_id changed:
use strict;
chomp($myhostname = `hostname --fqdn`);
1;  # ensure a defined return

/etc/amavis/conf.d/15-av_scanners changed:
use strict;
@av_scanners = (
['Sophos SAVI', \&ask_daemon, ['{}','savi-perl:'] ],
  ### http://www.kaspersky.com/  (kav4mailservers)
  ['KasperskyLab AVP - aveclient',
    ['/usr/local/kav/bin/aveclient','/usr/local/share/kav/bin/aveclient',
     '/opt/kav/5.5/kav4mailservers/bin/aveclient','aveclient'],
    '-p /var/run/aveserver -s {}/*',
    [0,3,6,8], qr/\b(INFECTED|SUSPICION|SUSPICIOUS)\b/m,
    qr/(?:INFECTED|WARNING|SUSPICION|SUSPICIOUS) (.+)/m,
  ],
  # NOTE: one may prefer [0],[2,3,4,5], depending on how suspicious,
  # currupted or protected archives are to be handled
  ### http://www.kaspersky.com/
  ['KasperskyLab AntiViral Toolkit Pro (AVP)', ['avp'],
    '-* -P -B -Y -O- {}', [0,3,6,8], [2,4],    # any use for -A -K   ?
    qr/infected: (.+)/m,
    sub {chdir('/opt/AVP') or die "Can't chdir to AVP: $!"},
    sub {chdir($TEMPBASE) or die "Can't chdir back to $TEMPBASE $!"},
  ],
  ### The kavdaemon and AVPDaemonClient have been removed from Kasperky
  ### products and replaced by aveserver and aveclient
  ['KasperskyLab AVPDaemonClient',
    [ '/opt/AVP/kavdaemon',       'kavdaemon',
      '/opt/AVP/AvpDaemonClient', 'AvpDaemonClient',
      '/opt/AVP/AvpTeamDream',    'AvpTeamDream',
      '/opt/AVP/avpdc', 'avpdc' ],
    "-f=$TEMPBASE {}", [0,8], [3,4,5,6], qr/infected: ([^\r\n]+)/m ],
    # change the startup-script in /etc/init.d/kavd to:
    #   DPARMS="-* -Y -dl -f=/var/amavis /var/amavis"
    #   (or perhaps:   DPARMS="-I0 -Y -* /var/amavis" )
    # adjusting /var/amavis above to match your $TEMPBASE.
    # The '-f=/var/amavis' is needed if not running it as root, so it
    # can find, read, and write its pid file, etc., see 'man kavdaemon'.
    # defUnix.prf: there must be an entry "*/var/amavis" (or whatever
    #   directory $TEMPBASE specifies) in the 'Names=' section.
    # cd /opt/AVP/DaemonClients; configure; cd Sample; make
    # cp AvpDaemonClient /opt/AVP/
    # su - vscan -c "${PREFIX}/kavdaemon ${DPARMS}"
  ### http://www.centralcommand.com/
  ['CentralCommand Vexira (new) vascan',
    ['vascan','/usr/lib/Vexira/vascan'],
    "-a s --timeout=60 --temp=$TEMPBASE -y $QUARANTINEDIR ".
    "--log=/var/log/vascan.log {}",
    [0,3], [1,2,5],
    qr/(?x)^\s* (?:virus|iworm|macro|mutant|sequence|trojan)\ found:\ ( [^\]\s']+ )\ \.\.\.\ /m ],
    # Adjust the path of the binary and the virus database as needed.
    # 'vascan' does not allow to have the temp directory to be the same as
    # the quarantine directory, and the quarantine option can not be disabled.
    # If $QUARANTINEDIR is not used, then another directory must be specified
    # to appease 'vascan'. Move status 3 to the second list if password
    # protected files are to be considered infected.
  ### http://www.avira.com/
  ### Avira AntiVir (formerly H+BEDV) or (old) CentralCommand Vexira Antivirus
  ['Avira AntiVir', ['antivir','vexira'],
    '--allfiles -noboot -nombr -rs -s -z {}', [0], qr/ALERT:|VIRUS:/m,
    qr/(?x)^\s* (?: ALERT: \s* (?: \[ | [^']* ' ) |
         (?i) VIRUS:\ .*?\ virus\ '?) ( [^\]\s']+ )/m ],
    # NOTE: if you only have a demo version, remove -z and add 214, as in:
    #  '--allfiles -noboot -nombr -rs -s {}', [0,214], qr/ALERT:|VIRUS:/,
  ### http://www.commandsoftware.com/
  ['Command AntiVirus for Linux', 'csav',
    '-all -archive -packed {}', [50], [51,52,53],
    qr/Infection: (.+)/m ],
  ### http://www.symantec.com/
  ['Symantec CarrierScan via Symantec CommandLineScanner',
    'cscmdline', '-a scan -i 1 -v -s 127.0.0.1:7777 {}',
    qr/^Files Infected:\s+0$/m, qr/^Infected\b/m,
    qr/^(?:Info|Virus Name):\s+(.+)/m ],
  ### http://www.symantec.com/
  ['Symantec AntiVirus Scan Engine',
    'savsecls', '-server 127.0.0.1:7777 -mode scanrepair -details -verbose {}',
    [0], qr/^Infected\b/m,
    qr/^(?:Info|Virus Name):\s+(.+)/m ],
    # NOTE: check options and patterns to see which entry better applies
  ### http://www.f-secure.com/products/anti-virus/  version 5.52
   ['F-Secure Antivirus for Linux servers',
    ['/opt/f-secure/fsav/bin/fsav', 'fsav'],
    '--virus-action1=report --archive=yes --auto=yes '.
    '--dumb=yes --list=no --mime=yes {}', [0], [3,4,6,8],
    qr/(?:infection|Infected|Suspected|Riskware): (.+)/m ],
    # NOTE: internal archive handling may be switched off by '--archive=no'
    #   to prevent fsav from exiting with status 9 on broken archives
  ['CAI InoculateIT', 'inocucmd',  # retired product
    '-sec -nex {}', [0], [100],
    qr/was infected by virus (.+)/m ],
  # see: http://www.flatmtn.com/computer/Linux-Antivirus_CAI.html
  ### http://www3.ca.com/Solutions/Product.asp?ID=156  (ex InoculateIT)
  ['CAI eTrust Antivirus', 'etrust-wrapper',
    '-arc -nex -spm h {}', [0], [101],
    qr/is infected by virus: (.+)/m ],
    # NOTE: requires suid wrapper around inocmd32; consider flag: -mod reviewer
    # see http://marc.theaimsgroup.com/?l=amavis-user&m=109229779912783
  ### http://mks.com.pl/english.html
  ['MkS_Vir for Linux (beta)', ['mks32','mks'],
    '-s {}/*', [0], [1,2],
    qr/--[ \t]*(.+)/m ],
  ### http://mks.com.pl/english.html
  ['MkS_Vir daemon', 'mksscan',
    '-s -q {}', [0], [1..7],
    qr/^... (\S+)/m ],
  ### http://www.eset.com/, version 3.0
  ['ESET Software ESETS Command Line Interface',
    ['/usr/bin/esets_cli', 'esets_cli'],
    '--subdir {}', [0], [2,3],
    qr/:\s*action="(?!accepted)[^"]*"\n.*:\s*virus="([^"]*)"/m ],
  ## http://www.nod32.com/,  NOD32LFS version 2.5 and above
  ['ESET NOD32 for Linux File servers',
    ['/opt/eset/nod32/sbin/nod32','nod32'],
    '--files -z --mail --sfx --rtp --adware --unsafe --pattern --heur '.
    '-w -a --action=1 -b {}',
    [0], [1,10], qr/^object=.*, virus="(.*?)",/m ],
  ### http://www.norman.com/products_nvc.shtml
  ['Norman Virus Control v5 / Linux', 'nvcc',
    '-c -l:0 -s -u -temp:$TEMPBASE {}', [0,10,11], [1,2,14],
    qr/(?i).* virus in .* -> \'(.+)\'/m ],
  ### http://www.pandasoftware.com/
  ['Panda CommandLineSecure 9 for Linux',
    ['/opt/pavcl/usr/bin/pavcl','pavcl'],
    '-auto -aex -heu -cmp -nbr -nor -nos -eng -nob {}',
    qr/Number of files infected[ .]*: 0+(?!\d)/m,
    qr/Number of files infected[ .]*: 0*[1-9]/m,
    qr/Found virus :\s*(\S+)/m ],
  # NOTE: for efficiency, start the Panda in resident mode with 'pavcl -tsr'
  # before starting amavisd - the bases are then loaded only once at startup.
  # To reload bases in a signature update script:
  #   /opt/pavcl/usr/bin/pavcl -tsr -ulr; /opt/pavcl/usr/bin/pavcl -tsr
  # Please review other options of pavcl, for example:
  #  -nomalw, -nojoke, -nodial, -nohackt, -nospyw, -nocookies
  ### http://www.nai.com/
  ['NAI McAfee AntiVirus (uvscan)', 'uvscan',
    '--secure -rv --mime --summary --noboot - {}', [0], [13],
    qr/(?x) Found (?:
        \ the\ (.+)\ (?:virus|trojan)  |
        \ (?:virus|trojan)\ or\ variant\ ([^ ]+)  |
        :\ (.+)\ NOT\ a\ virus)/m,
  # sub {$ENV{LD_PRELOAD}='/lib/libc.so.6'},
  # sub {delete $ENV{LD_PRELOAD}},
  ],
  # NOTE1: with RH9: force the dynamic linker to look at /lib/libc.so.6 before
  # anything else by setting environment variable LD_PRELOAD=/lib/libc.so.6
  # and then clear it when finished to avoid confusing anything else.
  # NOTE2: to treat encrypted files as viruses replace the [13] with:
  #  qr/^\s{5,}(Found|is password-protected|.*(virus|trojan))/
  ### http://www.virusbuster.hu/en/
  ['VirusBuster', ['vbuster', 'vbengcl'],
    "{} -ss -i '*' -log=$MYHOME/vbuster.log", [0], [1],
    qr/: '(.*)' - Virus/m ],
  # VirusBuster Ltd. does not support the daemon version for the workstation
  # engine (vbuster-eng-1.12-linux-i386-libc6.tgz) any longer. The names of
  # binaries, some parameters AND return codes have changed (from 3 to 1).
  # See also the new Vexira entry 'vascan' which is possibly related.
  ### http://www.cyber.com/
  ['CyberSoft VFind', 'vfind',
    '--vexit {}/*', [0], [23], qr/##==>>>> VIRUS ID: CVDL (.+)/m,
  # sub {$ENV{VSTK_HOME}='/usr/lib/vstk'},
  ],
  ### http://www.avast.com/
  ['avast! Antivirus', ['/usr/bin/avastcmd','avastcmd'],
    '-a -i -n -t=A {}', [0], [1], qr/\binfected by:\s+([^ \t\n\[\]]+)/m ],
  ### http://www.ikarus-software.com/
  ['Ikarus AntiVirus for Linux', 'ikarus',
    '{}', [0], [40], qr/Signature (.+) found/m ],
  ### http://www.bitdefender.com/
  ['BitDefender', 'bdscan',  # new version
    '--action=ignore --no-list {}', qr/^Infected files *:0+(?!\d)/m,
    qr/^(?:Infected files|Identified viruses|Suspect files) *:0*[1-9]/m,
    qr/(?:suspected|infected): (.*)(?:\033|$)/m ],
  ### http://www.bitdefender.com/
  ['BitDefender', 'bdc',  # old version
    '--arc --mail {}', qr/^Infected files *:0+(?!\d)/m,
    qr/^(?:Infected files|Identified viruses|Suspect files) *:0*[1-9]/m,
    qr/(?:suspected|infected): (.*)(?:\033|$)/m ],
  # consider also: --all --nowarn --alev=15 --flev=15.  The --all argument may
  # not apply to your version of bdc, check documentation and see 'bdc --help'
  ### ArcaVir for Linux and Unix http://www.arcabit.pl/
  ['ArcaVir for Linux', ['arcacmd','arcacmd.static'],
    '-v 1 -summary 0 -s {}', [0], [1,2],
    qr/(?:VIR|WIR):[ \t]*(.+)/m ],
);
@av_scanners_backup = (
  ### http://www.clamav.net/   - backs up clamd or Mail::ClamAV
  ['ClamAV-clamscan', 'clamscan',
    "--stdout --no-summary -r --tempdir=$TEMPBASE {}",
    [0], qr/:.*\sFOUND$/m, qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ],
  ### http://www.f-prot.com/   - backs up F-Prot Daemon, V6
  ['F-PROT Antivirus for UNIX', ['fpscan'],
    '--report --mount --adware {}',  # consider: --applications -s 4 -u 3 -z 10
    [0,8,64],  [1,2,3, 4+1,4+2,4+3, 8+1,8+2,8+3, 12+1,12+2,12+3],
    qr/^\[Found\s+[^\]]*\]\s+<([^ \t(>]*)/m ],
  ### http://www.f-prot.com/   - backs up F-Prot Daemon (old)
  ['FRISK F-Prot Antivirus', ['f-prot','f-prot.sh'],
    '-dumb -archive -packed {}', [0,8], [3,6],   # or: [0], [3,6,8],
    qr/(?:Infection:|security risk named) (.+)|\s+contains\s+(.+)$/m ],
  ### http://www.trendmicro.com/   - backs up Trophie
  ['Trend Micro FileScanner', ['/etc/iscan/vscan','vscan'],
    '-za -a {}', [0], qr/Found virus/m, qr/Found virus (.+) in/m ],
  ### http://www.sald.com/, http://drweb.imshop.de/   - backs up DrWebD
  ['drweb - DrWeb Antivirus',  # security LHA hole in Dr.Web 4.33 and earlier
    ['/usr/local/drweb/drweb', '/opt/drweb/drweb', 'drweb'],
    '-path={} -al -go -ot -cn -upn -ok-',
    [0,32], [1,9,33], qr' infected (?:with|by)(?: virus)? (.*)$'m ],
   ### http://www.kaspersky.com/
   ['Kaspersky Antivirus v5.5',
     ['/opt/kaspersky/kav4fs/bin/kav4fs-kavscanner',
      '/opt/kav/5.5/kav4unix/bin/kavscanner',
      '/opt/kav/5.5/kav4mailservers/bin/kavscanner', 'kavscanner'],
     '-i0 -xn -xp -mn -R -ePASBME {}/*', [0,10,15], [5,20,21,25],
     qr/(?:INFECTED|WARNING|SUSPICION|SUSPICIOUS) (.*)/m,
   ],
);
1;  # ensure a defined return

/etc/amavis/conf.d/15-content_filter_mode changed:
use strict;
@bypass_virus_checks_maps = (
   \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);
1;  # ensure a defined return

-- debconf information:
  amavisd-new/outdated_config_style_warning: