remote login to smtp and ftp takes forever

[singlespeed]

I've got 2 RH8 servers running qmail as a mail relay

they're both behind a firewall and have an additional IPTABLES firewall running on them.

The mail servers are qmail, the ftp is wu-ftp (hardened, not anom login)

both are running RAV antivirus for mail servers.

server1 has qmail and ftp
server2 has just qmail

this morning some wierd stuff started happening, smtp on server2 is taking forever to acknowledge requests and ftp on server1 is doing the same thing.

Also if I ssh to either maching i get a login prompt right away but then a long delay before a password prompt. This has not historically been the case.

if I telnet to port 25 on server 2, it takes forever. If I ftp to server1, connection is right away but prompt for login takes forever.

This all just started happening this morning. Nothing strange in the logs.

Any Ideas?

Server 1

[root@mail /]# ps -ax
PID TTY STAT TIME COMMAND
1 ? S 0:06 init
2 ? SW 0:00 [keventd]
3 ? SW 0:00 [kapmd]
4 ? SWN 0:00 [ksoftirqd_CPU0]
5 ? SW 0:00 [kswapd]
6 ? SW 0:00 [bdflush]
7 ? SW 0:00 [kupdated]
8 ? SW 0:00 [mdrecoveryd]
12 ? SW 0:00 [kjournald]
88 ? SW 0:00 [khubd]
212 ? SW 0:00 [kjournald]
674 ? S 0:00 syslogd -m 0
679 ? S 0:00 klogd -2
791 ? S 0:00 /usr/sbin/apmd -p 10 -w 5 -W -P /etc/sysconfig/apm-scripts/apmscript
812 ? S 0:00 /usr/sbin/sshd
845 ? S 0:00 xinetd -stayalive -reuse -pidfile /var/run/xinetd.pid
865 ? S 0:00 ravmd: supervise RAV scanning process...
867 ? S 0:01 ravmd: accepting connections ...
900 ? S 0:00 gpm -t ps/2 -m /dev/mouse
918 ? S 0:00 crond
954 ? S 0:00 [atd]
976 ? S 0:00 rhnsd --interval 120
980 tty1 S 0:00 /sbin/mingetty tty1
981 tty2 S 0:00 /sbin/mingetty tty2
982 tty3 S 0:00 /sbin/mingetty tty3
983 tty4 S 0:00 /sbin/mingetty tty4
984 tty5 S 0:00 /sbin/mingetty tty5
985 tty6 S 0:00 /sbin/mingetty tty6
986 ? S 0:00 /bin/sh /command/svscanboot
990 ? S 0:00 svscan /service
991 ? S 0:00 readproctitle service errors: .................................................. ...............
993 ? S 0:00 supervise log
995 ? S 0:00 supervise log
996 ? S 0:00 /usr/local/bin/multilog t /var/log/qmail
997 ? S 0:00 /usr/local/bin/multilog t /var/log/qmail/smtpd
2349 ? S 0:00 /usr/sbin/sshd
2377 pts/2 S 0:00 -bash
3146 ? S 0:00 supervise qmail-smtpd
3303 ? S 0:00 supervise qmail-send
3694 ? S 0:00 [qmail-send]
3697 ? S 0:00 qmail-lspawn ./Mailbox
3698 ? S 0:00 [qmail-rspawn]
3699 ? S 0:00 [qmail-clean]
3703 ? S 0:00 [tcpserver]
4596 ? SN 0:00 in.ftpd -l -a
4614 pts/2 R 0:00 ps -ax

[Icarus]

Probally DNS is not working correctly.

What is the DNS server?
Check the /etc/resolv.conf and verify the addrees is correct. Do the machine do DNS caching? If so, also check the /etc/nsswitch.conf and make sure it uses files before dns
(check this anyway even if it doesn't do caching, it should always check file first)

[singlespeed]

DNS is what I first thought but NOTHING has changes on our side. just for giggles I changed the DNS server on the boxes in resolv.conf and there was no change in the behavior.

It's almost as if some other process is running when a password is expected. Could these boxes have been hacked with some password grabbing app?

resolv.conf is fine, nothings changed...

nsswitch.conf:

passwd: files nisplus
shadow: files nisplus
group: files nisplus

#hosts: db files nisplus nis dns
hosts: files nisplus dns

# Example - obey only what nisplus tells us...
#services: nisplus [NOTFOUND=return] files
#networks: nisplus [NOTFOUND=return] files
#protocols: nisplus [NOTFOUND=return] files
#rpc: nisplus [NOTFOUND=return] files
#ethers: nisplus [NOTFOUND=return] files
#netmasks: nisplus [NOTFOUND=return] files

bootparams: nisplus [NOTFOUND=return] files

ethers: files
netmasks: files
networks: files
protocols: files nisplus
rpc: files
services: files nisplus

netgroup: files nisplus

publickey: nisplus

automount: files nisplus
aliases: files nisplus

[singlespeed]

also, when I do a "netstat -a" on either machine, it's pretty slow returning a response, it halts about half way down on both.

example:

[root@mail root]# netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:ftp *:* LISTEN
tcp 0 0 *:ssh *:* LISTEN
tcp 0 0 *:smtp *:* LISTEN

****paused here*****

tcp 0 0 mail.telaurus.net:ssh 10.16.1.7:2582 ESTABLISHED
tcp 0 0 mail.telaurus.net:smtp irresistable.cnch:40897 TIME_WAIT
tcp 0 0 mail.telaurus.net:smtp excellent.cnchost:44447 TIME_WAIT
tcp 0 0 mail.telaurus.net:smtp zealous.concentri:52714 ESTABLISHED
tcp 0 0 mail.telaurus.net:smtp excellent.cnchost:44447 TIME_WAIT
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 1470 /dev/gpmctl
unix 10 [ ] DGRAM 854 /dev/log
unix 2 [ ACC ] STREAM LISTENING 1445 /usr/local/rav8/private/_ravmdcom
unix 2 [ ] DGRAM 2198
unix 2 [ ] DGRAM 1540
unix 2 [ ] DGRAM 1485
unix 2 [ ] DGRAM 1286
unix 2 [ ] DGRAM 1285
unix 2 [ ] DGRAM 1024
unix 2 [ ] DGRAM 968
unix 2 [ ] DGRAM 866
unix 2 [ ] STREAM CONNECTED 464

[JohnT]

Can you try turning off your anti-virus app for test?

[singlespeed]

Ok, it may in fact have something to do with dns...

if I add a machine to the hosts file of mail server1 and then ssh to mail server1 from that machine, I get an instant password prompt. If I remove the host record from mailserver1 and ssh again... long delay for password prompt..

smtp and ftp however still are slow even with the host record in the /etc/hosts file..

[singlespeed]

It was my IPTABLES scripts on both servers...

The primarydns and secondary dns settings were pointing to old dns servers that were not congruent with the resolv.conf file.

once I changed my IPTABLES scritp settings, everything fell in line.

The think is, they have been like that for a while but according to my up2date logs, iptables were upgraded in march as part of a kernel dependancy. This may have caused the problem to arise.