ISP Shut Me Down--Someone Hacked my Mail Server!

[pushback]

Hello! I am running red Hat 7.3 with sendmail. I called my ISP (Speakeasy) this morning becuase I could not connect. Turns out thy shut me down. Rightfully so! I was told they got a lot of complaints from spam and phisching recipients and traced it to my IP. I checked my mail queue--18,000 emails waiting to go out!

So ... they turned me back on when I promised to shutdown sendmail until I sort it out.

Any email experts out there that can tell me how to determine the intrusion method and block it? To give you an idea of the volume, just check out this week's mail log size (only three days old) with the previous weeks:

-rwxr-xr-x 1 root root 30426653 Jun 2 16:55 maillog*
-rwxr-xr-x 1 root root 2699489 May 29 04:01 maillog.1*
-rwxr-xr-x 1 root root 3047093 May 22 04:01 maillog.2*
-rwxr-xr-x 1 root root 3210344 May 15 04:01 maillog.3*
-rwxr-xr-x 1 root root 3268453 May 8 04:01 maillog.4*

Thanks in advance for your help!

John

[je_fro]

I already have a lifetime supply of VIAGRA, SO STOP IT ALREADY!!!

In all seriousness, your box was compromised so a total wipe and reinstall is in order.
Try a recent distro with up to date packages...I'm sure whatever version of sendmail you had was insecure.
Try postfix next time.

[pushback]

NOT what I wanted to hear--but thanks.


Arrrrgh.

[Gertrude]

Was sendmail the only listening service on that box or were there other publicly available services running as well? Did you keep the system patched? What version of Sendmail are you using? Is it operating as a open relay? You may not have been hacked, sendmail may just be configured to allow anyone to relay mail off your server with no authentication, or firewalls rules in place to prevent it.

[pushback]

No, there are a few other listeners, like POP, ftp but those ports are filtered so only my LAN can get to them. I did configure it so it is not a public relay (domain specific) but that was a couple of years ago--I don't even know which files to check to see that configuration now.

Can you tell me which files to check? Thanks.

J

[teyon]

sounds like your mail server was being used as a relay server. if that enabled in the options then ppl were just sendingmail thru your server, if its off then its possible that you've been rooted and you'd have to wipe. http://www.sendmail.org/antispam has the settings you should be using to disable relaying

[Seven up]

If you do have to reload give this a try.

http://www.qmailrocks.org/

Its been working for me so far.

[pushback]

Well, the post-mortum for those who are interested.

It's a funny feeling you get when you log in as root and see something like

Last login from 64-12-54-23-blahblah.ro at 11:23:14

That's what I saw today when I logged in.

so, I ran 'last'

Sure enough, someone had been logging into my server with a username 'admins'. Upon checking the /etc/passwd file, the last entry was 'admins'. UID=0, GID=0.

Then came the email from my ISP, which was forwarded from their abuse mailbox--it was from ebay--a cease and desist letter telling the ISP to shut me down. Turns out not only was my server being used to send 18000 spams, but those spams were being used to direct the recipients to an ebay phishing page--guess who's server was hosting the page ...

Upon looking in /home/admins I found the .bash_history file which led me to all his/her files. Interesting stuff, to say the least. The machine is currently shutdown (I went to best buy and bought a linksys router to replace the routing services that used to be provided by the server--that's how I'm connecting now--nice that the ISP (Speakeasy--Highly recommended) believed me when I said I didn't do it! But I've been with them for 3-4 years)

If anyone is interested in seeing these evil Romanian's wares I will post them after I rebuild the machine. I'll gladly turn over all the files to ebay and/or my ISP as well.

This guy/gal was quick. S/he logged in from two different IPs (apparently did not care about covering his/her tracks as all log files seem to be intact) Only 1/2 dozen logins, most 1-2 minutes. The longest was 7 minutes. S/he just executed a bunch of wgets to grab his/her files from the server in Romania, untarred them and ran them--the whole set
up was scripted. Also found was the text file with thousands of email addresses for the spamming part.

I must say, I am appreciative that the gal/gal was not out to corrupt my system maliciously (other than doing the intended phishing/spamming deed), there are a lot of data/mail/websites/media content on the machine that s/he could have deleted.

The one thing I have not been able to figure out is how s/he got in the first time to create the admins username. That would be nice to know.

Thanks to those who had suggestions. You were right--the machine's been compromised--wipe it.

Fedora is downloading right now.

[Gertrude]

There is no way anyone is going to be able to tell you how it happened based on the info you have provided so far. Having access to the machine to be able to look through old logs, find out what versions the applications on the box were running, checking for user/kernel root kits would be very helpful (I'm not asking for that, just saying there are a lot of things that need to be looked at, and relaying that info over a forum is not the most efficient way).

It could have just been one of the automated brute force ssh bots that are in the wild(you did use a secure(non dictionary) type password for root didn't you?). It could have been from a vulnerable version of sendmail where the cracker was able to get a user account then use that as a base to exploit some other local vulnerability on the system to then escalate to root, and create that other account.

In the future you may look into setting a host based IDS like AIDE. Also keep everything updated especially any publicly services. Use TCP Wrappers (/etc/hosts.allow/deny) and firewall of your choice (IPChains/Tables ) to only allow access to your box from trusted hosts if you can. For example don't leave sshd open to "evil Romanian's". If you connect from work or another ISP, add just their subnets or specific IPs if you can.

[Gertrude]

edit double post

[bwkaz]

Yeah, that does sound like it's close to the stuff that was downloaded when the people that did the SSH brute-force attacks got into a honeypot server:

http://www.security.org.sg/gtec/honeynet/viewdiary.php?diary=20041102

It's not exactly the same (this honeypot capture found that the attacker was downloading IRC backdoors), but that was 7 months ago, so maybe they've changed tactics since then. They did wget a bunch of stuff from some other site and then install it (although I supposed that may be more "standard post-breakin procedure", I don't really know).