Login Troubles

[Mhaddy]

My RH7 web server box was running perfectly last night - went to bed, woke up and all the problems started to arise. First, without looking at the box I notice I can't login through FTP so I immediately goto take a look at it and notice big time 'weird' errors on screen in the form of:

235908203598 a092309802398 asdfl30293 asd303929u9 l32092340 lkajdd code 000004 etc.

I got some of it on tape (however clear that may be) and I'm going to get it back on the PC for you guys to take a look at (if you think it's worth it).

Anyhow, so I boot the machine (CTRL+ALT+DEL) and it says there isn't enough run processes left to run this command (or something similar... disappeared from the screen), so I hard booted it. While it was going through the Linux checks everything checked out until it got closer to the end bunch where things were going too fast for me to make sense of any of it but there were more of those 2390582093502 029509as.3009328 errors flashing on the screen.

Then it brings me to the login prompt - try to login as root, nogo... after I enter my pass and hit enter it just flashes something really fast on the screen and goes back to the prompt - try to login as other users, same thing.

Any idea what happened? I, and one other trustworthy friend are the only ones that had access to the box through FTP and the server was hosting one domain, which only a handful of people knew about. Oh - and a bunch of people on the dslreports.com forum knew about it while I was having trouble with DNS.

*sigh* Please help me out... I really don't want to start from scratch :(

[bdg1983]

Do you have a bootdisk you can use to access your system and try to find out what has changed or if someone has cracked your system?

Sounds bad to me. Hope you can somehow get it fixed without having to reinstall.

At least try to backup as much of your data as you can if you end up reinstalling.

[posterboy]

A mdwatts said, sounds bad. This is a pretty classic "I been hacked" scenario. You may have limited time in which to get your data, too. DO that as quick as possible, and then try to spend some time with the logging mechanisms. They usually reveal nothing, as the black-hats are pretty thorough in removing their own traces (this is illegal, after all) but, sometimes, you can find tell-tails of when it happened, like a restart of klogd or syslogd thats unusual.
It's a mistake to try to "fix" this, a re-install is the only viable answer. Really sorry, man, I know how it feels, based on experience. Good Luck ! Ray

[Mhaddy]

Hi there, thanks for your responses (however unfortunate they may be). All this time the box has been shutdown the RJ45 cable unplugged. When I boot back up, I'm assuming I have to do this with a boot disk (which, luckily I have) - what then?

I have all of my data that I was FTP'ing to it on my Win2K PC so I'm not worried about that. The only thing I really want to back up would be my httpd.conf, proftpd.conf and, most importantly my mySQL databases. I can copy my httpd.conf and proftpd.conf files to a floppy but how do I go about backing up my databases?

Man, who would want to do this :(

[Dark Ninja]

I'm not sure about backing up your databases, however, I must make one suggestion. Be careful about which files you carry over to your new system. A classic black-hat trick is to edit certain "important" files to allow access back into the system. (Much like a trojan horse, but, not quite.) However, when you unknowingly carry the old files to your new system...you'll once again open the door for people everywhere.


Dark Ninja

[Mhaddy]

I would be backing up the files simply for their information, not to replace them back on my system.

If I can't save the system, then I will have to format and start from fresh (might be the best idea actually). If I do so, I'm also going to put on RH7.1 instead of RH7.0. Not going to bother with bind either... just going to rely on dyndns.org services and take it from there since DNS never really worked on my system either (long story).

[Mhaddy]

Well, put in the boot disk, and at the prompt typed in 'linux -1' so it booted up normally...

Then it went through the start up steps and I selected the interactive set up and here are a few things I jotted down as I was going through the steps:

finding module dependencies: depmod: not an ELF file
setting the filetype for entry: 'fs7100' in /tmp/.font-unix(55) to 6
setting the filetype for entry: 'log' in /dev (24097) to 6
setting the filetype for entry: 'gpmctl' in /dev (24097) to 6
deleted inode 28148 has zero dtime. FIXED.
setting the filetype for entry: 'mysql.sock' in /lib/mysql (32151) to 6
setting the filetype for entry: 'pump.sock' in /run (38153) to 6
rmmod: module cleaner is not loaded
Start Service sshd: starting sshd: /etc/sshd_config: no such file or directory. error: fatal: could not load host key: /etc/ssh_host_key check path and permissions [FAILED]

Then after all of this it comes to:

Start service mysql: [Y]
Start service xfs [Y] Starting mysql daemon with databases from /var/lib/mysql

And it just hangs there. However, because httpd was running and now this... I tried to goto phpMyAdmin through the web and I was allowed in! So I backed up databases! :) That's one hurtle overcome... oh - these databases are okay to use again, right?

*Going to keep experimenting with different things here... going to see what I can save or get back up... I do not that if I didn't go step by step and when I'm back at the prompt and try to goto the web server through the browser PHP and mySQL are broken.

[Mhaddy]

Umm... all right, this is really weird. After that hang there at the mysql part (see above post), I hit CTRL+ALT+DEL and it rebooted normally (one error message in there that I didn't catch) and then it started up with the boot disk and so I typed in 'Linux -2' (don't know what that does, but I tried -1 before, heh) and guess what? It went through all the steps, no errors... and now I can login...

Going to try backing up a few things here, then will reboot again...

[Mhaddy]

And I think I have come down to the conclusion that I was hacked. While browsing through the /home dir I noticed two users that were not there before - 'danutz' and 'cap'. 'danutz' fits right in with the hackers arrogant nature that I expected to find sooner or later...

So I guess that settles it, eh? Backup some files (not replacing ANYTHING, but for referrence) and then format and start over. Also... isn't syslog supposed to be in /var/log? If it is, I can't find it...

[Mhaddy]

AH HAH! After searching in /var/log/messages I found these lines (amongst others):
Aug 8 07:18:39 mdsnexus adduser[2787]: new group: name=cap, gid=505
Aug 8 07:18:39 mdsnexus adduser[2787]: new user: name=cap, uid=505, gid=505, home=/home/cap, shell=/bin/bash
Aug 8 07:18:56 mdsnexus adduser[2795]: new group: name=danutz, gid=506
Aug 8 07:18:56 mdsnexus adduser[2795]: new user: name=danutz, uid=506, gid=506, home=/home/danutz, shell=/bin/bash
Aug 8 07:19:44 mdsnexus PAM_unix[2798]: (system-auth) session opened for user danutz by (uid=0)
Aug 8 07:19:44 mdsnexus -- danutz[2798]: LOGIN ON pts/0 BY danutz FROM 217.156.72.228 ARGH!

217.156.72.228: Can someone trace / lookup this IP for me?

[ 11 August 2001: Message edited by: Mhaddy ]

[posterboy]

That's kind of a silly hack. A decent hack would have eliminated ANY logging of their activity. Anyway, here we go........

inetnum: 217.156.72.224 - 217.156.72.239
netname: AXEE-STILL-MODERN
descr: SC AXEE STILL MODERN SRL
descr: PITESTI STR.PASAJUL
descr: EGALITATII NR.2 JUD.ARGES
country: ro
admin-c: SC489-RIPE
tech-c: SC489-RIPE
status: ASSIGNED PA
mnt-by: AS3233-MNT
notify: domain-admin@listserv.rnc.ro
changed: cristih@rnc.ro 20010726
source: RIPE

person: SERB CORNEL
address: REPUBLICII BL.E3a,SC.F,AP.22
phone: +40-95-829879
fax-no: +40-48-251150
e-mail: dino@digitallinux.org
nic-hdl: SC489-RIPE
notify: domain-admin@listserv.rnc.ro
mnt-by: AS3233-MNT
changed: cristih@rnc.ro 20010726

That's just a piece of it, you can get the rest from RIPE, if you need it. DO bear in mind that this may well be a spoofed IP, or, another compromised box, and the folks this leads to may indeed be innocent victims like yourself. I have not found email very effective in these matters, rather, registered certified reciept U. S. mail tends to do better, though I don't know about Romania. Include the log snips you found and what time zone you are in, so they can match things, like GMT -5 or whatever it is for you. Good Luck, Ray

[ 11 August 2001: Message edited by: posterboy ]

[Mhaddy]

Thanks, posterboy. Did he happen to delete my syslog? Isn't it supposed to be in /var/log?

[posterboy]

I believe that in RH7, which I don't run, (there's a warning) klogd and syslogd are both by default directed into /var/log/messages. It's user configurable, but I think they ship it set up like that. This is the case on RH6.2 which I do know about. Ray