Monday morning my colleagues have found the mail server broken. When I've arrived at the office they've had already restored the situation by restarting the server from the console. Something's broken the sockets of the networking of the OS. Strange... very strange for a Linux kernel.
I've started my inspections reading the logs, obviously. Reading and reading, getting back line after line, I've found the time when the sockets got broken: I was horrified reading that someone, at the same time, two minutes before, logged in via ssh with the zimbra user (a program's user, that normally don't have any password, in order to disable remote ssh logins ).
Deep inspection of access logs reported that someone has used this user to login remotely, for a month or two. That was very bad. The IP used for the login changed every time, but the class was the same: all the IP's was pulled from the pool of the Romtelecom company, a rumanian company.
That was a very stupid security flaw: someone of us putted a too simple password for the zimbra user during a test and after the tests he forgotten to remove it.
The result was that a simple scanner was able to discover the too short password.
The next question was: what the hell does the hacker with my mail server?
I've found the answer in the /var/tmp directory: the only one opened for writing to all users. Our bad guy used this folder as a place to store, download and compile various scanner/knocking softwares, that was done in order to use our email server as a base for scanning very large set of ip's classes.
Our bad guy doesn't have any other interest, so, in order to hide his presence, he hadn't touched anything else: in this way he could perform his tasks for days, weeks... until he made the bad step that broken the sockets.
He was ingenuous too, and he doesn't removed the history, he never cleared anything, so now I can reconstruct everything: what was installed, what was his activities, what sites he used to download software: everything. I don't have interests to continue my inspections, but If I would... especially helped with a claim presented to the italian's police... I could easily follow him, and maybe one or two of the groups behind that.
I will report some of the software he downloaded, useful if someone want to study a little:
If you want to know something more, such as user and passwords lists, classes of scanned ip,
or if you want the full programs, in order to study them: feel free to ask me.
Oh, I've also wrote to the romanian telecommunication company. Here's their reply:
Thanks for Contacting Romtelecom's Abuse Team . Our customer will be notified .
Very pleased to see all this interest to stop eastern's cracking groups.