FW 2.00.01 - modules broken, skynet alerted

[patrolwomanjayne]

Well, I think I had better tell everyone what happened. Hopefully this will avoid some problems for people in the future.

Basically, we've had a Thecus here in my department for about 6 months. It has 20 users and 2.5TB of storage space. Everything works fine on it and people seem to love it. We have had many problems with NFS however, so I have been looking forward to patches and firmware updates.

I installed 2.00.03 on the system. Everything seemed fine, especially NFS. We only export 1 folder, and that's /raid/data, so we never ran into any issues with being able to export more than one folder. Most of the problems were related to random disconnections and slowness when 3 or more machines shared the NFS link.

It all looked good and nobody complained. Then, I received a phone call from security.

Turns out, someone with a perl script and way too much time on their hands had gotten into the Thecus and made it part of a botnet.

While I thought it hilarious that someone would add a NAS to a botnet (akin to making a Beowulf out of Commodore64's), it was really shocking and disturbing to hear. I looked at netstat and indeed, it was pwned with a running service on 6667 TCP. Of course, I had the SYSuser module installed, but I changed the password on it. Nobody could have accessed the server that way, right? Wrong.

I watched very closely when I changed the password on the sys user. the hash in /etc/shadow never changed. perplexed, I removed the hash, uninstalled the module, then reinstalled. Same hash. Oh crap. Default password. panicked, I tried changing ANYONE's password. I couldn't.

I guess I would back it down to 1.00.10 because at least then I could use the RSYNC module. Now I have to back stuff up using insecure FTP, which I'm sure will make everyone in network security love me so very much.

Can anyone provide detailed instructions on how to back down the firmware or at least fix this "I hate modules" attitude that my Thecus seems to have taken up lately?

So I don't know what has happened. I can install and remove modules just fine. The only problem is that the passwords can't be modified and I can't change the port SSH is listening on.

[gideon007]

Very disturbing indeed, I only wonder about this as well:

I installed 2.00.03 on the system.

The subject says 2.00.01 then you state ...03, which one is it and does it matter or are both firmwares behaving the same way?

[patrolwomanjayne]

The subject says 2.00.01 then you state ...03, which one is it and does it matter or are both firmwares behaving the same way?

Good eye. I upgraded from 1.00.10 to 2.00.01 about three days ago. That's when the mayhem started. I mistakenly put 2.00.03 in because I had a freudian slip -- i was going to install that if nobody could aid me in fixing this thing.

Thanks for pointing that out ;D

[omega]

Hi patrolwomanjayne,

I really don't understand why people are installing firmware 2.00.03 (which is a BETA firmware) and ignore all the warnings about this firmware versions. For a long time now others are reporting a lot of problems with this firmware.....
Ok, you did it by mistake, bad luck

Even though it might be too late for many of you: don't install firmware 2.00.03!!

I've checked the behaviour of the SYSUSER module together with FW 2.00.01 and what happens when changing the password.
The password entry in /etc/shadow changes everytime the password is changed so this behaviour you describe is not showing up in 2.00.01.

But what interest me most is what program is establishing the botnet? What process opens the port 6667 and how is it started?
This would be the most important information about your issue, so I hope you can tell us.....

Andreas

[patrolwomanjayne]

omega said:

Hi patrolwomanjayne,

I really don't understand why people are installing firmware 2.00.03 (which is a BETA firmware) and ignore all the warnings about this firmware versions. For a long time now others are reporting a lot of problems with this firmware.....
Ok, you did it by mistake, bad luck

Even though it might be too late for many of you: don't install firmware 2.00.03!!

I've checked the behaviour of the SYSUSER module together with FW 2.00.01 and what happens when changing the password.
The password entry in /etc/shadow changes everytime the password is changed so this behaviour you describe is not showing up in 2.00.01.

But what interest me most is what program is establishing the botnet? What process opens the port 6667 and how is it started?
This would be the most important information about your issue, so I hope you can tell us.....

Andreas

Hi Omega,

Thanks for writing all of these modules, by the way. If you didn't take the time to help us all, I'm sure Thecus would be out a lot of users. They should give you royalty checks ;D

Anyway, I have 2.00.01 on there right now. When I go to Status > about -- this is what it says:

anufacturer Thecus
Product No. N5200
Firmware Version 2.00.01
Up Time 15 hours 58 minutes

The module pages are empty now. Before I removed the SSH module for the final time, I moved the /raid/data/modules folder to /raid/data/modules.backup and removed the mentions of a sysuser account from /etc/passwd and /etc/shadow. I then rebooted the server and reinstalled the module. It still does the same thing, which is this:

When I go to the page for the module properties of sysuser, the option to change the password is there. Let's hypothetically say that I tell it to use password "foobar". I click the button, the page refreshes. I would think that it changes, but for some reason, at the very top of the page, I get this message:

"Cannot change password to "foobar" for user "sys"!"

Thinking that it could just be a weird problem with that page or the module, I logged in via SSH and issued the command "passwd sys". It prompted me to change the password and so I did it. However, while said that it was successful, I would log out, try to log back in, and the password was still 'sys'. lol wut?

The message from Network Security said that netflow traffic was coming from 6667. When I nmapped it from my core server, I saw that the port WAS open. When I checked the router's logs, there was a bunch of traffic heading to a .de domain. As a municipal office in the US, we'd have no reason for that.

Do you have any idea of what I can do to fix this? I know ultimately what would fix it -- the total wipeout. However, if I do that, users will be extremely irate. I need to figure out a way to do this that involves preserving the users and the data...even if that means temporarily opening up the unit to vulnerabilities, nsync'ing the contents to the core server, then copying the /etc/passwd and /etc/shadow.