Dan Mahoney (gushi) wrote,
Dan Mahoney

Tracking down compromised email accounts

Mitigating a Mail Server Attack


A few days ago, I noticed an unusual number of bounceback messages from one specific user directed at yahoo.com email addresses. When I looked inside the mail queue, I noticed that each message had dozens of recipients.

The troubling thing is, looking at the mail logs, I saw the dreaded line

maillog.7.bz2:Aug 18 21:33:08 <mail.info> prime sm-mta[80096]: 
AUTH=server, relay=[], authid=vivian@littlestdomain.com, 
mech=PLAIN, bits=0

The AUTH=server bit tells me that rather than a rogue script running here (a not-uncommon thing that happens when you let users run PHP scripts), this was an actual password that got leaked, and was being used to send mail just as a regular user would.

I quickly concluded "compromised account", changed the user's password, and contacted them out-of-band with a new password. Life seemed good.

...then I noticed a second account doing the same thing. Okay, that's weird. Maybe my users have started falling for a really effective phishing scam?

When it happened a third time, a few days later, I recalled the old Ian Fleming quote:

"Once is happenstance. Twice is coincidence. Three times, it's enemy action."

Combat Perl

So, faced with the task of seeing if there were any other users who were affected, I wrote a short little bit of perl code to analyze my mail logs, and spit out each login, as well as the number of times each IP had logged in.

The code looks like this. No, there's no fancy "use Strict" or anything like that. I used YAML as an output format because writing "Dump" is easier than writing a foreach loop to iterate over the hashes.


use YAML;
open FOO, "/usr/bin/bzgrep -i \"auth=server\" /var/log/maillog.0.bz2|";
my @lines = <FOO>;
my %thing;
foreach my $line (@lines) {
  chomp $line;
  if ($line =~ /\[(\d+\.\d+\.\d+\.\d+)\].*authid=(\S+),/) {
        print "ip address $1 authid $2 found in $line\n";

print Dump %thing;

Combat perl output

The code above produced output like what I have here. Note that I've altered all the logins and none of the below actually exist on my system. The IP addresses and counts, however, are real.

--- jim
--- 1 4
--- bob
--- 1
--- moe
--- 10
--- curly@curly.net
--- 9 10 3 1 2 3 2 6 1 4 9 2 7 4 2 3 3 4 5 1 2 4 6 4 2 5 2 1 10 3 8 1 4 1 4 3 7 7 7 5 6 2 3 4 3 2 7 13 4 4 1 2 8 5 12 3 8 7 8 2 1 3 3 2 4 6 1 4 5 5 4 4 3 3 5 4 7 1 7 1 8
--- thing
--- 1
--- stuff
--- 2
--- steve
--- 1 1
--- joe
--- 1

So, one of these things is not like the others. It's understandable that a person may have two or three ips in a given period. Their ips change, they're logging on from multiple computers.

Remember that these are ONLY the ip addresses pulled from the sendmail logs. Only on connections where a piece of mail is sent, using SMTP auth.

So, that curly@curly.net entry? Yeah, that is that what security researchers call a "snowshoe" attack -- not one server sending hundreds of mails (which would be easy to block), but instead, it's spread out, and even though I now have a list of ips I could block, what we're looking at here is a botnet of otherwise compromised machines on a dozen or more ISPs.

The other thing to note about the curly@curly.net entry is that it's a full user@domain entry. Put another way, it's an email address -- one where the LHS (left hand side) just so happens to match the user's actual login.

What was going on here is that the way I (and most people) do SMTP auth in sendmai, there can be a concept of multiple "realms" defined -- for example, to log in against different authentication databases. As I'm not using this feature, the realm and everything after is ignored (but still logged).

As I normally instruct my users to only use the barename to log in, any login using a full realm must be a compromised account.

Notifying the User

So, there's a problem here. While I can easily change the user's password and send them mail, this effectively locks them out of their account and keeps them from getting anything done until we touch base.

What I wanted to do was find a way to block the users who were using the "bad" format, while letting good users go on. I wanted a quick, guilt-free way to block the sending of mail, without breaking the communication link.

What I discovered was a ten year old post in the old usenet group comp.mail.sendmail, here.

With a little bit of tweaking, I had applied that same config to my own sendmail, and had configured a line in the access database to block a test user. The account still worked, but it wouldn't let them send mail. Perfect. I could block "curly@curly.com" without blocking "curly". (And yes, this relies on a little bit of obscurity -- but it's a botnet, not monkeys at typewriters, it's only going to try what it knows).

Identifying the source

So, three accounts with relatively secure passwords compromised in a week. What was the common thread? Could these people have all used the same insecure passwordless wifi networks? Is there some newfangled router exploit that mails your traffic all off to the highest bidder?

I spoke to all the users. None of them had fallen for any phishing emails. They were running different OSes, so a password-stealing virus was out. And then it hit me. Like a ton of bricks.

I've recently seen a surge of spam to addresses like macromedia-at-gushi.org, adobe-at-gushi.org, adobe2012-at-gushi.org.
The reason?

Well, it's all because adobe sucks at securing your data.
Sometime last year, people were able to download 150 million usernames and passwords from adobe's backend servers. And, as the article I just linked will tell you, those passwords were encrypted weakly, and in a way that all the users had the same encrypted password string, for a given password.

While I'm not 100 percent sure this was the attack vector -- there's been several other leaks that happened (last.fm, Linkedin, E-harmony), and about 90 percent sure that this is the likely cause, even if I don't know which site it was that ultimately spilled my users' beans.

Tags: freebsd, security, sysadmin
  • Post a new comment


    Comments allowed for friends only

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded