Son of Spam

A week has passed since my last episode, for which I am profoundly sorry. Happily, young Ms. Shaw from the previous episode (I picture her as a college student with the unenviable job of combing through responses to emails that robots send out with her name attached) wrote a follow-up letter (well, a robot did, anyway) which inspired me to compose another response.

This time I actually sent it to the poor benighted young lady, to give her a little smile, a brief ray of sunshine as she toils in her corner of the sub-basement of a decaying building, her only sources of light her flickering computer screen and a feeble incandescent swinging naked from a wire, while water drips from a large pipe that runs horizontally through the middle of her “space”. The only thing that breaks up the monotony of her job are visits from her cigar-smoking, foul-mouthed ogre of a boss.

I’m pretty sure, if you read between the lines of the original message, that all that is in there. And more. But this isn’t about poor Katie, who really just needs to earn enough money to pay for her mother’s new kidney before she’s out of there for the bright lights of Hollywood. This is about me. Here’s what she will be reading when she comes in to the office tomorrow (at 6am, after the early shift at Dunkin Donuts, with just enough time to study for her Quantum Electrodynamics exam):

Dear Ms. Shaw,

Indeed I do remember your previous email. I get messages like this from time to time, but yours struck a particular chord with me. I think it was the phrase “professionally written in line with your site’s theme and voice.” An intriguing dialectic, that.

First, this thing you call “theme”. The theme of Muddled Ramblings and Half-Baked Ideas is much like Bigfoot or the Loch Ness Monster; while there may be a few crackpots who believe a theme exists, the more level-headed among us realize their ravings are just a cry for attention. We smile and nod and move on, trying not to encourage them, but we remain mildly worried what they might do if we too readily dismiss their silliness or roll our eyes once too often.

Second, your humorous use of “professional” and “my … voice” in the same sentence did indeed give me a little laugh. Trust me, Katie (may I call you Katie?) there’s nothing professional about MR&HBI. On a good day I might achieve “whimsical” or more often “snarky”, but professional is right out. The site’s been active for over ten years, is approaching a million words of content, yet “professional” remains a distant dream, my Xanadu, if you will; glimpsed in a fevered vision only to shatter on the jagged shore of reality.

My metaphors aren’t very tight, either.

Ironically, despite all that I have just said, cher Katie, you have already provided me with content for Muddled Ramblings and Half-Baked Ideas. You see, I was tickled enough by your first request that I devoted a small episode to it, including another, briefer hypothetical response that contains no references to opiate-addled Romantic poets. So I guess I owe you one.

Yours in Perpetuity,
Jerry Seeger

Note: for veracity I left in the improper semicolon.

4

Fun With Spam

I get messages like this on a regular basis, so I thought I’d share one with y’all, followed by my (unsent) response:

From: Katie Shaw
Subject: Guest editorial on muddledramblings.com

Hello,

We are interested in working with you and producing editorial content for your site, muddledramblings.com.

The content will be professionally written in line with your site’s theme and voice. If you are interested in exclusive content for your site, please let me know.

I appreciate your consideration and look forward to hearing from you.

Sincerely,
Katie Shaw
Marketing Assistant

Dear Ms. Shaw,

Thank you for your kind offer, but I’m afraid I’m a bit confused. Professionally written in line with my site’s theme and voice? That’s inherently contradictory! Even if you could identify Muddled Ramblings’ theme (I sure as hell can’t), I assure you there’s nothing remotely resembling professionalism going on here.

TOTALLY Sincerely,
Jerry Seeger
Editor in Chief, Muddled Ramblings and Half-Baked Ideas

1

An Email I Just Sent – Updated!

An email I just sent:

You guys left comment spam on my blog! Not the sort of behavior I expect from a hosting provider — probably against your own terms of service (which I suspect you don’t enforce). Ironically, since the topic of my blog episode was colocation, you could have left an honest message about your services, something like “we can beat the price of the company you endorsed.” (You can’t, though. Not even close.) Or maybe, “One thing your readers might want to consider is…” I would have welcomed that. All you had to do was be honest about who you were.

But you didn’t, so, stop it. If it happens again I shall shake my tiny fist in public, and have no negative impact on your business whatsoever because I’m just some obscure back-water blogger to whom exactly no one turns for colocation advice. But the next guy you annoy may not be.

Jerry Seeger
muddledramblings.com

So, yeah, I said I would shake my tiny fist if they did it again, and here I am tiny-fist-shaking anyway. I decided that by not giving the name of the hosting company I’m still within the spirit of the threat. (‘Threat’ used in the broadest sense of the word.)

UPDATE:

I find that when I complain to companies about their spamming practices, I get one of two responses. Either I’m ignored, or I get a request for more information. Then there’s these guys, who both took my tiny-fist-shaking seriously, and flattered me in the process. Here’s the message I got in return for my above rant, in its entirety except for contact info redacted at the request of Mr. Welbourn:

Jerry,

Thank you so much for bringing this to my attention. It was not I who “comment spammed” your blog but I suspect it was a company hired by me to provide search engine optimization services.

If possible, it would be greatly appreciated if you could forward me the comment. .

This is my first foray into the realm of SEO companies. In the past, I have been very hesitant to engage with these firms as they all claim to have some “secret sauce” and are reluctant to share their strategies and tactics. Unfortunately, while we are an extremely high quality colo provider in the Chicago area, we are somewhat of a well-kept secret amongst the Ubergeeks of the world and I really need to boost my web traffic. After some extensive research, follow up with several references and completing what I felt was substantial due-diligence, I entered into an agreement with a vendor who will remain nameless until I am able to get to the bottom of this.

Their services were described to me as a combination of on-site and off-site components. The on-site modifications were pretty simple text revisions to make the key words I am targeting more prevalent in the copy of our website. The off-site component, which is handled 100% by the vendor, consists of establishing external links to our website.

I assume this is where muddleramblings.com became involved.

If their off-site methodology includes “comment spamming” blogs then I imagine our relationship will be a short one. I spent an enormous amount of time with them explaining what we do, how we do it and how we would like to be represented. While I have not yet seen the content of the comment, since it annoyed you and prompted you to shake your fist, I am sure it must be obvious spam and the sort of thing I would not want to be associated with.

Any additional information you can provide would be helpful.

By the way, I spent some time reading some of the muddled ramblings. I like your style.

Regards,

Don Welbourn

Director of Sales / Account Relations
360 Technology Center Solutions

It’s interesting on a lot of levels. First, even people who try really hard to do the right thing have to trust the companies marketing their product. In his letter giving me permission to publish the above, Mr. Welbourn said he had clarified with his marketing company what was and was not acceptable. He also said that Munchies would make a great movie.

Pardon me while I pat myself on the back with my tiny fist — I helped a company maintain its ethics and microscopically reduced the amount of spam in the world. And hats off to Mr. Welbourn, for taking the issue so seriously. I like the way those guys conduct business. If you’re in the market for enterprise-level colocation services (and who isn’t?), maybe you should drop by 360tcs.com.

Thesaurus bot in action!

Over a stretch of a few hours, my spam blocker flagged messages with the following content:

  • I intended to send you the tiny word to finally say thank you the moment again on your awesome
  • I wanted to send you the little observation so as to thank you very much again regarding the pleasant
  • Needed to compose you a very little remark to help say thanks a lot yet again over the lovely
  • I needed to create you this bit of observation to say thanks over again with the marvelous

There were probably more, but you get the idea. There is a template sentence that might actually be grammatically correct, in which certain words are marked for replacement by thesaurus. For instance, in the above, every line has a replacement for “note”.

Two questions present themselves: What is the actual template, and (more fun) what is the most ridiculous version of the template?

My humble contribution:
“I am pathologically compelled to fire at you this wee missive to once and for all pay you the respect you are due once more for the unbelievable.”

4

More Thoughts on Spam

A recent attempt at comment spam on my blog was a message heavy with phrases designed to get a search engine riled up: Attorney Personal Injury Las Vegas, Attorney Personal Injury, Lawyer Personal Injury, Our lawyer handles all the legal matters professionally!

By putting those phrases here, not connected to the Web site of the sleazy lawyer resorting to illegal practices to promote his business, I weaken the search engine power by diluting the phrase. I think. That or I get blacklisted by the Goog.

But it seems like there should be more I can do. Here, on my blog, is a law firm breaking the law. Let me say that again, so you get the full feel of it. A group of people who are bound to upholding the law, are breaking the law right here and now.

From their Web site (careful not to actually click any link in the spam), I sent them this message:

You guys are lawyers. Yet you, or agents employed by you, are engaging in illegal spamming. Really, you guys should be smarter than that.

Fix it.

No reply, though days have passed now. There won’t be a reply. But I’m watching my spam bin with a little extra diligence right now; the next one is going to the Nevada Bar Association.

In the meantime, I got a glut of comment spam from a Forex trading site. Forex (foreign exchange) is the practice of trading currency, a high-risk practice of predicting the perceived values of global currencies, and the pool is filled with sharks ready to fleece ordinary joes who somehow get the impression that there’s quick money in those markets. The brokers brag that they have a can’t-lose system, and the unsophisticated suckers buy in, lose their money, the brokers pocket the profits, and the system worked. It really is a can’t-lose system — for the brokers.

So, when I got a heapin’ helpin’ of spam from a Forex site, I decided once more to play an activist role. I went to the site (as always, careful not to use the link in the spam directly) and it seemed to be devoted to exposing the bad guys. They’re called the Forex Peace Army, or FPA. Still, a spamming asshole is a spamming asshole. I sent them a message:

While your site makes it appear that you want to be one of the good guys, you are engaged in illegal spam activities. That is disappointing, and hypocritical. Please stop.

And they wrote back! To paraphrase (and infer just a bit): Sorry, but some jerks we pissed off have started a spam smear campaign. Any data you can give us might help us bring them down.

Alas, it looks like the jerks outsourced their libelous campaign; the spams I got came from China. Still, I’m sending them the data, in hopes that maybe somewhere along they way the FPA will catch a break and get the evidence they need. And you have to like an organization named Forex Peace Army. I picture a shark in tie-die.

2

It’s Inside the Building!

You know in that horror movie where the girl is on the phone and there’s some crazy mofo who’s freaking her out but for some reason she doesn’t hang up and eventually it turns out the crazy mofo is already inside the house and really has no reason to call? I had a moment like that tonight. I’ve had a rash of spam lately, all using my Facebook identities. I waited for my spam-catchers to get a clue, but the comments kept coming. “Fine,” thought I, “I’ll just block the addresses they’re coming from.”

I fired up my diagnostics, and found the source. localhost. My server thought the comments were coming from itself! Double-plus ungood, to quote Orwell. Extra double-plus. My spam-detecting software, it turns out, recognized the evil of the comments, but was immediately overridden by the administrator. By me, or a vile piece of software pretending to be me.

I just changed a lot of passwords. I hope I can remember them later. I also set a switch that requires that all comments be approved before they go live. Alas, this is likely more an inconvenience to legit comment traffic, as the evil robot has already proven capable of emulating me and giving permission.

I also spastically updated all my wordpress plugins (I do this fairly often anyway) — including, perhaps significantly or not, the one that passes comments between here and Facebook. Later, going back, I see nothing in that plugin’s update info to the tune of “closed egregious spam hole.” But the attack vector seems to be through my Facebook identities. It may be that the conduit trusted the origin of the messages too much.

So now I wait and watch, and your comments will take a little longer to reach the page. Hopefully I can loosen things up soon.

A Little Web Irony

A while back I posted that this blog was blocked by the Great Firewall of China. Most likely that was because some other site that shared my server had annoyed them, but I had recently blamed China for a surfeit of hyphens, so you never know.

In an interesting (to me) turnabout, I just blocked several million Chinese IP addresses from accessing my site, due to a Chinese deluge of spam. Even spam that’s blocked by my filters costs me server performance and bandwidth, so when things get bad I just prevent spam sources from reaching my server at all (thanks, CloutFlare!). Lately that’s been China.

I’m now erecting a wall to keep China out, when once I joked about them keeping me out.

1

At Last, the Recognition I’ve Sought all these Years

Tonight I was named Top Rambler of the Day by not once, but twice! Wow!

Yep, Top Rambler. Second to none. There are many who aspire to these heights, but out of the millions of blogs out there that do little more than ramble, none compares to this one. Bow down before me, those who would ramble, and learn from the master! I AM TRoD!

For some reason my spam software blocked both notifications of my major awards (from two different places), hiding them from the eyes of the general public — along with a comment that said, “Why’s presently there this kind of fine publish!”

Why’s indeed?

1

Ubiquity Solutions: Evil or merely Overwhelmed?

Note: Wow. This got long, and somewhat technical. For today, some of you might want to look at cute pictures of cats instead. I won’t mind.

I noticed the other day a huge rush of spam comments from ip addresses starting 108.62. I did a lookup and found that the whole block is owned by an outfit called Nobis Technology Group. Most of the addresses also mentioned Ubiquity Server Solutions. They are a massive hosting and colocation service. Basically, they supply the hardware and infrastructure, and their customers set up Web servers and whatnot.

Some of those customers (or the customers of the customers) send out a lot of spam. A truckload. In some cases the customer of a customer of a customer might have been lax and his server got hacked and turned into an unwitting spambot. In other cases the people using Ubiquity’s servers are likely institutional spammers.

Brief aside: Why does comment spam even exist in the first place? Google plays a big role there, with a number called Page Rank. Part of Page Rank (at least historically) was that more links pointing to a page make it land higher in Google searches. So, the spam comment isn’t to get readers of a blog to buy Doc Marten shoes, it’s to get that particular site to land higher in Google’s results when someone searches for them.

The thing is, Google doesn’t publish page rank numbers anymore, and they steadfastly maintain that the comment spamming actually hurts your results in a search. That hasn’t stopped many companies from promising higher sales and taking people’s money in return for smearing their name all over the Internet.

Google could go a long way toward eliminating this sort of spam by publishing page rank again, only now include the amount the rank was hurt by spamming activities. My shoe salesman above is not going to keep paying when Google shows the opposite of the desired result.

So anyway, using CloudFlare’s threat control, I blocked an entire range of ip addresses allocated to Ubiquity’s servers. Then another. I didn’t like this solution; I had no idea how many legitimate potential blog visitors I was blocking. After reading more, the answer surprised me.

The folks at Ubiquity point out that they have terms of service that prohibit using their infrastructure to spam people. When I sent them a complaint, they were professional and courteous. They asked for more specifics, then said they’d sent a complaint to the culprit. Only after they’d asked what my domain name was.

Question: Did they send a message to the culprit saying ‘stop spamming people’ or did it say ‘stop spamming that guy?’

On other blogs where people have ranted about Ubiquity, representatives of the company have responded with measured, rational responses, explaining what a huge uphill battle it is for them, and asking the community to keep sending reports when spam comes from their range. Those reports make it possible for them to put sanctions on clients who are in violation of their terms of service. It is a huge problem and not easily solved.

And yet. Other hosting companies don’t seem as bad, from where I’m sitting.

One of those responses from a Ubiquity representative threw out the argument (I’m paraphrasing from this) “While it’s theoretically possible to monitor all data to weed out the 500MB/s of spam from the 2GB/s of legitimate traffic, that would be really expensive and we wouldn’t be able to compete in this market.” My first takeaway: they think 20% of the traffic from their servers is unethical. Wow. Now, that’s reading a lot into a statement like that, so take it with a grain of salt. Also, it was in a comment to a blog post and may well have been a typo in the first place.

But still, it makes me wonder. And a request coming in to a server for data (legitimate traffic like a request to load a Web page) is fundamentally different than robots on a server sending unrequested data OUT (a high percentage of which will be spam), and sending emails (almost all of which will be spam). A small random sampling of GET and PUT messages outbound from their data centers would probably smoke out the most egregious violators pretty quickly, and not require a lot of hardware to implement. (Not sure how I feel about this from a privacy standpoint.)

Once I got the message that Ubiquity had sent their complaint to the spammer involved, I unblocked that range. Sure enough, in a few minutes more spam came through. I sent the report and back up went the blockade. In my casting around the Internet I read assertions that were not contradicted (so must be true!) that said that NO legitimate traffic would come from those IP’s anyway; they were the addresses of big servers and not IP’s that would appear when Joe User is surfing. So there’s no downside to blocking them. (I’ll put the blocked ranges in a comment below, if you want to follow suit.)

Although, as I put the blockade back up, I had a thought: If I complain about every violation, and cc Google, then the cost of NOT clamping down more effectively on the host’s clients goes up. At some point, if enough people complain enough times, the cost of fixing the problem at the source becomes less than the cost of continuing to do business they way they are now.

That goes not just for Ubiquity, but for all hosts, and for Google and the other search engines. There is no incentive for them to play nice unless we create one.

Yep, I’m proposing fighting spam with a deluge of emails, and I’m probably too lazy to do it effectively.

Of course, this blog is hosted at a data center that almost inevitably will have spammers. Do I want to pay more for my own hosting because my data center has to install a bunch of spam detectors? In my case, I’d be willing to pay a bit more to know my host is doing the right thing, but I think I’d be in the minority. That makes it really difficult for one host to unilaterally decide to take the high road. And you’d be alienating about 20% of your customers, if Ubiquity’s off-the-cuff numbers are an indication.

2

Mmm… Honey

I just installed a honey pot on this site. The idea of a honey pot (or honey trap) is to create a tempting target that attracts wrongdoers, but once they put their hand in the honey pot they leave sticky fingerprints everywhere they go.

In Internet terms, the honey is a seemingly-innocent email address placed on a Web site, invisible to humans but easy for robots to find. When the spam harvesters scrape the email address off the site and use it, both the harvester and the spammer are caught and blacklisted, which reduces their ability to run robots and get their mail through.

The more people who participate, the more trouble spammers have spotting the honey pots. How can you help? Even if you don’t have control of your site or run a blog through one of the major services, you can pitch in. Go to Project Honey Pot and sign up. You can provide invisible-to-humans links to honey pots on other sites, if nothing else, and it doesn’t cost you diddley-doo.

If you click on the “swag” link in the header, you will see that they could also use a graphic designer. I imagine a spam-bear with his head stuck in a honey pot. How you communicate that it’s a spam-bear and not an ordinary bear I leave as an exercise for the visually talented.

Once Project Honey Pot compiles its list of villains and ne’er-do-wells, what happens next? Many major services use the list, and I also use a program called Bad Behavior which blocks blacklisted bots and spammers from reaching my site. Recently I added another layer called CloudFlare which is awesome enough for me to devote a separate episode to it. So, you have that to look forward to.

In the meantime, I encourage you to join the crusade to make life more difficult for those who want to use the Internet for evil.

1

Trying a Different Spam Filter

Every day, literally hundreds of spam comments are sent to this blog. I have a a couple lines of defense, and generally they work pretty well. My first defense is a product called Bad Behavior, which inspects incoming messages and blocks the ones that look malicious before the WordPress code is even started up. Stopping evil at this stage can save a lot of server resources, as well as prevent this site from being hijacked by an unknown WordPress vulnerability.

Comments that get through that layer are then inspected to see if they look suspicious. Ones that the inspection service doesn’t like get thrown into a bucket behind the scenes where I can inspect them and approve innocent comments that were mistakenly flagged as spam.

I have been using Akismet for that, and in general I’ve been pleased with the results. The only downside is that now there are so many suspicious comments that I’m afraid that I’ll miss actual legit comments that were improperly flagged. Scanning through a list of hundreds of comments each day is not effective and, really, not a good use of my time. So, I began to look for alternatives.

Defensio is similar to Akismet, in that comments are shipped off to some service somewhere and then returned with a grade. The main difference is in the administration interface that I see, where Defensio sorts the rejected spam comments to allow me to more quickly spot legitimate comments that were falsely flagged as spam.

You may have noticed a surge in the amount of spam around here. This is (I hope) a learning phase for Defensio, and eventually it will stop allowing 3% of the spam comments to get through. (Akismet is still running, but mostly in a “see? I told you so” capacity right now.) I’m a little confused, because some of the comments Defensio displays are rated at 100% spamminess by Defensio’s own service.

Please bear with me through this somewhat-more-spammy-than-usual phase. I’ll be checking for spam comments regularly, and watching to see if Defensio’s performance improves. Also, this is a particularly good time to leave comments, from a training-the-filter perspective.