Spam Control to Major Paul

Hi ho, your friendly spam flagging hobbyist here, annoying you once again.

I think it would be a good idea to implement a captcha in the registration process. The majority of spam I'm seeing is a first post from a newly registered user that is obviously a bot. When the bots can register user accounts and you have to flag their posts and ban them one by one, you're fighting a losing battle.

I like reCAPTCHA, because at least the modicum of effort that goes into typing the two words is actually used somewhere else--to help digitize books.

Permies, would it have kept you from registering to use this site if to do so you had to type two words that were shown to you in a distorted image (or if you prefer played over your computer speakers)? How much of a burden do you think it would be if you had to do this every time you posted? or just the first time you posted after you registered? Paul, what's feasible?

Well,

I think we're not getting to many spammers.  And I'm not sure if they are bots. 

The new forum software will support captcha.

Huh. I feel like I'm flagging around two a day, or as often as I show up to the site. And the pattern looks like bot attacks to me. Your site, your call, though.

Didn't know there was new software in the works.

We used to get a hundred a day.  So two a day is pretty good!

Hi. I've been lurking here for a short while. Paul has an interesting website. I'm an administrator on a small cabin owner-builder forum. Anti-Spam has become a sort of a hobby with me, an obsession maybe. 

I'm curious; when you find spammers here do you report them to websites that track forum spammers, such as http://www.stopforumspam.com/ ?

In approximately three years they have amassed a database of over 750,000 spammers. User ID, email and IP info. I find it useful when trying to decide if recent signups are spammers or not, before they actually have a chance to spam.

I would be interested in exchanging info with Paul and/or the mods here, if there is any interest.

I didn't know there was such a thing. Do they provide a subscription to the IP address so the IPs of known spammers can be blocked from registering? That would be a useful service! Er, not that they won't just go get another IP address, of course...

MountainDon!  Thanks for stopping by!

And thanks for the link. 

I like the idea of checking with their database before allowing signups.  In fact, I think it would be really great to figure something out where I can filter out my existing database using their database.

As for signups - I like the idea that if somebody comes here, they can get through registration in under a minute and then post. 

Thanks for the welcome. 

StopForumSpam itself is simply a database anyone can access. Using an email, ID or IP one can check to see if the person has been reported by others as a forum spammer. Sometimes I've found new arrivals at CountryPlans to have 500 reports.    Many times they can be banned before they come back to post spam.

Their database is also used by some third party "mods" that can be added to the SMF software. The addon mod checks IP's, emails and ID's being used by new signups to cut the spammer off before their registration is completed. It's a very big hammer. There are at least a couple versions available.

Another database of spammers is Project Honeypot. They collect data on all types of spammers; email through forum spammers, including the actions of email harvesters. A webmaster can add scripts to their pages that automate the reporting of suspicious activity from harvesters and the like. That can be handy as an insurance of sorts... If a mail harvester takes the bait address it produces a record of the exact time and date of the harvesting and notification of suspicious activity is sent. They have an IP lookup available for use as well.

There is also a spammer blocking addon available that uses the Honeypot database. That makes two very large hammers. We don't use either automated systems at CountryPlans, although there is a certain appeal to being able to stop a spammer before they can do anything. We have built a good database of our own, based on the folks who spam us. Using that for our filering/blocking seems qyite effective for the most part.

For the past year or so we have been more targets of the signature line spammer than the active comment spammer. I have a simple solution that is quite effective in blocking their attempts to add signature spam, yet let contributing members add to their signature. It does not stop registration, just premature signature activity. Some figure it out, but surprisingly, most do not. They go away and do leave inactive signups which is a nuisance, but they can be dealt with in bulk at a later time. If you want info on what I do, please ask.

Soitenly!  What's the trick?

The signature spammer operates on the premise that if they get their spam URL in their profile signature block the search engine bots will record it and that will increase the ranking of the website they are pointing to. If a real human views and clicks on the link that is a bonus.

When contemplating the surge in signature spam last year, since I didn't seem to be able to stop the spammers, I wondered if I could undermine their purpose. Realizing the primary point of signature spam was to be seen by the search bots I decided to make it invisible to the bots. The default settings of the SMF software allows guests to view the profiles of all members. Search engine bots are in the guest class. I set the guest permissions to make all member profiles invisible to guests. That was only satisfying to my own sense of right and wrong; knowing that the spammers efforts were for naught was soothing to my OCD. However, the spammers were still at work, doing their thing and I assumed, feeling like they were getting their job done.


Then I had another idea. I'm not sure how many membership levels you use here; the SMF software default setup has 5 beginning with “newbie”. I began with inserting a new membership class between “newbie” and the next level up. I named it “apprentice”. I set the permissions for this new group to be a duplicate of the “newbie” class to begin with. Then I adjusted the “newbie” permissions to allow them to view their own member profile, but to disallow editing of the profile. Thay can see it, they just can't do anything to it, like add a signature spam. All other permissions remain mostly the same; they can post, start topics and so on.

Next I set the required number of posts to advance automatically from the “newbie” class to the “apprentice” class to 1. Once they make a single post they automatically advance up the hierarchy and can access their own profile. If they are a signature spammer, the first thing they want to do is add their spam to their signature. The 'no edit' permission prevents them from doing so. Most often they depart at that point and leave us alone, for a while at least. 

This still lets any new member make a new post or reply to a topic. Once they have done that one thing they have full normal access to their profile. Most new members will never run into any issues with forum use. Recently a would be spammer actually posted a question as to why they could not access their profile. Once they did that they discovered they could edit their profile, added a spam signature and proceeded to make 5 comment spams. Within a couple hours they were caught and everything cleaned up.

I can tell by the large number of new members that lay idle and spam free, and are matches to the spammers recorded in StopForumSpam's database, that the majority are stopped from their purpose of adding signature spam by this simple trick. It dose not get rid of them and they do clutter up the membership roll, but at least I have a small personal sense of satisfaction and don't feel the urgency to get that stuff deleted as soon as possible.


I also have a special membergroup titled “limited” where I toss those whom appear 'gray'. This membergroup has much the same permissions as the “newbie”. Mainly they have to prove themselves to be real life useful forum contributors before being returned to the mainline membership list. I don't use it too often.


Some small scale research of mine has shown that the people out there running SEO outfits sincerely believe that they are doing no wrong in hiring low wage comment and signature spammers to plant there links wherever they can. I've also found that some folks who employ SEO's are unaware of just what underhanded techniques many of the SEO's use. Some actually have told me they are distressed by the low level of ethics. I don't know what if any, good has come of it though.

That's some of the 'trickery' I've developed. I've probably rambled on enough about that for now.

Smart!

I have been reading up on SEO a lot in the last few weeks.  What a complicated space.  I can see what you are talking about with folks hiring low wage folks to spam forums.  I suspect that most of my spam traffic is human and not bots.  But I am also concerned that some of my new users are stumbling over the spam control stuff. 

Things are so complicated now. 

Sometimes I think about making things tougher, but right now I delete a few spams a day and figure I am okay with doing that in exchange for the signup process being easier for newbies.

I would really like to write a little app that will read in all of the spammer data and then see if any of those are in my database.

And maybe I should rig my stuff up so that every time I detect a spammer I automatically tell the spammer database about it!

I believe most of the spammers that hit CPF are humans, although there is one with an Israeli IP that I suspect is a bot. Either a bot or a very determined individual. There's an array of IP's with just the last 3 digits of the dot address changing. Since last November we've recorded over 400 hits from the addresses in 212.235.107.* The first few contained bona fide spam links. StopForumSpam has hundreds of reports from the same range. So I have a wildcard ban on that group.

Do you run the suspects email address through Google? That frequently will bring up a list of that email address showing at a list of other forums. The forums on that list are ones that allow guests to read the member profiles. I look at the referenced profiles and if they show things in the signature block that could be considered spam I take that into account.

This AM we had two that turned up in a half dozen other forums. I figure an IP based in India, with signature links to a Toronto Canada airport limo service placed on a forum in Germany that's all about bittorrent software might be up to no good. Especially when you can find the email address on several other diverse forums with registrations all occurring the same night. The other was planting cell phone service links on a variety of forums. As well both came up on StopForumSpam.

I do believe there is a mod written for the SMF software that will read your database. I haven't bothered with investigating. If I recall where I might have seen it I'll let you know. The SMF forum has loads of mods for their software. Things for blocking spammers using database info, as well as mods that will turn the first X number of links posted by a member into a non clickable link. After X is passed things go back to normal.



I find  http://en.utrace.de  a handy IP lookup. It shows the IP location on a map graphic, plus will provide all the info about the owner.

Over the past few months I selected a handful of companies that had their spam appearing in comment spam on CPF. The comment spam was the usual nonsensical type, not quite on the topic and often poorly written. Yet the products were either useful or interesting to someone hanging out on a small cabin forum. However I could also find the same links on a wider variety of other forums.

I decided to contact the website that was being promoted. Example: a week ago we got several inane comments from the same new member, all with a link to an Australian architect website. Lots of pictures showing off completed homes in AU. Very expensive upscale homes for the most part. They were very nice. The IP was Indian. So I contacted the guy named Peter. I told him I loved his designs and use of open space. I asked him if he had recently employed a SEO firm. I sent screen shots of the web pages displaying the spam before I deleted them. I briefly explained how this sort of thing was viewed by many forum operators. I also sent him links to two other forums with similar posts, one a 'green cars' site and the other on a Spanish language web developer forum.

He replied in a very friendly manner and responded that yes, he had started up with an SERO about 6 months before. He said he'd run my comments by his SEO consultant. After a day or so he wrote back and included the text of his email exchanges with his SEO firm. They tried to run a smoke screen by him, stating they were following Google's best practices, they were good guys, not bad guys. They also did admit to using a sub contractor in India. They said they would instruct the Indian firm to exclude our forum from future activity.

I also told Peter how ineffectual this sort of thing can be. When forums that care about what they present to their members and guests delete such posts it is money ill spent. In the end I liked his designs so much I posted a little blurb for him and linked to him. So he wins on that and if the IN spammer stops spamming us I guess we win too. A lot of bother though.

This was the fifth such experiment this year that I've done. I came away from each with a small glimmer of hope as the companies being promoted were all unaware of some of the machinations that go on behind the guise of SEO.

Then again I'm probably fooling myself.

Have a good one.

Wow!  Don!  That is an awesome story! 

I would guess that the architecture firm is still employing that SEO company and the indian firm is still getting money to spam forums?

I really like the SMF stuff for being able to pan an IP range for a certain amount of time.  I think a lot of spammers are probably on one IP for several months.  So if I just ban that one IP for two months, I probably won't hear from them for two months.

paul wheaton wrote:

I would guess that the architecture firm is still employing that SEO company and the indian firm is still getting money to spam forums?

No doubt everything is running the same as before. The only difference might be that that particular SEO won't send their spammer to the CPF, if the person there is true to their word.

paul wheaton wrote:
  I think a lot of spammers are probably on one IP for several months.  So if I just ban that one IP for two months, I probably won't hear from them for two months.

Right, the SMF folks have done a good job on writing the software. They also have a sense of humor, a somewhat different sense of humor at times. For the 2.0 RC version someone wrote an "annoy user" mod.
From the mod page http://custom.simplemachines.org/mods/index.php?mod=2232 

Quote: "This mod allows the admin to pick trouble users and apply what amounts to a passive-aggressive method to encourage them to leave, by applying various 'problems' to them, for example a percentage chance of them seeing 'this has been disabled' error messages."




Re Spammers......
I am somewhat surprised at the number of spammers that actually use the same IP for 6 months or even over a year. On the other hand there are those others who change at will. Some are extremely hard to track or block as their user ID's and emails are all over the place too. At least I have seen the same IP with 100 or more spam reports and 50+ ID's that appear to be randomly generated, like  uzwahxmbacdz  who registered today. Email from Poland and IP from Spain.

That is excellent!

I've been getting a lot of spam from russia and china.

More activity today, it seems like. I don't know why I get so ticked off about these asshats.

Spammers are to forums as mosquitoes are to camping. More concentrated at times than others, but always around somewhere.
 

I think the "report to moderator" thing is a huge help.

I've been sick the last couple of days and trying to do some of my work.  Getting an email that says "click here to see spam" makes things way easier on me.

I wish the software had something so that it could do all the stuff I do in two clicks:

1) show me a screen with three types of bans all pre-set (email, IP and server domain).  And it shows the country of origin for the spam.  95% of the time it will all be just right.  For 5% of the time I can change things a little.

2)  Click on "ok" and then the specified bans are done, the account is deleted and the post is deleted. 

As is, it's something like a 15 step process for each spam.

Yes it would be nice if the ban user screen could be adjusted to our own selected defaults. That would be very handy. SMF is always working on new versions and tweaks.  I'll make a suggestion in the SMF forum unless you want to claim authorship. Your call, Paul.

It's all yours!

Sorry that you've been sick, Paul—but it's good to hear my compulsive flagging is actually helpful and not just getting old. Hope you feel better soon.

I filed a request.

Hope you are feeling better or at least on the road to recovery.

Still trying to shake this funky cold.  Yuck.

I do feel a little more productive/alive today. 

As for the new version - I sent you email about this earlier today, but I might as well mention it here.

I've tweaked this software many, many times.  I probably have over a hundred hours of tinkering in on this that would not carry over to a newer version.  I kinda hate to ditch that. 

Although the new stuff does have a lot of nifty looking bits and bobs!

I sent you an email; you can go have a look under the hood now.

Our SMF tinkering has been minimal. A change to V2 for us offers more to be gained potentially for us than not.

I do like the ability to build in as much up front anti spam as one chooses during the registration. On the SMF forum, running under version 2 for over a year I think, they use the captcha set fairly difficult. They do not use any of the 3rd party add ons that are available through them. They claim to have been spam free since switching their live forum to version 2.  THey use the captcha on every post until after the member counter hits 10, I think.

I recently tried to register for another forum and it said that don't allow spammers to register.  So apparently they somehow think I am a spammer. 

I think there are a lot of people that are very tech savvy and forum savvy .... and for every one of those there are a dozen that are not. 

I think they way you have that demo set up is quite good - only I would take out the part where it looks like I would have to fill out a captcha for every post (only later it turns out to be only for the first post).

spammer?  Have you tried running your IP through one of those spam tools? They are not perfect especially when an IP or whatever only shows one or two hits.

Yes, I'm not sure about that captcha on message posting. Mainly I'm just trying out the new features. I have seen that used on a few forums.