Internet being clogged by bots.

[A Nonny Moose 8,261]

Web sites traffic is more than half bots.

 

This is getting to be a nuisance.  Web administrators must strategize to hold off this flood of useless hits.

[hym 358]

Link directs to hanging of war criminal.

[Easy Bakes 253]

Link directs to hanging of war criminal.

probably a bot

[catty-cb 4,381]

I think he meant this link here

 

http://www.bbc.co.uk/news/technology-25346235

 

and you can stop good bots from visiting, that's what I did at CB, you use a robots.txt to tell them not to visit your site

 

http://en.wikipedia.org/wiki/Robots_exclusion_standard

 

it doesn't stop the bad bots, but you can IP ban those ones when they turn up

 

-catty

[Cyclone Boom 16,829]

This is quite apparent on new websites, especially those with a forum or blog. With all these bots, it's now much more challenging to establish a thriving online community. While the legitimate bots are useful for search indexing, the others are just a real pain. Not only increasing server load (in large volumes), but creating registrations and posting spam.

There are measures to limit this, but no real solution to block them completely. As they're becoming more clever, some will always find a way through. Blocking IP ranges is one way, but there's a risk of preventing real users from accessing the site.

EDIT: Sorry Catty, it looks like we posted at the same time. Didn't notice your reply as I was typing.

  Edited December 13, 2013 by Cyclone Boom  

[catty-cb 4,381]

Blocking IP ranges is one way, but there's a risk of preventing real users from accessing the site. 

 

I don't block IP ranges, just a single IP address at a time, it makes for a bit more work at my end, but you are less likely to block a real user that way.

[Duke87 1,070]

Bots that follow the rules won't crawl your site if your robots.txt prohibits it. Here, for example, is ST's.

 

Malicious bots, of course, can easily be programmed to ignore these instructions so you have to take other measures (such as the aforementioned IP bans) to stop them.

[A Nonny Moose 8,261]

Link directs to hanging of war criminal.

Fixed.  Guess I grabbed the wrong tab on the browser.

[AkemiAzula 280]

Surely all that is needed to protect a website is a password or captcha system. I may not be the brightest spark in the programming world, but I still think that a bot that can understand captchas must be a future entity, surely?

 

Still the way I see it the answer is to develop good bots that find and capture bad bots and 'eat' them at their source, then deactivate the bad bots source computer.

[A Nonny Moose 8,261]

Surely all that is needed to protect a website is a password or captcha system. I may not be the brightest spark in the programming world, but I still think that a bot that can understand captchas must be a future entity, surely?

 

Still the way I see it the answer is to develop good bots that find and capture bad bots and 'eat' them at their source, then deactivate the bad bots source computer.

There is far more to it than that.  Many bots are very clever indeed.

[Duke87 1,070]

CAPTCHA is effective at preventing bots from creating accounts, but it does nothing to prevent a bot from browsing a page as a guest unless you want to require that all users complete a CAPTCHA every time they access the site (or make it so you have to be registered and logged in to view anything but the main page).

[NCGAIO 562]

Bots that follow the rules won't crawl your site if your robots.txt prohibits it. Here, for example, is ST's.

 

 

Unfortunately also prevents access of the "'Wayback Machine' which sometimes is very useful for things that were lost.

[A Nonny Moose 8,261]

"What one fool can do, another can undo".

Sylvanus P. Thompson.

 

Why do you suppose some people just want to use the Internet for nefarious purposes?  Evil exists in the world for the purpose of doing evil.

[catty-cb 4,381]

In theory my robots.txt allows googlebots and the wayback machine and asks everyone else to keep out of everything

[A Nonny Moose 8,261]

How well  does it work, Catty?  Have you any good evidence that crawlers like Bing don't get past this?

[AkemiAzula 280]

At my Virtucity proboard I require users to register before posting and the registration requires email verification, possibly including a captcha I can't recall. I get a large number of 'visitors' but no spam.

[catty-cb 4,381]

How well  does it work, Catty?  Have you any good evidence that crawlers like Bing don't get past this?

 

I was going to post an answer this morning my time, in fact was in the middle of writing it when I was hit by a big dose of RL interruptus, so hopefully the second attempt at responding to this goes better than the first

 

... anyway to answer your question, the first year I was running the website I didn't have a robots.txt, just relied on firewalls, anti-spam, etc and had a quite high visitor count, the second year I finally added a robots.txt and watched my visitor numbers go down by about two thirds, I can't say for certain if Bing is obeying the KEEP OUT sign, but something is