[OzzModz] vBulletin Spiders List Hits 1000 Spiders!

[OzzModz] vBulletin Spiders List Hits 1000 Spiders! 2016-09-26

No permission to download

ozzy47

Tazmanian Master
Joined
Oct 18, 2013
Messages
8,989

Delfi_R

Neophyte
Joined
Nov 6, 2016
Messages
3
The list it's great, but I've found today two new bots: AhrefsBot SemrushBot
 

I A 1

Enthusiast
Joined
Jun 7, 2015
Messages
137
I made a new plugin few months ago to identify all bots automatically. It basically searches for certain keywords in the useragent and marks them as spider.
Below is the code:

Hook location: online_bit_complete
Code:
if (!$userinfo['spider']) {
if ((strpos($userinfo['useragent'], 'bot') !== false) OR (strpos($userinfo['useragent'], 'crawler') !== false) OR (strpos($userinfo['useragent'], 'spider') !== false) OR (strpos($userinfo['useragent'], 'Scanner') !== false) OR (strpos($userinfo['useragent'], 'Analyzer') !== false))
{
$userinfo['spider'] = true;
}
}

When I view Who's Online, it shows them as "1 Spider".
 

mysiteguy

Migration Expert
Joined
Feb 20, 2007
Messages
3,215
allow google

block everything else

There's a problem with that....

Bing/Yahoo search.

Plus if you use Adsense, Google forwards many of their 3rd party networks the URL in real time, and they then crawl the page just as the Adsense crawler does. Block those and the ads won't be as relevant and higher paying. I have a program which dynamically writes out robots.txt depending on which bot is crawling. I have a couple hundred defined, about a third are ad crawling related bots which I allow to fetch content related pages.
 
Top