Allowing ad crawlers/networks in robots.txt.

[Xtreme2damax]

I have some pages that are off limits to search engine crawlers due to duplicate content, but I want to allow access to the crawlers for ad networks so the appropriate ad can display on the page.

What are the appropriate user agents or strings I can add to my robots.txt file to allow access to Google Adsense, Chitika content links/eminimalls, and Kontera in text advertising?

[Xtreme2damax]

Just wondering if anyone may know how to do this or what user agents I add to my robots.txt to allow Adsense Crawlers, Chitika, and Kontera access to crawl all pages in order to display proper ads?

[vlauria]

From google:
If you would like to grant our crawler access your pages, you can do so without granting permission to any other bots. Simply add the following two lines to the top of your robots.txt file:

User-agent: Mediapartners-Google*
Disallow:

--
And on a side note, if anybody wants to let an adsense crawler in behind a login, here's how to do it - https://www.google.com/adsense/suppo...n&answer=37081

[Xtreme2damax]

What about for other ad networks?

I've also tried adding that to my robots.txt, but Google still shows public service ads on pages I disallowed crawlers from accessing due to duplicate content such as the urls for getnew posts etc..

I also want to know how I can allow access for other advertisers besides Google, I couldn't find the information I was seeking via a Google search.

[vlauria]

Quote:

Originally Posted by Xtreme2damax What about for other ad networks? ... I also want to know how I can allow access for other advertisers besides Google, I couldn't find the information I was seeking via a Google search.

As for what you need to include in robots for each ad provider, you will have to contact them directly or search google with their network name and 'allow robots crawl', like 'adbrite allow robots crawl' and the 1st result on google is - http://www.tw3o.com/robotstxt-guide/

Quote:

Originally Posted by Xtreme2damax I've also tried adding that to my robots.txt, but Google still shows public service ads on pages I disallowed crawlers from accessing due to duplicate content such as the urls for getnew posts etc..

You'll need to find out if the google adsense crawler is actually crawling your site or not, there's way to debug this and test that out with tools, but it's a bit advanced.

[Xtreme2damax]

I needed to exclude/delete the asterisk at the end of the user agent, I don't know why they added that there since it's wrong and causes problems.