Banning by IP range

C

crazyfruitbat

Aspirant
Joined
Sep 5, 2011
Messages
12
119.63.196.49 is a Baidu spider (China search engine) not complying to robots.txt so I want to ban it - actually I have banned it some of the IPs via iptables server side - problem is that now I get loads of these buggers coming from 119.63.196.xx. Can I safely ban that whole range 119.63.196.xx without affecting other people?

The reason of concern is that my server runs websites based in Japan (where this Baidu search is), so I don't want to block normal folk by accident. Biggest issue is now that I have Russian spiders and other Chinese search people crawling my site - its only a matter of time before I get spammed like mad - any ideas?

Also if I want to ban an ip range - how do I write that in SSH or webmin? is it 119.63.196.0/xxx (xxx= the max IP I can find?)

Cheers,
C
 
Judge Dredd

Judge Dredd

Old Guy
Joined
Apr 20, 2011
Messages
3,144
Usually this can be done within the admin control panel of your board software. Whart script are you running?
 
djbaxter

djbaxter

Tazmanian Veteran
Joined
Jun 6, 2006
Messages
10,473
Or, at the server level, you block either 119.63.196.0/16 or 119.63.196.0/24. (i.e., don't use the highest IP you find - it's a different syntax to block ranges of addresses.)
 
C

crazyfruitbat

Aspirant
Joined
Sep 5, 2011
Messages
12
Hi guys, cheers for the replies. My board software in Xenforo but I want to block on a server level because it still takes up the resources on the board and seems a bit pointless as that is what I want to avoid, I don't mind for the odd IP address but today I have 40 individual spiders all from Baidu at one time - all ignoring the robots blocking I set up.

minstrel said:
Or, at the server level, you block either 119.63.196.0/16 or 119.63.196.0/24. (i.e., don't use the highest IP you find - it's a different syntax to block ranges of addresses.)
Click to expand...

I think this is the best way and block in webmin. What is the rule about 0/16 0/24? Is this a random number or a specific thing based on blocking all ip ranges? Sorry for the questions - interesting stuff.
 
B

bigkev93

Aspirant
Joined
Sep 4, 2011
Messages
16
Some peoples Ip changes in the last 6 digits aswell, so sometimes i ban the first 6 XXX.XXX.
 
emilykrys

emilykrys

Aspirant
Joined
Aug 12, 2011
Messages
17
Usually you can set an exception like 119.63.196.* which should ban the range but banning like that would kill anyone within that IP network. Also if its a proxy it wouldn't matter they would just get a new proxy list and come back another day. The best thing is to have a good system for registration and stop them from making accounts.
 
C

couponsitescript

Participant
Joined
Oct 1, 2011
Messages
61
crazyfruitbat said:
119.63.196.49 is a Baidu spider (China search engine) not complying to robots.txt so I want to ban it - actually I have banned it some of the IPs via iptables server side - problem is that now I get loads of these buggers coming from 119.63.196.xx. Can I safely ban that whole range 119.63.196.xx without affecting other people?

The reason of concern is that my server runs websites based in Japan (where this Baidu search is), so I don't want to block normal folk by accident. Biggest issue is now that I have Russian spiders and other Chinese search people crawling my site - its only a matter of time before I get spammed like mad - any ideas?

Also if I want to ban an ip range - how do I write that in SSH or webmin? is it 119.63.196.0/xxx (xxx= the max IP I can find?)

Cheers,
C
Click to expand...
How do you know you will be spammed? Baidu is an important search engine. For me what are you trying to do is like banning Google from your site.
Sure if you ban everyone you will get no spam to your site.
My point: you can't prevent spam. It will be. However letting spiders from know search engines will benefit you. You can consider banning known spiders from there you might have the spam problem.
 
U

Upside

Aspirant
Joined
Oct 26, 2010
Messages
23
couponsitescript said:
How do you know you will be spammed? Baidu is an important search engine. For me what are you trying to do is like banning Google from your site.
Click to expand...

Have you had a good experience with Baidu? I have observed Baidu does not obey robots.txt and I've never seen measurable traffic from Baidu even when they are permitted to crawl over time.
 
C

crazyfruitbat

Aspirant
Joined
Sep 5, 2011
Messages
12
couponsitescript said:
How do you know you will be spammed? Baidu is an important search engine. For me what are you trying to do is like banning Google from your site.
Sure if you ban everyone you will get no spam to your site.
My point: you can't prevent spam. It will be. However letting spiders from know search engines will benefit you. You can consider banning known spiders from there you might have the spam problem.
Click to expand...

Upside said:
Have you had a good experience with Baidu? I have observed Baidu does not obey robots.txt and I've never seen measurable traffic from Baidu even when they are permitted to crawl over time.
Click to expand...

Yes exactly, I see googlebot, amazon, yahoo etc all collect data on the forums, they come do the job and move on and it's only one at a time. Now I have Baidu and some other random Chinese TV station all searching my boards at the same time - i'm getting like 80 bots at one time, it's really not a joke as it eats your bandwidth
 
J

johnfoster

Neophyte
Joined
Nov 18, 2011
Messages
6
I own some websites but never knew that too much crawling by spiders can be a problem. Well I learnt a new thing today...I guess I should go and check my server log..Thanks!
 
Cre8tin

Cre8tin

Aspirant
Joined
Aug 25, 2011
Messages
21
The7thSage said:
No racist agenda, just some countries are a nuisance in the online world.
Click to expand...
which ones? e.g. United States always occupying #1 of spammers reports?

Besides, it is not that important who does the spam but who orders, finances and provides financial channels/mechanisms for this business.
And USA is indisputable leader in spam business

More links to Spam Countries of Origin:
 
The7thSage

The7thSage

Adherent
Joined
Aug 7, 2011
Messages
298
Well USA traffic still generates nice revenue so its a 50-50 problem. China and some other countries do not generate much revenue, the traffic is not much of good quality and there are always some kind of attacks from them. Again, no offense intended.
 
DigNap15

DigNap15

Adherent
Joined
Sep 14, 2019
Messages
405
I just read that entering say
217.0.0.0/8
Would ban the whole range of 217 Ip adresses
Does that sound correct?
 
Last edited:
Paul M

Paul M

Limeade Addict
Joined
Jun 26, 2006
Messages
3,966
Assuming it understands that notation, then yes.
Whether doing so is advisable ? Thats another matter.
 
mysiteguy

mysiteguy

Migration Expert
Joined
Feb 20, 2007
Messages
3,152
Cre8tin said:
which ones? e.g. United States always occupying #1 of spammers reports?

Besides, it is not that important who does the spam but who orders, finances and provides financial channels/mechanisms for this business.
And USA is indisputable leader in spam business

More links to Spam Countries of Origin:
Click to expand...

China is the top country for spam harvesting. The USA is 2nd. This isn't the same as sending spam.
China is the top country for spam-sending. The USA is 2nd.
China is the 2nd country for dictionary attacks. India is 1st, the USA is third.
China is the top country to comment spamming. The USA is 2nd.
China brings little to no "real" human traffic to most sites outside of China. The USA brings enormous amounts of real traffic to countries all over the world (much like the EU does, the UK and Germany are my #2 and #3 sources of traffic after the USA).
- This information is from projecthoneypot.org, which has close ties with the folks at Cloudflare, and they probably have a better idea of traffic trends than just about anyone. Projecthoneypot is one of the largest and oldest sources of harvesting and bad behavior tracking.

I get it, you're sensitive about China for some reason. But facts are facts, China is notorious for low-quality traffic. Its Internet ecosystem is to a large degree "in country" due to language barriers, the Great Firewall of China, and homegrown search and social networks. There's a valid reason large numbers of people have been asking how to block China for 20 years, because it's a large problem. You'll rarely see people ask how to block the EU, USA, Australia, etc. You do see them asking how to block China, Ukraine, Russia and other sites which have little "positive" traffic.

And frankly, given China's horrific track record with stealing intellectual property, I'd block them on principle alone even if they weren't a scraping source. I have little use for a government that looks the other way while it's businesses rip off hundreds of billions of dollars of intellectual property from the rest of the world. That's not a remark about the people of China, those I've met have been very nice.
 
You must log in or register to reply here.
Top