China bots - oh how they love me...

[Jon123456]

In the last couple of weeks, I do seem to have a rather large number of guests. Like on a Sunday, I might have typically about 200 guests, but this time it is 2,000! I am getting a lot of vBulletin error messages in my email, saying session table is full. But the IP addresses are all from China Telecom. And the urls that they are on tend to be things like calendar, top posters, tags etc., rather than normal threads.

How do you deal with these? What is the best solution?

Someone pointed me to this: https://www.google.com/recaptcha/intro/v3.html

Anyone used that?

 

[Cyburbia]

Just tweak your .htaccess

Blocking aggressive Chinese crawlers/scrapers/bots - John Large .co.uk

Over the last few days I’ve had a massive increase in traffic from Chinese data centres & ISPs. The traffic has been relentless & the CPU usage on my server kept spiking enough to cause a fault in my cPanel hosting. I’m on a great hosting package with UKHOST4U and the server is fast & […]

www.johnlarge.co.uk

My experience: https://www.cyburbia.org/forums/threads/aggressive-bots-from-chinanet-and-china-unicom-ips.56183/

 

[Jon123456]

Thanks for the link. I got onto my host and they have implemented modsecurity. They said it might have an impact on site functionality, but they are not sure, so I am monitoring the situation.

 

[zappaDPJ]

Thanks for the link. I got onto my host and they have implemented modsecurity. They said it might have an impact on site functionality, but they are not sure, so I am monitoring the situation.

If your host has configured it properly you shouldn't encounter too many issues. If they haven't it can cause all manner of problems including '403' or 'access denied' errors, '404' errors, login issues and so on.

 

[Jon123456]

Oh wow, so some significant stuff then. How do I find out if it is causing problems?

 

[zappaDPJ]

It's really just something to keep very much in mind when someone reports an error because once installed it's so easy to forget about it.

If someone reports an error it's best to first establish if they are using an ad-blocker because they can cause similar errors. If there's no ad-blocker involved and nothing much has changed in terms of your installation then it's fairly safe to assume it's an issue with ModSecurity. To be sure you can turn it off in your .htaccess.txt to check that fixes the issue.

 

[Jon123456]

zappaDPJ, thanks for that. As a side note and perhaps a bit off topic from the thread, was TAZ using vBulletin 3.8 before and if so, did you notice any drop off in traffic after switching over? Did user engagement change at all? Just doing my due diligence!

 

[Kyrie]

zappaDPJ, thanks for that. As a side note and perhaps a bit off topic from the thread, was TAZ using vBulletin 3.8 before and if so, did you notice any drop off in traffic after switching over? Did user engagement change at all? Just doing my due diligence!

The important thing to remember is your niche. Is your community programmers/gamers from the year 99? They may not like XF...

 

[Jon123456]

They are programmers actually. Hah!

 

[mysiteguy]

I block China at the firewall level with ipset.

 

[Skybound]

Had same issue. I blocked all China IPs in the htaccess.

 

[Jon123456]

Does blocking China with ipset have a negative impact on performance?

 

[mysiteguy]

Does blocking China with ipset have a negative impact on performance?

No. Blocking them in .htaccess will. Apache must scan every single deny rule for every page, image, javascript, css and other items fetched.

IPset works at the firewall level before it even get's to Apache. And it uses a binary tree to match up rules, so it's extremely quick - you can have a hundred thousand rules in it and it will add virtually nothing to overhead.

 

[TLChris]

Couldn't you use a service like CloudFlare to block countries as well?

 

[Dan18]

I block China at the firewall level with ipset.

https://lite.ip2location.com/china-ip-address-ranges

China has a total of 344,274,750 IP address assigned.
That's huge amount of IP :|

 

[mysiteguy]

https://lite.ip2location.com/china-ip-address-ranges

China has a total of 344,274,750 IP address assigned.
That's huge amount of IP :|

You don't list each IP, you list the CIDR IP ranges. It comes to about 34,000 to cover not only China but several other troublesome countries, TOR exit nodes, and the stopforumspam list.

 

[mysiteguy]

Couldn't you use a service like CloudFlare to block countries as well?

Cloudflare hurts TTFB of page content when it comes to dynamic content so I don't use them. For CDN purposes (js, images, css) I find MaxCDN to be a better solution, and I use them for those types of files, since they are static and will be cached as a result, decreasing their TTFB.

 

[Kyrie]

Cloudflare hurts TTFB of page content when it comes to dynamic content so I don't use them. For CDN purposes (js, images, css) I find MaxCDN to be a better solution, and I use them for those types of files, since they are static and will be cached as a result, decreasing their TTFB.

After reading a review this part startled me via Digitial.com

Fewer security options than competitors like CloudFlare, which focus more on security than speed.

Do you consider this to be the case?

 

[mysiteguy]

Digital.com doesn't show any testing on their "review". They are merely repeating/quoting cloudflare's claims. A CDN won't speed up dynamic content - it can't. It can only speed up static content.

Digital.com's "review" is merely a front to earn affiliate commissions through Cloudflare's partner program, and they are going through extraordinary steps to hide this by opening in a new window, through a go-between URL and javascript trying to obfuscate the referring URL on the cloudflare destination page.

 

[Kyrie]

Digital.com doesn't show any testing on their "review". They are merely repeating/quoting cloudflare's claims. A CDN won't speed up dynamic content - it can't. It can only speed up static content.

Digital.com's "review" is merely a front to earn affiliate commissions through Cloudflare's partner program, and they are going through extraordinary steps to hide this by opening in a new window, through a go-between URL and javascript trying to obfuscate the referring URL on the cloudflare destination page.

Couldn't it, hypothetically, slow-down the content though? Also, I didn't realize what I was reading so thanks for that... Doesn't that seem illegal to you?