Nginx What is the best log file analyzer in case of attacks?

  • Thread starter
  • Admin
  • #1

Alfa1

Administrator
Joined
May 28, 2007
Messages
3,931
I like to analyze my access log and my server log to find abuse and issues with my site. Some important data to find is:
  • Top IP's ordered by number of requests.
  • Top IP's ordered by bandwidth.
  • Top IP's ordered by number of errors.
  • Top errors
I'm currently using Deep Log Analyzer, but I'm not really happy with it. I have seen an old mention by eva2000 about GoAccess web log analyzer tool. I'm not sure if that's still relevant.
Does anyone have suggestions for this?
 

eva2000

Habitué
Joined
Jan 11, 2004
Messages
1,769
usually i use pure shell scripting to analyse logs for finer grain analysis i.e. calculating number of requests per second, minute, hour etc and then charting those in graphs

but ngxtop can help see https://community.centminmod.com/threads/ngxtop-real-time-metrics-for-nginx.285/ which has 3 pages of posts with examples

example of top requests by average bytes sent size
Code:
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow --order-by 'avg(bytes_sent) * count'
running for 0 seconds, 6011 records processed: 51469.79 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|    6011 |        24728.661 |  6006 |     0 |     0 |     5 |

Detailed:
| request_path   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|----------------+---------+------------------+-------+-------+-------+-------|
| /              |    6000 |        24773.940 |  5995 |     0 |     0 |     5 |
| /wp-cron.php   |      11 |           31.000 |    11 |     0 |     0 |     0 |
by status codes
Code:
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow top status -n10
running for 0 seconds, 6011 records processed: 53862.51 req/sec

top status
|   status |   count |
|----------+---------|
|      200 |    6006 |
|      500 |       5 |
by user agents with HTTP status code = 200
Code:
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow top http_user_agent -i 'status == 200' -n10
running for 0 seconds, 6006 records processed: 34858.99 req/sec

top http_user_agent
| http_user_agent                                  |   count |
|--------------------------------------------------+---------|
| Mozilla/5.0 (pc-x86_64-linux-gnu) Siege/4.0.4    |    5995 |
| WordPress/5.3.2; http://cache-enabler.domain.com |      11 |
by IP just filtered for Googlebot useragent
Code:
cat /home/nginx/domains/$domain/log/access.log | grep 'Googlebot' | ngxtop --no-follow --group-by remote_addr
running for 0 seconds, 2598 records processed: 16824.80 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|    2598 |        12314.505 |  1670 |   881 |    47 |     0 |

Detailed:
| remote_addr    |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|----------------+---------+------------------+-------+-------+-------+-------|
| 66.249.79.61   |    1231 |        14078.539 |   852 |   363 |    16 |     0 |
| 66.249.79.223  |     525 |         8911.514 |   292 |   232 |     1 |     0 |
| 66.249.79.63   |     311 |        13264.614 |   208 |    95 |     8 |     0 |
| 66.249.66.179  |     230 |         8986.509 |   119 |   102 |     9 |     0 |
| 66.249.79.65   |     191 |        14356.131 |   139 |    50 |     2 |     0 |
| 66.249.79.230  |      34 |         8199.324 |    18 |    16 |     0 |     0 |
| 66.249.66.182  |      29 |         9298.276 |    16 |     9 |     4 |     0 |
| 66.249.66.185  |      19 |         5710.632 |     8 |     6 |     5 |     0 |
| 66.249.79.237  |      16 |         7718.500 |     9 |     7 |     0 |     0 |
| 209.58.130.199 |       3 |          795.000 |     1 |     0 |     2 |     0 |
 
  • Thread starter
  • Admin
  • #3

Alfa1

Administrator
Joined
May 28, 2007
Messages
3,931
Aha. Apparently I have this included already. The wonders of CentminMod. :)
Im going to try get this to work and have it give me the top IP addresses that his site root today.
 
Last edited:
Top