In the apache access.log all ip addresses are stored complete. This is neither necessary for statistcs nor it´s needed to store these data.
IPv4 addresses can be quite simple anonymous, if the last part is changed. I.e. 176.9.33.188 goes to 176.9.33.0
The most statistic-tools can handle the data furthermore.
As the apache-logs are already running through syslog-ng laufen (please read ISPConfig – Apache-Logfiles and syslog first), i use syslog-ng to do the job.
I use the pattern-db in the following. The CSV-Parser is on my systems simply not fast enough. But can easily use the csv-parser described in my posted noted above.
Just add rewrite(r_ano-apache); to the log-statment:
log {
source(src);
parser(pattern_db);
filter(f_apache2);
rewrite(r_ano-apache);
destination (d_apache-logs);
destination (d_ispconfig_apache-stat);
};
rewrite(r_ano-apache) contains is defined like this:
rewrite r_ano-apache {
subst("([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})","$1.$2.$3.0",
value(".apache.client_ip"),flags("pcre") condition(tags("IPv4")));
subst("([0-9a-f]{1,4})\:([0-9a-f]{1,4})\:([0-9a-f]{1,4})\:([0-9a-f]{1,4})","$1:$2:$3",
value(".apache.client_ip"),flags("pcre") condition(tags("IPv6")));
};
To use tools like fail2ban you need full logs. I do this with a second log-statment. This results in two logs – the complete one is deleted after a few days, the otherone can stay much longer.
log {
source(src);
parser(pattern_db);
filter(f_apache2);
destination (d_apache-logs-admin);
};
And a second destination:
destination d_apache-logs-admin {
file("/var/log/full/${.apache.domain}-access.log"
template("${.apache.client_ip} ${.apache.ident_name} ${.apache.user_name} [${.apache.timestamp}] \"${.apache.request_url}\" ${.apache.request_status} ${.apache.content_length} \"${.apache.referer}\" \"${.apache.user_agent}\"\n")
template_escape(yes));
};
Download: