309 lines
10 KiB
Markdown
309 lines
10 KiB
Markdown
# StatoolInfos
|
|
|
|
StatoolInfos is a simple statistic tool software.
|
|
|
|
|
|
## LICENSE
|
|
|
|
StatoolInfos is released under the GNU AGPL+ license. Enjoy!
|
|
|
|
Authors:
|
|
* Christian Pierre MOMON <christian.momon@devinsy.fr>
|
|
|
|
Contributors : Angie, MrFlos, Quentin Duchemin, Fabrice61, Thomas TConstans, labecasse, Antoine Jaba, setop, Jérémy Collot, Pilou, Kepon, Laurent Sleto, Rodinux…
|
|
|
|
### Cat Avatar Generator images
|
|
|
|
I am glad to use the beautiful artwork of David Revoy (http://www.peppercarrot.com).
|
|
* Graphics CC-BY http://creativecommons.org/licenses/by/4.0/
|
|
* Service: https://www.peppercarrot.com/extras/html/2016_cat-generator/
|
|
* Source: https://framagit.org/Deevad/cat-avatar-generator
|
|
* Originally inspired of the code for "MonsterID" by Andreas Gohr http://www.splitbrain.org/go/monsterid.
|
|
|
|
### Circle-icons
|
|
|
|
I am glad to use the generous artwork of ElegantThemes.com :
|
|
|
|
* https://www.elegantthemes.com/blog/freebie-of-the-week/beautiful-flat-icons-for-free
|
|
* "These icons are completely free and Open Source under the GPL".
|
|
|
|
|
|
### Logo
|
|
|
|
Author: Christian Pierre MOMON <christian.momon@devinsy.fr>
|
|
|
|
License: Creative Commons CC BY-SA last version.
|
|
|
|
|
|
### Other stuff
|
|
|
|
License of other stuff:
|
|
* csv-logo.svg, json-logo.svg, ods-icon.svg:
|
|
* Author: Christian P. Momon
|
|
* License: CC BY-SA 3+.
|
|
* other: without any specific information, default license of images is CC BY-SA 3+.
|
|
|
|
|
|
## DOCUMENTATION
|
|
|
|
Read CONCEPT.md and ONTOLOGY.md files.
|
|
|
|
## Requirements
|
|
|
|
- Java 11
|
|
- Eclipse 4.16 (202006).
|
|
|
|
## INSTALL
|
|
|
|
Install Java :
|
|
|
|
```
|
|
apt-get install openjdk-11-jre-headless
|
|
```
|
|
|
|
Download the last release: https://forge.devinsy.fr/devinsy/statoolinfos/releases
|
|
|
|
Unzip the package and move the `statoolinfos.jar` and `statoolinfo.sh` files in `/srv/statoolinfos/bin/`.
|
|
|
|
Create link:
|
|
|
|
```
|
|
cd /srv/statoolinfos/bin/ ; ln -s statoolinfos.sh statoolinfos
|
|
```
|
|
|
|
|
|
## Usage
|
|
|
|
```
|
|
Usage:
|
|
statoolinfos [ -h | -help | --help ]
|
|
statoolinfos [ -v | -version | --version ]
|
|
|
|
statoolinfos build <configurationfile> build property files from conf and input
|
|
statoolinfos clear <configurationfile> remove property files from conf
|
|
statoolinfos crawl <configurationfile> crawl all file from conf and input
|
|
statoolinfos format <fileordirectory> format property files in tiny way
|
|
statoolinfos htmlize <configurationfile> generate web pages from conf
|
|
statoolinfos probe OPTION [<directory>|<configurationfile>] generate metrics files from conf
|
|
OPTION = [-full|-today|-previousday|-NN] with NN a day count
|
|
statoolinfos tagdate <fileordirectory> update the file.datetime file
|
|
statoolinfos uptime <configurationfile> update uptime journal
|
|
|
|
statoolinfos list file <logfilesorconfigfile> display http access log files
|
|
statoolinfos list log [-bot|-nobot] <logfilesorconfigfile> display http access log lines
|
|
statoolinfos list ip [-bot|-nobot] <logfiles> generate ip list from http log file
|
|
statoolinfos list ua [-bot|-nobot] <logfiles> generate user agent list from http log file
|
|
statoolinfos list visitor [-bot|-nobot] <logfiles> generate visitors (ip+ua) list from http log file
|
|
statoolinfos stat ip [-bot|-nobot] <logfiles> generate stats about ip from http log file
|
|
statoolinfos stat ua [-bot|-nobot] <logfiles> generate stats about user agent from http log file
|
|
statoolinfos stat visitor [-bot|-nobot] <logfiles> generate stats about visitor (ip+ua) from http log file
|
|
```
|
|
|
|
|
|
## Generate metrics files
|
|
|
|
### Basics
|
|
|
|
Create a configuration file `/srv/statoolInfos/conf/foo.bar.org.conf`:
|
|
|
|
```
|
|
conf.probe.types=<metrictype1>, <metrictype2>
|
|
conf.probe.metrictype1param1.file=<value>
|
|
conf.probe.metrictype1param2.file=<value>
|
|
conf.probe.metrictype2param1.file=<value>
|
|
conf.probe.metrictype2param2.file=<value>
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
Generate the metric file for the first time:
|
|
|
|
```
|
|
/srv/statoolinfos/bin/statoolinfos probe -full /srv/statoolinfos/conf/foo.bar.org.conf
|
|
```
|
|
|
|
Create a cron file in `/etc/cron.d/statoolinfos` to update the metric file everyday:
|
|
```
|
|
1 0,12 * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/foo.bar.org.conf >> /srv/statoolinfos/cron.log
|
|
```
|
|
|
|
In case of several configuration files, just set the configuration directory (without wildcard):
|
|
```
|
|
1 0,12 * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/ >> /srv/statoolinfos/cron.log
|
|
```
|
|
|
|
Warning: in previous day mode, the metrics generated are overwrited for the last month, the last week and the last day. So, **six weeks in logs are required**.
|
|
|
|
|
|
### Etherpad metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Etherpad
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
|
|
conf.probe.httpaccesslog.pattern=
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
# This parameter requires real ip proxy setting and log output setting.
|
|
conf.probe.etherpad.logs=/var/log/etherpad/output.log
|
|
# Database URL. Example of the most frequent cases:
|
|
# jdbc:mariadb://localhost:1234/databasename
|
|
# jdbc:mysql://localhost:1234/databasename
|
|
# jdbc:postgresql://localhost:1234/databasename
|
|
# jdbc:sqlite:/foo/bar/databasename.sqlite
|
|
conf.probe.etherpad.database.url=
|
|
conf.probe.etherpad.database.user=
|
|
conf.probe.etherpad.database.password=
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### Framdadate metrics (coming soon)
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Framadate
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### Gitea metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Gitea
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
|
|
conf.probe.httpaccesslog.pattern=
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
conf.probe.gitea.data=/opt/gitea/data
|
|
conf.probe.gitea.api.url=https://forge.libre-service.eu/
|
|
conf.probe.gitea.token=b6598c616b1cd350b834258205da4e5e8b951005
|
|
# jdbc:mariadb://localhost:1234/databasename
|
|
# jdbc:mysql://localhost:1234/databasename
|
|
# jdbc:postgresql://localhost:1234/databasename
|
|
# jdbc:sqlite:/foo/bar/databasename.sqlite
|
|
conf.probe.gitea.database.url=
|
|
conf.probe.gitea.database.user=
|
|
conf.probe.gitea.database.password=
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### GSL metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=GSL
|
|
conf.probe.gsl.stats=/foo/bar/stats.properties
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### LibreQR metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=LibreQR
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
|
|
conf.probe.httpaccesslog.pattern=
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
conf.probe.libreqr.datafiles=/var/www/foo.bar.org/temp/
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### Minetest metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Minetest
|
|
conf.probe.minetest.logs=/home/cpm/Projets/StatoolInfos/EnvTest/minetest/minetest.log*
|
|
|
|
conf.probe.minetest.players.database.url=jdbc:postgresql://localhost:5432/minetestdb
|
|
conf.probe.minetest.players.database.user=minetestdba
|
|
conf.probe.minetest.players.database.password=XXXXXXXXXXX
|
|
|
|
conf.probe.minetest.worlds.database.url=jdbc:postgresql://localhost:5432/minetestdb
|
|
conf.probe.minetest.worlds.database.user=minetestdba
|
|
conf.probe.minetest.worlds.database.password=XXXXXXXXXXX
|
|
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
|
|
```
|
|
|
|
### Mumble metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Mumble
|
|
conf.probe.mumble.logs=/var/log/mumble-server/mumble-server.log*
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### Nextcloud metrics (coming soon)
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Nextcloud
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
|
|
conf.probe.httpaccesslog.pattern=
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### PrivateBin metrics (partial)
|
|
|
|
Warning: works fine if database, image and comment options are disabled.
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=PrivateBin
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
|
|
conf.probe.httpaccesslog.pattern=
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
conf.probe.privatebin.data=/var/www/paste.libre-service.eu/data/
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### Web metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=HttpAccessLog, HttpErrorLog
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*,/var/log/apache2/foo.bar.org-extraaccess.log*
|
|
conf.probe.httperrorlog.file=/var/log/apache2/foo.bar.org-error.log*,/var/log/apache2/foo.bar.org-extraerror.log*
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
|
|
# Custom access log pattern with Java regex.
|
|
# Default: ^(?<remoteAddress>[a-fA-F0-9\:\.]+) - (?<remoteUser>[^\[]+) \[(?<time>[^\]]+)\] "(?<request>.*)" (?<status>\d+) (?<bodyBytesSent>\d+) "(?<referer>.*)" "(?<userAgent>[^"]*)".*$
|
|
conf.probe.httpaccesslog.pattern=
|
|
|
|
# Custom datetime access log pattern (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns).
|
|
# Default: dd/MMM/yyyy:HH:mm:ss Z|EN
|
|
conf.prob.httpaccesslog.datetimepattern=
|
|
|
|
# Filter request with Java regex (without the initial '^' and without the final '$', see https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html).
|
|
# Example: pathfilter=/.*well-known.*
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
```
|
|
|
|
Warning: to have month metric values, you have to set **retention log to 7 weeks or 50 days** in Nginx or Apache logrotate configuration.
|
|
|
|
For Nginx, éditer le fichier `/etc/logrotate.d/nginx` :
|
|
```
|
|
/var/log/nginx/*.log {
|
|
weekly
|
|
missingok
|
|
rotate 7
|
|
```
|
|
|
|
For Apache, éditer le fichier `/etc/logrotate.d/apache2` :
|
|
```
|
|
/var/log/apache2/*.log {
|
|
weekly
|
|
missingok
|
|
rotate 7
|
|
```
|