307 lines
11 KiB
Markdown
307 lines
11 KiB
Markdown
# StatoolInfos
|
|
|
|
StatoolInfos is a simple statistic tool software.
|
|
|
|
|
|
## LICENSE
|
|
|
|
StatoolInfos is released under the GNU AGPL+ license. Enjoy!
|
|
|
|
Authors:
|
|
* Christian Pierre MOMON <christian.momon@devinsy.fr>
|
|
|
|
Contributors : Angie, MrFlos, Quentin Duchemin, Fabrice61, Thomas TConstans, labecasse, Antoine Jaba, setop, Jérémy Collot, Pilou, Kepon, Laurent Sleto, Rodinux…
|
|
|
|
### Cat Avatar Generator images
|
|
|
|
I am glad to use the beautiful artwork of David Revoy (http://www.peppercarrot.com).
|
|
* Graphics CC-BY http://creativecommons.org/licenses/by/4.0/
|
|
* Service: https://www.peppercarrot.com/extras/html/2016_cat-generator/
|
|
* Source: https://framagit.org/Deevad/cat-avatar-generator
|
|
* Originally inspired of the code for "MonsterID" by Andreas Gohr http://www.splitbrain.org/go/monsterid.
|
|
|
|
|
|
### Logo
|
|
|
|
Author: Christian Pierre MOMON <christian.momon@devinsy.fr>
|
|
|
|
License: Creative Commons CC BY-SA last version.
|
|
|
|
|
|
### Other stuff
|
|
|
|
License of other stuff: without any specific information, default license of images is CC BY-SA 3+.
|
|
|
|
|
|
## DOCUMENTATION
|
|
|
|
Read CONCEPT.md and ONTOLOGY.md files.
|
|
|
|
## Requirements
|
|
|
|
- Java 17
|
|
- Eclipse 4.29 (2023-09).
|
|
|
|
## INSTALL
|
|
|
|
Install Java :
|
|
|
|
```
|
|
apt-get install openjdk-17-jre-headless
|
|
```
|
|
|
|
Download the last release: https://forge.devinsy.fr/statool/statoolinfos/releases
|
|
|
|
Unzip the package and move the `statoolinfos.jar` and `statoolinfo.sh` files in `/srv/statoolinfos/bin/`.
|
|
|
|
Create link:
|
|
|
|
```
|
|
cd /srv/statoolinfos/bin/ ; ln -s statoolinfos.sh statoolinfos
|
|
```
|
|
|
|
|
|
## Usage
|
|
|
|
```
|
|
Usage:
|
|
statoolinfos [ -h | -help | --help ]
|
|
statoolinfos [ -v | -version | --version ]
|
|
|
|
statoolinfos build <configurationfile> build property files from conf and input
|
|
statoolinfos clear <configurationfile> remove property files from conf
|
|
statoolinfos crawl <configurationfile> crawl all file from conf and input
|
|
statoolinfos format <fileordirectory> format property files in tiny way
|
|
s̶t̶a̶t̶o̶o̶l̶i̶n̶f̶o̶s̶ ̶h̶t̶m̶l̶i̶z̶e̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶<̶c̶o̶n̶f̶i̶g̶u̶r̶a̶t̶i̶o̶n̶f̶i̶l̶e̶>̶ ̶ ̶ ̶ ̶ ̶g̶e̶n̶e̶r̶a̶t̶e̶ ̶w̶e̶b̶ ̶p̶a̶g̶e̶s̶ ̶f̶r̶o̶m̶ ̶c̶o̶n̶f̶ REMOVED since splitweb
|
|
statoolinfos probe OPTION [<directory>|<configurationfile>] generate metrics files from conf
|
|
OPTION = [-full|-today|-previousday|-NN] with NN a day count
|
|
statoolinfos tagdate <fileordirectory> update the file.datetime file
|
|
statoolinfos uptime <configurationfile> update uptime journal
|
|
|
|
statoolinfos list file <logfilesorconfigfile> display http access log files
|
|
statoolinfos list log [-bot|-nobot] <logfilesorconfigfile> display http access log lines
|
|
statoolinfos list ip [-bot|-nobot] <logfilesorconfigfile> generate ip list from http log file
|
|
statoolinfos list ua [-bot|-nobot] <logfilesorconfigfile> generate user agent list from http log file
|
|
statoolinfos list visitor [-bot|-nobot] <logfilesorconfigfile> generate visitors (ip+ua) list from http log file
|
|
statoolinfos stat ip [-bot|-nobot] <logfilesorconfigfile> generate stats about ip from http log file
|
|
statoolinfos stat ua [-bot|-nobot] <logfilesorconfigfile> generate stats about user agent from http log file
|
|
statoolinfos stat visitor [-bot|-nobot] <logfilesorconfigfile> generate stats about visitor (ip+ua) from http log file
|
|
statoolinfos list errfile <logfilesorconfigfile> display http error log files
|
|
statoolinfos list errlog <logfilesorconfigfile> display http error log lines
|
|
```
|
|
|
|
### htmlize removed
|
|
The htmlize command is removed since 2024-07.
|
|
|
|
As a reminder, the htmlize command generated absolutely all the pages needed for a static site.
|
|
|
|
As the number of pages generated became too large (> 10000 with 100 organizations), the decision was made to separate the html valuation into a separate project, StatoolInfosWeb.
|
|
|
|
A dynamic web site allows to generate page only when it is necessary.
|
|
|
|
## Generate metrics files
|
|
|
|
### Basics
|
|
|
|
Create a configuration file `/srv/statoolInfos/conf/foo.bar.org.conf`:
|
|
|
|
```
|
|
conf.probe.types=<metrictype1>, <metrictype2>
|
|
conf.probe.metrictype1param1.file=<value>
|
|
conf.probe.metrictype1param2.file=<value>
|
|
conf.probe.metrictype2param1.file=<value>
|
|
conf.probe.metrictype2param2.file=<value>
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
Generate the metric file for the first time:
|
|
|
|
```
|
|
/srv/statoolinfos/bin/statoolinfos probe -full /srv/statoolinfos/conf/foo.bar.org.conf
|
|
```
|
|
|
|
Create a cron file in `/etc/cron.d/statoolinfos` to update the metric file everyday:
|
|
```
|
|
1 0,12 * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/foo.bar.org.conf >> /srv/statoolinfos/cron.log
|
|
```
|
|
|
|
In case of several configuration files, just set the configuration directory (without wildcard):
|
|
```
|
|
1 0,12 * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/ >> /srv/statoolinfos/cron.log
|
|
```
|
|
|
|
Warning: in previous day mode, the metrics generated are overwrited for the last month, the last week and the last day. So, **six weeks in logs are required**.
|
|
|
|
|
|
### Etherpad metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Etherpad
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
|
|
conf.probe.httpaccesslog.pattern=
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
# This parameter requires real ip proxy setting and log output setting.
|
|
conf.probe.etherpad.logs=/var/log/etherpad/output.log
|
|
# Database URL. Example of the most frequent cases:
|
|
# jdbc:mariadb://localhost:1234/databasename
|
|
# jdbc:mysql://localhost:1234/databasename
|
|
# jdbc:postgresql://localhost:1234/databasename
|
|
# jdbc:sqlite:/foo/bar/databasename.sqlite
|
|
conf.probe.etherpad.database.url=
|
|
conf.probe.etherpad.database.user=
|
|
conf.probe.etherpad.database.password=
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### Framdadate metrics (coming soon)
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Framadate
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### Gitea metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Gitea
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
|
|
conf.probe.httpaccesslog.pattern=
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
conf.probe.gitea.data=/opt/gitea/data
|
|
conf.probe.gitea.api.url=https://forge.libre-service.eu/
|
|
conf.probe.gitea.token=b6598c616b1cd350b834258205da4e5e8b951005
|
|
# jdbc:mariadb://localhost:1234/databasename
|
|
# jdbc:mysql://localhost:1234/databasename
|
|
# jdbc:postgresql://localhost:1234/databasename
|
|
# jdbc:sqlite:/foo/bar/databasename.sqlite
|
|
conf.probe.gitea.database.url=
|
|
conf.probe.gitea.database.user=
|
|
conf.probe.gitea.database.password=
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### LibreQR metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=LibreQR
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
|
|
conf.probe.httpaccesslog.pattern=
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
conf.probe.libreqr.datafiles=/var/www/foo.bar.org/temp/
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### Minetest metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Minetest
|
|
conf.probe.minetest.logs=/home/cpm/Projets/StatoolInfos/EnvTest/minetest/minetest.log*
|
|
|
|
conf.probe.minetest.players.database.url=jdbc:postgresql://localhost:5432/minetestdb
|
|
conf.probe.minetest.players.database.user=minetestdba
|
|
conf.probe.minetest.players.database.password=XXXXXXXXXXX
|
|
|
|
conf.probe.minetest.worlds.database.url=jdbc:postgresql://localhost:5432/minetestdb
|
|
conf.probe.minetest.worlds.database.user=minetestdba
|
|
conf.probe.minetest.worlds.database.password=XXXXXXXXXXX
|
|
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
|
|
```
|
|
|
|
### Mumble metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Mumble
|
|
conf.probe.mumble.logs=/var/log/mumble-server/mumble-server.log*
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### Nextcloud metrics (coming soon)
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Nextcloud
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
|
|
conf.probe.httpaccesslog.pattern=
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### PrivateBin metrics (partial)
|
|
|
|
Warning: works fine if database, image and comment options are disabled.
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=PrivateBin
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
|
|
conf.probe.httpaccesslog.pattern=
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
conf.probe.privatebin.data=/var/www/paste.libre-service.eu/data/
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### Web metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=HttpAccessLog, HttpErrorLog
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*,/var/log/apache2/foo.bar.org-extraaccess.log*
|
|
conf.probe.httperrorlog.file=/var/log/apache2/foo.bar.org-error.log*,/var/log/apache2/foo.bar.org-extraerror.log*
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
|
|
# Custom access log pattern with Java regex.
|
|
# Default: ^(?<remoteAddress>[a-fA-F0-9\:\.]+) - (?<remoteUser>[^\[]+) \[(?<time>[^\]]+)\] "(?<request>([^\"]|\")*)" (?<status>\d+) (?<bodyBytesSent>\d+) "(?<referer>([^\"]|\")*)" "(?<userAgent>([^\"]|\")*)".*$
|
|
conf.probe.httpaccesslog.pattern=
|
|
|
|
# Custom datetime access log pattern (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns). The language ID is mandatory.
|
|
# Default: dd/MMM/yyyy:HH:mm:ss Z|EN
|
|
conf.prob.httpaccesslog.datetimepattern=
|
|
|
|
# Filter request with Java regex (without the initial '^' and without the final '$', see https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html).
|
|
# Example: pathfilter=/.*well-known.*
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
|
|
# Custom error log pattern with Java regex. By default, automatic detection.
|
|
# Default Apache: ^\[(?<time>[^\]]+)\]\s\[(?<level>[^\]]*)\]\s(?<message>.*)$
|
|
# Default NGinx: ^(?<time>\S+\s\S+)\s\[(?<level>[^\]]*)\]\s(?<message>.*)$
|
|
conf.probe.httperrorlog.pattern=
|
|
|
|
# Custom datetime access log pattern (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns). The language ID is mandatory.
|
|
# Default: dd/MMM/yyyy:HH:mm:ss Z|EN
|
|
conf.prob.httperrorlog.datetimepattern=
|
|
```
|
|
|
|
Warning: to have month metric values, you have to set **retention log to 7 weeks or 50 days** in Nginx or Apache logrotate configuration.
|
|
|
|
For Nginx, éditer le fichier `/etc/logrotate.d/nginx` :
|
|
```
|
|
/var/log/nginx/*.log {
|
|
weekly
|
|
missingok
|
|
rotate 7
|
|
```
|
|
|
|
For Apache, éditer le fichier `/etc/logrotate.d/apache2` :
|
|
```
|
|
/var/log/apache2/*.log {
|
|
weekly
|
|
missingok
|
|
rotate 7
|
|
```
|