504 lines
17 KiB
Markdown
504 lines
17 KiB
Markdown
# StatoolInfos
|
|
|
|
StatoolInfos is a simple statistic tool software.
|
|
|
|
|
|
## LICENSE
|
|
|
|
StatoolInfos is released under the GNU AGPL+ license. Enjoy!
|
|
|
|
Authors:
|
|
* Christian Pierre MOMON <christian.momon@devinsy.fr>
|
|
|
|
Contributors : Angie, MrFlos, Quentin Duchemin, Fabrice61, Thomas TConstans, labecasse, Antoine Jaba, setop, Jérémy Collot, Pilou, Kepon, Laurent Sleto, Rodinux…
|
|
|
|
### Cat Avatar Generator images
|
|
|
|
I am glad to use the beautiful artwork of David Revoy (http://www.peppercarrot.com).
|
|
* Graphics CC-BY http://creativecommons.org/licenses/by/4.0/
|
|
* Service: https://www.peppercarrot.com/extras/html/2016_cat-generator/
|
|
* Source: https://framagit.org/Deevad/cat-avatar-generator
|
|
* Originally inspired of the code for "MonsterID" by Andreas Gohr http://www.splitbrain.org/go/monsterid.
|
|
|
|
|
|
### Logo
|
|
|
|
Author: Christian Pierre MOMON <christian.momon@devinsy.fr>
|
|
|
|
License: Creative Commons CC BY-SA last version.
|
|
|
|
|
|
### Other stuff
|
|
|
|
License of other stuff: without any specific information, default license of images is CC BY-SA 3+.
|
|
|
|
|
|
## DOCUMENTATION
|
|
|
|
Read CONCEPT.md and ONTOLOGY.md files.
|
|
|
|
## Requirements
|
|
|
|
- Java 17
|
|
- Eclipse 4.29 (2023-09).
|
|
|
|
## INSTALL
|
|
|
|
Install Java on Debian:
|
|
|
|
```
|
|
apt-get install openjdk-17-jre-headless
|
|
```
|
|
|
|
Download the last release: `https://forge.devinsy.fr/statool/statoolinfos/releases` and unzip the package.
|
|
|
|
The `srv/` directory is a example of deployment tree:
|
|
|
|
```
|
|
srv/
|
|
srv/statoolinfos
|
|
srv/statoolinfos/bin
|
|
srv/statoolinfos/bin/refresh
|
|
srv/statoolinfos/bin/probe
|
|
srv/statoolinfos/bin/uptime
|
|
srv/statoolinfos/bin/statoolinfos.sh
|
|
srv/statoolinfos/bin/crawl
|
|
srv/statoolinfos/conf
|
|
srv/statoolinfos/conf/sample-probe-http.conf
|
|
srv/statoolinfos/conf/log4j2-default.properties
|
|
srv/statoolinfos/conf/sample-probe-minetest.conf
|
|
srv/statoolinfos/conf/sample-federation.conf
|
|
srv/statoolinfos/conf/sample-probe-qrcode.conf
|
|
srv/statoolinfos/cache
|
|
srv/statoolinfos/statoolinfos-cron.log
|
|
srv/statoolinfos/well-known
|
|
srv/statoolinfos/well-known/statoolinfos
|
|
srv/statoolinfos/well-known/statoolinfos/organization-template.properties
|
|
srv/statoolinfos/well-known/statoolinfos/services.properties
|
|
srv/statoolinfos/well-known/statoolinfos/BOM
|
|
srv/statoolinfos/well-known/statoolinfos/federation-template.properties
|
|
srv/statoolinfos/well-known/statoolinfos/service-template.properties
|
|
```
|
|
|
|
Then, move the `statoolinfos.jar` file in `/srv/statoolinfos/bin/`.
|
|
|
|
Create link:
|
|
|
|
```
|
|
cd /srv/statoolinfos/bin/ ; ln -s statoolinfos.sh statoolinfos
|
|
```
|
|
|
|
Edit script in `/srv/statoolinfos/bin/` to target your federation configuration file.
|
|
|
|
|
|
In case of bad language output, add these lines on top of cron file:
|
|
|
|
```
|
|
LANGUAGE=en_EN.UTF8
|
|
LC_ALL=en_EN.UTF-8
|
|
```
|
|
|
|
## Usage
|
|
|
|
```
|
|
Usage:
|
|
statoolinfos [ -h | -help | --help ]
|
|
statoolinfos [ -v | -version | --version ]
|
|
|
|
statoolinfos build <configurationfile> build property files from conf and input
|
|
statoolinfos clear <configurationfile> remove property files from conf
|
|
statoolinfos crawl <configurationfile> crawl all file from conf and input
|
|
statoolinfos format <fileordirectory> format property files in tiny way
|
|
s̶t̶a̶t̶o̶o̶l̶i̶n̶f̶o̶s̶ ̶h̶t̶m̶l̶i̶z̶e̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶<̶c̶o̶n̶f̶i̶g̶u̶r̶a̶t̶i̶o̶n̶f̶i̶l̶e̶>̶ ̶ ̶ ̶ ̶ ̶g̶e̶n̶e̶r̶a̶t̶e̶ ̶w̶e̶b̶ ̶p̶a̶g̶e̶s̶ ̶f̶r̶o̶m̶ ̶c̶o̶n̶f̶ REMOVED since splitweb
|
|
statoolinfos probe OPTION [<directory>|<configurationfile>] generate metrics files from conf
|
|
OPTION = [-full|-today|-previousday|-NN] with NN a day count
|
|
statoolinfos tagdate <fileordirectory> update the file.datetime file
|
|
statoolinfos uptime <configurationfile> update uptime journal
|
|
|
|
statoolinfos list file <logfilesorconfigfile> display http access log files
|
|
statoolinfos list log [-bot|-nobot] <logfilesorconfigfile> display http access log lines
|
|
statoolinfos list ip [-bot|-nobot] <logfilesorconfigfile> generate ip list from http log file
|
|
statoolinfos list ua [-bot|-nobot] <logfilesorconfigfile> generate user agent list from http log file
|
|
statoolinfos list visitor [-bot|-nobot] <logfilesorconfigfile> generate visitors (ip+ua) list from http log file
|
|
statoolinfos stat ip [-bot|-nobot] <logfilesorconfigfile> generate stats about ip from http log file
|
|
statoolinfos stat ua [-bot|-nobot] <logfilesorconfigfile> generate stats about user agent from http log file
|
|
statoolinfos stat visitor [-bot|-nobot] <logfilesorconfigfile> generate stats about visitor (ip+ua) from http log file
|
|
statoolinfos list errfile <logfilesorconfigfile> display http error log files
|
|
statoolinfos list errlog <logfilesorconfigfile> display http error log lines
|
|
```
|
|
|
|
## Simple federation configuration file
|
|
|
|
Example of `libre-service.eu.conf`:
|
|
|
|
```
|
|
# [Configuration] conf + organization + services + machines + metrics
|
|
conf.class=federation
|
|
conf.protocol=StatoolInfos-0.6.0
|
|
|
|
conf.crawl.input=https://www.libre-service.eu/.well-known/statoolinfos/libre-service.eu.properties
|
|
conf.crawl.cache=/srv/statoolinfos/cache/
|
|
|
|
```
|
|
|
|
The field `input` gives the federation properties URL file. Reminder, the StatoolInfos Protocol requires that the federation properties file is exposed on the Web.
|
|
|
|
The field `cache` gives local directory used as data cache.
|
|
|
|
|
|
## Crawl
|
|
|
|
The `statoolinfos crawl` command loads the federation configuration file, then downloads the federation properties file and download all files set in.
|
|
|
|
In case of use with StatoolInfosWeb, never forget to give read permissions to Tomcat server. So, create a dedicated script would help.
|
|
|
|
Example of script `/srv/statoolinfos/bin/crawl`:
|
|
|
|
```
|
|
#!/bin/bash
|
|
|
|
echo "========== CRAWL at $(date --iso-8601=seconds)"
|
|
cd $(dirname "$0")
|
|
./statoolinfos crawl ../conf/federation.conf
|
|
chmod -R g+r /srv/statoolinfos/cache
|
|
chgrp -R tomcat /srv/statoolinfos/cache
|
|
```
|
|
|
|
Example of cron configuration `/etc/cron.d/statoolinfos`:
|
|
|
|
```
|
|
4 * * * * root /srv/statoolinfos/bin/crawl >> /srv/statoolinfos/statoolinfos-cron.log
|
|
```
|
|
|
|
Adjust your frequency according to your need. In case of large federation, once an hour is a good compromise.
|
|
|
|
## Uptime
|
|
|
|
The `statoolinfos uptime` command is very useful to collect the uptime of services. It's just to share which services are down and since how many time. It is not a monitoring feature because an one hour check is enough for this. Actually, only website are checked.
|
|
|
|
In case of use with StatoolInfosWeb, never forget to give read permissions to Tomcat server. So, create a dedicated script would help.
|
|
|
|
Example of script `/srv/statoolinfos/bin/uptime`:
|
|
|
|
|
|
```
|
|
#!/bin/bash
|
|
|
|
echo "========== UPTIME at $(date --iso-8601=seconds)"
|
|
cd $(dirname "$0")
|
|
./statoolinfos uptime ../conf/federation.conf
|
|
chmod -R g+r /srv/statoolinfos/cache
|
|
chgrp -R tomcat /srv/statoolinfos/cache
|
|
```
|
|
|
|
Example of cron configuration `/etc/cron.d/statoolinfos`:
|
|
|
|
```
|
|
4 * * * * root /srv/statoolinfos/bin/uptime >> /srv/statoolinfos/statoolinfos-cron.log
|
|
```
|
|
|
|
One check per hour is a good idea. In the uptime page, the mouse over a yellow status icon show the list of the check results.
|
|
|
|
In case of crawl, uptime and probe cron on the same computer, you can have different cron frequencies. It is better to have only script for call all them:
|
|
|
|
```
|
|
#!/bin/bash
|
|
|
|
echo "========== REFRESH at $(date --iso-8601=seconds)"
|
|
cd $(dirname "$0")
|
|
./probe
|
|
./crawl
|
|
./uptime
|
|
#./htmlize
|
|
chmod -R g+r /srv/statoolinfos/cache
|
|
chgrp -R tomcat /srv/statoolinfos/cache
|
|
|
|
```
|
|
|
|
And so your cron can be like that:
|
|
|
|
```
|
|
4 0,12,23 * * * root /srv/statoolinfos/bin/refresh >> /srv/statoolinfos/statoolinfos-cron.log
|
|
4 1-11,13-22 * * * root /srv/statoolinfos/bin/uptime >> /srv/statoolinfos/statoolinfos-cron.log
|
|
```
|
|
|
|
|
|
## Htmlize command **removed**
|
|
|
|
The `statoolinfos htmlize` command is removed since 2024-07.
|
|
|
|
As a reminder, the htmlize command generated absolutely all the pages needed for a static website.
|
|
|
|
As the number of pages generated became too large (> 10000 with 100 organizations), the decision was made to separate the html valuation into a separate project, StatoolInfosWeb.
|
|
|
|
A dynamic web site allows to generate page only when it is necessary.
|
|
|
|
|
|
## Probe (Generate metrics files)
|
|
|
|
### Basics
|
|
The `statoolinfos probe` command computes data sources to generate metric properties files. In case of already existings file, the data are loaded before the computation to be updated.
|
|
|
|
In case of use with StatoolInfosWeb, never forget to give read permissions to Tomcat server. So, create a dedicated script would help.
|
|
|
|
Example of script `/srv/statoolinfos/bin/probe`:
|
|
|
|
```
|
|
#!/bin/bash
|
|
|
|
echo "========== PROBE at $(date --iso-8601=seconds)"
|
|
cd $(dirname "$0")
|
|
./statoolinfos probe -previousday ../conf/
|
|
chmod -R g+r /srv/statoolinfos/cache
|
|
chgrp -R tomcat /srv/statoolinfos/cache
|
|
```
|
|
|
|
Note: set a specific configuration file is not necessary, because you can have multiple service to probe. Only probe configuration files will be manage.
|
|
|
|
Example of cron configuration `/etc/cron.d/statoolinfos`:
|
|
|
|
```
|
|
4 * * * * root /srv/statoolinfos/bin/probe >> /srv/statoolinfos/statoolinfos-cron.log
|
|
```
|
|
|
|
### HTTP Server configuration
|
|
|
|
The probe feature generates properties files to expose on web. StatoolInfos protocol recommends to follow the well-known convention.
|
|
|
|
Example of `/etc/apache2/statoolinfos.conf` generic configuration file:
|
|
```
|
|
Alias "/.well-known/statoolinfos/" "/srv/statoolinfos/well-known/statoolinfos/"
|
|
<Directory "/srv/statoolinfos/well-known/statoolinfos">
|
|
Options +Indexes
|
|
Require all granted
|
|
</Directory>
|
|
```
|
|
|
|
Example of include directive in `/etc/apache2/sites-enabled/myservice.conf`:
|
|
```
|
|
<VirtualHost *:443>
|
|
[…]
|
|
Include statoolinfos.conf
|
|
[…]
|
|
</VirtualHost>
|
|
```
|
|
|
|
### Sample configuration file
|
|
|
|
Create a configuration file `/srv/statoolInfos/conf/foo.bar.org.conf`:
|
|
|
|
```
|
|
conf.probe.types=<metrictype1>, <metrictype2>
|
|
conf.probe.metrictype1param1.file=<value>
|
|
conf.probe.metrictype1param2.file=<value>
|
|
conf.probe.metrictype2param1.file=<value>
|
|
conf.probe.metrictype2param2.file=<value>
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
Generate the metric file for the first time:
|
|
|
|
```
|
|
/srv/statoolinfos/bin/statoolinfos probe -full /srv/statoolinfos/conf/foo.bar.org.conf
|
|
```
|
|
|
|
Create a cron file in `/etc/cron.d/statoolinfos` to update the metric file everyday:
|
|
```
|
|
4 0,12 * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/foo.bar.org.conf >> /srv/statoolinfos/cron.log
|
|
```
|
|
|
|
In case of several configuration files, just set the configuration directory (without wildcard):
|
|
```
|
|
1 0,12 * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/ >> /srv/statoolinfos/cron.log
|
|
```
|
|
|
|
Warning: in previous day mode, the metrics generated are overwrited for the last month, the last week and the last day. So, **six weeks in logs are required**.
|
|
|
|
|
|
### Etherpad metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Etherpad
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
|
|
conf.probe.httpaccesslog.pattern=
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
# This parameter requires real ip proxy setting and log output setting.
|
|
conf.probe.etherpad.logs=/var/log/etherpad/output.log
|
|
# Database URL. Example of the most frequent cases:
|
|
# jdbc:mariadb://localhost:1234/databasename
|
|
# jdbc:mysql://localhost:1234/databasename
|
|
# jdbc:postgresql://localhost:1234/databasename
|
|
# jdbc:sqlite:/foo/bar/databasename.sqlite
|
|
conf.probe.etherpad.database.url=
|
|
conf.probe.etherpad.database.user=
|
|
conf.probe.etherpad.database.password=
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### Framdadate metrics (coming soon)
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Framadate
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### Gitea metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Gitea
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
|
|
conf.probe.httpaccesslog.pattern=
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
conf.probe.gitea.data=/opt/gitea/data
|
|
conf.probe.gitea.api.url=https://forge.libre-service.eu/
|
|
conf.probe.gitea.token=b6598c616b1cd350b834258205da4e5e8b951005
|
|
# jdbc:mariadb://localhost:1234/databasename
|
|
# jdbc:mysql://localhost:1234/databasename
|
|
# jdbc:postgresql://localhost:1234/databasename
|
|
# jdbc:sqlite:/foo/bar/databasename.sqlite
|
|
conf.probe.gitea.database.url=
|
|
conf.probe.gitea.database.user=
|
|
conf.probe.gitea.database.password=
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### LibreQR metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=LibreQR
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
|
|
conf.probe.httpaccesslog.pattern=
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
conf.probe.libreqr.datafiles=/var/www/foo.bar.org/temp/
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### Minetest metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Minetest
|
|
conf.probe.minetest.logs=/home/cpm/Projets/StatoolInfos/EnvTest/minetest/minetest.log*
|
|
|
|
conf.probe.minetest.players.database.url=jdbc:postgresql://localhost:5432/minetestdb
|
|
conf.probe.minetest.players.database.user=minetestdba
|
|
conf.probe.minetest.players.database.password=XXXXXXXXXXX
|
|
|
|
conf.probe.minetest.worlds.database.url=jdbc:postgresql://localhost:5432/minetestdb
|
|
conf.probe.minetest.worlds.database.user=minetestdba
|
|
conf.probe.minetest.worlds.database.password=XXXXXXXXXXX
|
|
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
|
|
```
|
|
|
|
### Mumble metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Mumble
|
|
conf.probe.mumble.logs=/var/log/mumble-server/mumble-server.log*
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### Nextcloud metrics (coming soon)
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=Nextcloud
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
|
|
conf.probe.httpaccesslog.pattern=
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### PrivateBin metrics (partial)
|
|
|
|
Warning: works fine if database, image and comment options are disabled.
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=PrivateBin
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
|
|
conf.probe.httpaccesslog.pattern=
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
conf.probe.privatebin.data=/var/www/paste.libre-service.eu/data/
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
```
|
|
|
|
### Web metrics
|
|
|
|
Configuration template:
|
|
|
|
```
|
|
conf.probe.types=HttpAccessLog, HttpErrorLog
|
|
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*,/var/log/apache2/foo.bar.org-extraaccess.log*
|
|
conf.probe.httperrorlog.file=/var/log/apache2/foo.bar.org-error.log*,/var/log/apache2/foo.bar.org-extraerror.log*
|
|
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
|
|
|
|
# Custom access log pattern with Java regex.
|
|
# Default: ^(?<remoteAddress>[a-fA-F0-9\:\.]+) - (?<remoteUser>[^\[]+) \[(?<time>[^\]]+)\] "(?<request>([^\"]|\")*)" (?<status>\d+) (?<bodyBytesSent>\d+) "(?<referer>([^\"]|\")*)" "(?<userAgent>([^\"]|\")*)".*$
|
|
conf.probe.httpaccesslog.pattern=
|
|
|
|
# Custom datetime access log pattern (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns). The language ID is mandatory.
|
|
# Default: dd/MMM/yyyy:HH:mm:ss Z|EN
|
|
conf.prob.httpaccesslog.datetimepattern=
|
|
|
|
# Filter request with Java regex (without the initial '^' and without the final '$', see https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html).
|
|
# Example: pathfilter=/.*well-known.*
|
|
conf.probe.httpaccesslog.pathfilter=
|
|
|
|
# Custom error log pattern with Java regex. By default, automatic detection.
|
|
# Default Apache: ^\[(?<time>[^\]]+)\]\s\[(?<level>[^\]]*)\]\s(?<message>.*)$
|
|
# Default NGinx: ^(?<time>\S+\s\S+)\s\[(?<level>[^\]]*)\]\s(?<message>.*)$
|
|
conf.probe.httperrorlog.pattern=
|
|
|
|
# Custom datetime access log pattern (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns). The language ID is mandatory.
|
|
# Default: dd/MMM/yyyy:HH:mm:ss Z|EN
|
|
conf.prob.httperrorlog.datetimepattern=
|
|
```
|
|
|
|
Warning: to have month metric values, you have to set **retention log to 7 weeks or 50 days** in Nginx or Apache logrotate configuration.
|
|
|
|
For Nginx, éditer le fichier `/etc/logrotate.d/nginx` :
|
|
|
|
```
|
|
/var/log/nginx/*.log {
|
|
weekly
|
|
missingok
|
|
rotate 7
|
|
```
|
|
|
|
For Apache, éditer le fichier `/etc/logrotate.d/apache2` :
|
|
|
|
```
|
|
/var/log/apache2/*.log {
|
|
weekly
|
|
missingok
|
|
rotate 7
|
|
```
|
|
|
|
### Cron all task
|
|
|
|
A situation you will encounter is the need to do several tasks from the same machine. In this case, avoid multiple cron in the same time. Put all in Bash script and call it with only one cron.
|
|
|
|
Example:
|
|
|
|
```
|
|
4 0,12,23 * * * root /srv/statoolinfos/bin/refresh >> /srv/statoolinfos/statoolinfos-cron.log
|
|
4 1-11,13-22 * * * root /srv/statoolinfos/bin/statoolinfos uptime /srv/statoolinfos/conf/libre-service.eu.conf >> /srv/statoolinfos/statoolinfos-cron.log
|
|
|
|
```
|
|
|