statool/statoolinfos
statool/statoolinfos
1
0
Fork 0
Code Issues Pull requests Releases 8 Activity
statoolinfos/README.md

366 lines
13 KiB
Markdown
Raw Blame History

# StatoolInfos
StatoolInfos is a simple statistic tool software.
## LICENSE
StatoolInfos is released under the GNU AGPL+ license. Enjoy!
Authors:
* Christian Pierre MOMON <christian.momon@devinsy.fr>
Contributors : Angie, MrFlos, Quentin Duchemin, Fabrice61, Thomas TConstans, labecasse, Antoine Jaba, setop, Jérémy Collot, Pilou, Kepon, Laurent Sleto, Rodinux…
### Cat Avatar Generator images
I am glad to use the beautiful artwork of David Revoy (http://www.peppercarrot.com).
* Graphics CC-BY http://creativecommons.org/licenses/by/4.0/
* Service: https://www.peppercarrot.com/extras/html/2016_cat-generator/
* Source: https://framagit.org/Deevad/cat-avatar-generator
* Originally inspired of the code for "MonsterID" by Andreas Gohr http://www.splitbrain.org/go/monsterid.
### Logo
Author: Christian Pierre MOMON <christian.momon@devinsy.fr>
License: Creative Commons CC BY-SA last version.
### Other stuff
License of other stuff: without any specific information, default license of images is CC BY-SA 3+.
## DOCUMENTATION
Read CONCEPT.md and ONTOLOGY.md files.
## Requirements
- Java 17
- Eclipse 4.29 (2023-09).
## INSTALL
Install Java :
```
apt-get install openjdk-17-jre-headless
```
Download the last release: https://forge.devinsy.fr/statool/statoolinfos/releases
Unzip the package and move the `statoolinfos.jar` and `statoolinfo.sh` files in `/srv/statoolinfos/bin/`.
Create link:
```
cd /srv/statoolinfos/bin/ ; ln -s statoolinfos.sh statoolinfos
```
## Usage
```
Usage:
statoolinfos [ -h | -help | --help ]
statoolinfos [ -v | -version | --version ]
statoolinfos build <configurationfile> build property files from conf and input
statoolinfos clear <configurationfile> remove property files from conf
statoolinfos crawl <configurationfile> crawl all file from conf and input
statoolinfos format <fileordirectory> format property files in tiny way
s̶t̶a̶t̶o̶o̶l̶i̶n̶f̶o̶s̶ ̶h̶t̶m̶l̶i̶z̶e̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶<̶c̶o̶n̶f̶i̶g̶u̶r̶a̶t̶i̶o̶n̶f̶i̶l̶e̶>̶ ̶ ̶ ̶ ̶ ̶g̶e̶n̶e̶r̶a̶t̶e̶ ̶w̶e̶b̶ ̶p̶a̶g̶e̶s̶ ̶f̶r̶o̶m̶ ̶c̶o̶n̶f̶ REMOVED since splitweb
statoolinfos probe OPTION [<directory>|<configurationfile>] generate metrics files from conf
OPTION = [-full|-today|-previousday|-NN] with NN a day count
statoolinfos tagdate <fileordirectory> update the file.datetime file
statoolinfos uptime <configurationfile> update uptime journal
statoolinfos list file <logfilesorconfigfile> display http access log files
statoolinfos list log [-bot|-nobot] <logfilesorconfigfile> display http access log lines
statoolinfos list ip [-bot|-nobot] <logfilesorconfigfile> generate ip list from http log file
statoolinfos list ua [-bot|-nobot] <logfilesorconfigfile> generate user agent list from http log file
statoolinfos list visitor [-bot|-nobot] <logfilesorconfigfile> generate visitors (ip+ua) list from http log file
statoolinfos stat ip [-bot|-nobot] <logfilesorconfigfile> generate stats about ip from http log file
statoolinfos stat ua [-bot|-nobot] <logfilesorconfigfile> generate stats about user agent from http log file
statoolinfos stat visitor [-bot|-nobot] <logfilesorconfigfile> generate stats about visitor (ip+ua) from http log file
statoolinfos list errfile <logfilesorconfigfile> display http error log files
statoolinfos list errlog <logfilesorconfigfile> display http error log lines
```
### Simple configuration file
Example of `libre-service.eu.conf`:
```
# [Configuration] conf + organization + services + machines + metrics
conf.class=federation
conf.protocol=StatoolInfos-0.6.0
conf.crawl.input=https://www.libre-service.eu/.well-known/statoolinfos/libre-service.eu.properties
conf.crawl.cache=/srv/statoolinfos/cache/
```
The field `input` gives the federation properties URL file. Reminder, the StatoolInfos Protocol requires that the federation properties file is exposed on the Web.
The field `cache` gives local directory used as data cache.
## Crawl
Example of cron configuration:
```
55 * * * * root /srv/statoolinfos/bin/statoolinfos crawl /srv/statoolinfos/conf/libre-service.eu.conf >> /srv/statoolinfos/statoolinfos-cron.log
```
## Uptime
This command is very useful to collect the uptime of services. It's just to share which services are down and since how many time. It is not a monitoring feature because an one hour check is enough for this. Actually, only website are checked.
Example of cron configuration:
```
4 * * * * root /srv/statoolinfos/bin/statoolinfos uptime /srv/statoolinfos/conf/libre-service.eu.conf >> /srv/statoolinfos/statoolinfos-cron.log
```
One check per hour is a good idea. In the uptime page, the mouse over a yellow status icon show the list of the check results.
## Htmlize removed
The htmlize command is removed since 2024-07.
As a reminder, the htmlize command generated absolutely all the pages needed for a static website.
As the number of pages generated became too large (> 10000 with 100 organizations), the decision was made to separate the html valuation into a separate project, StatoolInfosWeb.
A dynamic web site allows to generate page only when it is necessary.
## Probe (Generate metrics files)
### Basics
Create a configuration file `/srv/statoolInfos/conf/foo.bar.org.conf`:
```
conf.probe.types=<metrictype1>, <metrictype2>
conf.probe.metrictype1param1.file=<value>
conf.probe.metrictype1param2.file=<value>
conf.probe.metrictype2param1.file=<value>
conf.probe.metrictype2param2.file=<value>
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
Generate the metric file for the first time:
```
/srv/statoolinfos/bin/statoolinfos probe -full /srv/statoolinfos/conf/foo.bar.org.conf
```
Create a cron file in `/etc/cron.d/statoolinfos` to update the metric file everyday:
```
1 0,12 * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/foo.bar.org.conf >> /srv/statoolinfos/cron.log
```
In case of several configuration files, just set the configuration directory (without wildcard):
```
1 0,12 * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/ >> /srv/statoolinfos/cron.log
```
Warning: in previous day mode, the metrics generated are overwrited for the last month, the last week and the last day. So, **six weeks in logs are required**.
### Etherpad metrics
Configuration template:
```
conf.probe.types=Etherpad
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
# This parameter requires real ip proxy setting and log output setting.
conf.probe.etherpad.logs=/var/log/etherpad/output.log
# Database URL. Example of the most frequent cases:
# jdbc:mariadb://localhost:1234/databasename
# jdbc:mysql://localhost:1234/databasename
# jdbc:postgresql://localhost:1234/databasename
# jdbc:sqlite:/foo/bar/databasename.sqlite
conf.probe.etherpad.database.url=
conf.probe.etherpad.database.user=
conf.probe.etherpad.database.password=
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
### Framdadate metrics (coming soon)
Configuration template:
```
conf.probe.types=Framadate
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
### Gitea metrics
Configuration template:
```
conf.probe.types=Gitea
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.gitea.data=/opt/gitea/data
conf.probe.gitea.api.url=https://forge.libre-service.eu/
conf.probe.gitea.token=b6598c616b1cd350b834258205da4e5e8b951005
# jdbc:mariadb://localhost:1234/databasename
# jdbc:mysql://localhost:1234/databasename
# jdbc:postgresql://localhost:1234/databasename
# jdbc:sqlite:/foo/bar/databasename.sqlite
conf.probe.gitea.database.url=
conf.probe.gitea.database.user=
conf.probe.gitea.database.password=
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
### LibreQR metrics
Configuration template:
```
conf.probe.types=LibreQR
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.libreqr.datafiles=/var/www/foo.bar.org/temp/
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
### Minetest metrics
Configuration template:
```
conf.probe.types=Minetest
conf.probe.minetest.logs=/home/cpm/Projets/StatoolInfos/EnvTest/minetest/minetest.log*
conf.probe.minetest.players.database.url=jdbc:postgresql://localhost:5432/minetestdb
conf.probe.minetest.players.database.user=minetestdba
conf.probe.minetest.players.database.password=XXXXXXXXXXX
conf.probe.minetest.worlds.database.url=jdbc:postgresql://localhost:5432/minetestdb
conf.probe.minetest.worlds.database.user=minetestdba
conf.probe.minetest.worlds.database.password=XXXXXXXXXXX
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
### Mumble metrics
Configuration template:
```
conf.probe.types=Mumble
conf.probe.mumble.logs=/var/log/mumble-server/mumble-server.log*
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
### Nextcloud metrics (coming soon)
Configuration template:
```
conf.probe.types=Nextcloud
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
### PrivateBin metrics (partial)
Warning: works fine if database, image and comment options are disabled.
Configuration template:
```
conf.probe.types=PrivateBin
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.privatebin.data=/var/www/paste.libre-service.eu/data/
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
### Web metrics
Configuration template:
```
conf.probe.types=HttpAccessLog, HttpErrorLog
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*,/var/log/apache2/foo.bar.org-extraaccess.log*
conf.probe.httperrorlog.file=/var/log/apache2/foo.bar.org-error.log*,/var/log/apache2/foo.bar.org-extraerror.log*
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
# Custom access log pattern with Java regex.
# Default: ^(?<remoteAddress>[a-fA-F0-9\:\.]+) - (?<remoteUser>[^\[]+) \[(?<time>[^\]]+)\] "(?<request>([^\"]|\")*)" (?<status>\d+) (?<bodyBytesSent>\d+) "(?<referer>([^\"]|\")*)" "(?<userAgent>([^\"]|\")*)".*$
conf.probe.httpaccesslog.pattern=
# Custom datetime access log pattern (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns). The language ID is mandatory.
# Default: dd/MMM/yyyy:HH:mm:ss Z|EN
conf.prob.httpaccesslog.datetimepattern=
# Filter request with Java regex (without the initial '^' and without the final '$', see https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html).
# Example: pathfilter=/.*well-known.*
conf.probe.httpaccesslog.pathfilter=
# Custom error log pattern with Java regex. By default, automatic detection.
# Default Apache: ^\[(?<time>[^\]]+)\]\s\[(?<level>[^\]]*)\]\s(?<message>.*)$
# Default NGinx: ^(?<time>\S+\s\S+)\s\[(?<level>[^\]]*)\]\s(?<message>.*)$
conf.probe.httperrorlog.pattern=
# Custom datetime access log pattern (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns). The language ID is mandatory.
# Default: dd/MMM/yyyy:HH:mm:ss Z|EN
conf.prob.httperrorlog.datetimepattern=
```
Warning: to have month metric values, you have to set **retention log to 7 weeks or 50 days** in Nginx or Apache logrotate configuration.
For Nginx, éditer le fichier `/etc/logrotate.d/nginx` :
```
/var/log/nginx/*.log {
weekly
missingok
rotate 7
```
For Apache, éditer le fichier `/etc/logrotate.d/apache2` :
```
/var/log/apache2/*.log {
weekly
missingok
rotate 7
```
### Cron all task
A situation you will encounter is the need to do several tasks from the same machine. In this case, avoid multiple cron in the same time. Put all in Bash script and call it with only one cron.
Example:
```
4 0,12,23 * * * root /srv/statoolinfos/bin/refresh >> /srv/statoolinfos/statoolinfos-cron.log
4 1-11,13-22 * * * root /srv/statoolinfos/bin/statoolinfos uptime /srv/statoolinfos/conf/libre-service.eu.conf >> /srv/statoolinfos/statoolinfos-cron.log
```
Reference in a new issue View git blame Copy permalink