statool/statoolinfos
statool/statoolinfos
1
0
Fork 0
Code Issues Pull requests Releases 8 Activity
statoolinfos/README.md

307 lines
11 KiB
Markdown
Raw Blame History

# StatoolInfos
StatoolInfos is a simple statistic tool software.
## LICENSE
StatoolInfos is released under the GNU AGPL+ license. Enjoy!
Authors:
* Christian Pierre MOMON <christian.momon@devinsy.fr>
Contributors : Angie, MrFlos, Quentin Duchemin, Fabrice61, Thomas TConstans, labecasse, Antoine Jaba, setop, Jérémy Collot, Pilou, Kepon, Laurent Sleto, Rodinux…
### Cat Avatar Generator images
I am glad to use the beautiful artwork of David Revoy (http://www.peppercarrot.com).
* Graphics CC-BY http://creativecommons.org/licenses/by/4.0/
* Service: https://www.peppercarrot.com/extras/html/2016_cat-generator/
* Source: https://framagit.org/Deevad/cat-avatar-generator
* Originally inspired of the code for "MonsterID" by Andreas Gohr http://www.splitbrain.org/go/monsterid.
### Logo
Author: Christian Pierre MOMON <christian.momon@devinsy.fr>
License: Creative Commons CC BY-SA last version.
### Other stuff
License of other stuff: without any specific information, default license of images is CC BY-SA 3+.
## DOCUMENTATION
Read CONCEPT.md and ONTOLOGY.md files.
## Requirements
- Java 17
- Eclipse 4.29 (2023-09).
## INSTALL
Install Java :
```
apt-get install openjdk-17-jre-headless
```
Download the last release: https://forge.devinsy.fr/statool/statoolinfos/releases
Unzip the package and move the `statoolinfos.jar` and `statoolinfo.sh` files in `/srv/statoolinfos/bin/`.
Create link:
```
cd /srv/statoolinfos/bin/ ; ln -s statoolinfos.sh statoolinfos
```
## Usage
```
Usage:
statoolinfos [ -h | -help | --help ]
statoolinfos [ -v | -version | --version ]
statoolinfos build <configurationfile> build property files from conf and input
statoolinfos clear <configurationfile> remove property files from conf
statoolinfos crawl <configurationfile> crawl all file from conf and input
statoolinfos format <fileordirectory> format property files in tiny way
s̶t̶a̶t̶o̶o̶l̶i̶n̶f̶o̶s̶ ̶h̶t̶m̶l̶i̶z̶e̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶<̶c̶o̶n̶f̶i̶g̶u̶r̶a̶t̶i̶o̶n̶f̶i̶l̶e̶>̶ ̶ ̶ ̶ ̶ ̶g̶e̶n̶e̶r̶a̶t̶e̶ ̶w̶e̶b̶ ̶p̶a̶g̶e̶s̶ ̶f̶r̶o̶m̶ ̶c̶o̶n̶f̶ REMOVED since splitweb
statoolinfos probe OPTION [<directory>|<configurationfile>] generate metrics files from conf
OPTION = [-full|-today|-previousday|-NN] with NN a day count
statoolinfos tagdate <fileordirectory> update the file.datetime file
statoolinfos uptime <configurationfile> update uptime journal
statoolinfos list file <logfilesorconfigfile> display http access log files
statoolinfos list log [-bot|-nobot] <logfilesorconfigfile> display http access log lines
statoolinfos list ip [-bot|-nobot] <logfilesorconfigfile> generate ip list from http log file
statoolinfos list ua [-bot|-nobot] <logfilesorconfigfile> generate user agent list from http log file
statoolinfos list visitor [-bot|-nobot] <logfilesorconfigfile> generate visitors (ip+ua) list from http log file
statoolinfos stat ip [-bot|-nobot] <logfilesorconfigfile> generate stats about ip from http log file
statoolinfos stat ua [-bot|-nobot] <logfilesorconfigfile> generate stats about user agent from http log file
statoolinfos stat visitor [-bot|-nobot] <logfilesorconfigfile> generate stats about visitor (ip+ua) from http log file
statoolinfos list errfile <logfilesorconfigfile> display http error log files
statoolinfos list errlog <logfilesorconfigfile> display http error log lines
```
### htmlize removed
The htmlize command is removed since 2024-07.
As a reminder, the htmlize command generated absolutely all the pages needed for a static site.
As the number of pages generated became too large (> 10000 with 100 organizations), the decision was made to separate the html valuation into a separate project, StatoolInfosWeb.
A dynamic web site allows to generate page only when it is necessary.
## Generate metrics files
### Basics
Create a configuration file `/srv/statoolInfos/conf/foo.bar.org.conf`:
```
conf.probe.types=<metrictype1>, <metrictype2>
conf.probe.metrictype1param1.file=<value>
conf.probe.metrictype1param2.file=<value>
conf.probe.metrictype2param1.file=<value>
conf.probe.metrictype2param2.file=<value>
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
Generate the metric file for the first time:
```
/srv/statoolinfos/bin/statoolinfos probe -full /srv/statoolinfos/conf/foo.bar.org.conf
```
Create a cron file in `/etc/cron.d/statoolinfos` to update the metric file everyday:
```
1 0,12 * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/foo.bar.org.conf >> /srv/statoolinfos/cron.log
```
In case of several configuration files, just set the configuration directory (without wildcard):
```
1 0,12 * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/ >> /srv/statoolinfos/cron.log
```
Warning: in previous day mode, the metrics generated are overwrited for the last month, the last week and the last day. So, **six weeks in logs are required**.
### Etherpad metrics
Configuration template:
```
conf.probe.types=Etherpad
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
# This parameter requires real ip proxy setting and log output setting.
conf.probe.etherpad.logs=/var/log/etherpad/output.log
# Database URL. Example of the most frequent cases:
# jdbc:mariadb://localhost:1234/databasename
# jdbc:mysql://localhost:1234/databasename
# jdbc:postgresql://localhost:1234/databasename
# jdbc:sqlite:/foo/bar/databasename.sqlite
conf.probe.etherpad.database.url=
conf.probe.etherpad.database.user=
conf.probe.etherpad.database.password=
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
### Framdadate metrics (coming soon)
Configuration template:
```
conf.probe.types=Framadate
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
### Gitea metrics
Configuration template:
```
conf.probe.types=Gitea
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.gitea.data=/opt/gitea/data
conf.probe.gitea.api.url=https://forge.libre-service.eu/
conf.probe.gitea.token=b6598c616b1cd350b834258205da4e5e8b951005
# jdbc:mariadb://localhost:1234/databasename
# jdbc:mysql://localhost:1234/databasename
# jdbc:postgresql://localhost:1234/databasename
# jdbc:sqlite:/foo/bar/databasename.sqlite
conf.probe.gitea.database.url=
conf.probe.gitea.database.user=
conf.probe.gitea.database.password=
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
### LibreQR metrics
Configuration template:
```
conf.probe.types=LibreQR
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.libreqr.datafiles=/var/www/foo.bar.org/temp/
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
### Minetest metrics
Configuration template:
```
conf.probe.types=Minetest
conf.probe.minetest.logs=/home/cpm/Projets/StatoolInfos/EnvTest/minetest/minetest.log*
conf.probe.minetest.players.database.url=jdbc:postgresql://localhost:5432/minetestdb
conf.probe.minetest.players.database.user=minetestdba
conf.probe.minetest.players.database.password=XXXXXXXXXXX
conf.probe.minetest.worlds.database.url=jdbc:postgresql://localhost:5432/minetestdb
conf.probe.minetest.worlds.database.user=minetestdba
conf.probe.minetest.worlds.database.password=XXXXXXXXXXX
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
### Mumble metrics
Configuration template:
```
conf.probe.types=Mumble
conf.probe.mumble.logs=/var/log/mumble-server/mumble-server.log*
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
### Nextcloud metrics (coming soon)
Configuration template:
```
conf.probe.types=Nextcloud
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
### PrivateBin metrics (partial)
Warning: works fine if database, image and comment options are disabled.
Configuration template:
```
conf.probe.types=PrivateBin
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.privatebin.data=/var/www/paste.libre-service.eu/data/
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```
### Web metrics
Configuration template:
```
conf.probe.types=HttpAccessLog, HttpErrorLog
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*,/var/log/apache2/foo.bar.org-extraaccess.log*
conf.probe.httperrorlog.file=/var/log/apache2/foo.bar.org-error.log*,/var/log/apache2/foo.bar.org-extraerror.log*
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
# Custom access log pattern with Java regex.
# Default: ^(?<remoteAddress>[a-fA-F0-9\:\.]+) - (?<remoteUser>[^\[]+) \[(?<time>[^\]]+)\] "(?<request>([^\"]|\")*)" (?<status>\d+) (?<bodyBytesSent>\d+) "(?<referer>([^\"]|\")*)" "(?<userAgent>([^\"]|\")*)".*$
conf.probe.httpaccesslog.pattern=
# Custom datetime access log pattern (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns).
# Default: dd/MMM/yyyy:HH:mm:ss Z|EN
conf.prob.httpaccesslog.datetimepattern=
# Filter request with Java regex (without the initial '^' and without the final '$', see https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html).
# Example: pathfilter=/.*well-known.*
conf.probe.httpaccesslog.pathfilter=
# Custom error log pattern with Java regex.
# Default Apache: ^\[(?<time>[^\]]+)\]\s\[(?<level>[^\]]*)\]\s(?<message>.*)$
# Default NGinx: ^(?<time>\S+\s\S+)\s\[(?<level>[^\]]*)\]\s(?<message>.*)$
conf.probe.httperrorlog.pattern=
# Custom datetime access log pattern (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns).
# Default: dd/MMM/yyyy:HH:mm:ss Z|EN
conf.prob.httperrorlog.datetimepattern=
```
Warning: to have month metric values, you have to set **retention log to 7 weeks or 50 days** in Nginx or Apache logrotate configuration.
For Nginx, éditer le fichier `/etc/logrotate.d/nginx` :
```
/var/log/nginx/*.log {
weekly
missingok
rotate 7
```
For Apache, éditer le fichier `/etc/logrotate.d/apache2` :
```
/var/log/apache2/*.log {
weekly
missingok
rotate 7
```
Reference in a new issue View git blame Copy permalink