statoolinfos/README.md

# StatoolInfos

StatoolInfos is a simple statistic tool software.


## LICENSE

StatoolInfos is released under the GNU AGPL+ license. Enjoy!

Authors:
* Christian Pierre MOMON <christian.momon@devinsy.fr>

Contributors : Angie, MrFlos, Quentin Duchemin, Fabrice61, Thomas TConstans, labecasse, Antoine Jaba, setop, Jérémy Collot, Pilou, Kepon, Laurent Sleto, Rodinux…

### Cat Avatar Generator images

I am glad to use the beautiful artwork of David Revoy (http://www.peppercarrot.com).
* Graphics CC-BY http://creativecommons.org/licenses/by/4.0/
* Service: https://www.peppercarrot.com/extras/html/2016_cat-generator/
* Source: https://framagit.org/Deevad/cat-avatar-generator
* Originally inspired of the code for "MonsterID" by Andreas Gohr http://www.splitbrain.org/go/monsterid.

### Circle-icons

I am glad to use the generous artwork of ElegantThemes.com :

* https://www.elegantthemes.com/blog/freebie-of-the-week/beautiful-flat-icons-for-free
* "These icons are completely free and Open Source under the GPL".


### Logo

Author: Christian Pierre MOMON <christian.momon@devinsy.fr>

License: Creative Commons CC BY-SA last version.


### Other stuff

License of other stuff:
* csv-logo.svg, json-logo.svg, ods-icon.svg:
  * Author: Christian P. Momon
  * License: CC BY-SA 3+.
* other: without any specific information, default license of images is CC BY-SA 3+.


## DOCUMENTATION

Read CONCEPT.md and ONTOLOGY.md files.

## Requirements

- Java 11
- Eclipse 4.16 (202006).

## INSTALL

Install Java :

```
apt-get install openjdk-11-jre-headless
```

Download the last release: https://forge.devinsy.fr/devinsy/statoolinfos/releases

Unzip the package and move the `statoolinfos.jar` and `statoolinfo.sh` files in `/srv/statoolinfos/bin/`.

Create link:

```
cd /srv/statoolinfos/bin/ ; ln -s statoolinfos.sh statoolinfos
```


## Usage

```
Usage:
    statoolinfos [ -h | -help | --help ]
    statoolinfos [ -v | -version | --version ]

    statoolinfos build                       <configurationfile>     build property files from conf and input
    statoolinfos clear                       <configurationfile>     remove property files from conf
    statoolinfos crawl                       <configurationfile>     crawl all file from conf and input
    statoolinfos format                      <fileordirectory>       format property files in tiny way
    statoolinfos htmlize                     <configurationfile>     generate web pages from conf
    statoolinfos probe   OPTION [<directory>|<configurationfile>]    generate metrics files from conf
           OPTION = [-full|-today|-previousday|-NN] with NN a day count
    statoolinfos tagdate                     <fileordirectory>       update the file.datetime file
    statoolinfos uptime                      <configurationfile>     update uptime journal

    statoolinfos list file                   <logfilesorconfigfile>  display http access log files
    statoolinfos list log      [-bot|-nobot] <logfilesorconfigfile>  display http access log lines
    statoolinfos list ip       [-bot|-nobot] <logfilesorconfigfile>  generate ip list from http log file
    statoolinfos list ua       [-bot|-nobot] <logfilesorconfigfile>  generate user agent list from http log file
    statoolinfos list visitor  [-bot|-nobot] <logfilesorconfigfile>  generate visitors (ip+ua) list from http log file
    statoolinfos stat ip       [-bot|-nobot] <logfilesorconfigfile>  generate stats about ip from http log file
    statoolinfos stat ua       [-bot|-nobot] <logfilesorconfigfile>  generate stats about user agent from http log file
    statoolinfos stat visitor  [-bot|-nobot] <logfilesorconfigfile>  generate stats about visitor (ip+ua) from http log file
    statoolinfos list errfile                <logfilesorconfigfile>  display http error log files
    statoolinfos list errlog                 <logfilesorconfigfile>  display http error log lines
```


## Generate metrics files

### Basics

Create a configuration file `/srv/statoolInfos/conf/foo.bar.org.conf`:

```
conf.probe.types=<metrictype1>, <metrictype2>
conf.probe.metrictype1param1.file=<value>
conf.probe.metrictype1param2.file=<value>
conf.probe.metrictype2param1.file=<value>
conf.probe.metrictype2param2.file=<value>
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```

Generate the metric file for the first time:

```
/srv/statoolinfos/bin/statoolinfos probe -full /srv/statoolinfos/conf/foo.bar.org.conf
```

Create a cron file in `/etc/cron.d/statoolinfos` to update the metric file everyday:
```
1 0,12 * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/foo.bar.org.conf >> /srv/statoolinfos/cron.log
```

In case of several configuration files, just set the configuration directory (without wildcard):
```
1 0,12 * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/ >> /srv/statoolinfos/cron.log
```

Warning: in previous day mode, the metrics generated are overwrited for the last month, the last week and the last day. So, **six weeks in logs are required**.


### Etherpad metrics

Configuration template:

```
conf.probe.types=Etherpad
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
# This parameter requires real ip proxy setting and log output setting.
conf.probe.etherpad.logs=/var/log/etherpad/output.log
# Database URL. Example of the most frequent cases:
# jdbc:mariadb://localhost:1234/databasename
# jdbc:mysql://localhost:1234/databasename
# jdbc:postgresql://localhost:1234/databasename
# jdbc:sqlite:/foo/bar/databasename.sqlite
conf.probe.etherpad.database.url=
conf.probe.etherpad.database.user=
conf.probe.etherpad.database.password=
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```

### Framdadate metrics (coming soon)

Configuration template:

```
conf.probe.types=Framadate
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```

### Gitea metrics

Configuration template:

```
conf.probe.types=Gitea
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.gitea.data=/opt/gitea/data
conf.probe.gitea.api.url=https://forge.libre-service.eu/
conf.probe.gitea.token=b6598c616b1cd350b834258205da4e5e8b951005
# jdbc:mariadb://localhost:1234/databasename
# jdbc:mysql://localhost:1234/databasename
# jdbc:postgresql://localhost:1234/databasename
# jdbc:sqlite:/foo/bar/databasename.sqlite
conf.probe.gitea.database.url=
conf.probe.gitea.database.user=
conf.probe.gitea.database.password=
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```

### GSL metrics

Configuration template:

```
conf.probe.types=GSL
conf.probe.gsl.stats=/foo/bar/stats.properties
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```

### LibreQR metrics

Configuration template:

```
conf.probe.types=LibreQR
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.libreqr.datafiles=/var/www/foo.bar.org/temp/
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```

### Minetest metrics

Configuration template:

```
conf.probe.types=Minetest
conf.probe.minetest.logs=/home/cpm/Projets/StatoolInfos/EnvTest/minetest/minetest.log*

conf.probe.minetest.players.database.url=jdbc:postgresql://localhost:5432/minetestdb
conf.probe.minetest.players.database.user=minetestdba
conf.probe.minetest.players.database.password=XXXXXXXXXXX

conf.probe.minetest.worlds.database.url=jdbc:postgresql://localhost:5432/minetestdb
conf.probe.minetest.worlds.database.user=minetestdba
conf.probe.minetest.worlds.database.password=XXXXXXXXXXX

conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

```

### Mumble metrics

Configuration template:

```
conf.probe.types=Mumble
conf.probe.mumble.logs=/var/log/mumble-server/mumble-server.log*
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```

### Nextcloud metrics (coming soon)

Configuration template:

```
conf.probe.types=Nextcloud
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```

### PrivateBin metrics (partial)

Warning: works fine if database, image and comment options are disabled.

Configuration template:

```
conf.probe.types=PrivateBin
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.privatebin.data=/var/www/paste.libre-service.eu/data/
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties
```

### Web metrics

Configuration template:

```
conf.probe.types=HttpAccessLog, HttpErrorLog
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*,/var/log/apache2/foo.bar.org-extraaccess.log*
conf.probe.httperrorlog.file=/var/log/apache2/foo.bar.org-error.log*,/var/log/apache2/foo.bar.org-extraerror.log*
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

# Custom access log pattern with Java regex.
# Default: ^(?<remoteAddress>[a-fA-F0-9\:\.]+) - (?<remoteUser>[^\[]+) \[(?<time>[^\]]+)\] "(?<request>([^\"]|\")*)" (?<status>\d+) (?<bodyBytesSent>\d+) "(?<referer>([^\"]|\")*)" "(?<userAgent>([^\"]|\")*)".*$
conf.probe.httpaccesslog.pattern=

# Custom datetime access log pattern (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns).
# Default: dd/MMM/yyyy:HH:mm:ss Z|EN
conf.prob.httpaccesslog.datetimepattern=

# Filter request with Java regex (without the initial '^' and without the final '$', see https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html).
# Example: pathfilter=/.*well-known.*
conf.probe.httpaccesslog.pathfilter=

# Custom error log pattern with Java regex.
# Default Apache: ^\[(?<time>[^\]]+)\]\s\[(?<level>[^\]]*)\]\s(?<message>.*)$
# Default NGinx:  ^(?<time>\S+\s\S+)\s\[(?<level>[^\]]*)\]\s(?<message>.*)$
conf.probe.httperrorlog.pattern=

# Custom datetime access log pattern (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns).
# Default: dd/MMM/yyyy:HH:mm:ss Z|EN
conf.prob.httperrorlog.datetimepattern=
```

Warning: to have month metric values, you have to set **retention log to 7 weeks or 50 days** in Nginx or Apache logrotate configuration.

For Nginx, éditer le fichier `/etc/logrotate.d/nginx` :
```
/var/log/nginx/*.log {
    weekly
    missingok
    rotate 7
```

For Apache, éditer le fichier `/etc/logrotate.d/apache2` :
```
/var/log/apache2/*.log {
    weekly
    missingok
    rotate 7
```