StatoolInfos

StatoolInfos is a simple statistic tool software.

LICENSE

StatoolInfos is released under the GNU AGPL+ license. Enjoy!

Authors:

• Christian Pierre MOMON christian.momon@devinsy.fr

Contributors : Angie, MrFlos, Quentin Duchemin, Fabrice61, Thomas TConstans, labecasse, Antoine Jaba, setop, Jérémy Collot, Pilou, Kepon, Laurent Sleto, Rodinux…

Cat Avatar Generator images

I am glad to use the beautiful artwork of David Revoy (http://www.peppercarrot.com).

• Graphics CC-BY http://creativecommons.org/licenses/by/4.0/
• Service: https://www.peppercarrot.com/extras/html/2016_cat-generator/
• Source: https://framagit.org/Deevad/cat-avatar-generator
• Originally inspired of the code for "MonsterID" by Andreas Gohr http://www.splitbrain.org/go/monsterid.

Circle-icons

I am glad to use the generous artwork of ElegantThemes.com :

• https://www.elegantthemes.com/blog/freebie-of-the-week/beautiful-flat-icons-for-free
• "These icons are completely free and Open Source under the GPL".

Logo

Author: Christian Pierre MOMON christian.momon@devinsy.fr

License: Creative Commons CC BY-SA last version.

Other stuff

License of other stuff:

• csv-logo.svg, json-logo.svg, ods-icon.svg:
  • Author: Christian P. Momon
  • License: CC BY-SA 3+.
• other: without any specific information, default license of images is CC BY-SA 3+.

DOCUMENTATION

Read CONCEPT.md and ONTOLOGY.md files.

Requirements

• Java 17
• Eclipse 4.29 (2023-09).

INSTALL

Install Java :

apt-get install openjdk-17-jre-headless

Download the last release: https://forge.devinsy.fr/devinsy/statoolinfos/releases

Unzip the package and move the statoolinfos.jar and statoolinfo.sh files in /srv/statoolinfos/bin/.

Create link:

cd /srv/statoolinfos/bin/ ; ln -s statoolinfos.sh statoolinfos

Usage

Usage:
    statoolinfos [ -h | -help | --help ]
    statoolinfos [ -v | -version | --version ]

    statoolinfos build                       <configurationfile>     build property files from conf and input
    statoolinfos clear                       <configurationfile>     remove property files from conf
    statoolinfos crawl                       <configurationfile>     crawl all file from conf and input
    statoolinfos format                      <fileordirectory>       format property files in tiny way
    s̶t̶a̶t̶o̶o̶l̶i̶n̶f̶o̶s̶ ̶h̶t̶m̶l̶i̶z̶e̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶<̶c̶o̶n̶f̶i̶g̶u̶r̶a̶t̶i̶o̶n̶f̶i̶l̶e̶>̶ ̶ ̶ ̶ ̶ ̶g̶e̶n̶e̶r̶a̶t̶e̶ ̶w̶e̶b̶ ̶p̶a̶g̶e̶s̶ ̶f̶r̶o̶m̶ ̶c̶o̶n̶f̶  REMOVED since splitweb
    statoolinfos probe   OPTION [<directory>|<configurationfile>]    generate metrics files from conf
           OPTION = [-full|-today|-previousday|-NN] with NN a day count
    statoolinfos tagdate                     <fileordirectory>       update the file.datetime file
    statoolinfos uptime                      <configurationfile>     update uptime journal

    statoolinfos list file                   <logfilesorconfigfile>  display http access log files
    statoolinfos list log      [-bot|-nobot] <logfilesorconfigfile>  display http access log lines
    statoolinfos list ip       [-bot|-nobot] <logfilesorconfigfile>  generate ip list from http log file
    statoolinfos list ua       [-bot|-nobot] <logfilesorconfigfile>  generate user agent list from http log file
    statoolinfos list visitor  [-bot|-nobot] <logfilesorconfigfile>  generate visitors (ip+ua) list from http log file
    statoolinfos stat ip       [-bot|-nobot] <logfilesorconfigfile>  generate stats about ip from http log file
    statoolinfos stat ua       [-bot|-nobot] <logfilesorconfigfile>  generate stats about user agent from http log file
    statoolinfos stat visitor  [-bot|-nobot] <logfilesorconfigfile>  generate stats about visitor (ip+ua) from http log file
    statoolinfos list errfile                <logfilesorconfigfile>  display http error log files
    statoolinfos list errlog                 <logfilesorconfigfile>  display http error log lines

htmlize removed

The htmlize command is removed since 2024-07.

As a reminder, the htmlize command generated absolutely all the pages needed for a static site.

As the number of pages generated became too large (> 10000 with 100 organizations), the decision was made to separate the html valuation into a separate project, StatoolInfosWeb.

A dynamic web site allows to generate page only when it is necessary.

Generate metrics files

Basics

Create a configuration file /srv/statoolInfos/conf/foo.bar.org.conf:

conf.probe.types=<metrictype1>, <metrictype2>
conf.probe.metrictype1param1.file=<value>
conf.probe.metrictype1param2.file=<value>
conf.probe.metrictype2param1.file=<value>
conf.probe.metrictype2param2.file=<value>
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

Generate the metric file for the first time:

/srv/statoolinfos/bin/statoolinfos probe -full /srv/statoolinfos/conf/foo.bar.org.conf

Create a cron file in /etc/cron.d/statoolinfos to update the metric file everyday:

1 0,12 * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/foo.bar.org.conf >> /srv/statoolinfos/cron.log

In case of several configuration files, just set the configuration directory (without wildcard):

1 0,12 * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/ >> /srv/statoolinfos/cron.log

Warning: in previous day mode, the metrics generated are overwrited for the last month, the last week and the last day. So, six weeks in logs are required.

Etherpad metrics

Configuration template:

conf.probe.types=Etherpad
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
# This parameter requires real ip proxy setting and log output setting.
conf.probe.etherpad.logs=/var/log/etherpad/output.log
# Database URL. Example of the most frequent cases:
# jdbc:mariadb://localhost:1234/databasename
# jdbc:mysql://localhost:1234/databasename
# jdbc:postgresql://localhost:1234/databasename
# jdbc:sqlite:/foo/bar/databasename.sqlite
conf.probe.etherpad.database.url=
conf.probe.etherpad.database.user=
conf.probe.etherpad.database.password=
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

Framdadate metrics (coming soon)

Configuration template:

conf.probe.types=Framadate
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

Gitea metrics

Configuration template:

conf.probe.types=Gitea
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.gitea.data=/opt/gitea/data
conf.probe.gitea.api.url=https://forge.libre-service.eu/
conf.probe.gitea.token=b6598c616b1cd350b834258205da4e5e8b951005
# jdbc:mariadb://localhost:1234/databasename
# jdbc:mysql://localhost:1234/databasename
# jdbc:postgresql://localhost:1234/databasename
# jdbc:sqlite:/foo/bar/databasename.sqlite
conf.probe.gitea.database.url=
conf.probe.gitea.database.user=
conf.probe.gitea.database.password=
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

GSL metrics

Configuration template:

conf.probe.types=GSL
conf.probe.gsl.stats=/foo/bar/stats.properties
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

LibreQR metrics

Configuration template:

conf.probe.types=LibreQR
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.libreqr.datafiles=/var/www/foo.bar.org/temp/
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

Minetest metrics

Configuration template:

conf.probe.types=Minetest
conf.probe.minetest.logs=/home/cpm/Projets/StatoolInfos/EnvTest/minetest/minetest.log*

conf.probe.minetest.players.database.url=jdbc:postgresql://localhost:5432/minetestdb
conf.probe.minetest.players.database.user=minetestdba
conf.probe.minetest.players.database.password=XXXXXXXXXXX

conf.probe.minetest.worlds.database.url=jdbc:postgresql://localhost:5432/minetestdb
conf.probe.minetest.worlds.database.user=minetestdba
conf.probe.minetest.worlds.database.password=XXXXXXXXXXX

conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

Mumble metrics

Configuration template:

conf.probe.types=Mumble
conf.probe.mumble.logs=/var/log/mumble-server/mumble-server.log*
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

Nextcloud metrics (coming soon)

Configuration template:

conf.probe.types=Nextcloud
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

PrivateBin metrics (partial)

Warning: works fine if database, image and comment options are disabled.

Configuration template:

conf.probe.types=PrivateBin
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.privatebin.data=/var/www/paste.libre-service.eu/data/
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

Web metrics

Configuration template:

conf.probe.types=HttpAccessLog, HttpErrorLog
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*,/var/log/apache2/foo.bar.org-extraaccess.log*
conf.probe.httperrorlog.file=/var/log/apache2/foo.bar.org-error.log*,/var/log/apache2/foo.bar.org-extraerror.log*
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

# Custom access log pattern with Java regex.
# Default: ^(?<remoteAddress>[a-fA-F0-9\:\.]+) - (?<remoteUser>[^\[]+) \[(?<time>[^\]]+)\] "(?<request>([^\"]|\")*)" (?<status>\d+) (?<bodyBytesSent>\d+) "(?<referer>([^\"]|\")*)" "(?<userAgent>([^\"]|\")*)".*$
conf.probe.httpaccesslog.pattern=

# Custom datetime access log pattern (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns).
# Default: dd/MMM/yyyy:HH:mm:ss Z|EN
conf.prob.httpaccesslog.datetimepattern=

# Filter request with Java regex (without the initial '^' and without the final '$', see https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html).
# Example: pathfilter=/.*well-known.*
conf.probe.httpaccesslog.pathfilter=

# Custom error log pattern with Java regex.
# Default Apache: ^\[(?<time>[^\]]+)\]\s\[(?<level>[^\]]*)\]\s(?<message>.*)$
# Default NGinx:  ^(?<time>\S+\s\S+)\s\[(?<level>[^\]]*)\]\s(?<message>.*)$
conf.probe.httperrorlog.pattern=

# Custom datetime access log pattern (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns).
# Default: dd/MMM/yyyy:HH:mm:ss Z|EN
conf.prob.httperrorlog.datetimepattern=

Warning: to have month metric values, you have to set retention log to 7 weeks or 50 days in Nginx or Apache logrotate configuration.

For Nginx, éditer le fichier /etc/logrotate.d/nginx :

/var/log/nginx/*.log {
    weekly
    missingok
    rotate 7

For Apache, éditer le fichier /etc/logrotate.d/apache2 :

/var/log/apache2/*.log {
    weekly
    missingok
    rotate 7