statool/statoolinfos

Fork 0

Christian P. MOMON 4c0fc7d6a5 Deleted a no longer targeted service.

2024-08-14 23:43:11 +02:00

11 KiB

Raw Blame History

StatoolInfos

StatoolInfos is a simple statistic tool software.

LICENSE

StatoolInfos is released under the GNU AGPL+ license. Enjoy!

Authors:

Christian Pierre MOMON christian.momon@devinsy.fr

Contributors : Angie, MrFlos, Quentin Duchemin, Fabrice61, Thomas TConstans, labecasse, Antoine Jaba, setop, Jérémy Collot, Pilou, Kepon, Laurent Sleto, Rodinux…

Cat Avatar Generator images

I am glad to use the beautiful artwork of David Revoy (http://www.peppercarrot.com).

Graphics CC-BY http://creativecommons.org/licenses/by/4.0/
Service: https://www.peppercarrot.com/extras/html/2016_cat-generator/
Source: https://framagit.org/Deevad/cat-avatar-generator
Originally inspired of the code for "MonsterID" by Andreas Gohr http://www.splitbrain.org/go/monsterid.

Logo

Author: Christian Pierre MOMON christian.momon@devinsy.fr

License: Creative Commons CC BY-SA last version.

Other stuff

License of other stuff: without any specific information, default license of images is CC BY-SA 3+.

DOCUMENTATION

Read CONCEPT.md and ONTOLOGY.md files.

Requirements

Java 17
Eclipse 4.29 (2023-09).

INSTALL

Install Java :

apt-get install openjdk-17-jre-headless

Download the last release: https://forge.devinsy.fr/statool/statoolinfos/releases

Unzip the package and move the statoolinfos.jar and statoolinfo.sh files in /srv/statoolinfos/bin/.

Create link:

cd /srv/statoolinfos/bin/ ; ln -s statoolinfos.sh statoolinfos

Usage

Usage:
    statoolinfos [ -h | -help | --help ]
    statoolinfos [ -v | -version | --version ]

    statoolinfos build                       <configurationfile>     build property files from conf and input
    statoolinfos clear                       <configurationfile>     remove property files from conf
    statoolinfos crawl                       <configurationfile>     crawl all file from conf and input
    statoolinfos format                      <fileordirectory>       format property files in tiny way
    s̶t̶a̶t̶o̶o̶l̶i̶n̶f̶o̶s̶ ̶h̶t̶m̶l̶i̶z̶e̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶ ̶<̶c̶o̶n̶f̶i̶g̶u̶r̶a̶t̶i̶o̶n̶f̶i̶l̶e̶>̶ ̶ ̶ ̶ ̶ ̶g̶e̶n̶e̶r̶a̶t̶e̶ ̶w̶e̶b̶ ̶p̶a̶g̶e̶s̶ ̶f̶r̶o̶m̶ ̶c̶o̶n̶f̶  REMOVED since splitweb
    statoolinfos probe   OPTION [<directory>|<configurationfile>]    generate metrics files from conf
           OPTION = [-full|-today|-previousday|-NN] with NN a day count
    statoolinfos tagdate                     <fileordirectory>       update the file.datetime file
    statoolinfos uptime                      <configurationfile>     update uptime journal

    statoolinfos list file                   <logfilesorconfigfile>  display http access log files
    statoolinfos list log      [-bot|-nobot] <logfilesorconfigfile>  display http access log lines
    statoolinfos list ip       [-bot|-nobot] <logfilesorconfigfile>  generate ip list from http log file
    statoolinfos list ua       [-bot|-nobot] <logfilesorconfigfile>  generate user agent list from http log file
    statoolinfos list visitor  [-bot|-nobot] <logfilesorconfigfile>  generate visitors (ip+ua) list from http log file
    statoolinfos stat ip       [-bot|-nobot] <logfilesorconfigfile>  generate stats about ip from http log file
    statoolinfos stat ua       [-bot|-nobot] <logfilesorconfigfile>  generate stats about user agent from http log file
    statoolinfos stat visitor  [-bot|-nobot] <logfilesorconfigfile>  generate stats about visitor (ip+ua) from http log file
    statoolinfos list errfile                <logfilesorconfigfile>  display http error log files
    statoolinfos list errlog                 <logfilesorconfigfile>  display http error log lines

htmlize removed

The htmlize command is removed since 2024-07.

As a reminder, the htmlize command generated absolutely all the pages needed for a static site.

As the number of pages generated became too large (> 10000 with 100 organizations), the decision was made to separate the html valuation into a separate project, StatoolInfosWeb.

A dynamic web site allows to generate page only when it is necessary.

Generate metrics files

Basics

Create a configuration file /srv/statoolInfos/conf/foo.bar.org.conf:

conf.probe.types=<metrictype1>, <metrictype2>
conf.probe.metrictype1param1.file=<value>
conf.probe.metrictype1param2.file=<value>
conf.probe.metrictype2param1.file=<value>
conf.probe.metrictype2param2.file=<value>
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

Generate the metric file for the first time:

/srv/statoolinfos/bin/statoolinfos probe -full /srv/statoolinfos/conf/foo.bar.org.conf

Create a cron file in /etc/cron.d/statoolinfos to update the metric file everyday:

1 0,12 * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/foo.bar.org.conf >> /srv/statoolinfos/cron.log

In case of several configuration files, just set the configuration directory (without wildcard):

1 0,12 * * * root /srv/statoolinfos/bin/statoolinfos probe -previousday /srv/statoolinfos/conf/ >> /srv/statoolinfos/cron.log

Warning: in previous day mode, the metrics generated are overwrited for the last month, the last week and the last day. So, six weeks in logs are required.

Etherpad metrics

Configuration template:

conf.probe.types=Etherpad
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
# This parameter requires real ip proxy setting and log output setting.
conf.probe.etherpad.logs=/var/log/etherpad/output.log
# Database URL. Example of the most frequent cases:
# jdbc:mariadb://localhost:1234/databasename
# jdbc:mysql://localhost:1234/databasename
# jdbc:postgresql://localhost:1234/databasename
# jdbc:sqlite:/foo/bar/databasename.sqlite
conf.probe.etherpad.database.url=
conf.probe.etherpad.database.user=
conf.probe.etherpad.database.password=
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

Framdadate metrics (coming soon)

Configuration template:

conf.probe.types=Framadate
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

Gitea metrics

Configuration template:

conf.probe.types=Gitea
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.gitea.data=/opt/gitea/data
conf.probe.gitea.api.url=https://forge.libre-service.eu/
conf.probe.gitea.token=b6598c616b1cd350b834258205da4e5e8b951005
# jdbc:mariadb://localhost:1234/databasename
# jdbc:mysql://localhost:1234/databasename
# jdbc:postgresql://localhost:1234/databasename
# jdbc:sqlite:/foo/bar/databasename.sqlite
conf.probe.gitea.database.url=
conf.probe.gitea.database.user=
conf.probe.gitea.database.password=
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

LibreQR metrics

Configuration template:

conf.probe.types=LibreQR
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.libreqr.datafiles=/var/www/foo.bar.org/temp/
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

Minetest metrics

Configuration template:

conf.probe.types=Minetest
conf.probe.minetest.logs=/home/cpm/Projets/StatoolInfos/EnvTest/minetest/minetest.log*

conf.probe.minetest.players.database.url=jdbc:postgresql://localhost:5432/minetestdb
conf.probe.minetest.players.database.user=minetestdba
conf.probe.minetest.players.database.password=XXXXXXXXXXX

conf.probe.minetest.worlds.database.url=jdbc:postgresql://localhost:5432/minetestdb
conf.probe.minetest.worlds.database.user=minetestdba
conf.probe.minetest.worlds.database.password=XXXXXXXXXXX

conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

Mumble metrics

Configuration template:

conf.probe.types=Mumble
conf.probe.mumble.logs=/var/log/mumble-server/mumble-server.log*
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

Nextcloud metrics (coming soon)

Configuration template:

conf.probe.types=Nextcloud
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

PrivateBin metrics (partial)

Warning: works fine if database, image and comment options are disabled.

Configuration template:

conf.probe.types=PrivateBin
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*
conf.probe.httpaccesslog.pattern=
conf.probe.httpaccesslog.pathfilter=
conf.probe.privatebin.data=/var/www/paste.libre-service.eu/data/
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

Web metrics

Configuration template:

conf.probe.types=HttpAccessLog, HttpErrorLog
conf.probe.httpaccesslog.file=/var/log/apache2/foo.bar.org-access.log*,/var/log/apache2/foo.bar.org-extraaccess.log*
conf.probe.httperrorlog.file=/var/log/apache2/foo.bar.org-error.log*,/var/log/apache2/foo.bar.org-extraerror.log*
conf.probe.target=/srv/statoolinfos/well-known/statoolinfos/foo.bar.org-metrics.properties

# Custom access log pattern with Java regex.
# Default: ^(?<remoteAddress>[a-fA-F0-9\:\.]+) - (?<remoteUser>[^\[]+) \[(?<time>[^\]]+)\] "(?<request>([^\"]|\")*)" (?<status>\d+) (?<bodyBytesSent>\d+) "(?<referer>([^\"]|\")*)" "(?<userAgent>([^\"]|\")*)".*$
conf.probe.httpaccesslog.pattern=

# Custom datetime access log pattern (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns).
# Default: dd/MMM/yyyy:HH:mm:ss Z|EN
conf.prob.httpaccesslog.datetimepattern=

# Filter request with Java regex (without the initial '^' and without the final '$', see https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html).
# Example: pathfilter=/.*well-known.*
conf.probe.httpaccesslog.pathfilter=

# Custom error log pattern with Java regex.
# Default Apache: ^\[(?<time>[^\]]+)\]\s\[(?<level>[^\]]*)\]\s(?<message>.*)$
# Default NGinx:  ^(?<time>\S+\s\S+)\s\[(?<level>[^\]]*)\]\s(?<message>.*)$
conf.probe.httperrorlog.pattern=

# Custom datetime access log pattern (https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns).
# Default: dd/MMM/yyyy:HH:mm:ss Z|EN
conf.prob.httperrorlog.datetimepattern=

Warning: to have month metric values, you have to set retention log to 7 weeks or 50 days in Nginx or Apache logrotate configuration.

For Nginx, éditer le fichier /etc/logrotate.d/nginx :

/var/log/nginx/*.log {
    weekly
    missingok
    rotate 7

For Apache, éditer le fichier /etc/logrotate.d/apache2 :

/var/log/apache2/*.log {
    weekly
    missingok
    rotate 7

11 KiB Raw Blame History

StatoolInfos

LICENSE

Cat Avatar Generator images

Logo

Other stuff

DOCUMENTATION

Requirements

INSTALL

Usage

htmlize removed

Generate metrics files

Basics

Etherpad metrics

Framdadate metrics (coming soon)

Gitea metrics

LibreQR metrics

Minetest metrics

Mumble metrics

Nextcloud metrics (coming soon)

PrivateBin metrics (partial)

Web metrics

11 KiB

Raw Blame History