Compare commits

..

4 commits

Author SHA1 Message Date
cec190a723 Build 0.5.1 2023-05-09 15:01:42 +02:00
00486a4710 Reviewed Javadoc. 2023-05-09 15:01:06 +02:00
5e7ec3e6f2 Reviewed Javadoc. 2023-05-09 14:34:34 +02:00
189f8471ac Added CHANGELOG file. 2023-05-09 14:21:50 +02:00
21 changed files with 358 additions and 59 deletions

125
CHANGELOG.md Normal file
View file

@ -0,0 +1,125 @@
# StatoolInfos Changelog
All notable changes of StatoolInfos will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## [Unreleased]
Changes for metrics:
- …
Changes for web building:
- …
## [0.5.1] 2023-05-09
Changes for metrics:
- Added multi sources for Http log file search.
- Added wrong password metrics for Minetest.
- Fixed property label in Privatebin patch probe.
- Added CLI commands to test HttpAccessLog files and lines.
- Moved HttpErrorLog analyzer in dedicated package.
- Refactored prober call from CLI.
- Added new metrics analyzers.
- Added pathFilter parameter in httpAccessLog read.
- Fixed HTTP access log pattern.
- Fixed HTTP pattern management from metric conf file.
- Added Kuma in Bot detection file.
- Added not authorized HTTP status code.
- Added chat metrics for Minetest.
- Set 401 uptime case as OK.
- Improved bot pattern list with TinyStatus.
- Added metrics for GSL.
- Improved documentation (README)
Changes for web building:
- Improved property stats page with only active organizations.
- Improved PropertyStats page with only active services.
- Fixed hosting properties file management.
- Added Z support in DATETIME property check.
- Added optional parameter IPV6ONLY in statoolinfos.sh.
- Changed metrics menu entry from 2020 to 2022.
- Fixed bad category list building.
- Removed debug trace.
- Fixed categories compare.
- Fixed wording.
- Fixed separate case characters in property stats.
- Improved federation service count.
- Improved date management.
- Improved startDate and endDate management for services.
- Improved startDate and endDate management for organizations.
- Improved Federation getStartDate management.
- Added provider hypervisor charts for service stats.
- Added type chart and provider hypervisor charts for organization stats.
- Added federation organization type chart. Improved federation charts.
- Refactored federation stats. Added provider hypervisor chart.
- Added organization.type check entry.
- Added host.provider.hypervisor check entry.
- Changed color of ServiceCountYearChart.
- Fixed bad display of htmlizeHostNamePieChart.
- Swap columns in htmlizeOrganizationInOutChart.
- Fixed colors.
## [0.5.0] 2022-01-23
This release contains many improvments:
- Extended http access log regex.
- Restricted federation stats to active organizations only.
- Improved status management in organizations and services display.
- Added Let's encrypt detection for bot filtering.
- Added date tips in header column with monthly label.
- Replaced 0 value with emoji in organizations view and services view.
- Added visits.ipv4 and visits.ipv6 management.
- Added metrics configuration templates in README.
- Changed visit and visitor definition.
- Fixed bug in error nginx time parsing.
- Fixed label display in 2 data graph.
- Added bot pattern for mod_jk dummy connection in http logs.
- Added asterisk month in organization list view.
- Added asterisk month in service list view.
- Fixed DataTables paths.
- Fixed former member display.
- Improved userAgent analyze.
- Fixed empty http log file bug.
- Added first veresion of Mumble probing (see README.md).
- Fixed bug in TimeMark code.
- Added first version of PrivateBin probing (see README.md).
- Migrated to Log4j2 (with Log4shell fix).
- Made code improvments.
Warning: visit and visitors definition has changed. Now it is more restricted: humans + method GET + page + status SUCCESS.
So some metrics have to be generate again (with -full) and some properties have to deleted:
- Added
- metrics.http.requesters : nombre de requêteurs.
- metrics.http.requesters.ipv4 : nombre de requêteurs IPv4.
- metrics.http.requesters.ipv6 : nombre de requêteurs IPv6.
- metrics.http.requesters.humans : nombre de requêteurs humains.
- metrics.http.requesters.bots : nombre de requêteurs bots.
- metrics.http.requesters.bots.walks : nombre de passages de requêteurs bots.
- metrics.http.metrics.methods.XXX : nombre de requête avec la méthode XXX.
- metrics.http.visits.ipv4 : nombre de visites en IPv4.
- metrics.http.visits.ipv6 : nombre de visites en IPv6.
- Changed
- metrics.http.visits : définition plus restrictive (humains + GET + page + SUCCESS).
- metrics.http.visitors : définition de visiteur plus restrictive.
- metrics.http.visitors.ipv4 : définition de visiteur plus restrictive.
- metrics.http.visitors.ipv6 : définition de visiteur plus restrictive.
- Removed
- metrics.http.visitors.humans : remplacé par metrics.http.visitors (nouvelle définition d'une visite).
- metrics.http.visitors.bots : supprimé (nouvelle définition d'une visite).
- metrics.https.visits.bots : supprimé (nouvelle définition d'une visite).
## [0.4.1] 2021-12-23
This release contains small improvments:
- Fixed DataTables (CVE-2021-23445).
- Improved wildcard feature in probe command (#1 thx Kepon).
- Improved sample config file.
- Added beta Minetest specific probing feature.
## [0.4.0] 2021-12-15
This release contains probe feature to generate metrics files.

View file

@ -127,7 +127,7 @@
<copy file="resources/scripts/${product.name}.sh" todir="${dist.dir}/" overwrite="true" />
<chmod file="${dist.dir}/${product.name}.sh" perm="ugo+rx" />
<copy file="LICENSE" todir="${dist.dir}/" overwrite="true" failonerror="false" />
<copy file="README" todir="${dist.dir}/" overwrite="true" failonerror="false" />
<!--copy file="README" todir="${dist.dir}/" overwrite="true" failonerror="false" /-->
<copy file="README.md" todir="${dist.dir}/" overwrite="true" failonerror="false" />
<!--copy file="scripts/log4j.properties" todir="${dist.dir}/" overwrite="true" /-->

View file

@ -1,3 +1,3 @@
#Build Number for ANT. Do not edit!
#Sun Jan 23 03:49:04 CET 2022
build.number=1
#Tue May 09 15:01:37 CEST 2023
build.number=2

View file

@ -143,6 +143,7 @@ public class Configuration extends PathPropertyList
*
* @return the category file
* @throws StatoolInfosException
* the statool infos exception
*/
public File getCategoryFile() throws StatoolInfosException
{
@ -337,10 +338,11 @@ public class Configuration extends PathPropertyList
}
/**
* Gets the htmlize input path.
* Gets the htmlize input URL.
*
* @return the htmlize input path
* @return the htmlize input URL
* @throws MalformedURLException
* the malformed URL exception
*/
public URL getHtmlizeInputURL() throws MalformedURLException
{
@ -400,8 +402,6 @@ public class Configuration extends PathPropertyList
/**
* Gets the probe http access log sources.
*
* @param source
* the source
* @return the probe http access log sources
*/
public StringList getProbeHttpAccessLogSources()

View file

@ -1,5 +1,5 @@
/*
* Copyright (C) 2021-2022 Christian Pierre MOMON <christian@momon.org>
* Copyright (C) 2021-2023 Christian Pierre MOMON <christian@momon.org>
*
* This file is part of StatoolInfos, simple service statistics tool.
*
@ -39,16 +39,31 @@ public class DatabaseConfig
this.password = null;
}
/**
* Gets the password.
*
* @return the password
*/
public String getPassword()
{
return this.password;
}
/**
* Gets the url.
*
* @return the url
*/
public String getUrl()
{
return this.url;
}
/**
* Gets the user.
*
* @return the user
*/
public String getUser()
{
return this.user;
@ -76,16 +91,34 @@ public class DatabaseConfig
return result;
}
/**
* Sets the password.
*
* @param password
* the new password
*/
public void setPassword(final String password)
{
this.password = password;
}
/**
* Sets the url.
*
* @param url
* the new url
*/
public void setUrl(final String url)
{
this.url = url;
}
/**
* Sets the user.
*
* @param user
* the new user
*/
public void setUser(final String user)
{
this.user = user;

View file

@ -164,12 +164,10 @@ public class StatoolInfos
}
/**
* Test http access log files.
* List files.
*
* @param configurationFile
* the configuration file
* @throws IOException
* @throws StatoolInfosException
*/
public static void listFiles(final File configurationFile)
{
@ -206,14 +204,10 @@ public class StatoolInfos
}
/**
* List http access log files.
* List files.
*
* @param files
* the files
* @throws StatoolInfosException
* the statool infos exception
* @throws IOException
* Signals that an I/O exception has occurred.
*/
public static void listFiles(final Files files)
{
@ -234,7 +228,8 @@ public class StatoolInfos
*
* @param source
* the source
* @throws IOException
* @param filter
* the filter
*/
public static void listIps(final Files source, final BotFilter filter)
{

View file

@ -147,8 +147,11 @@ public class CrawlCache
}
/**
* @return
* Restore crawl journal.
*
* @return the crawl journal
* @throws IOException
* Signals that an I/O exception has occurred.
*/
public CrawlJournal restoreCrawlJournal() throws IOException
{
@ -261,9 +264,13 @@ public class CrawlCache
}
/**
* Restore to properties.
*
* @param url
* @return
* the url
* @return the path properties
* @throws IOException
* Signals that an I/O exception has occurred.
*/
public PathProperties restoreToProperties(final URL url) throws IOException
{
@ -446,10 +453,11 @@ public class CrawlCache
}
/**
* Gets the journal URL.
* Gets the crawl journal URL.
*
* @return the journal URL
* @return the crawl journal URL
* @throws MalformedURLException
* the malformed URL exception
*/
public static URL getCrawlJournalURL() throws MalformedURLException
{

View file

@ -1,5 +1,5 @@
/*
* Copyright (C) 2020-2021 Christian Pierre MOMON <christian@momon.org>
* Copyright (C) 2020-2023 Christian Pierre MOMON <christian@momon.org>
*
* This file is part of StatoolInfos, simple service statistics tool.
*
@ -80,6 +80,7 @@ public class BreadcrumbTrail extends ArrayList<Breadcrumb>
* the label
* @param link
* the link
* @return the breadcrumb trail
*/
public BreadcrumbTrail add(final String label, final String link)
{

View file

@ -1,5 +1,5 @@
/*
* Copyright (C) 2020-2022 Christian Pierre MOMON <christian@momon.org>
* Copyright (C) 2020-2023 Christian Pierre MOMON <christian@momon.org>
*
* This file is part of StatoolInfos, simple service statistics tool.
*
@ -47,9 +47,11 @@ public class CategoriesPage
/**
* Builds the.
*
*
* @throws StatoolInfosException
* the statool infos exception
* @throws IOException
* Signals that an I/O exception has occurred.
*/
public static void build() throws StatoolInfosException, IOException
{

View file

@ -46,11 +46,6 @@ public class ExportsPage
/**
* Builds the.
*
* @throws StatoolInfosException
* the statool infos exception
* @throws IOException
* Signals that an I/O exception has occurred.
*/
public static void build()
{

View file

@ -1,5 +1,5 @@
/*
* Copyright (C) 2020-2022 Christian Pierre MOMON <christian@momon.org>
* Copyright (C) 2020-2023 Christian Pierre MOMON <christian@momon.org>
*
* This file is part of StatoolInfos, simple service statistics tool.
*
@ -18,8 +18,6 @@
*/
package fr.devinsy.statoolinfos.htmlize;
import java.io.IOException;
import org.apache.commons.lang3.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@ -42,14 +40,13 @@ public class FederationHeaderView
private static Logger logger = LoggerFactory.getLogger(FederationHeaderView.class);
/**
* Builds the.
* Htmlize.
*
* @param federation
* the organization
* the federation
* @return the string
* @throws StatoolInfosException
* the statool infos exception
* @throws IOException
*/
public static String htmlize(final Federation federation) throws StatoolInfosException
{

View file

@ -1,5 +1,5 @@
/*
* Copyright (C) 2020-2021 Christian Pierre MOMON <christian@momon.org>
* Copyright (C) 2020-2023 Christian Pierre MOMON <christian@momon.org>
*
* This file is part of StatoolInfos, simple service statistics tool.
*
@ -135,6 +135,11 @@ public class BarChart
return result;
}
/**
* Gets the datasets.
*
* @return the datasets
*/
public BarChartDatasets getDatasets()
{
return this.datasets;
@ -165,21 +170,42 @@ public class BarChart
return this.title;
}
/**
* Checks if is animated.
*
* @return true, if is animated
*/
public boolean isAnimated()
{
return this.animated;
}
/**
* Checks if is display title.
*
* @return true, if is display title
*/
public boolean isDisplayTitle()
{
return this.displayTitle;
}
/**
* Checks if is stacked.
*
* @return true, if is stacked
*/
public boolean isStacked()
{
return this.stacked;
}
/**
* Sets the animated.
*
* @param animated
* the new animated
*/
public void setAnimated(final boolean animated)
{
this.animated = animated;

View file

@ -1,5 +1,5 @@
/*
* Copyright (C) 2020 Christian Pierre MOMON <christian@momon.org>
* Copyright (C) 2020-2023 Christian Pierre MOMON <christian@momon.org>
*
* This file is part of StatoolInfos, simple service statistics tool.
*
@ -49,31 +49,64 @@ public class BarChartData
this.color = color;
}
/**
* Gets the color.
*
* @return the color
*/
public ChartColor getColor()
{
return this.color;
}
/**
* Gets the label.
*
* @return the label
*/
public String getLabel()
{
return this.label;
}
/**
* Gets the value.
*
* @return the value
*/
public double getValue()
{
return this.value;
}
/**
* Sets the color.
*
* @param color
* the new color
*/
public void setColor(final ChartColor color)
{
this.color = color;
}
/**
* Sets the label.
*
* @param label
* the new label
*/
public void setLabel(final String label)
{
this.label = label;
}
/**
* Sets the value.
*
* @param value
* the new value
*/
public void setValue(final double value)
{
this.value = value;

View file

@ -1,5 +1,5 @@
/*
* Copyright (C) 2020-2021 Christian Pierre MOMON <christian@momon.org>
* Copyright (C) 2020-2023 Christian Pierre MOMON <christian@momon.org>
*
* This file is part of StatoolInfos, simple service statistics tool.
*
@ -47,6 +47,11 @@ public class BarChartDataset extends ArrayList<BarChartData>
this.borderWidth = 1;
}
/**
* Gets the border width.
*
* @return the border width
*/
public int getBorderWidth()
{
return this.borderWidth;
@ -72,6 +77,11 @@ public class BarChartDataset extends ArrayList<BarChartData>
return result;
}
/**
* Gets the name.
*
* @return the name
*/
public String getName()
{
return this.name;
@ -99,11 +109,23 @@ public class BarChartDataset extends ArrayList<BarChartData>
return result;
}
/**
* Sets the border width.
*
* @param borderWidth
* the new border width
*/
public void setBorderWidth(final int borderWidth)
{
this.borderWidth = borderWidth;
}
/**
* Sets the name.
*
* @param name
* the new name
*/
public void setName(final String name)
{
this.name = name;

View file

@ -1,5 +1,5 @@
/*
* Copyright (C) 2020 Christian Pierre MOMON <christian@momon.org>
* Copyright (C) 2020-2023 Christian Pierre MOMON <christian@momon.org>
*
* This file is part of StatoolInfos, simple service statistics tool.
*
@ -42,21 +42,43 @@ public class DateCount
this.count = count;
}
/**
* Gets the count.
*
* @return the count
*/
public long getCount()
{
return this.count;
}
/**
* Gets the date.
*
* @return the date
*/
public String getDate()
{
return this.date;
}
/**
* Sets the count.
*
* @param count
* the new count
*/
public void setCount(final long count)
{
this.count = count;
}
/**
* Sets the date.
*
* @param date
* the new date
*/
public void setDate(final String date)
{
if (StringUtils.isBlank(date))
@ -68,5 +90,4 @@ public class DateCount
this.date = date;
}
}
}

View file

@ -1,5 +1,5 @@
/*
* Copyright (C) 2021 Christian Pierre MOMON <christian@momon.org>
* Copyright (C) 2021-2023 Christian Pierre MOMON <christian@momon.org>
*
* This file is part of StatoolInfos, simple service statistics tool.
*
@ -53,6 +53,7 @@ public class CSVFile
* @param source
* the source
* @throws IOException
* Signals that an I/O exception has occurred.
*/
public static void save(final File file, final Organizations source) throws IOException
{

View file

@ -1,5 +1,5 @@
/*
* Copyright (C) 2021 Christian Pierre MOMON <christian@momon.org>
* Copyright (C) 2021-2023 Christian Pierre MOMON <christian@momon.org>
*
* This file is part of StatoolInfos, simple service statistics tool.
*
@ -64,9 +64,9 @@ public class CSVWriter implements SpreadsheetWriter
/**
* Close.
*
* @throws FileNotFoundException
*
* @throws FileNotFoundException
* the file not found exception
* @throws IOException
* Signals that an I/O exception has occurred.
*/
@ -138,8 +138,6 @@ public class CSVWriter implements SpreadsheetWriter
/**
* Write end row.
*
* @throws IOException
*/
@Override
public void writeEndRow()

View file

@ -99,8 +99,6 @@ public class MinetestLogAnalyzer
* the file
* @throws IOException
* Signals that an I/O exception has occurred.
* @throws StatoolInfosException
* the statool infos exception
*/
public void probe(final File file) throws IOException
{

View file

@ -1,5 +1,5 @@
/*
* Copyright (C) 2021-2022 Christian Pierre MOMON <christian@momon.org>
* Copyright (C) 2021-2023 Christian Pierre MOMON <christian@momon.org>
*
* This file is part of StatoolInfos, simple service statistics tool.
*
@ -105,8 +105,6 @@ public class MumbleLogAnalyzer
* the file
* @throws IOException
* Signals that an I/O exception has occurred.
* @throws StatoolInfosException
* the statool infos exception
*/
public void probe(final File file) throws IOException
{

View file

@ -1,5 +1,5 @@
/*
* Copyright (C) 2021-2022 Christian Pierre MOMON <christian@momon.org>
* Copyright (C) 2021-2023 Christian Pierre MOMON <christian@momon.org>
*
* This file is part of StatoolInfos, simple service statistics tool.
*
@ -25,7 +25,6 @@ import org.apache.commons.io.FileUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import fr.devinsy.statoolinfos.core.StatoolInfosException;
import fr.devinsy.statoolinfos.metrics.PathCounters;
import fr.devinsy.statoolinfos.util.FilesUtils;
import fr.devinsy.strings.StringList;
@ -52,8 +51,6 @@ public class DatafilesProber
* @return the path counters
* @throws IOException
* Signals that an I/O exception has occurred.
* @throws StatoolInfosException
* the statool infos exception
*/
public static PathCounters probe(final File directory) throws IOException
{
@ -69,12 +66,12 @@ public class DatafilesProber
* Probe.
*
* @param directory
* the data directory
* the directory
* @param prefix
* the prefix
* @return the path counters
* @throws IOException
* Signals that an I/O exception has occurred.
* @throws StatoolInfosException
* the statool infos exception
*/
public static PathCounters probe(final File directory, final String prefix) throws IOException
{

View file

@ -1,5 +1,5 @@
/*
* Copyright (C) 2020 Christian Pierre MOMON <christian@momon.org>
* Copyright (C) 2020-2023 Christian Pierre MOMON <christian@momon.org>
*
* This file is part of StatoolInfos, simple service statistics tool.
*
@ -46,26 +46,51 @@ public class CategoryStat
this.visitCount = 0;
}
/**
* Gets the category.
*
* @return the category
*/
public Category getCategory()
{
return this.category;
}
/**
* Gets the organization count.
*
* @return the organization count
*/
public int getOrganizationCount()
{
return this.organizationCount;
}
/**
* Gets the service count.
*
* @return the service count
*/
public int getServiceCount()
{
return this.serviceCount;
}
/**
* Gets the user count.
*
* @return the user count
*/
public int getUserCount()
{
return this.userCount;
}
/**
* Gets the visit count.
*
* @return the visit count
*/
public int getVisitCount()
{
return this.visitCount;
@ -125,21 +150,45 @@ public class CategoryStat
this.visitCount += value;
}
/**
* Sets the organization count.
*
* @param organizationCount
* the new organization count
*/
public void setOrganizationCount(final int organizationCount)
{
this.organizationCount = organizationCount;
}
/**
* Sets the service count.
*
* @param serviceCount
* the new service count
*/
public void setServiceCount(final int serviceCount)
{
this.serviceCount = serviceCount;
}
/**
* Sets the user count.
*
* @param userCount
* the new user count
*/
public void setUserCount(final int userCount)
{
this.userCount = userCount;
}
/**
* Sets the visit count.
*
* @param visitCount
* the new visit count
*/
public void setVisitCount(final int visitCount)
{
this.visitCount = visitCount;