Improved bot detection.

2022-02-15 01:31:42 +01:00 · 2022-02-15 01:31:42 +01:00 · e64872aa9a
commit e64872aa9a
parent 224470e609
1 changed files with 7 additions and 0 deletions
--- a/src/fr/devinsy/statoolinfos/metrics/http/userAgentBotDetectorData.txt
+++ b/src/fr/devinsy/statoolinfos/metrics/http/userAgentBotDetectorData.txt
@ -1,4 +1,6 @@
+AhrefsBot
 ^Apache-HttpClient/
+ Barkrowler/
 bot
 BingPreview
 crawler
@ -17,6 +19,7 @@ internal dummy connection
 ^Java/
 ^JGit/
 Let's Encrypt validation server;
+ltx71 
 ^MastoPeek 
 ^mattermost-
 ^Misskey/
@ -24,6 +27,7 @@ internal dummy connection
 monitoring
 ^newspaper/
 ^node-fetch/
+Nutch-
 ^okhttp/
 ^PeerTube/
 ^PHP/
@ -31,6 +35,9 @@ monitoring
 ^python-requests/
 ^python/
 ^Python/
+^Python-urllib/
+ SemrushBot
 ^Synapse/
 ^Tusky/
 YisouSpider
+ zgrab/