The Architecture has two main components: the web traffic classification server and the Yona server. The web traffic classification server is SmoothWall, running on custom Linux distribution. This server classifies all HTTP requests and writes this classification in a log file. The requests in categories that are linked to goals in the Yona server need to be passed on to the Yona server analysis engine. To do this, we need a small program that reads the log file and passes the relevant requests to the Yona server. Given the custom Linux distribution, there is only one dependable technology for this program: Perl.
This page describes the Perl script.
The Perl script operates in the following context:
The SmoothWall server produces a log file (
/var/log/dansguardian3/access.log) with a (JSON) record for each HTTP request. We will replace the log file with a named pipe, written by SmoothWall and read by the Perl script. To know what categories are relevant, the script pulls the list of goals from the Yona server. For each relevant requests, it performs a post on the analysis engine.
The script has the following usage:
goalsURL: the URL from where the goals can be pulled. Default value: http://localhost:8080/goals/
analysisEngineURL: the URL where the relevant messages are to be posted. Default value: http://localhost:8080/analysisEngine/
- input file: the file to be read. If the file is a named pipe, the script will enter an endless loop, thus resuming the read operation after a restart of SmoothWall. Default value: STDIN
Latest version can be found at https://github.com/yonadev/yona-server/blob/master/scripts/smoothwall/HandleDansGuardianLog.pl
Installing on the Smoothwall server
- Copy the script to /usr/local/sbin/HandleDansGuardianLog.pl
- Make the directory /usr/local/sbin if it doesn't exist yet.
- In the repository location https://github.com/yonadev/yona-server/tree/master/scripts/smoothwall you will see a
yona-log-parserbash script - This is an init file. Place this as:
- Change the header of the above init file to make sure that the details are correct for OPTIONS (ip of the Analysis engine, and the log file name
- On the smoothwall server. Make sure the executable bit is set with
chmod +x yona-log-parser
- The second file is
9399yona-log-parser- This is the 'hook' file to start the init during the boot process. The 9399 places it in a specific order. It is possible that future changes to Smoothwall product might require the order to change. Place this file as:
- Also make sure the executable bit is set with
chmod +x 9399yona-log-parser
- On boot, the system will start the parser.
- You can manually start/stop with
/etc/init.d/yona-log-parser [ start | stop | restart ]
In the repository location https://github.com/yonadev/yona-server/tree/master/scripts/smoothwall you will see a yona-logrotate configuration file. Place this as :
Monitoring the HandleDansGuardianLog process
To be detailed, but in general the process will be watching the stdout of the script for the timestamped entries that a category refresh was done. This will happen every categories_refresh_interval defined in the perl script. It defaults to 300 seconds (5 minutes).
It will output a log line like this
Thu Sep 7 22:09:52 2017 INFO Finished loading relevant categories
Parse out the date from the last instance of this in the logs, compare with current and if > categories_refresh_interval + some fudge factor for how long a category pull migth take, triger some kind of alert event.