Skip to end of metadata
Go to start of metadata

Introduction

The Architecture has two main components: the web traffic classification server and the Yona server. The web traffic classification server is SmoothWall, running on custom Linux distribution. This server classifies all HTTP requests and writes this classification in a log file. The requests in categories that are linked to goals in the Yona server need to be passed on to the Yona server analysis engine. To do this, we need a small program that reads the log file and passes the relevant requests to the Yona server. Given the custom Linux distribution, there is only one dependable technology for this program: Perl.

This page describes the Perl script.

Context

The Perl script operates in the following context:

Context

The SmoothWall server produces a log file (/var/log/dansguardian3/access.log) with a (JSON) record for each HTTP request. We will replace the log file with a named pipe, written by SmoothWall and read by the Perl script. To know what categories are relevant, the script pulls the list of goals from the Yona server. For each relevant requests, it performs a post on the analysis engine.

Command line

The script has the following usage:

Usage: HandleDansGuardianLog.pl [--goalsURL <URL>] [--analysisEngineURL <URL>] [<input file>]
  • goalsURL: the URL from where the goals can be pulled. Default value: http://localhost:8080/goals/
  • analysisEngineURL: the URL where the relevant messages are to be posted. Default value: http://localhost:8080/analysisEngine/
  • input file: the file to be read. If the file is a named pipe, the script will enter an endless loop, thus resuming the read operation after a restart of SmoothWall. Default value: STDIN

Script

Latest version can be found at https://github.com/yonadev/yona-server/blob/master/scripts/smoothwall/HandleDansGuardianLog.pl

Installing on the Smoothwall server

  1. Copy the script to /usr/local/sbin/HandleDansGuardianLog.pl
  2. Make the directory /usr/local/sbin if it doesn't exist yet.
  3. In the repository location https://github.com/yonadev/yona-server/tree/master/scripts/smoothwall you will see a yona-log-parser bash script - This is an init file.  Place this as:
    /etc/init.d/yona-log-parser
  4. Change the header of the above init file to make sure that the details are correct for OPTIONS (ip of the Analysis engine, and the log file name
  5. On the smoothwall server.  Make sure the executable bit is set with chmod +x yona-log-parser
  6. The second file is 9399yona-log-parser - This is the 'hook' file to start the init during the boot process. The 9399 places it in a specific order.  It is possible that future changes to Smoothwall product might require the order to change.  Place this file as:
    /etc/actions/secondboot/9399yona-log-parser
  7. Also make sure the executable bit is set with chmod +x 9399yona-log-parser
  8. On boot, the system will start the parser.  
  9. You can manually start/stop with 
    /etc/init.d/yona-log-parser [ start | stop | restart ]

Log Rotation

In the repository location https://github.com/yonadev/yona-server/tree/master/scripts/smoothwall you will see a yona-logrotate configuration file. Place this as :

/etc/logrotate.d/yona-logrotate

Monitoring the HandleDansGuardianLog process

To be detailed, but in general the process will be watching the stdout of the script for the timestamped entries that a category refresh was done.  This will happen every categories_refresh_interval defined in the perl script.  It defaults to 300 seconds (5 minutes).

It will output a log line like this

Thu Sep  7 22:09:52 2017 INFO Finished loading relevant categories

in /var/log/HandleDansGuardianLog.log

Parse out the date from the last instance of this in the logs, compare with current and if > categories_refresh_interval + some fudge factor for how long a category pull migth take, triger some kind of alert event.


 

TODO list

Key Summary T Created Updated Due Assignee Reporter P Status Resolution
Loading...
Refresh

  • No labels