Skip to end of metadata
Go to start of metadata



For the request interception strategy, we have concluded that using VPN is the only viable option. Given that choice, there are still options to consider, for instance regarding the VPN protocol to use, where to end the VPN tunnel, what product to use to implement the VPN, etc. This page discusses these options and ends with a conclusion.

(info) Note that the below text does not distinguish HTTP from HTTPS. In theory, we can handle both protocols equally well, through the MITM approach described on the page Intercepting HTTPS traffic, but in practice we are not using this today because of issues with mobile apps.

Solution options

This section describes a number of solution options:

Local VPN

The purpose of the VPN is to force all network traffic through the classification engine (practically: Smoothwall). This approach has the benefit of making everything available to the classification engine (not just the URL, but also the full request and response, including payload and headers). This comes at the cost of having to tunnel all network traffic of the user through the VPN tunnel and the Yona server running the classification engine. This puts a considerable bandwidth demand on the Yona servers, causes a potential bottleneck for the user and a considerable battery drain, due to the encryption that VPNs apply.

On Android, it is possible use tun2socks to set up a VPN service on the device that forwards all traffic to a SOCKS server that could be running on the device itself. That SOCKS server could work in two ways:

  1. Route the traffic through the classification engine
    The SOCKS server would forward all requests to two destinations:
    • To the actual destination, for regular processing
    • To the classification engine, for classification. The classification engine should not serve the responses, to limit the network bandwidth.
  2. Send only the URLs to the classification engine
    The SOCKS server would need to do two things:
    • Interpret the requests and send all URLs to the classification engine for classification (can be done asynchronously).
    • Forward all requests to the actual destination

Instead of sending all requests or URLs to the classification engine, we could sample only a percentage. That could give a drastic reduction in load and still give a high probability of a good classification.

Comparison of options ((lightbulb) is positive, (grey lightbulb) is negative):

Traffic through classification engineOnly URLs sent to classification engineComment

Sufficient info for classification

(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(grey lightbulb)If the traffic flows through the classification engine, it gets the full requests and responses, so has all possible data to make the right classification decision. With only URLs, it cannot do much..
Reduced network bandwidth(lightbulb)(grey lightbulb)(grey lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)If the traffic flows through the classification engine, the Yona server needs to do all requests, resulting in the same amount of network traffic as before (actually slightly more, as all requests are sent twice).
Functionally correct(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(grey lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)(lightbulb)If the traffic flows through the classification engine, all requests are executed twice. This will lead to issues with some web servers.
Ease of implementation(lightbulb)(lightbulb)(lightbulb)(lightbulb)(grey lightbulb)(grey lightbulb)(grey lightbulb)(grey lightbulb)(grey lightbulb)(grey lightbulb)The difficulty of the first option is to prevent from returning all responses from the classification engine. The difficulty with the second one is that the app needs to interpret and understand the network requests. That looks like a very difficult task.

Concluding: The first option retains our current classification strength and is relatively easy to implement, but has very little benefits. The second one will reduce the classification accuracy and is very difficult to implement. Neither seems particularly attractive.

Nonencrypting VPN

The tun2socks solution mentioned above could also be used to send the network traffic from the device to a SOCKS endpoint on the Smoothwall server (assuming supports this). This would take away the encryption burden from the device, thus reducing the battery drain.

Efficient VPN protocol

At the moment, we use SSL VPN with OpenVPN. This has several downsides. On Android, we are now embedding OpenVPN in the app. This works well but makes the app bigger than necessary and spreads the GPL license to the app. On iOS, we cannot bundle OpenVPN because of the GPL license, so users need to download and install it manually, which us cumbersome. Besides the issues with OpenVPN, the battery drain would be less and the speed more if we could use a VPN protocol that is native to the devices.

The alternative would be to use IPSec and on Android then use the IKEv2 key exchange, as that is known to be very efficient. Unfortunately, Smoothwall does not support IKEv2. To get around that, we could consider not to use the VPN option of Smoothwall but instead set up different VPN provider, but that brings the difficulty of passing the user identity from the VPN tunnel to Smoothwall


To be done.

  • No labels


  1. I will be able to review and comment more next week.   I am preparing for my talk at LinuxFest Northwest.

  2. I do not understand all technical details and would like to suggest to involve Jelte from Kliksafe and possibly somebody from Smoothwall.

    1. Yes, we should, but I'd rather first get the reactions of Richard Clark and Anton. Can you ask Anton to respond?

  3. 1) Do we know that Smoothwall is doing classification on the 'content' layer of pages ?   If so, any idea how much of the classification accuracy is dependent on that, or is the URL database from previous knowledge so vast, that the content is only a small factor in classification ?

    If we need deep content classification, then the 'fake' side requests are going to have to happen without any user context.  This will probably often result in just 'login' page like material, and to the vendor website, might make Yona servers look hostile (trying to it URLS that make no sense for it to hit, not being a logged in user).

    This could also deal with the HTTPS issue though I think as we're 'the client'.   Though would still need to MITM with Smoothwall if that's the only way we can get classification instead of some undocumented API submission (send in page content, url, etc. and get classification result).

    (update - Yes, they do deep content analysis.)

    2) HTTPS / MITM We're not doing this now.  If we don't do it going forward, then smoothwall content classification is pointless anyway.  Its getting rare to see non-https websites.  At least of we do a local socks proxy system, we would at least get the SNI URL to pass on for classification.

    (Update - Since we want deep content, we need to do MITM, but since SNI doesn't give us the full url needed to do deep content on say, a erotic seller on ebay, we would need to do MITM at the first proxy layer, on device)

    3) I think there needs to be a software component between the Device proxy software, and the Smoothwall classification.  This could also be the system with the "vpn" endpoint software (IPSec, etc.).  It would just take the data from the Device proxy, ACK it so device can continue on with life, then this layer will deal with Smoothwall classification.

    4) Caching.   We could probably easily cache classification results based on URL - building our own LRU cache so we don't have to fully crawl to the target site.  This might help mitigate Yona looking like an attacker.

    We need to do some proof of concepts I think.