What is a Web Application Firewall (WAF)

web application firewall (WAF) is an application firewall for HTTP applications. It applies a set of rules to an HTTP conversation. Generally, these rules cover common attacks vectors. While proxies generally protect clients, WAFs protect servers. Hence, a WAF protects a specific web application or set of web applications. Engineers consider a WAF a reverse proxy. WAFs may come in different forms, and the effort to perform this customization can be significant and as the application changes. In this post, we will explain how to use NGINX and OpenResty combined with Apility.io anti-abuse IP addresses service to deploy an NGINX WAF that will automatically deny access to the resources behind the firewall without the hassle of maintaining and customizing NGINX.

A simple NGINX WAF with Apility.io

We will show an example of how to integrate Apility.io API with NGINX to build a Web Application Firewall to secure the access to resources like web applications, API, and static content. We will integrate all these technologies:

NGINX is a Web Server which can also be used as a reverse proxy, load balancer, and HTTP cache. Nginx is free and open source software, released under the terms of a BSD-like license.

Together with the NGINX Web Server, we will use the OpenResty project. OpenResty is a full-fledged web platform by integrating the standard Nginx core, LuaJIT, many carefully written Lua libraries, lots of high-quality 3rd-party Nginx modules, and most of their external dependencies. Its design help developers easily build scalable web applications, web services, and dynamic web gateways.

Lua is a powerful, efficient, lightweight, embeddable scripting language. It supports procedural programming, object-oriented programming, functional programming, data-driven programming, and data description.

lua-resty-http is a Lua library to perform HTTP requests, used by the file filter.lua that implements the caching logic, combined with the nginx.conf configuration file and the default.conf example.

How to Install OpenResty

We will use a Linux Ubuntu Server 16.04. OpenResty bundles the NGINX web server, the OpenResty libraries and the LuaJIT. The community provides official pre-built packages for some of the common Linux distributions (Ubuntu, Debian, CentOS, RHEL, Fedora, and Amazon Linux). Moreover, there are pre-built Win32 packages and OpenResty for Mac OS X or macOS systems via homebrew package manager.

Now add the APT repository to your Ubuntu system. To add the repository, just run the following commands:

    # import our GPG key:
    wget -qO - https://openresty.org/package/pubkey.gpg | sudo apt-key add -

    # for installing the add-apt-repository command
    # (you can remove this package and its dependencies later):
    sudo apt-get -y install software-properties-common

    # add the our official APT repository:
    sudo add-apt-repository -y "deb http://openresty.org/package/ubuntu $(lsb_release -sc) main"

    # to update the APT index:
    sudo apt-get update

Then you can install the package OpenResty, like this:

    sudo apt-get install openresty

In order to test if OpenResty is ready, you can print a hello world output to the console thanks to the resty script shipped with OpenResty:

    resty -e 'print("hello, world!")'

It should print out the following line to the stdout device:

    hello, world

How to install Luarocks

LuaRocks is the package manager for Lua modules. It allows you to create and install Lua modules as self-contained packages called rocks. We need Luarocks to install the package lua-resty-http to perform HTTP requests to the Apility.io API endpoints. So, to install the package manager in Ubuntu we have to enter this command:

sudo apt-get install luarocks

In addition to luarocks installation, we now install the lua-resty-http as follows:

sudo luarocks install lua-resty-http

Given that we have installed all the packages successfully we can move on to the configuration step.

Configure nginx.conf and default.conf file

The way NGINX and its modules work is determined in the configuration file. By default, the configuration file is named nginx.conf and placed in the directory /usr/local/nginx/conf/etc/nginx, or /usr/local/etc/nginxBut this is not a standard NGIN. This is an OpenResty NGINX. Hence, the configuration file is in the directory /usr/local/openresty/nginx/conf/nginx.conf. This file contains the configuration needed to enable Lua and the caching objects to NGINX.

Now we are going to overwrite the nginx.conf file with our own nginx.conf file:

worker_processes 1;
error_log stderr notice;
events {
    worker_connections 1024;
}
env APILITYIO_URL;
env APILITYIO_LOCAL_CACHE_TTL;
env APILITYIO_API_KEY;

http {
    variables_hash_max_size 1024;
    access_log off;
    include /usr/local/openresty/nginx/conf/mime.types;
    real_ip_header X-Real-IP;
    charset utf-8;
    lua_shared_dict ip_cache 1m;
    init_by_lua '
        require "resty.core"
    ';

    include /etc/nginx/conf.d/*.conf;
}

This file has the minimal NGINX WAF configuration. We can see how it starts the resty.core with the init_by_lua directive. The directive lua_shared_dict inits the cache of IP addresses to 1 megabyte (you can increase or decrease this value if you want). The script also defines three environment variables needed by the filter.lua, APILITY_URL, APILITYIO_LOCAL_CACHE_TTL and APILITY_API_KEY.

At the bottom of the file, we can see the includes of local configuration files in /etc/nginx/conf.d/. This local files will include the configuration of each of the different sites protected by your NGINX WAF.

Create the directory /etc/nginx/conf.d/, and copy this file inside with the name default.conf:

server {
    listen 80;
    lua_code_cache on;
    error_log /usr/local/openresty/nginx/logs/filter-err.log;

    # Sample API gateway with access filter to externally hosted API (http://mockbin.org)
    location ~* ^/apility/(.*) {
        access_by_lua_file      "filter.lua";
        resolver                8.8.8.8;  # use Google's open DNS server for example

        set $backend            "api.apility.net";
        set $url_full           '$1';
        proxy_pass              http://$backend/$url_full;
        proxy_set_header        Host $backend;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_redirect          off;
        proxy_read_timeout      90;
    }
}

Don’t forget to save it and move to the next step.

Filter.lua

filter.lua implements the logic that queries the Apility.io Abuse IP addresses services everytime the NGINX WAF handles a new request. This function implements also a caching logic to avoid unnecessary requests to our API. Each entry in this cache has a Time to Live (TTL) that can be defined with the environment variable APILITYIO_LOCAL_CACHE_TTL. The function also uses APILITYIO_URL to store the endpoint of the Apility.io Services http://api.apility.net and APILITYIO_API_KEY to keep the API Key of the service. You can obtain a valid API Key if you register in Apility.io (it’s free).

You must place the file filter.lua in the folder /usr/local/openresty/nginx/:

local endpoint_url    = os.getenv("APILITYIO_URL") .. "/badip/"
local cache_ttl     = tonumber(os.getenv("APILITYIO_LOCAL_CACHE_TTL"))
local x_auth_token = os.getenv("APILITYIO_API_KEY")
local all_headers = {}

if x_auth_token then
    all_headers = { ["X-Auth-Token"] = x_auth_token }
end

local ip = ngx.var.remote_addr
local ip_cache = ngx.shared.ip_cache

-- check first the local blacklist
ngx.log(ngx.DEBUG, "ip_cache: Look up in local cache "..ip)
local cache_result = ip_cache:get(ip)
if cache_result then
  ngx.log(ngx.DEBUG, "ip_cache: found result in local cache for "..ip.." -> "..cache_result)
  if cache_result == 200 then
    ngx.log(ngx.DEBUG, "ip_cache: (local cache) "..ip.." is blacklisted")
    return ngx.exit(ngx.HTTP_FORBIDDEN)
  else
    ngx.log(ngx.DEBUG, "ip_cache: (local cache) "..ip.." is whitelisted")
    return
  end
else
  ngx.log(ngx.DEBUG, "ip_cache: not found in local cache "..ip)
end

-- Nothing in local cache, go and do a roundtrip to apility.io API
local http = require "resty.http"
local httpc = http.new()
      local res, err = httpc:request_uri(endpoint_url .. ip,  {
        method = "GET",
        ssl_verify = false,
        headers = all_headers
      })

-- Something went wrong...
if not res then
  ngx.say("failed to request: ", err)
  return
end

local status = res.status

ip_cache:set(ip, status, cache_ttl)

if res.status == 404 then
  ngx.log(ngx.DEBUG, "blacklist: lookup returns nothing "..ip..":"..res.status)
  return
end

if res.status == 200 then
  ngx.log(ngx.DEBUG, "whitelist: lookup returns something "..ip..":"..res.status)
  return ngx.exit(ngx.HTTP_FORBIDDEN)
end

if res.status == 409 then
  ngx.log(ngx.DEBUG, "whitelist: lookup run out of quota "..ip..":"..res.status)
  return ngx.exit(ngx.HTTP_FORBIDDEN)
end

-- You should never be here...
ngx.log(ngx.ERR, "ip_cache: "..ip.." something went wrong...")
return ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)

Set the environment variables

To set the environment variables for the OpenResty NGINX service, create the file /etc/default/openresty and paste the following variables:

export APILITYIO_URL=http://api.apility.net
export APILITYIO_LOCAL_CACHE_TTL=SECONDS
export APILITYIO_API_KEY=USER_API_KEY

Don’t forget to modify SECONDS for the TTL for each entry of the cache, and USER_API_KEY with your own API Key. If you leave APILITY_API_KEY empty it will work but the rate-limit and the daily limit will be smaller than a free API Key or a paid plan.

Start and Stop the OpenResty NGINX WAF

To start Openresty:

# . /etc/init.d/openresty start

And to stop Openresty:

# . /etc/init.d/openresty stop

Test the NGINX WAF

The sample configuration points to the Apility.io endpoint through the NGINX WAF we configured but through the  URI /apility/*. Once the service is up and running, open a browser and enter the following URL https://<YOUR_SERVER>/apility/geoip. The servicegeoip without parameters returns the IP of the remote origin. The browser should display the JSON object with the information:

NGINX WAF Apility.io Sample

If your Origin IP is clean and does not belong to any blacklist, then you will get the full response. But if the IP is blacklisted (for example connecting using the TOR network) then the access to the API will be forbidden:

NGINX WAF Tor forbidden

Now go and open the IP Activity Dashboard you will see the requests made from the NGINX WAF.

NGINX WAF Activity

Source code and Docker image

You can consider this post as an example of how to integrate Apility.io with your services. You can find the source code of the example and a fully functional image in the Docker Hub. Enjoy!