Why Apility.io on-premise?

When I created Apility.io I decided that I didn’t want to follow the traditional start-up growth process. There are several reasons: my personal experience in previous companies, the change in how to invest in the companies… But I think I am good at seeing the trends in the technology market but very bad at guessing the right timing. I believe that how services are going to be consumed will be increasing ‘as a Service’ but more and more abstracted from the peculiarities of the infrastructure and the underlying technological solution. In other words, the’serverless’ concept will gradually become the new norm and will become the norm in a few years’ time.

Apility.io is a serverless database designed to make searching IP address lists, domains, and emails as fast and easy as possible. My vision is to offer what you would get with a Content Delivery Network but with dynamic data. If you are close to one of our ‘satellite’ nodes the response times are in milliseconds, which makes unnecessary to deploy your own infrastructure. But as I said before, this is a vision, and only a very selected group of early adopters are ready in 2018 to fully adopt this model.

However, during these months the question I have been answering the most is ‘Can I install an Apility.io “satellite” in my datacenter?’ And yes, it is possible, but there are security implications that cannot be solved in the short term. But there is a middle ground: downloading an updated version of our databases.

Who can download the databases?

Customers who have signed up for the Enterprise or Professional plans can download the full databases of IP addresses or domains that we use at Apility.io in our malicious content search processes. The difference between the Enterprise and the Professional plan is related to the ’freshness’ of the data of the information that can be exported:

  • The Professional Plan allows you to download the databases corresponding to the dump carried out on the current day at 00:00 UTM. That is, the data has a maximum lifespan of 24 hours.
  • The Enterprise Plan allows you to download the databases of the dump carried out every hour on the hour. That is, the data has a maximum lifespan of 60 minutes.

The Enterprise plan is almost an exact copy of the operational databases used in Apility.io’s processes and APIs and is the ideal choice for users with a high volume of requests and low latency needs

How to export a database

To export blacklists databases we can do it in two ways:

  • From the web interface of the management dashboard.
  • Using the Service API.

To enter the Export screen we look for the Export Databases option in the left menu (not to be confused with Export Your Data):

Apility Export Databases Menu

 

After opening the screen with the export options, at the top we can see the different parameters. On the left, the two available databases:

  • IP Addresses: IP addresses and CIDR blocks found in our abuse databases.
  • Domains: which are found in our domain databases and are also used for abuse detection. in email.

To the right are the three formats available for download:

  • CSV: Comma Separated Values, which separate the key (IP address or Domain) from the list of lists where that key is found.
  • JSON: JSON format where we have an array containing objects composed of the key and the list of key lists.
  • SQL: SQL file containing the INSERT commands to add the keys and their list lists to any relational engine.

Command generation

Immediately below the filters, we have a window where we can see how to modify the CURL command needed to download the file with the selected database in our system. To get the command we only have to press the button Copy Command to Clipboard or if the permissions of our browser are limited cut and paste the contents of the dark window with the green text.

Apility Export databases Curl

Download file with the browser

If the user wants to download the latest version of the database to his or her local computer using the browser, this can be done by clicking on the Download button below.

File formats

Zipped file

All files come compressed with the ZIP format. The user will have to uncompress the file before accessing the content.

CSV

Both IP addresses and Domains share the same format. There is no header describing the content, so the file will have thousands of rows. Each row will have two comma-separated fields:

  • Key field: It can be an IP, a CIDR block or a domain.
  • List of lists field: It will contain a list of the names of the blacklists separated by a semicolon.

Example:

1.10.140.106,UDGER-WEB-PROXY
1.10.140.20,ALIENVAULT-REPUTATION
1.10.141.109,UCEPROTECT-LEVEL1
1.10.141.121,STOPFORUMSPAM-1;STOPFORUMSPAM-90;STOPFORUMSPAM-30;STOPFORUMSPAM-7;STOPFORUMSPAM-180;STOPFORUMSPAM-365;UDGER-WEB-PROXY
1.10.141.31,UCEPROTECT-LEVEL1

JSON

The JSON format returns an array that contains two different type of JSON objects depending on the data type.

The format of the IP addresses object is IP addresses or CIDR blocks and lists for the list of blacklists. The list of lists is a semicolon separated list of blacklist names:

[
...
    {
        "ip": "1.10.140.106",
        "lists": "UDGER-WEB-PROXY"
    },
    {
        "ip": "1.10.140.20",
        "lists": "ALIENVAULT-REPUTATION"
    },
    {
        "ip": "1.10.141.109",
        "lists": "UCEPROTECT-LEVEL1"
    },
...
]

Same applies for the domains:

[
...
    {
        "domain": "0-google.com",
        "lists": "SQUIDBLACKLIST-MALICIOUS-DOMAINS"
    },
    {
        "domain": "0-mail.com",
        "lists": "FREEMAIL;IVOLO-DED;MARTENSON-DED;LISINGE-DED"
    },
    {
        "domain": "0.2090000.ru",
        "lists": "SQUIDBLACKLIST-MALICIOUS-DOMAINS"
    },
...
]

SQL

Both IP addresses and Domains share the same format. Each INSERT command will try to insert into two fields in a table named ‘DB_<DATA_TYPE>’, where data_type can be the values IP or DOMAIN. The columns of the table should have string columns with names ‘KEY’ and ‘LISTS’:

  • Key column (KEY): It can be an IP, a CIDR block or a domain.
  • List of lists column (LISTS): It will contain a list of the names of the blacklists separated by a semicolon.

Examples:

For IP addresses:

INSERT INTO db_ip ('key','lists') VALUES ('1.10.140.106','UDGER-WEB-PROXY');
INSERT INTO db_ip ('key','lists') VALUES ('1.10.140.20','ALIENVAULT-REPUTATION');
INSERT INTO db_ip ('key','lists') VALUES ('1.10.141.109','UCEPROTECT-LEVEL1');
INSERT INTO db_ip ('key','lists') VALUES ('1.10.141.121','STOPFORUMSPAM-1;STOPFORUMSPAM-90;STOPFORUMSPAM-30;STOPFORUMSPAM-7;STOPFORUMSPAM-180;STOPFORUMSPAM-365;UDGER-WEB-PROXY');
INSERT INTO db_ip ('key','lists') VALUES ('1.10.141.31','UCEPROTECT-LEVEL1');

For domains:

INSERT INTO db_domain ('key','lists') VALUES ('0-google.com','SQUIDBLACKLIST-MALICIOUS-DOMAINS');
INSERT INTO db_domain ('key','lists') VALUES ('0-mail.com','LISINGE-DED;IVOLO-DED;MARTENSON-DED;FREEMAIL');
INSERT INTO db_domain ('key','lists') VALUES ('0.2090000.ru','SQUIDBLACKLIST-MALICIOUS-DOMAINS');

Sample files

To try the file formats you can download some sample files here:

These files only contain a few dozens of elements. The final files contain hundreds of thousands of rows and you will need enough computing power to ingest this data.

A live example

If you want to see how to import our database and use it in your own projects you can have a look at our GitHub repository and check our Cloud Provides Abusers sample. This script search for IP addresses of services providers in the downloaded blacklists of Apility.io. Due to the size of the IP addresses computing pool of these providers (several million) a downloaded database of IP addresses and blacklists is the right solution.