Export blacklists databases of IP addresses or Domains

You are here:

What is Export blacklists feature?

Customers who have signed up for the Enterprise or Professional plans can download the full databases of IP addresses or domains that we use at Apility.io in our malicious content search processes. The difference between the Enterprise and the Professional plan is related to the’freshness’ of the data of the information that can be exported:

  • The Professional Plan allows you to download the databases corresponding to the dump carried out on the current day at 00:00 UTM. That is, the data has a maximum lifespan of 24 hours.
  • The Enterprise Plan allows you to download the databases of the dump carried out every hour on the hour. That is, the data has a maximum lifespan of 60 minutes.

The Enterprise plan is almost an exact copy of the operational databases used in Apility.io’s processes and APIs and is the ideal choice for users with a high volume of requests and low latency needs.

How to export a database

To export blacklists databases we can do it in two ways:

  • From the web interface of the management dashboard.
  • Using the Service API.

To enter the Export screen we look for the Export Databases option in the left menu (not to be confused with Export Your Data):

Apility Export Databases Menu

 

After opening the screen with the export options, at the top we can see the different parameters. On the left, the two available databases:

  • IP Addresses: IP addresses and CIDR blocks found in our abuse databases.
  • Domains: which are found in our domain databases and are also used for abuse detection. in email.

To the right are the three formats available for download:

  • CSV: Comma Separated Values, which separate the key (IP address or Domain) from the list of lists where that key is found.
  • JSON: JSON format where we have an array containing objects composed of the key and the list of key lists.
  • SQL: SQL file containing the INSERT commands to add the keys and their list lists to any relational engine.

Command generation

Immediately below the filters, we have a window where we can see how to modify the CURL command needed to download the file with the selected database in our system. To get the command we only have to press the button Copy Command to Clipboard or if the permissions of our browser are limited cut and paste the contents of the dark window with the green text.

Apility Export databases Curl

Download file with the browser

If the user wants to download the latest version of the database to his or her local computer using the browser, this can be done by clicking on the Download button below.

File formats

Zipped file

All files come compressed with the ZIP format. The user will have to uncompress the file before accessing the content.

CSV

Both IP addresses and Domains share the same format. There is no header describing the content, so the file will have thousands of rows. Each row will have two comma-separated fields:

  • Key field: It can be an IP, a CIDR block or a domain.
  • List of lists field: It will contain a list of the names of the blacklists separated by a semicolon.

Example:

1.10.140.106,UDGER-WEB-PROXY
1.10.140.20,ALIENVAULT-REPUTATION
1.10.141.109,UCEPROTECT-LEVEL1
1.10.141.121,STOPFORUMSPAM-1;STOPFORUMSPAM-90;STOPFORUMSPAM-30;STOPFORUMSPAM-7;STOPFORUMSPAM-180;STOPFORUMSPAM-365;UDGER-WEB-PROXY
1.10.141.31,UCEPROTECT-LEVEL1

JSON

The JSON format returns an array that contains two different type of JSON objects depending on the data type.

The format of the IP addresses object is IP addresses or CIDR blocks and lists for the list of blacklists. The list of lists is a semicolon separated list of blacklist names:

[
...
    {
        "ip": "1.10.140.106",
        "lists": "UDGER-WEB-PROXY"
    },
    {
        "ip": "1.10.140.20",
        "lists": "ALIENVAULT-REPUTATION"
    },
    {
        "ip": "1.10.141.109",
        "lists": "UCEPROTECT-LEVEL1"
    },
...
]

Same applies for the domains:

[
...
    {
        "domain": "0-google.com",
        "lists": "SQUIDBLACKLIST-MALICIOUS-DOMAINS"
    },
    {
        "domain": "0-mail.com",
        "lists": "FREEMAIL;IVOLO-DED;MARTENSON-DED;LISINGE-DED"
    },
    {
        "domain": "0.2090000.ru",
        "lists": "SQUIDBLACKLIST-MALICIOUS-DOMAINS"
    },
...
]

SQL

Both IP addresses and Domains share the same format. Each INSERT command will try to insert into two fields in a table named ‘DB_<DATA_TYPE>’, where data_type can be the values IP or DOMAIN. The columns of the table should have string columns with names ‘KEY’ and ‘LISTS’:

  • Key column (KEY): It can be an IP, a CIDR block or a domain.
  • List of lists column (LISTS): It will contain a list of the names of the blacklists separated by a semicolon.

Examples:

For IP addresses:

INSERT INTO db_ip ('key','lists') VALUES ('1.10.140.106','UDGER-WEB-PROXY');
INSERT INTO db_ip ('key','lists') VALUES ('1.10.140.20','ALIENVAULT-REPUTATION');
INSERT INTO db_ip ('key','lists') VALUES ('1.10.141.109','UCEPROTECT-LEVEL1');
INSERT INTO db_ip ('key','lists') VALUES ('1.10.141.121','STOPFORUMSPAM-1;STOPFORUMSPAM-90;STOPFORUMSPAM-30;STOPFORUMSPAM-7;STOPFORUMSPAM-180;STOPFORUMSPAM-365;UDGER-WEB-PROXY');
INSERT INTO db_ip ('key','lists') VALUES ('1.10.141.31','UCEPROTECT-LEVEL1');

For domains:

INSERT INTO db_domain ('key','lists') VALUES ('0-google.com','SQUIDBLACKLIST-MALICIOUS-DOMAINS');
INSERT INTO db_domain ('key','lists') VALUES ('0-mail.com','LISINGE-DED;IVOLO-DED;MARTENSON-DED;FREEMAIL');
INSERT INTO db_domain ('key','lists') VALUES ('0.2090000.ru','SQUIDBLACKLIST-MALICIOUS-DOMAINS');

Sample files

To try the file formats you can download some sample files here:

These files only contain a few dozens of elements. The final files contain hundreds of thousands of rows and you will need enough computing power to ingest this data.

Leave a Reply

Your email address will not be published. Required fields are marked *