Component: Logstash, Elasticsearch, Kibana

Logstash

Install Logstash with this tutorial. The configuration files for logstash (like 10-logstash.conf) are included in this repository.

Create a symlink to the configuration file in the cloned git project with: sudo ln -s <location_configfile> /etc/logstash/conf.d/<link_name>

Logstash can be executed with one of following commands (after installing Elasticsearch also):

• sudo -E /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/10-logstash-shodan.conf
  Executes Logstash with the specified configuration file.
• sudo -E /usr/share/logstash/bin/logstash --path.settings=/etc/logstash/conf.d
  Executes Logstash with all configuration files in /etc/logstash/conf.d/

Command flags which may come in handy:

• -t Test given configuration file on syntax errors;
• --verbose Extra console output for debugging.

Logstash may take a long time (> 30 seconds) to start. This could possibly be fixed with these instructions.

You can see Logstash output in the console or in /var/log/logstash/logstash-plain.log

Elasticsearch

Install Elasticsearch with this tutorial.

In elasticsearch.yml the following values are changed for this project:

Using 1 VM

• cluster.name: <cluster_name>
• network.host: <localhost>

Using multiple VMs

• cluster.name: <cluster_name>
• network.host: <IP_of_VM>
  WARNING: make sure port 9200-9400 is open only to the other VMs of this project!
• discovery.zen.ping.unicast.hosts: ["<IP>", "<IP>"] Set this value with the other VMs IP-addresses
• node.name: <node_name>

Kibana

Install Kibana with this tutorial.

• Make sure Kibana (default port 5601) is only reachable via localhost, and via the NGINX proxy for remote access.
• Elasticsearch needs a specific mapping to import the data from the scripts. Use the mapping on the page "Required Elasticsearch mapping for indices".
• TIP: Kibana's "Dev Tools" page is very useful to perform actions on the underlying Elasticsearch cluster.
• More info about Kibana's functionalities can be found here.