ELK Log Analysis System: A Comprehensive Guide to Installation and Configuration

Devops
1

ELK Log Analysis System: A Comprehensive Guide to Installation and Configuration

System Requirements

The ELK (Elasticsearch, Logstash, Kibana) log analysis system is a powerful tool for collecting, processing, and visualizing log data. To install and configure the ELK system, you will need a server with the following specifications:

  • More than 2 CPU cores
  • A stable and secure network connection

Step 1: Server Preparation

Before installing the ELK system, you need to prepare your server by installing the necessary packages. The following commands will install the required packages:

yum install java-1.8.0
yum install unzip

Additionally, you need to install the GCC compiler and the TCL package:

yum install gcc
yum install tcl

Step 2: Firewalld Configuration

To ensure that the local browser can access the virtual machine, you need to disable the firewalld service:

systemctl stop firewalld
systemctl mask firewalld

Step 3: ELK Package Download

The ELK package can be downloaded from the following locations:

Download the latest version of Elasticsearch, Logstash, and Kibana, and extract the files to the /ELK directory in the virtual machine.

Step 4: ELK Software Installation

To install the ELK system, follow these steps:

  • Install Logstash:
tar zxf logstash-6.0.0.tar.gz -C /usr/local/
echo "export PATH=\$PATH:/usr/local/logstash-6.0.0/bin"> /etc/profile.d/logstash.sh
. /etc/profile.d/logstash.sh
  • Install Redis:
tar zxf redis-4.0.2.tar.gz
cd redis-4.0.2
make MALLOC=libc
make test
make install
  • Start Redis:
cd utils
./install_server.sh
netstat -tnlp | grep redis
  • Start Redis dynamic monitoring:
cd ../src
./redis-cli monitor
  • Configure Logstash to write to Redis:
vi logstash-to-redis.conf
input {
  stdin {
  }
}
output {
  stdout {
    codec => rubydebug
  }
  redis {
    host => '192.168.13.245'
    data_type => 'list'
    key => 'logstash:redis'
  }
}
  • Run Logstash:
logstash -f logstash-to-redis.conf --verbose
  • Test Logstash and Redis:
./redis-cli monitor

Step 5: Elasticsearch Installation

To install Elasticsearch, follow these steps:

  • Install Elasticsearch:
tar zxf elasticsearch-6.0.0.tar.gz -C /usr/local/
  • Modify the Elasticsearch configuration file:
vi /usr/local/elasticsearch-6.0.0/config/elasticsearch.yml
  • Start Elasticsearch:
nohup /usr/local/elasticsearch-6.0.0/bin/elasticsearch> /ELK/logs/es.log 2> & 1 &
  • Verify that Elasticsearch is running:
netstat -tnlp | grep java
  • Configure Logstash to write to Elasticsearch:
vi logstash-simple.conf
input {
  stdin {
  }
}
output {
  elasticsearch {
    hosts => ['192.168.13.245:9200']
  }
  stdout {
    codec => rubydebug
  }
}
  • Run Logstash:
/usr/local/logstash-6.0.0/bin/logstash -f /ELK/logstash-simple.conf
  • Verify that Logstash is writing to Elasticsearch:
curl http://192.168.13.245:9200/_search?pretty

Step 6: Cerebro Installation

To install Cerebro, follow these steps:

  • Download Cerebro:
wget https://github.com/lmenezes/cerebro/releases/download/v0.7.2/cerebro-0.7.2.zip
  • Unzip Cerebro:
unzip cerebro-0.7.2.zip
  • Start Cerebro:
nohup bin/cerebro> /ELK/logs/cerebro.log 2> & 1 &
  • Access Cerebro:
http://192.168.13.245:9000

Step 7: Kibana Installation

To install Kibana, follow these steps:

  • Download Kibana:
wget https://www.elastic.co/downloads/kibana/kibana-6.0.0-linux-x86_64.tar.gz
  • Extract Kibana:
tar zxf kibana-6.0.0-linux-x86_64.tar.gz -C /usr/local/
  • Modify the Kibana configuration file:
vi /usr/local/kibana-6.0.0-linux-x86_64/config/kibana.yml
  • Start Kibana:
nohup /usr/local/kibana-6.0.0-linux-x86_64/bin/kibana> /ELK/logs/kibana.log 2> & 1 &
  • Access Kibana:
http://192.168.13.245:5601

Congratulations! You have successfully installed and configured the ELK log analysis system.