Integrating Spring Boot with ELK

Web Technologies
1

Integrating Spring Boot with ELK

In this article, we will delve into the integration of Spring Boot with the ELK (Elasticsearch, Logstash, and Kibana) stack. We will cover the configuration of the ELK stack, the creation of a Spring Boot project, and the logging of Spring Boot applications to Elasticsearch using Logstash.

ELK Stack Overview

The ELK stack is an open-source, distributed logging system that provides a scalable and flexible way to collect, process, and visualize log data. The components of the ELK stack are:

  • Elasticsearch: An open-source, distributed search engine that provides a robust and scalable way to store and search log data.
  • Logstash: A fully open-source tool that can collect log data from various sources, filter it, and store it in Elasticsearch for later use.
  • Kibana: An open-source, free tool that provides a user-friendly web interface for analyzing and visualizing log data stored in Elasticsearch.

ELK Installation

To install the ELK stack, we will use Docker images. We will first pull the Elasticsearch image and then run it in a container with the necessary memory settings.

# Pull the Elasticsearch image
docker pull sebp/elk

# Run the Elasticsearch image in a container with the necessary memory settings
docker run -e ES_JAVA_OPTS="-Xms256m -Xmx256m" -p 5601:5601 -p 5044:5044 -p 9200:9200 -p 9300:9300 -it --name elk 2fbf0a30426d

Next, we will modify the Logstash configuration to collect log data from the Spring Boot application.

# Enter the container through the exec command
docker exec -it elk /bin/bash

# Modify the Logstash configuration
input {
  tcp {
    port => 5044
    codec => json_lines
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

Once the Logstash configuration is modified, we will restart the container to apply the changes.

# Restart the container
docker restart elk

Creating a Spring Boot Project

To create a Spring Boot project, we will create a new project with the following dependencies:

  • Spring Boot Starter Web: Provides a simple way to create a web application using Spring Boot.
  • Spring Boot Starter Test: Provides a simple way to create unit tests for Spring Boot applications.
  • Logback: Provides a logging framework for Spring Boot applications.
  • Logstash Logback Encoder: Provides a Logstash encoder for Logback.

Here is the pom.xml file for the Spring Boot project:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.gf</groupId>
  <artifactId>springboot-elk</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>
  <name>springboot-elk</name>
  <description>Demo project for Spring Boot</description>
  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.1.1.RELEASE</version>
    <relativePath/> <!-- lookup parent from repository -->
  </parent>
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    <java.version>1.8</java.version>
  </properties>
  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-test</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>ch.qos.logback</groupId>
      <artifactId>logback-classic</artifactId>
    </dependency>
    <dependency>
      <groupId>net.logstash.logback</groupId>
      <artifactId>logstash-logback-encoder</artifactId>
      <version>5.2</version>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
        <configuration>
          <fork>true</fork>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

Here is the SpringbootElkApplication class:

@RestController
@SpringBootApplication
public class SpringbootElkApplication {
  private final static Logger logger = LoggerFactory.getLogger(SpringbootElkApplication.class);
  
  public static void main(String[] args) {
    SpringApplication.run(SpringbootElkApplication.class, args);
  }
  
  @GetMapping("/ {name}")
  public String hi(@PathVariable(value = "name") String name) {
    logger.info("name = {}", name);
    return "hi," + name;
  }
}

Here is the logback-spring.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
  <springProperty scope="context" name="springAppName" source="spring.application.name"/>
  
  <!-- Log output location -->
  <property name="LOG_FILE" value="${BUILD_FOLDER:-build}/$ {springAppName}"/>
  
  <!-- Console log output style -->
  <property name="CONSOLE_LOG_PATTERN" value="%clr(%d {yyyy-MM-dd HH:mm:ss.SSS}) {faint}%clr($ {LOG_LEVEL_PATTERN:-%5p})%clr($ {PID:-}) {magenta} %clr(---) {faint}%clr([%15.15t]) {faint}%m%n $ {LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}"/>
  
  <!-- Console output -->
  <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
      <level>INFO</level>
    </filter>
    <encoder>
      <pattern>${CONSOLE_LOG_PATTERN}</pattern>
      <charset>utf8</charset>
    </encoder>
  </appender>
  
  <!-- JSON format logstash outputted Appender -->
  <appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
    <destination>127.0.0.1:5044</destination>
    <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
      <providers>
        <timestamp>
          <timeZone>UTC</timeZone>
        </timestamp>
        <pattern>
          <pattern>{"Severity": "%level", "Service": "$ {springAppName:-}", "Trace": "%X {X-B3-TraceId:-}", "Span": "%X {X-B3-SpanId:-}", "Exportable": "%X {X-Span-Export:-}", "Pid": "$ {PID:-}", "Thread": "%thread", "Class": "%logger{40}", "Rest": "%message"}</pattern>
        </pattern>
      </providers>
    </encoder>
  </appender>
  
  <!-- Log output level -->
  <root level="INFO">
    <appender-ref ref="console"/>
    <appender-ref ref="logstash"/>
  </root>
</configuration>

Once the Spring Boot project is created, we can start it and log into Elasticsearch using Kibana web interface. We can see that the log data is being collected and stored in Elasticsearch.

Input Pattern Configuration

To configure the input pattern, we need to select all data that matches the pattern. We can do this by selecting the @timestamp field, which will allow us to see the log data in time sequence.

Conclusion

In this article, we have demonstrated how to integrate Spring Boot with the ELK stack. We have created a Spring Boot project, configured the ELK stack, and logged into Elasticsearch using Kibana web interface. We have also demonstrated how to configure the input pattern to select all data that matches the pattern.