1. Introduction

Filebeat is a Lightweight shipper that helps you forward and centralize your log data. The tool is part of the Elastic Beats family and can be installed as an agent on your servers. Filebeat can forward your logs directly to Elasticsearch or Logstash for further processing. In this tutorial, you will learn how to install and configure Filebeat using Docker.

2. Prerequisites

• You are expected to have basic knowledge of Docker. This tutorial uses Docker Desktop for Windows, but you can use any other Docker installation depending on your Operating System. Moving forward, we will assume that you have a working installation of Docker in your workstation.
• Filebeat will be sending its data to Elasticsearch. Ensure you have Elasticsearch up and running.

3. Directory Structure

First things first. Let’s start by creating a directory filebeat where we will keep the Filebeat configuration file and any other file related to the installation.

C:\apps\filebeat>tree /f
Folder PATH listing for volume Windows
Volume serial number is xxx-xxx
C:.
├───config
│       filebeat.yml
│
└───sample-logs
        demo-app.log

• filebeat.yml is the Filebeat configuration file. You can create an empty file for now.
• demo-app.log is a simple text file that we will use to test our installation.

4. Filebeat Configuration file filebeat.yml

Filebeat requires a configuration file in YAML format. There are many settings that you can add to the filebeat.yml file. We will focus here on Inputs and Outputs only.

Let’s create the file and save it under “C:\apps\filebeat\config\filebeat.yml”, with the following content:

filebeat.inputs:
- type: filestream
  id: demo-app
  paths:
    - /var/log/demo-app.log

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]

• Our input is of type filestream. With this input, we can stream the content of a text file. You may specify multiple inputs in the same config file. You can find the full list of input types here.
• The output is of type elasticsearch. This allows Filebeat to send the input content directly to an Elasticsearch backend. The data will be sent to the default index filebeat-*. You may only specify one single output in a config file. You can find the full list of Filebeat output types here.

5. Create the Network

Because Filebeat will communicate with Elasticsearch, let’s start by creating a network interface.

From a command prompt(or terminal), enter the following command:

docker network create elk

With this command, we create a network named “elk”.
Your output will be similar to this:

759d536b180c54486af7414d1f9f7017f9df405ef41c991db5e3f011c20cd5ab

6. Pull the Filebeat Docker image

You can find the list of Docker images for Filebeat in the Docker Elastic Registry. For this tutorial, we will be installing the version 8.15.0.
Open a command prompt (or a terminal) and run the following command:

docker pull docker.elastic.co/beats/filebeat:8.15.0

7. Start the Container

Once the image is pulled, use the following command to start the Filebeat container:

docker run --name filebeat -h filebeat --net elk -it -m 1GB -v C:/apps/filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml -v C:/apps/filebeat/sample-logs/demo-app.log:/var/log/demo-app.log docker.elastic.co/beats/filebeat:8.15.0

• --name filebeat: the name we are giving to the container
• -h filebeat: the hostname of the docker container
• --net elk : to specify the network to which the container is attached
• -m 1GB : to limit the memory size
• -v ... : to bind a directory in the host machine to a directory in the container. We use it to pass the configuration file from our host machine to the container. For testing purposes only, we are also binding the dummy log file demo-app.log to a file in the container(/var/log/demo-app.log).

If everything goes well, Filebeat will start listening to changes in the file demo-app.log.

8. Install Filebeat: Verifications

To test your setup, you can do any of the following:
Check Filebeat logs in Docker

docker logs filebeat

Generate some data in demo-app.log
You can generate some data in the demo-app.log file on the host machine. Since the file is bound to /var/log/demo-app.log in the container, the generated data will be sent to Elasticsearch.
You can then query the filebeat index in Elasticsearch using Curl:

curl -u elastic:%ELASTIC_PASSWORD% http://localhost:9200/filebeat-*/_search

Where ELASTIC_PASSWORD refers to the Elasticsearch password saved as an environment variable.

9. Conclusion

In this quick tutorial, you learned how to install Filebeat using Docker.