April 2026 • Observability • 7 min read

Building a Real-Time Observability Pipeline: Installing Elasticsearch and Kibana on Ubuntu

In the world of data engineering and DevOps, the ELK Stack (Elasticsearch, Logstash, Kibana) stands as the gold standard for searching, analyzing, and visualizing big data in real-time. Whether you are monitoring kubernetes logs, analyzing security events, or building an enterprise search engine, a deep understanding of deploying this stack effectively is a fundamental engineering skill.

In this guide, I will walk you through the architecture of the stack and provide an advanced, production-grade installation guide for Elasticsearch and Kibana on an Ubuntu server. This is not just a quick start; we will focus on zero-trust networking parameters, OS fine-tuning for JVM memory allocation, and industry-standard security boundaries.

1. The Architecture

Before diving into the terminal, we must understand the topology of how data flows through our infrastructure.

Elasticsearch (The Engine): A distributed, RESTful search and analytics engine running on Apache Lucene. It serves as your primary data store and inverted index.
Kibana (The Viewport): Operational dashboard layer. Connects securely to the Elasticsearch cluster, allowing operators to build robust visualization pipelines, alert thresholds, and manage node health.
Beats / Logstash (The Pipeline): Lightweight edge shippers and robust ETL processors that normalize logs before persisting them to disk.

2. Prerequisites & Environment Specs

Modern versions of the Elastic Stack (8.x) require careful resource provisioning. A heavily utilized Elasticsearch node will easily exhaust an under-provisioned server.

OS: Ubuntu 22.04 LTS or 24.04 LTS.
Memory: Minimum 4GB RAM (8GB+ recommended). Elasticsearch JVM consumes 50% of available RAM by default.
Disk: Fast SSD storage (NVMe preferred) mapped to /var/lib/elasticsearch. IOPS performance dictates search speeds.
Network: Ports 9200 (Elasticsearch API) and 5601 (Kibana Web).

3. Step-by-Step Production Installation

Step 1: System Tuning & Dependencies

First, we configure the kernel. Elasticsearch utilizes `mmapfs` to map indices into memory. The default OS limits on mmap counts are too low and will cause out-of-memory errors on startup.

Bash

# Update package index and install TLS dependencies
sudo apt update && sudo apt upgrade -y
sudo apt install wget curl gnupg2 apt-transport-https -y

# Permanently increase highly critical max_map_count parameter
echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

Step 2: Add the Elastic Repository Securely

Modern Ubuntu drops support for apt-key. We follow the strict GPG dearmor pipeline to authorize Elastic signatures.

Bash

# Fetch and process the official Elastic Stack GPG Key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | gpg --dearmor | sudo tee /usr/share/keyrings/elasticsearch-keyring.gpg > /dev/null

# Register the 8.x stable repository pointing to our keyring
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

sudo apt update

Step 3: Install & Solidify Elasticsearch

Version 8.x ships with TLS (SSL) enabled by default for all HTTP and transport-layer communication.

Bash

sudo apt install elasticsearch -y

Crucial: Save the terminal output! Immediately after the package finishes, the installer prints the autogenerated password for the elastic superuser and the Kibana setup token.

Configure the Node for a Single-Server footprint (if not deployed in a cluster to avoid Bootstrap checks failing) and set the correct network bindings.

Configuration

# /etc/elasticsearch/elasticsearch.yml
cluster.name: prod-observability-cluster
node.name: node-1
network.host: localhost  # Keep localhost if Kibana is on same server, use VPN IP otherwise.
discovery.type: single-node

Tune the JVM Heap. Never exceed 50% of total system RAM, up to a maximum of 31GB.

Configuration

# /etc/elasticsearch/jvm.options.d/heap.options
-Xms2g
-Xmx2g

Bash

sudo systemctl daemon-reload
sudo systemctl enable --now elasticsearch

Step 4: Install Kibana & Complete Enrollment

The web layer, Kibana, uses the Elasticsearch token to instantly establish a secure connection using the pre-generated CA certificates.

Bash

sudo apt install kibana -y

# Configure Kibana for external access (Reverse Proxy recommended)
sudo nano /etc/kibana/kibana.yml

Configuration

# /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0" # Expose to network

Bash

sudo systemctl enable --now kibana

Missed the initial token? Regenerate it via cli:

Bash

sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
sudo /usr/share/kibana/bin/kibana-verification-code

Navigate to http://YOUR_SERVER_IP:5601, paste the token, verify with the secondary code, and authenticate using the elastic username.

4. Essential Production Best Practices

If you aim to run this in strict production environments, keep these practices in your playbook:

Reverse Proxying: Never expose port 5601 and absolutely never expose 9200 over the raw internet. Set Kibana to `server.host: localhost` and proxy traffic via NGINX using properly signed Let's Encrypt SSL Certs.
Firewall (UFW): sudo ufw block 9200. Only the internal servers (Logstash/Beats endpoints) should be allowed to hit the ingest port.
Index Lifecycle Management (ILM): Data explodes. Utilize Kibana's ILM UI to define strict rollover profiles. E.g., Move to "Warm" storage after 15 days, "Delete" after 30 days.
Kernel Swap: Swapping Elasticsearch memory pages to disk will destroy performance. Disable swap via sudo swapoff -a and add bootstrap.memory_lock: true in elasticsearch.yml.