log mgmt Introduced in L8

ELK Stack

Elasticsearch + Logstash + Kibana — the open-source log management trio. Now also "Elastic Stack" with Beats.

Mindmap

hover · click to navigate
this tech depends on / used by alternative Shipyard anchor
What it is

The plain-English version

The ELK Stack is Elasticsearch (search engine + storage), Logstash (ingest pipeline), and Kibana (UI for querying and dashboards). With Beats (lightweight shippers like Filebeat), it becomes the Elastic Stack. The pre-eminent open-source log platform.

Why it exists

The problem it solves

For self-hosted log management at scale, ELK is the open standard. Search is fast, the UI is good, integrations exist for everything. Alternatives — Loki, OpenSearch (the AWS fork), Splunk, Datadog Logs — have their place but ELK is what most regulated, on-prem, or cost-sensitive teams settle on.

What it competes with

Alternatives

AlternativeTypeWhen it wins
SentryerrorsThe error-tracking standard. Captures frontend and backend exceptions with full context. First tool teams add for production observability.
DatadogAPMComprehensive observability — metrics, logs, APM, RUM, all under one expensive roof.
PrometheusmetricsThe open-source metrics standard. Pull-based scraping, time-series database, the basis of most cloud-native observability.
Where it shows up in Shipyard

Deep links

Vocabulary

The words you'll hear

Elasticsearch
Distributed search/analytics engine. JSON documents in indices.
Logstash
Ingest pipeline. Parses, transforms, routes. Heavyweight.
Kibana
Web UI for query, dashboards, alerting.
Beats / Filebeat
Lightweight shippers. Tail logs and ship them.
Index / Shard
Logical grouping / its physical pieces.
ILM
Index Lifecycle Management. Hot → warm → cold → delete based on age.
OpenSearch
AWS's open-source fork after Elastic re-licensed. Mostly compatible.
Prompting

Bad vs. good prompt for ELK Stack

✕ Bad prompt
set up logging
✓ Good prompt
Set up an Elastic Stack on Docker Compose for development: Elasticsearch (single-node, security disabled for local), Kibana, Filebeat tailing /var/log/tasklane/*.log. Add an ILM policy: hot for 7 days, warm for 30, delete at 90. Show the docker-compose.yml and the filebeat.yml.

Why it works: Specifies single-node for dev (avoids the cluster setup pain), names the lifecycle policy explicitly, and asks for both files. Realistic dev setup, not the production-grade rabbit hole.

Pitfalls

What bites real teams

⚠ Out of disk

Logs grow forever without ILM. Set retention before going to production.

⚠ Cluster sizing

Elasticsearch loves RAM. Undersized clusters fall over under search load. Plan capacity.

⚠ Logstash heavyweight

Many use cases don't need full Logstash; Filebeat → Elasticsearch directly is leaner.

References

Official docs only