Designing a Scalable Log Aggregation Architecture

The Challenge

When you're running dozens of services across multiple environments, logs become both essential and overwhelming. grep-ing through files on individual servers doesn't scale. You need centralized logging.

But centralized logging comes with its own challenges: How do you handle millions of log events per day? How do you keep storage costs manageable? How do you make logs searchable without destroying query performance?

Architecture Overview

After evaluating several approaches, I settled on the ELK stack (Elasticsearch, Logstash, Kibana) with Filebeat for collection. Here's the high-level architecture:

• *Filebeat on each host - lightweight, low overhead
• *Logstash for parsing and enrichment - handles complex transformations
• *Elasticsearch cluster for storage and search - scales horizontally
• *Kibana for visualization and exploration - powerful query interface

┌─────────────┐    ┌─────────────┐    ┌─────────────────┐
│ Application │───▶│  Filebeat   │───▶│    Logstash     │
│   Servers   │    │  (shipper)  │    │   (processor)   │
└─────────────┘    └─────────────┘    └────────┬────────┘
                                               │
                                               ▼
                                    ┌─────────────────┐
                                    │  Elasticsearch  │
                                    │    (3 nodes)    │
                                    └────────┬────────┘
                                               │
                                               ▼
                                    ┌─────────────────┐
                                    │     Kibana      │
                                    └─────────────────┘

Sizing Considerations

Elasticsearch sizing is part art, part science. The main factors are daily ingest volume, retention period, and query patterns. Here's my approach:

• *Estimate daily log volume (usually 2-5GB per service)
• *Multiply by retention period (30 days typical)
• *Add 50% for replicas and overhead
• *Plan for 3x growth

Cost Management

Log storage can get expensive quickly. I use Index Lifecycle Management (ILM) to automatically tier data from hot (SSD) to warm (HDD) to cold (S3) storage:

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "s3-logs"
          }
        }
      },
      "delete": { 
        "min_age": "90d",
        "actions": { "delete": {} } 
      }
    }
  }
}

Key Design Decisions

• *JSON structured logging from applications (much easier to parse)
• *Separate indices per application (isolation, different retention)
• *Index templates for consistent mappings
• *Correlation IDs for tracing requests across services