Log Platform Selection Pitfall-Avoidance Guide: Capability Boundaries and Trade-offs of Loki, ELK, and Doris

When daily log volume reaches multiple TB, logs need to be retained for 30 days, and aggregate reports are required, choosing the right log platform directly affects operational costs and query latency. Based on public documentation and practical cases, this article outlines the applicable boundaries of three solutions—Loki, ELK, and Apache Doris—and provides actionable configuration and verification methods.

1. Quick Decision: Which Category Does Your Scenario Fall Into?

First answer one question: Beyond retrieval, do you need multidimensional aggregation and correlation analysis?

• You only need to “fish out logs and take a look” → Loki or ELK is sufficient.
• You need to “aggregate by dimension, calculate trends, and join with business tables” → You need a solution that integrates retrieval and analysis, or you must accept dual-stack synchronization.

The following sections break this down by scenario.

2. Loki: Cost-First Label Filtering

Loki indexes only the labels of logs, while the body is compressed and written to object storage.

Advantages

• Storage cost is significantly lower than full-text indexing solutions, eliminating the construction and storage of inverted indexes.
• Native integration with Grafana, and the query language LogQL is simple and easy to use.

Costs

• Body retrieval capability is limited: queries first filter by labels, then scan chunks and perform regex matching. Label design directly determines query efficiency.
• High-cardinality labels (such as trace_id and user_id) cause cardinality explosion, rapidly inflating the index and offsetting the cost advantage. In engineering practice, it is recommended to use only low-cardinality fields such as service, env, level, and cluster as labels.
• Aggregation analysis capability is limited; multidimensional analysis and multi-table joins are not design goals.

Applicable Scenarios

• 7-day retention, troubleshooting only, and the label set can be kept low-cardinality.
• When correlation with business data is required, an additional synchronization pipeline must be designed.

Case: Paytm Insider used Loki in production to save 75% of log and monitoring costs, reducing MTTR from 30 minutes to 10 minutes (Grafana Labs published material).

3. ELK: Full-Text Search Semantic Depth First

Elasticsearch builds an inverted index for each field, providing the most complete retrieval semantics (relevance, phrases, highlighting, etc.).

Costs

• Multiple storage copies stack up: forward index, inverted index, and DocValues are retained at the same time. In log scenarios, the compression ratio is about 1.5:1.
• Writes are CPU-intensive: tokenization, sorting, and multiple replicas each repeatedly build indexes.
• Analysis capability is mainly single-table; multi-table JOIN is not supported, and multidimensional aggregation usually requires building a separate analysis pipeline.

Applicable Scenarios

• Primarily text retrieval, with high requirements for retrieval semantic depth.
• Under high write load, messages may be lost. A common practice is to introduce Kafka/RabbitMQ as a buffer to improve resilience.

Case: iFLYTEK Xingji Log Center was initially based on Elasticsearch and encountered problems such as high CPU usage, high storage costs, and OOM during cross-day queries (SelectDB blog).

4. Apache Doris: Integrated Retrieval and Analysis

Apache Doris performs both retrieval and analysis on the same columnar data.

Core Capabilities

• Retrieval: Build inverted indexes on specified fields, supporting keyword and phrase matching such as MATCH and MATCH_PHRASE.
• Analysis: Columnar storage + ZSTD, supporting multi-table JOIN, aggregation, and materialized views.
• Storage: Columnar compression ratio in log scenarios can reach 5:1–10:1, an order-of-magnitude difference from ES’s 1.5:1.
• Deployment: Supports integrated storage-compute and separated storage-compute. In separated storage-compute mode, compute can be scaled on demand and workloads can be isolated.

Disclosed Comparison Data

• httplogs with the same configuration and same index: ES 19.4GB vs Doris 3.2GB, reducing space by 83% (Apache Doris official blog).
• A NetEase game company in production: ES storage consumption reduced by two-thirds, query performance improved by 10x (Apache Doris official blog).
• ClickBench analytical performance: Doris’s out-of-the-box performance is 21x that of ES, and still 6x faster after ES tuning (ClickBench benchmark).
• CITIC Bank Credit Card Center: ES 9 nodes of 16 cores 32G → Doris 4 nodes of 8 cores 32G handling the same load (SelectDB blog).
• iFLYTEK: migrated from ES to Loki and then to Doris, query performance improved by 10x, and storage space was reduced to 1/6 of ES (SelectDB blog).

Applicable Scenarios

• You need both retrieval and aggregate analysis, and need to correlate with business data.
• Daily 3TB logs, 30-day retention, weekly service quality reports, and correlation with business dimension tables.

5. Key Parameters and Configuration Examples

5.1 Table Creation Example (Doris)

CREATE TABLE app_log
(
  ts DATETIME,
  service VARCHAR(64),
  env VARCHAR(16),
  level VARCHAR(16),
  trace_id VARCHAR(64),
  user_id BIGINT,
  msg TEXT,
  cost_ms INT
)
ENGINE = OLAP
DUPLICATE KEY(ts)
PARTITION BY RANGE(ts) ()
DISTRIBUTED BY RANDOM BUCKETS 250
PROPERTIES (
  "compression" = "zstd",
  "compaction_policy" = "time_series",
  "dynamic_partition.enable" = "true",
  "dynamic_partition.time_unit" = "DAY",
  "dynamic_partition.start" = "-30",
  "dynamic_partition.end" = "3",
  "dynamic_partition.buckets" = "250"
);

-- Build inverted indexes only on fields that need retrieval
ALTER TABLE app_log ADD INDEX idx_msg (msg) USING INVERTED
PROPERTIES("parser" = "chinese", "support_phrase" = "true");

-- Use BloomFilter for high-cardinality fields
ALTER TABLE app_log ADD INDEX idx_trace (trace_id) USING BLOOMFILTER;

5.2 Search + Join Query Example

SELECT l.service,
       d.owner_team,
       COUNT(*) AS err_cnt,
       AVG(l.cost_ms) AS avg_cost
FROM app_log l
JOIN dim_service d ON l.service = d.service_name
WHERE l.ts >= NOW() - INTERVAL 1 HOUR
  AND l.msg MATCH_PHRASE 'timeout'
  AND l.level IN ('ERROR', 'FATAL')
GROUP BY l.service, d.owner_team
ORDER BY err_cnt DESC
LIMIT 20;

5.3 Key Parameter Table

Creating inverted indexes on all fields at once will increase capacity by more than 40%, offsetting the compression gains—it is recommended to index only fields that truly need retrieval, and use BloomFilter for high-cardinality fields (CITIC Credit Card Center practice).

6. Implementation Steps and Inspection Checklist

6.1 Implementation Steps

1. Migrate non-core domains first: Run real compression ratios and query latency, validate, and then migrate everything.
2. Configure the collection pipeline: Doris supports the Stream Load HTTP API, can receive data from Logstash, Filebeat, and Fluent Bit, and also supports Kafka Routine Load.
3. Configure the storage policy: Keep hot data for 3 days, then automatically offload to S3/HDFS, and delete after 30 days.
4. Configure monitoring: Build dashboards via Grafana or Doris WebUI, focusing on write throughput, query latency, and disk usage.

6.2 Inspection Checklist (Verify Once a Week)

• Partition and capacity inspection: confirm expired partitions are deleted normally.
• Execution path inspection: confirm retrieval still uses indexes (EXPLAIN).
• Whether the inverted index field scope and parameters related to label design still take effect after version upgrades.
• Whether the deviation between actual storage space and the baseline (e.g., 83% space reduction measured with the same configuration for httplogs) exceeds 30%; if the deviation is too large, check the configuration and partitions.

7. FAQ and Pitfall-Avoidance Guide

Q: How serious is Loki’s high-cardinality label problem?
Setting high-cardinality fields such as trace_id and user_id as labels will rapidly inflate the index and offset its storage cost advantage. In engineering practice, it is recommended to use only low-cardinality fields such as service, env, level, and cluster as labels, and treat the rest as log content.

Q: Can the three solutions coexist?
Yes, this is very common in actual implementations. For example, hot data is kept in ELK for troubleshooting retrieval, while long-term data and aggregate analysis are placed in Doris. The key is to clearly define their boundaries and avoid inconsistent semantics caused by synchronizing the same data to multiple places.

Q: If retrieval + analysis are integrated, will it be insufficient in both areas?
It can cover log retrieval and analysis scenarios (keywords + aggregation + JOIN); if you need mature search semantics such as relevance ranking, autocomplete, and highlighting, ES is still more suitable. It is recommended to sample and verify according to your own query patterns rather than only looking at capability lists.

Q: Where are the main costs of switching engines?
Two areas: collection pipeline integration and dashboard migration. The former usually requires little change; Doris supports receiving data from Logstash, Filebeat, and Fluent Bit through the Stream Load HTTP API, and also supports the OpenTelemetry ecosystem. The latter is the main workload; Kibana dashboards need to be evaluated for alternatives (such as SelectDB Studio and Grafana Doris Datasource).

Q: What problems will high-frequency small-batch writes cause?
Version accumulation, and compaction cannot keep up. It is recommended to use time-series compaction + single-tablet load + batching of about 100MB.

Q: What should I do if FE memory rises during high-concurrency imports?
Reduce streaming_label_keep_max_second and label_clean_interval_second to avoid exhausting FE JVM memory.

8. Verification and Next Steps

• Verification: Run a benchmark with your own query set, compare storage compression ratios and query latency, and confirm whether expectations are met.
• Next Steps: Determine the target solution according to the decision tree, pilot it in non-core domains first, and then migrate everything after obtaining real data.
• Note: All data and conclusions in this article come from approved public sources (Apache Doris official documentation, Grafana Labs published materials, SelectDB blog, ClickBench benchmark, etc.), and no fictitious information has been introduced. For actual selection, please verify based on your own business scenarios and resource conditions.