Observability is the property of a system whose internal state can be inferred from its external outputs. In modern distributed/microservice systems these outputs are commonly grouped into the "three pillars" — metrics (numeric time series such as CPU, latency,request rate), logs (semi-structured text events), and traces (causal request paths across services) — with events (deployments, config changes, alerts) as afourth, cross-cutting signal. The OpenTelemetry specification defines these signal typesand their data models directly (OpenTelemetry docs).

AIOps, a term popularized by Gartner (industry analyst, not a primary technical source — flagged), refers to platforms that apply analytics and ML to operations data to automate detection, correlation, and remediation. The practical driver for applying ML at scale is volume and velocity: cloud-native systems emit telemetry at cardinalities and rates that exceed human triage capacity, and static thresholds generalize poorly across seasonal, multi-tenant, and rapidly-deploying workloads. The AIOps problem is therefore dominated by unsupervised / weakly-supervised learning, because ground-truth incident labels are scarce, delayed, and noisy.

2. Anomaly Detection Methods

2.1 Statistical / classical

The oldest and still widely deployed family models the "normal" signal explicitly and flags deviations.

• Moving averages / EWMA: an exponentially weighted mean \(\hat{x}_t = \alpha x_t + (1-\alpha)\hat{x}_{t-1}\) with residual thresholding.
• Holt-Winters triple exponential smoothing captures level, trend, and seasonality; it underpins many production forecasters (e.g., historically in RRDtool/Graphite).
• ARIMA (\(ARIMA(p,d,q)\)) models autocorrelation of a differenced series; anomalies are points with large forecast residuals.
• Seasonal decomposition (classical and STL, Cleveland et al., 1990, J. Official Statistics) separates \(x_t = T_t + S_t + R_t\); anomalies are outliers in the remainder \(R_t\). Twitter's AnomalyDetection package built on STL + generalized ESD is a well-known open-source instance (repo, not peer-reviewed — flagged).
• 3-sigma / MAD: flag points beyond \(\mu \pm 3\sigma\), or robustly beyond a Median Absolute Deviation multiple.

These methods are cheap, interpretable, and strong baselines; they struggle with multivariate dependence and non-stationary regime changes.

2.2 Traditional ML

• Isolation Forest (Liu, Ting, Zhou, ICDM 2008, DOI:10.1109/ICDM.2008.17) isolates anomalies via random partitioning; anomalies require fewer splits (shorter path length).
• One-Class SVM (Schölkopf et al., 2001, Neural Computation, DOI:10.1162/089976601750264965) learns a boundary enclosing normal data in a kernel feature space.
• Clustering-based outlier detection with k-means or DBSCAN (Ester et al., KDD 1996) flags points far from any dense cluster; LOF (Breunig et al., SIGMOD 2000, DOI:10.1145/342009.335388) scores local density deviation.

2.3 Deep learning

• LSTM/GRU forecasting + residual thresholding for sequences; the LSTM approach to system-log anomaly detection is exemplified by DeepLog (see §2.5).
• Autoencoders / VAE: reconstruction error as anomaly score. OmniAnomaly (Su et al., KDD 2019, DOI:10.1145/3292500.3330672) uses a stochastic recurrent VAE for multivariate metrics and popularized the SMD/SMAP/MSL benchmarks.
• CNN for time series (dilated/temporal convolutions) and Seq2Seq encoder–decoder models for forecasting-based detection.
• LSTM-VAE and GAN-based detectors (e.g., MAD-GAN, Li et al., 2019) extend this reconstruction paradigm.

2.4 Modern / SOTA (2021–2025)

Transformers dominate long-horizon multivariate forecasting and, adapted, anomaly detection:

• Informer (Zhou et al., AAAI 2021, arXiv:2012.07436) — ProbSparse attention for long-sequence forecasting. (verified)
• Autoformer (Wu et al., NeurIPS 2021, arXiv:2106.13008) — decomposition + Auto-Correlation. (verified)
• PatchTST (Nie et al., ICLR 2023, arXiv:2211.14730) — patching + channel independence ("a time series is worth 64 words"). (verified)
• TimesNet (Wu et al., ICLR 2023, arXiv:2210.02186) — reshapes 1D series into 2D by periodicity for general analysis incl. anomaly detection. (verified)
• Anomaly Transformer (Xu et al., ICLR 2022, arXiv:2110.02642) — association-discrepancy criterion. (verified)
• TranAD (Tuli et al., VLDB 2022, arXiv:2201.07284) — deep transformer with adversarial training for multivariate anomaly detection. (verified)
• Diffusion / LLM-based: diffusion models and reprogrammed LLMs are emerging for forecasting/imputation (see §5 foundation models).
• Graph neural networks for multivariate metrics model inter-series dependency; GDN (Deng & Hooi, AAAI 2021, arXiv:2106.06947) learns a graph over sensors and flags edge-deviation. MTAD-GAT (Zhao et al., ICDM 2020) uses dual graph-attention.

Caveat on transformer AD benchmarks. Several works (e.g., "revisiting time-series anomaly detection" critiques, 2022–2023) argue that popular point-adjustment evaluation inflates scores; comparative rankings should be read with this in mind. I have not re-verified individual leaderboard numbers here.

2.5 Unsupervised log anomaly detection

• Drain (He et al., ICWS 2017, DOI:10.1109/ICWS.2017.13) — fixed-depth parse tree for online log-template extraction. (Venue ICWS 2017 verified via Crossref.)
• DeepLog (Du et al., CCS 2017, DOI:10.1145/3133956.3134015) — LSTM over log-key sequences; predicts next key and flags surprises.
• LogAnomaly (Meng et al., IJCAI 2019, DOI:10.24963/ijcai.2019/658) — template2vec semantic embeddings + sequential + quantitative modeling.
• LogRobust (Zhang et al., ESEC/FSE 2019, DOI:10.1145/3338906.3338931) — attention-BiLSTM robust to log instability/evolution.
• Log-template clustering (LogCluster, LKE) groups templates to reduce dimensionality before detection.

Comparison table.

3. Clustering and Signal Correlation

Clustering serves two operational goals: compression (turn millions of log lines into a few hundred templates) and noise reduction (collapse redundant alerts).

• Log clustering: Drain (parse-tree), LogCluster (Lin et al., ICSE 2016, DOI:10.1145/2889160.2889232) clusters logs to identify problems; Spell (Du & Li, ICDM 2016) uses longest-common-subsequence parsing.
• Metric clustering: grouping correlated time series (via correlation/DTW distance) identifies redundant signals and shared failure modes.
• Event correlation & alert aggregation: temporal + topological correlation groups alerts into incidents, reducing "alert storms." Much of this is vendor-implemented; primary technical treatments appear in AIOps-challenge literature rather than product docs.
• Trace/span clustering: clustering trace structures/latency profiles surfaces anomalous request paths in microservice systems.
• Incident clustering — grouping alerts believed to share a root cause — is a bridge to RCA: a well-formed cluster localizes where to look before why is determined.

4. Root Cause Analysis and Fault Localization

• Spectrum-based fault localization (SBFL) ranks components by suspiciousness from pass/fail execution spectra (e.g., Ochiai, Tarantula) — mature in software testing, adapted to services.
• Causal inference / causal graphs: learn a causal DAG over metrics/services (e.g., PC/Granger-based) and trace anomaly propagation. CloudRanger (Wang et al., CCGrid 2018, DOI:10.1109/CCGRID.2018.00076) builds an impact graph for cloud-native RCA.
• Microservice dependency graphs: MicroRCA (Wu et al., NOMS 2020, DOI:10.1109/NOMS47738.2020.9110353) localizes anomalous services on an attributed service graph.
• GNN-based RCA: graph neural nets over service/topology graphs propagate anomaly evidence to rank likely culprits.
• LLM-assisted RCA: recent work uses LLMs to summarize evidence, hypothesize causes, and draft mitigations. RCACopilot (Chen et al., 2023, arXiv:2305.15778 — ID verified: "Automatic Root Cause Analysis via Large Language Models for Cloud Incidents") and Microsoft studies on LLMs for incident RCA are representative. Treat efficacy claims cautiously: reported gains depend heavily on retrieval quality and evaluation setup.

5. Recent Trends (2023–2025)

• LLM / LLM-agent observability. Domain LLMs and agents parse logs, answer natural-language queries over telemetry, and drive investigation loops. OWL: A Large Language Model for IT Operations (Guo et al., 2023, arXiv:2309.09298) (verified) is an explicit ops-domain LLM with an Ops benchmark. "Observability copilots" are largely commercial (Datadog Bits AI, Grafana/Elastic AI assistants) — product docs, not peer-reviewed.
• Foundation / self-supervised models for time series. Zero-shot forecasters: TimesFM (decoder-only foundation model, Das et al., 2023, arXiv:2310.10688) (verified), TimeGPT-1 (Garza et al., 2023, arXiv:2310.03589) (verified), and LLM-reprogramming via Time-LLM (Jin et al., ICLR 2024, arXiv:2310.01728) (verified). These promise few/zero-shot deployment across heterogeneous metrics.
• Generative AI for synthetic telemetry: using generative models to synthesize logs/ traces for augmentation and rare-fault simulation — active but still emerging; strong peer-reviewed baselines are sparse (flagged as low-verification).
• eBPF-based observability: kernel-level, low-overhead instrumentation (Cilium/Hubble, Pixie, Parca) enables auto-generated metrics/traces without code changes. Primary sources are the projects' docs/kernel docs rather than papers.
• OpenTelemetry standardization: OTel (CNCF) is now the de facto vendor-neutral instrumentation and semantic-conventions standard, unifying the pillars (opentelemetry.io) — a primary spec source.
• Causality-driven AIOps: shift from correlation to intervention/causal models for RCA.
• Benchmarks & datasets:
• LogHub (He et al., collection of system-log datasets; arXiv:2008.06448).
• NAB — Numenta Anomaly Benchmark (Lavin & Ahmad, 2015, arXiv:1510.03336).
• TSB-AD — recent reproducible TS-anomaly benchmark (VLDB 2024/2025; associated reproducibility paper indexed on arXiv — exact ID not cleanly verified in this pass, flagged).
• GAIA dataset and AIOps Challenge datasets (KPI/multi-source) from the AIOps community — dataset repos, not peer-reviewed papers.

6. Production Tooling and Ecosystem

Open-source AIOps frameworks include Merlion (Salesforce, arXiv:2109.09265) and PyOD/TODS for anomaly-detection pipelines. Vendor "AI" features (Datadog Watchdog, Elastic ML jobs, Splunk MLTK) are documented in product docs but their internal algorithms are only partially disclosed — treat capability claims as vendor-sourced.

7. Open Challenges and Future Directions

• Evaluation validity: point-adjustment and leaky benchmarks inflate AD scores; the field is moving toward stricter protocols (TSB-AD, revisiting-AD critiques).
• Label scarcity & concept drift: production telemetry is unlabeled and non-stationary; online/continual learning remains hard.
• Multimodal fusion: jointly reasoning over metrics + logs + traces + topology is still immature; most methods handle one pillar.
• Causality vs. correlation: reliable causal RCA at scale is unsolved; causal graphs are sensitive to hidden confounders and sampling.
• LLM reliability: hallucination, cost, latency, and reproducibility limit LLM-agent RCA in production; grounding via retrieval and tool use is the current mitigation.
• Foundation-model transfer: zero-shot TS models are promising but under-validated on operational anomaly tasks specifically.

References

Verified against arXiv metadata in this pass are marked ✔.

1. ✔ Zhou et al. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, AAAI 2021. arXiv:2012.07436
2. ✔ Wu et al. Autoformer: Decomposition Transformers with Auto-Correlation, NeurIPS 2021. arXiv:2106.13008
3. ✔ Nie et al. A Time Series is Worth 64 Words (PatchTST), ICLR 2023. arXiv:2211.14730
4. ✔ Wu et al. TimesNet: Temporal 2D-Variation Modeling, ICLR 2023. arXiv:2210.02186
5. ✔ Xu et al. Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy, ICLR 2022. arXiv:2110.02642
6. ✔ Tuli et al. TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series, VLDB 2022. arXiv:2201.07284
7. ✔ Guo et al. OWL: A Large Language Model for IT Operations, 2023. arXiv:2309.09298
8. ✔ Das et al. A decoder-only foundation model for time-series forecasting (TimesFM), 2023. arXiv:2310.10688
9. ✔ Garza et al. TimeGPT-1, 2023. arXiv:2310.03589
10. ✔ Jin et al. Time-LLM: Time Series Forecasting by Reprogramming Large Language Models, ICLR 2024. arXiv:2310.01728
11. Liu, Ting, Zhou. Isolation Forest, ICDM 2008. DOI:10.1109/ICDM.2008.17
12. Schölkopf et al. Estimating the Support of a High-Dimensional Distribution (One-Class SVM), Neural Computation 2001. DOI:10.1162/089976601750264965
13. Breunig et al. LOF: Identifying Density-Based Local Outliers, SIGMOD 2000. DOI:10.1145/342009.335388
14. Ester et al. A Density-Based Algorithm for Discovering Clusters (DBSCAN), KDD 1996.
15. Cleveland et al. STL: A Seasonal-Trend Decomposition Procedure Based on Loess, J. Official Statistics 1990.
16. Su et al. OmniAnomaly: Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Networks, KDD 2019. DOI:10.1145/3292500.3330672
17. Deng & Hooi. Graph Neural Network-Based Anomaly Detection in Multivariate Time Series (GDN), AAAI 2021. arXiv:2106.06947
18. He et al. Drain: An Online Log Parsing Approach with Fixed Depth Tree, ICWS 2017. DOI:10.1109/ICWS.2017.13
19. Du et al. DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning, CCS 2017. DOI:10.1145/3133956.3134015
20. Meng et al. LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs, IJCAI 2019. DOI:10.24963/ijcai.2019/658
21. Zhang et al. Robust Log-Based Anomaly Detection on Unstable Log Data (LogRobust), ESEC/FSE 2019. DOI:10.1145/3338906.3338931
22. Lin et al. Log Clustering Based Problem Identification for Online Service Systems (LogCluster), ICSE 2016. DOI:10.1145/2889160.2889232
23. Wang et al. CloudRanger: Root Cause Identification for Cloud Native Systems, CCGrid 2018. DOI:10.1109/CCGRID.2018.00076
24. Wu et al. MicroRCA: Root Cause Localization of Performance Issues in Microservices, NOMS 2020. DOI:10.1109/NOMS47738.2020.9110353
25. He et al. LogHub: A Large Collection of System Log Datasets, 2020. arXiv:2008.06448
26. Lavin & Ahmad. Evaluating Real-Time Anomaly Detection Algorithms — the Numenta Anomaly Benchmark (NAB), 2015. arXiv:1510.03336
27. Bhatnagar et al. Merlion: A Machine Learning Library for Time Series, 2021. arXiv:2109.09265
28. OpenTelemetry Documentation. opentelemetry.io/docs (primary spec)
29. Prometheus Documentation. prometheus.io/docs (primary)
30. Grafana Documentation. grafana.com/docs (primary)

Unverified / flagged in text: TSB-AD exact arXiv ID (VLDB reproducibility paper —not cleanly resolved in this pass); GAIA and AIOps-Challenge dataset provenance (communitydataset repos, not peer-reviewed papers); generative-synthetic-telemetry peer-reviewedbaselines (sparse — emerging area); all vendor "AI" capability claims (Datadog Watchdog/BitsAI, Elastic ML, Splunk MLTK) are product-doc sourced, not peer-reviewed; "AIOps" definitionattributed to Gartner (analyst, not a primary technical source). (Drain ICWS 2017 venue andRCACopilot arXiv:2305.15778 were both verified post-hoc via Crossref/arXiv metadata and areno longer flagged.)

We operate a distributed system at scale. When it breaks, we know what broke and where — the affected service, the degraded operation, the time it started. This takes minutes.

We do not know why.

The answer exists somewhere in our telemetry — buried in a log line, encoded in a metric curve, hinted at in a trace fragment, or recorded in a change ticket. But these artifacts live in separate systems, speak different languages, and carry no common thread. Bridging them is a human act: an engineer opens half a dozen consoles, triangulates timestamps, forms a mental model of dependencies, and manually reasons backward from symptom to cause. This takes hours. Sometimes days. Every single time.

This is not a tooling gap. It is a reasoning gap. And we intend to close it.

I. The Aegis Thesis

Correlation is not causation, but it is the raw material from which causation can be inferred. If we can fuse change records with telemetry signals across all three pillars — logs, metrics, traces — and if we can construct even a partial map of how our services depend on one another and on their infrastructure, then we can automate the hypothesis-generation step of root cause analysis. We can give the operator a ranked, evidence-backed causal timeline in under five minutes, instead of a blank search bar and a prayer.

Aegis does not aim to replace the engineer. It aims to compress the mechanical, multi-system search-and-correlate phase — the part that burns time without burning insight — so that the engineer starts their investigation already holding a set of plausible explanations, each backed by evidence drawn from the systems they would otherwise query by hand.

II. The Signals We Have

We possess three families of telemetry, each with different character:

Logs

The richest and messiest signal. Some structured, some semi-structured, some nearly free-form text. They carry error messages, latency measurements, status codes, and — where maintained — ownership attribution and caller identity. They are the ground truth of what the application experienced.

Metrics

The cleanest signal. System metrics, application metrics, infrastructure metrics, trace-derived metrics. Well-typed, regularly sampled, pre-aggregated. They tell us that something changed and when, with precision. An existing anomaly pipeline already computes per-metric confidence intervals continuously.

Traces

The connective tissue — where they reach. Distributed tracing gives us span-level visibility into request flow, service-to-service call graphs, and latency propagation. But instrumentation is incomplete: critical infrastructure components, particularly database systems, are excluded from tracing due to agent stability concerns. The dependency graph has holes where it matters most.

Beyond Telemetry

We also possess a rich change record: every database modification, software deployment, and configuration alteration is recorded with precise timestamps through a formal approval pipeline. We know who changed what and when. What we lack is the connection from change to consequence — the blast radius, the dependent services, the expected behavioral signature.

III. The Gaps We Must Bridge

Aegis is defined as much by what is missing as by what is present:

1. No universal correlation identifier. Some signals carry trace IDs, request IDs, or transaction IDs. Most do not. Cross-signal correlation today relies on temporal proximity and manual search.

2. Traces amputated at the infrastructure boundary. The most critical dependency — the persistence layer — is invisible to our distributed tracing. When a database degrades, we see the symptoms in the services that call it but cannot see inside.

3. A fragmented dependency graph. No single source maps which services depend on which other services, databases, or infrastructure. Tracing provides partial edges. Logs encode caller-callee relationships in fields that are being deprecated. The configuration management database aspires to centrality but is not yet complete.

4. Change events carry no blast radius. A database administrator alters a parameter. The change is recorded. But nothing in that record says "this database serves these twelve services." That knowledge lives in human minds, scattered across teams.

5. Ownership decays. Service ownership fields, where they exist, are populated at creation time and rarely maintained. A field written five years ago may point to a team that no longer exists.

Aegis must work despite these gaps. It cannot wait for all data to be clean, all instrumentation to be complete, or all dependencies to be declared. It must make defensible inferences from partial information and express its uncertainty honestly.

IV. What Aegis Does

Given a degraded service and a time window, Aegis returns:

A ranked causal timeline. A chronologically ordered sequence connecting change events to anomaly onset to symptom emergence, with each candidate root cause scored by confidence and accompanied by supporting evidence drawn from logs, metrics, and traces.

The operator receives this as a report — pushed to a communication channel and persisted in our log aggregation system for visualization — within five minutes of trigger.

Aegis is triggered in two ways:

• Automatically, when the anomaly pipeline detects a metric exceeding a severity threshold.
• Manually, when a platform operator initiates an investigation during an incident or post-incident review.

In its first version, Aegis is non-interactive. It generates a complete report. Drill-down, hypothesis challenging, and iterative refinement are deferred to later iterations.

V. How Aegis Reasons

The Engine: Heuristics First

Aegis v1 uses explicit, auditable rules. Each rule encodes a causal pattern: a type of change event, a temporal window, an expected signature in telemetry, and a scoring function. Rules compete against each other for explanatory power. Aegis ranks candidates; it does not pronounce a single verdict.

We begin with rule-based reasoning because it is transparent, debuggable, and requires no training data. It can be deployed immediately. Statistical and machine learning approaches are natural extensions when we have accumulated a labeled corpus of resolved incidents.

Candidate Rule Families

Change-Induced Degradation. A change occurs on an entity. Within a bounded temporal window, dependent services exhibit anomaly onset, error rate increases, and novel error patterns in logs that were absent before the change.

No-Change Degradation. An anomaly with no proximate change event. Suggests gradual resource exhaustion, external dependency failure, or traffic anomaly. Evidence is drawn from metric saturation curves and upstream correlation.

Cascading Failure. Service A degrades. Service B, which depends on A, degrades with a temporal lag consistent with propagation. Service C follows. Evidence is drawn from trace spans and temporal ordering across the dependency graph.

Infrastructure Event. Multiple unrelated services degrade simultaneously. Correlated with a host-level or network-level anomaly. Evidence is drawn from infrastructure metrics and network flow data.

The Dependency Graph: Synthesized, Not Discovered

Since no single source provides a complete dependency graph, Aegis constructs one from fragments:

• Trace spans from distributed tracing systems, where instrumented.
• Caller-callee relationships inferred from log fields.
• Application-to-host mappings from the configuration management database.
• Network flow data, mapped to services.
• Operator-declared known dependencies, weighted as high-confidence edges.

The graph is constructed lazily, scoped to the investigation at hand. It is never expected to be complete — only useful. Every edge carries a confidence weight, and Aegis propagates uncertainty through its inferences.

Correlation vs. Causation

Aegis employs multiple strategies to avoid conflating coincidence with cause:

• Rank, do not decide. All plausible candidates are presented. The operator retains judgment.
• Intervention analysis. If a changed entity serves N dependent services and all N degrade, the causal link is strong. If only one degrades, the link is weak.
• Temporal precedence. Changes that occurred after symptom onset are excluded as causes.
• Counterfactual weakening. If the same or similar change occurred previously without degradation, the hypothesis is discounted.
• Signal convergence. Hypotheses supported by multiple independent telemetry pillars gain confidence; those supported by only one are penalized.

VI. What Aegis Does Not Do

Clarity about scope is as important as clarity about function.

Aegis does not:

• Replace the on-call engineer. It gives them a head start, not a final answer.
• Automate remediation. It identifies probable causes; it does not trigger rollbacks or configuration reversions.
• Require perfect data. It is designed to degrade gracefully with partial information, expressing lower confidence when signals are sparse.
• Operate continuously in v1. Investigations are triggered on-demand; continuous pre-computation of causal indices is a future optimization.
• Provide interactive drill-down in v1. The initial report is a complete artifact; iterative questioning comes later.

VII. The Measure of Success

Aegis v1 is successful if:

1. Platform operators receive a ranked causal hypothesis list in under five minutes from trigger.
2. The true root cause appears in the top three candidates in the majority of investigations, as measured against post-incident review outcomes.
3. Incidents resolved during the event increase, because root cause is identified while the incident is still active — reducing the volume of formal post-incident reviews.
4. Operators adopt Aegis as their first investigation step, displacing manual multi-system querying as the default workflow.

VIII. The Path Forward

This manifesto describes intent, not implementation. The next steps are concrete:

1. Ratify the scope. Confirm the heuristic-first approach, trigger modes, output format, and latency target.
2. Design the initial rule catalog. Convene domain experts — service owners, database administrators, incident commanders — to articulate the causal patterns they recognize today.
3. Prototype the dependency graph construction. Build a throwaway integration against real data sources and measure coverage against a sample of known service dependencies.
4. Define data contracts. Formalize the query interfaces and response schemas for each source system.
5. Establish an evaluation baseline. Instrument current manual investigation workflows to capture time-to-root-cause for a representative set of historical incidents, so we can measure Aegis against reality.

Aegis is not a dashboard. It is not an alerting system. It is a reasoning prosthesis — a machine that does the mechanical correlation work so that the human can do the thinking work. Our operators deserve to start their investigations with a hypothesis, not a blank page.

Picnic Decision Dataset:

A simple dataset.

The above toy dataset contains 14 samples of wind measurements and their respected picnic/no-picnic records. The first thing to do is to find the how much information does Picnic column have. There is 9 Evet and 5 Hayır records, total of 14 measurements. Thus, the entropy of Picnic variable is following.

Entropy of picnic.

Now, if we split the Picnic variable by the wind situation yes/none, the average reduction in the entropy would be the information gain. To do that, let's put the yes and no measurements next to each other in a row, and split it by the wind measurements.

graph TD
    Root[YYYYYYYYNNNNN] -->|Yes| Left[YYYNNNN]
    Root -->|None| Right[YYYYYYYYNN]

Split by wind.

Here, we should find the entropy in each leaf node. Then, we are going to calculate the weighted average of entropy by taking account of the E/H count in each node.

In the left most leaf node, the entropy is 1 since picking a letter results with the probability of 0.5 of Y and 0.5 of N. For the right node, we can calculate the entropy as follows.

Entropy of the right node.

Now, we can calculate the weighted average of entropy. There is 6 samples in the left, and 8 samples in the right. The following equation calculates the weighted average.

Weighted average of the entropies.

The last part is the calculating the reduction in the entropy. We just need to extract the average entropy from the first one.

Reduction in entropy.

This result tells us if we split the Picnic variable by the wind situations, there will be a 0.0561 reduction in the entropy. In another words, there will be a 0.0561 of information gain in this split.

In a decision tree, we do that for all of the independent variables. We calculate the each individual information gain for all of the independent variables. Then, we choose the feature that causes the largest reduction in entropy and split the tree into branches. We continue to the process until there is no split to make (entropy is 0) or to meet a stop condition (maximum split count or maximum depth of the tree). There are multiple ways to calculate the information gain for different purposes. Information gain is only one of them. Maybe, in another post I will write about some of them. Until then, see you in another day. Bye...

In this first story of mine, I will try my best to clear out the concepts of entropy and some other related concepts such as information gain and information gain ratio. I also want to suggest a video about entropy by Luis Serrano. You can find the video here. While this is an excellent video, it is not enough to fully understand the entropy. You need to make lots of reading sessions and maybe implement it with a programming language. Now, let's get back to the subject.

I am not a physicist but as far as I know, entropy in physics corresponds to the energy of particles in a media. If we give an example, the four phases of matter would be a good candidate.

The leftmost image (ice) has the lowest entropy since the particles are not moving as fast as the other two. The middle (water) has more energy than ice. So the entropy is higher than ice. The rightmost (water vapor) has the highest entropy since the particles move fastest.

This idea applies to the information theory in the sense of probability. Claude Shannon is the man who brings the idea in information theory. Suppose you have a computer. This computer can only output a sequence of characters. But some characters are permitted to repeat. The list of all possible characters that can present in the output is the dictionary of the computer. The question is here how much information do you have about the output of the computer?

A computer with a dictionary consists of the only letter "A" outputs a string of "AAAAAAAAAAAA".

The above computer has a dictionary \(D = \{ \text{"A"} \}\). So, we will have a guaranteed output of "A", right? This means that we know a lot about the output. We will get into how we will compute entropy, but at this point let's say that this output has an entropy value of 0. Because we are certain about the output. It will consist of nothing else but the letter "A" only.

What if our dictionary consists of more than one letter or even numbers? For example, dictionary \(D = \{ \text{"A"}, \text{"B"} \}\). Then the output of the computer might be as follows.

A computer with a dictionary consists of letter "A" and "B", outputs a string of "AAAAAABBBBBB".

Let's say from this output we need to pick a letter. Remember that each pick is independent of each other. So, what is the probability of producing an output that has the same order as in the original output sequence? The answer to this question is comes from the probability, right?

The probability of picking "A" or "B" independently is 0.5. There are 12 letters in the output. So we have multiplied 12 independent probability of picking different letters.

As you can see, the result is very small. This means that producing the same output as the computer is very difficult. We have little information about the output. Our chance to pick a letter and knowing what it is is the same as tossing a coin. This result suggests us we have a higher entropy value here.

But why do we need the entropy? We have the probability and it tells us something about the information amount of the data. Here is the reason. Actually, you can see it yourself, too. If we have fewer letters in the sequence, let's say 4 instead of 12; two for "A" and two for "B", the result will be 0.0625. But the entropy will still be high. So, what's change? You see that the amount of letters in the data affects the output of the probability exponentially. This is a problem for our computers, right? Smaller the floating point number, smaller the precision. You may be heard of the vanishing gradient problem in Recurrent Neural Networks (RNN). This is the reason for it and this is the starting point of entropy.

We need a way to not reducing the output that much but at the same time take care of all the letters in the sequence. What if we try to get rid of the exponential function (the repeated multiplication) by converting it to a linear function. How do we do that? Maybe the answer is in the summation. Because repeated summation is just a linear function, unlike multiplication. So, the conversion can be easily made by logarithms. Then let's take the logarithm of the above equation. There is only a little problem here. Which number should be used as the base of the logarithm? You can pick whatever number you want. However, we are from the area of computation. We love to express numbers and information as bits. A bit is a 1 or 0. So, a bit can be expressed in base 2. This means that we want to express the data with two symbols. Therefore, it is natural that we choose 2 as the base for the logarithm. Now, let's take the logarithm at base 2.

Logarithm base 2 of the above equation.

What a beautiful number here. But, hmm…, you see there are two little problems here. First, we have 12 letters in the sequence and the result is 12. Will this number increase as the letter count increases? Yes, it will increase. So we can normalize it with the letter count by dividing it to 12 in this case.

The result is normalized with the letter count.

This is much better. But, why the number is negative? There is no point to proceed with a negative number while trying to measure the information in a sequence. So, let's multiply the expression with -1.

The previous expression is multiplied by -1 to get rid of the redundant negative sign.

This is it. We made to the end. Almost. We still haven't converted the multiplication into the summation.

The multiplication is eliminated by taking advantage of the property of logarithms.

The first six expressions of the sequence are for the letter "A" and the remaining six is for the letter "B". Now, let's make this more obvious and distribute the multiplication over the whole expression.

The coefficient at the beginning of the whole expression is distributed. In the meantime, the summations are grouped together each for a letter (6 for A and 6 for B).

This is the entropy. For this sequence, as we change the number of letters, the probability of producing the exact same output will change. However, the entropy will never change as the proportion of letter counts are constant regardless of the total number of letters in the sequence. If we generalize the entropy, the following expression can be obtained.

The general formula for entropy.

\(P\) is the probability of picking ith distinct element from the sequence. The knowledge and entropy are opposites. If we have high knowledge about a sequence, the entropy is lower, and vice versa. Let's make another example with a different sequence.

An example sequence with a dictionary of D = {"A", "B", "C"}.

Here we have 3 distinct letters in a 12 letter sequence. Let's calculate the entropy with the above formula.

The entropy of the previous sequence.

Ok, we have calculated the entropy. But, what's the result even means? Actually, the result is the average number of yes/no questions to guess the next letter in the sequence. Be careful about that. You need to ask the questions in the smartest way possible.

Entropy is used in many ML methods. One of the popular ML methods is Decision Tree. In the decision tree, we try to find the independent variable that splits another dependent variable in the best way possible. How to find that variable? The answer is we split the target (dependent) variable with each of the other independent variables and check the reduction in entropy after each of the splits. The variable that causes the most reduction is the winner. This reduction is called information gain.

There are other types of entropy reduction metrics. For example, information gain ratio. It is a very close concept to information gain. I am planning to write about each of them later in separate stories. I hope this post is helpful. If I made a mistake in the post, please let me know. See you soon. Bye…