1. 150M series is feasible — it is a solved problem at the storage/platform layer, and the cheap algorithms scale too. VictoriaMetrics single-node is documented for up to 100M active series and the cluster version for billions (VM FAQ). Anodot ran anomaly detection over >120M series, >6B data points/day in production in 2017 (Anodot, PMLR 71). So the target is within demonstrated practice — not a "downgrade to 10M" situation. 10M is trivially within range.
2. The binding constraint is per-series model state and per-point compute, not raw storage. Keep per-series state in the hundreds of bytes to a few KB range (bounded sufficient statistics, seasonal coefficients, an anomaly threshold), and every series is cheap. This is exactly what the streaming/statistical methods do. Per-series deep networks or big forests break the budget at 150M.
3. The cheap tier that dominates = "residual + adaptive threshold": model the expected value (EWMA / Holt-Winters / z-score / median-seasonal), flag points whose residual exceeds an adaptively learned threshold (Extreme-Value-Theory SPOT/POT, or online MAD/quantile/z-score). O(1) per point, tiny state, online by construction, no GPU.
4. Online training is the default for these methods — it's an O(1) state update, not SGD. EWMA/CUSUM, Bayesian Online Changepoint Detection, SPOT/DSPOT, Random Cut Forest (streaming insert), and Holt-Winters recursive updates all update incrementally as points arrive. "Online" at 150M series must mean closed-form per-point state updates, not gradient descent per series.
5. Daily batch is fully supported and is a valid operating point. VictoriaMetrics vmanomaly and Merlion/Greykite all let you schedule fit_every: 1d (batch refit) or run truly online models with exponential decay. You can mix: a cheap online scorer every point + a heavier model refit daily on a filtered subset.
6. Recommended shape: a two-tier pipeline — (Tier 1) a cheap streaming statistical detector on every series/every point, (Tier 2) a heavier model (RRCF or shared/global model) escalated only on flagged windows. This is the same split Uber uses in production (outlier detector → outage detector) and is how you keep 150M-series cost bounded.

1. Constraints restated

• 150M unique series (10M fallback). Any per-series cost — parameters, state, compute, storage — is multiplied by ~1.5×10⁸.
• Cheap — CPU-only, low per-point cost, low memory, low ops overhead (no hand-tuning 150M models).
• Online training supported — the model must be able to learn/adapt incrementally as new data arrives.
• Batch (daily) acceptable but not strict — a daily refit/run cadence is fine; continuous streaming is a bonus, not a hard requirement.
• Highly extensible — pluggable model families, easy to add series and swap algorithms without re-architecting.

2. Feasibility verdict (the "150M vs 10M" question)

Storage/platform layer — 150M is proven. - VictoriaMetrics: single-node handles up to 100M active time series and 2M samples/s; the cluster version handles billions of active series and hundreds of millions of samples/s (based on real usage). — VictoriaMetrics FAQ, "scalability limits" - Anodot's commercial system (peer-reviewed description): "discovers anomalies today for about 50 different companies and over 120 million time series metrics", processing "over 6 billion data points per day", with ~20M of those series found to have seasonal patterns. — Toledano et al., PMLR 71:56–65, 2018 - Uber's Argos monitors "tens of millions of metrics" in real time. — Uber Engineering, "Identifying Outages with Argos"

Model layer — this is where the 150M vs 10M distinction actually lives.

• 10M series: you can afford ~1–100 KB of state per series (even a modest per-series forest or small adapter) within a few TB. Almost any "cheap" method fits.
• 150M series: you are essentially restricted to O(1)-state statistical methods (EWMA/CUSUM, online mean/variance, seasonal coefficients, a single EVT/quantile threshold) unless you spill state to disk. VictoriaMetrics vmanomaly supports an on-disk model mode for exactly this reason: "dumped to disk … significantly decreases RAM usage, particularly useful for larger setups" (vmanomaly models doc).

Conclusion: 150M is achievable, but only if per-series state is bounded to statistical summaries. The methods below are ordered by that budget.

3. The "cheap" method hierarchy (what to pick, and why)

3.1 Statistical streaming detectors — O(1) state, O(1)/point, online by construction

These are the cheapest and the ones that actually scale to 150M.

• EWMA / CUSUM / control charts. Classical statistical process control; maintain a running level (and optionally variance), flag points when the deviation exceeds a control limit. Closed-form, incremental, no history retained. (Background: the same exponential-weighting idea is the default "exp_avg_detector" in LinkedIn's Luminol, with a smoothing factor controlling adaptation — Luminol README.)
• Online z-score / MAD / quantile. Running mean+std (or median+MAD for robustness) with a fixed multiple-of-sigma threshold. These are built-in first-class "online" models in VictoriaMetrics vmanomaly (zscore_online, quantile_online, online MAD), and vmanomaly's docs explicitly recommend them as the cheap default for stationary/light-tailed data. — vmanomaly models doc
• Seasonality via online/forecast residual. Holt-Winters (triple exponential smoothing) / ETS forecast the expected value including daily/weekly seasonality; the anomaly score is the forecast residual. This is the core of:
• Yahoo EGADS — splits the problem into a Time-Series Model (forecaster: OlympicModel, TripleExponentialSmoothingModel, etc.) + an Anomaly Detection Model (KSigma, density-based, SimpleThresholdModel with adaptive sensitivity). EGADS's own README: "At Yahoo, our internal Yahoo Monitoring Service (YMS) processes millions of data-points every second." — EGADS README
• Anodot — per-series model chosen by a classifier (~40% of metrics are stationary; the rest sparse/step/discrete/multi-modal/irregular), parameters fit by MLE, then an online adaptive policy updates the model as points arrive. — Anodot, PMLR 71
• Greykite (LinkedIn) — forecast with Silverkite, flag points outside the forecast confidence interval; "Simple Anomaly Detection" auto-tunes the interval (by expected alert rate / APE) instead of hand-setting it. — Greykite simple AD docs

3.2 Adaptive thresholds (no manual threshold, no distribution assumption)

• SPOT / POT (Extreme Value Theory). SPOT adapts Peaks-Over-Threshold to streams: it fits a Generalized Pareto Distribution to the tail of the residual stream and computes a high quantile as the anomaly threshold, with no manual threshold and no distributional assumption, updating as data flows. The reference implementation, libspot, is a C99, -nostdlib library for "high-throughput streaming data," self-described as "the poor man's anomaly detector" and "a cheap algorithm that must be able to run on cheap systems." — libspot · SPOT, KDD 2017
• Bayesian Online Changepoint Detection (Adams & MacKay). Derives an exact online inference of the most recent changepoint / run-length via a simple message-passing algorithm — a principled, fully-online alternative to fixed thresholds that detects both point anomalies and regime shifts. — arXiv:0710.3742

3.3 Robust Random Cut Forest (RRCF) — streaming, but a "heavier" second stage

• RRCF is an unsupervised streaming detector: each tree scores a point by the expected change in tree complexity when inserted, and the model is updated incrementally (reservoir sampling, so it works on unbounded streams). "Inference time is proportional to the number of trees"; the docs recommend 100 trees to start. — Amazon SageMaker, "How RCF works" · Guha et al., ICML 2016
• Cost note: RRCF's per-series state is num_trees × num_samples_per_tree points — in Merlion's default config that's n_estimators=100, max_n_samples=512 ≈ 51k points/series, i.e., ~hundreds of KB per series, not bytes. At 150M series that is ~60 TB (see §2). So use RRCF as the escalated second stage on flagged windows, or with a very small sample size, not as the always-on 150M-series scorer. (Merlion's own default detector uses exactly this pattern — see §4.1.)

3.4 Subsequence/motif methods — note but don't default to them

• Matrix Profile / STUMPY finds discords (unusual subsequences) rather than point outliers. It is excellent for shape-based anomalies but is quadratic (or O(n log n) with SCRIMP-type approximations) per series and not "cheap" at 150M. Mentioned for completeness; not recommended as the 150M-series default. — UCR Matrix Profile page · STUMPY

4. Turnkey / semi-turnkey options (primary sources)

4.1 VictoriaMetrics vmanomaly (built for high-cardinality metrics)

• Built on VictoriaMetrics, which scales to 100M (single-node) / billions (cluster) series — the anomaly service is designed for exactly this cardinality regime. — VM FAQ
• Online models: zscore_online, quantile_online (online seasonal quantile), online MAD, and temporal_envelope (trend + calendar/holiday + seasonal patterns; the recommended online replacement for Prophet). — vmanomaly models
• Online adaptation knob: the decay parameter gives an exponential forgetting factor to online models so they "adapt to new data" without full refits (e.g., decay: 0.996 keeps ~1 day of data dominant at 1-min granularity). — vmanomaly models → Decay
• Batch/daily and hybrid scheduling: the scheduler defines fit_every / fit_window / infer_every. You can run offline models on a daily refit (fit_every: 1d) or online models with a bootstrap-only fit (fit_every: 1000d + infer_every: 5m) — i.e., daily batch and streaming are both first-class. — vmanomaly FAQ / scheduler
• All models emit a unified anomaly_score (>1 = anomalous), and alerting is decoupled via vmalert. Caveat: vmanomaly is an Enterprise-licensed component (free trial available). — vmanomaly FAQ

4.2 Salesforce Merlion (unified, AutoML, distributed)

• A Python framework unifying forecasting / anomaly / changepoint detection with a shared interface, AutoML, and a PySpark distributed backend for industrial scale. — Merlion README
• Its DefaultDetector (the "balances efficiency with performance" default) is, in source, an ensemble: univariate = AutoETS (error/trend/seasonality forecaster) + RandomCutForest (online_updates=True, 100 trees, 512 max samples) + ZMS (z-score moving statistic); multivariate = VAE + RRCF. This is a concrete, production-grade realization of "cheap forecaster + streaming RRCF + z-score" with online updates enabled. — Merlion models/defaults.py source
• Anomaly scores are calibrated to z-scores and passed through a trainable threshold/calibration post-rule to cut false positives. — Merlion anomaly docs

4.3 Yahoo EGADS (Java; forecasting + detection split, designed to scale)

• Open-source Java library; "a scalable, accurate and automated anomaly detection" system compiled to "a single light-weight jar and deployed easily at scale," backing Yahoo's monitoring service that processes millions of points/sec. Extensible by design: you can drop your own model into either the time-series-model or anomaly-detection-module component. — EGADS README

4.4 LinkedIn Luminol (lightweight, score-based)

• Lightweight Python AD+correlation lib; no predefined threshold — every point gets an anomaly score. Ships a default bitmap detector (good for large data) plus exp_avg (EWMA) and derivative detectors. Good as a reference for the "cheap, threshold-free" pattern. — Luminol README

4.5 Greykite (LinkedIn; forecast-interval AD)

• Forecast-based AD over Silverkite; auto-tuned confidence intervals for the alert rate you want. — Greykite simple AD docs

5. Online training — what it concretely means here

"Online training" at 150M series must be O(1) closed-form state updates, not per-series SGD. The methods that satisfy this natively:

Key nuance — adapting without being poisoned by anomalies. Anodot describes the production policy: during an anomaly, temporarily reduce the model's learning-rate parameters (Holt-Winters α, β, γ) so the anomaly doesn't get absorbed into "normal," then restore them after the anomaly passes; a persistent change eventually gets learned. — Anodot §3.3, PMLR 71. Uber's Argos uses a median-based robust update for the same reason ("past outages and outliers must not affect the outlier score"). — Uber Argos

Streaming evaluation is a real, benchmarked thing: the Numenta Anomaly Benchmark (NAB) is built specifically for detectors that "process data in real time (not batches)" and "learn while making predictions simultaneously." — NAB, arXiv:1510.03336 · github.com/numenta/nab

6. Batch (daily) is a first-class operating mode — not a limitation

• vmanomaly scheduling: fit_every controls refit cadence. fit_every: 1d = a classic daily batch (fit on a fit_window, infer); fit_every: 1000d + infer_every: 5m = bootstrap-once + stream. You can even run backtesting schedulers over a historical period to validate a config before production. — vmanomaly FAQ
• Uber's hybrid cadence: thresholds are recomputed hourly (batch-ish) while the online part only compares incoming points against those precomputed thresholds — "extremely fast and scalable." — Uber Argos
• Implication for the requirement "online training, but daily batch OK": you do not need per-point SGD. A daily (or hourly) batch refit of a cheap statistical model, or a bootstrap-once + online-decay model, both satisfy it. The requirement is best met by stateful incremental models with a decay/forgetting factor, which are cheap and give you "online" behavior even when the heavy recompute is batched.

7. Scaling to 150M series — the recommended architecture

1. Tier 1 — cheap always-on scorer (every series, every point). Online z-score/MAD/quantile or EWMA/CUSUM + seasonal coefficients + an EVT (SPOT/DSPOT) or online-quantile adaptive threshold. O(1) state (bytes–KB/series), O(1)/point, CPU-only, horizontally shardable by series hash. This is the only stage that must run at 150M scale.
2. Tier 2 — heavier model, escalated only on flagged windows. RRCF (small forest) or a shared/global model runs only on series/windows that Tier 1 flags. This is Uber's exact production split (fast per-series "outlier detector" → heavier "outage detector" on the small remainder) and Anodot's funnel (158k single-metric anomalies → ~910 significant → 147 grouped per 4M metrics/week). — Uber Argos · Anodot §6, PMLR 71
3. Bound state to the budget. At 150M, keep Tier-1 state ≤ ~1 KB/series (~150 GB RAM, or spill to disk via the vmanomaly on-disk mode). At 10M you have ~15× headroom for richer per-series state.
4. Make tuning learned, not manual. You cannot hand-tune 150M detectors. Anodot's answer: a classifier selects the best model type per metric (~40% stationary, else sparse/step/discrete/multi-modal/irregular) and fits parameters by MLE, fully unsupervised. EGADS's answer: an AutoSensitivity/adaptive-threshold mechanism. — Anodot §3.1 · EGADS
5. Extensibility point = the scorer interface. EGADS (TSM/ADM split), Merlion (unified ModelFactory + custom models), and vmanomaly (custom model guide) all make adding a new algorithm a plug-in, not a rewrite — this is the "highly extensible" part. — EGADS · Merlion · vmanomaly custom model guide
6. Where it runs with a metrics DB. If data is in InfluxDB/VictoriaMetrics/Prometheus: vmanomaly natively reads/writes VictoriaMetrics; for InfluxDB use a stream/batch processor (or InfluxData Kapacitor UDF) that applies Tier-1 scoring per point and writes anomaly_score back. (See the sibling note for the Kapacitor detail.)

8. Options matrix

9. Pragmatic recommendation

1. Start with the cheap tier, not a fancy model: online z-score/MAD (or EWMA) + seasonal residual (Holt-Winters/ETS or the vmanomaly temporal_envelope/quantile_online) + an EVT/online-quantile adaptive threshold. This single combination satisfies cheap + 150M + online + batch-daily simultaneously and is exactly what the proven systems (Anodot, Uber, EGADS, vmanomaly online models) do.
2. Run it daily-batch or streaming via a scheduler (fit_every: 1d for offline models, or bootstrap + decay for online models), writing a unified anomaly_score > 1 metric and alerting off it.
3. Escalate with RRCF (small forest) or a shared model only on flagged windows — buys down compute at 150M while keeping precision.
4. If 150M is truly needed, engineer the state budget first (~≤1 KB/series + horizontal sharding by series); if you land at 10M, you can afford richer per-series models (RRCF, Greykite, Merlion defaults) without special effort.
5. Validate on a stratified sample (tens–hundreds of series) — you can never hand-inspect 150M — and use NAB-style streaming evaluation to measure "learn while predicting" behavior.

10. Caveats on evidence

• Vendor claims vs. independent benchmarks: the scale numbers (VictoriaMetrics 100M/billions; Anodot 120M series; Uber "tens of millions"; Yahoo "millions of points/sec") are first-party statements — documented capabilities of the systems' owners, not independent third-party measurements. They are trustworthy as existence proofs ("this scale is operated in production") but not as apples-to-apples benchmarks.
• Anodot and Uber describe architecture and design considerations, not reproducible open code. They are strong primary sources for the pattern (cheap per-series scorer + heavier escalated stage + adaptive threshold + anomaly-aware learning-rate policy), but you'd reimplement the pieces.
• RRCF state estimates are computed from the documented defaults (num_trees≈100, num_samples_per_tree≈512) and are an order-of-magnitude illustration, not a vendor-stated per-series memory figure.
• SPOT is univariate (the paper is explicit); for multivariate/contextual anomalies you combine it with a forecaster residual or escalate to a multivariate model.
• Greykite's exact AD internals are lightly documented publicly (the docs describe what it auto-tunes, not the full statistical detail); cite it for the "auto-tuned forecast interval" capability, not for a specific algorithm.

References (all primary sources, verified)

Algorithms / papers - Robust Random Cut Forest: Guha, Mishra, Roy, Schrijvers, ICML 2016 — https://proceedings.mlr.press/v48/guha16.html - RRCF mechanics (streaming, reservoir sampling, 100-tree guidance): https://docs.aws.amazon.com/sagemaker/latest/dg/rcf_how-it-works.html - SPOT (EVT streaming threshold): Siffer et al., KDD 2017 — https://www.kdd.org/kdd2017/papers/view/anomaly-detection-in-streams-with-extreme-value-theory - libspot (SPOT reference implementation, C99): https://asiffer.github.io/libspot/ - Bayesian Online Changepoint Detection: Adams & MacKay 2007 — https://arxiv.org/abs/0710.3742 - Numenta Anomaly Benchmark (streaming/online evaluation): https://arxiv.org/abs/1510.03336 · https://github.com/numenta/nab - Matrix Profile: https://www.cs.ucr.edu/~eamonn/MatrixProfile.html · https://github.com/stumpy-dev/stumpy

Production systems / case studies - Anodot (120M series, 6B points/day, seasonality + adaptive online model): Toledano et al., PMLR 71:56–65, 2018 — https://proceedings.mlr.press/v71/toledano18a.html - Uber Argos (two-stage outlier/outage, hourly dynamic thresholds): https://www.uber.com/us/en/blog/argos-real-time-alerts/ - Yahoo EGADS (TSM/ADM split, scales to millions of points/sec): https://github.com/yahoo/egads - LinkedIn Luminol (lightweight, score-based, no fixed threshold): https://github.com/linkedin/luminol

Frameworks / platforms - VictoriaMetrics vmanomaly models (online/offline, decay, on-disk mode): https://docs.victoriametrics.com/anomaly-detection/components/models/ - vmanomaly FAQ (schedulers, anomaly_score, batch/backtest): https://docs.victoriametrics.com/anomaly-detection/faq/ - VictoriaMetrics scalability (100M single-node / billions cluster): https://docs.victoriametrics.com/faq/ - Salesforce Merlion (framework + DefaultDetector): https://github.com/salesforce/merlion · default detector source: https://github.com/salesforce/merlion/blob/main/merlion/models/defaults.py - LinkedIn Greykite simple anomaly detection: https://linkedin.github.io/greykite/docs/1.0.0/html/gallery/quickstart/0200_simple_anomaly_detection.html

1. The single most important finding: pretrained time-series foundation models (MOMENT, Chronos, TimesFM, Time-MoE, TSPulse) are weak at zero-shot anomaly detection — in independent benchmarks they do not beat trivial "one-liner" statistical baselines (moving-window variance / squared difference) (ONE-LINERS, ICLR 2026; arXiv 2412.19286). Do not build the system around the hope that an off-the-shelf foundation model will detect anomalies well. Use it as a feature extractor / initializer, and get accuracy from per-series adaptation.

2. Your two constraints are in tension if taken absolutely. Online anomaly detection requires a model of "normal." If you store no data, the model state itself becomes the only memory of normal. The resolution is: don't store raw series, but DO keep a small bounded per-series state (rolling sufficient statistics, seasonal coefficients, an anomaly threshold, and optionally a tiny adapter/embedding). Every detected anomaly is scored against that compressed summary.

3. Recommended architecture = two tiers:

4. Tier 1 (generalization): a shared/global model pretrained once on the 150M-series kick-start — used as a representation extractor / meta-prior (the "generalized" part).

5. Tier 2 (tuning + evolution): a per-series, online, bounded-memory detector with an adaptive threshold (Extreme Value Theory / SPOT-POT on the residual stream) that is updated incrementally as each new point arrives (the "tuned per series" + "evolves" part).

6. 150M per-series states is a feasibility problem. Keep per-series state tiny (hundreds of bytes to a few KB), prefer closed-form O(1) online updates, and use a tiered/filter stage (cheap statistical detector scores every point; a heavier model runs only on flagged windows).

7. You cannot feasibly hand-tune 150M detectors. Tuning must be learned: the global model predicts/instantiates per-series detector parameters (a meta-learning / hypernetwork pattern), and online updates keep them fresh.

1. Constraints restated

• 150M unique series — the "many-series" regime. Any per-series cost (params, compute, state) is multiplied by ~1.5×10⁸.
• Kick-start only — full historical data available once (for the initial/training pass); afterwards the raw data should not be retained.
• Generalized model, tunable per series, quickly — want a single model family that adapts to each series cheaply, not 150M hand-built models.
• Evolves with new data — online/incremental (streaming) learning, not periodic offline retraining on stored history.

2. Key finding: foundation models are not great anomaly detectors out of the box

2.1 The evidence

• ONE-LINERS (ICLR 2026), "When Foundation Models are One-Liners" evaluates MOMENT, Chronos, TimesFM, Time-MoE, and TSPulse on anomaly detection: zero-shot performance does not significantly differ from simple one-line baselines (moving-window variance for reconstruction-based, squared difference for forecasting-based). Regardless of family or size. Root cause identified: the assumption that "anomalies are harder to reconstruct/forecast" does not hold for these models. Source: OpenReview — One-Liners; companion benchmark repo necdetduruk/tsfm-anomaly-bench.

• arXiv 2412.19286, a separate critical evaluation across five manufacturing/space datasets: traditional statistical (weighted XGBoost) and deep (autoencoder) models match or outperform TSFMs on anomaly detection/prediction, while TSFMs are far more computationally expensive and do not help in few-/zero-shot. Source: arXiv 2412.19286.

• MOMENT is the notable positive counter-example, but modest and adaptation-dependent: zero-shot AD = "second best F₁", but with linear probing (fine-tuning the final layer) it reaches best F₁ — i.e., the value shows up after adaptation, not zero-shot. Source: MOMENT README (ICML 2024, arXiv 2402.03885).

• TimesFM (Google, 200M/2.5) is a forecasting model with no built-in anomaly detection; AD is done by flagging points outside quantile forecast bands. HuggingFace/PEFT (LoRA) fine-tuning example exists. Source: google-research/timesfm.

2.2 Implication

The "generalized" value of foundation models here is transferable representation learning, not built-in detection. That strongly favors the two-tier split: a shared representation/encoder (pretrained on the kick-start) + a lightweight per-series detector that is actually where detection happens.

3. Recommended two-tier architecture

Tier 1 — the "generalized" model (pretrained once on the kick-start)

Train/share one model over the full 150M-series kick-start. Candidate roles:

1. Representation/feature extractor — e.g., an autoencoder or a TSFM used as an encoder. Detection then operates on reconstruction/forecast error (anomaly score = residual magnitude). Demonstrated by the ChronosAD line of work ("Leveraging Time Series Foundation Models for Accurate Anomaly Detection") and CHARM, which use TSFMs specifically as feature extractors for AD. Sources: ChronosAD, arXiv 2606.01300; Chronos repo.

2. Meta-learner / hypernetwork — the global model outputs the per-series detector's parameters or hyperparameters, so "tuning" becomes an inference call rather than a hand-rolled optimization. This is the pattern that scales "tuned per series quickly" to 150M. Sources: iADCPS — incremental meta-learning for evolving CPS time-series AD, arXiv 2504.04374; Meta-Learning for Fast Model Recommendation, NeurIPS/HLMR 2023.

Tier 2 — the per-series online detector (the "tunable" + "evolves" part)

Per series, keep a tiny bounded state and update it O(1) per point:

• Online mean/variance / EWMA control-chart anomaly detection (e.g., AnEWMA), adapted incrementally with no batch retraining and little history. Source: AnEWMA — SCITEPRESS paper.
• Seasonality handled via online seasonal decomposition or per-series seasonal coefficients (e.g., Holt-Winters style), so contextual/seasonal anomalies are caught.
• Adaptive threshold via Extreme Value Theory (EVT) — SPOT / POT compute a high quantile from the residual stream to set the anomaly threshold dynamically, with no manual threshold and no strong distribution assumptions, and no meaningful historical retention. Sources: SPOT — libspot; Anomaly Detection in Streams with Extreme Value Theory, KDD 2017.
• Continuous-learning streaming detectors as a more expressive drop-in, e.g., Numenta HTM (learns continuously, real-time, unsupervised) and streaming-first libraries (ABERRANT, StreamAD). Sources: Numenta HTM paper; NAB benchmark; ABERRANT; StreamAD, GitHub.

The typical detection loop is: score(point | state) → compare to adaptive threshold → update(state, point) (assuming normal) — i.e., exactly an online "score-and-adapt," which matches "evolves whenever new data is available."

4. The "don't store data" constraint — what it really means

• You can't avoid some memory of normal. Online AD needs a model of normal; the model weights + compact sufficient statistics are that memory. What you can avoid is storing raw historical series. Keep each series' state bounded (e.g., a short in-memory rolling window + summary stats), not exhaustive history.
• Hard trade-off: the less history you retain, the worse you detect slow / concept-drift / regime anomalies versus point anomalies. Point anomalies are cheap (a threshold on a residual). Slow or seasonal-shift anomalies may need a longer implicit context (which is, effectively, stored data). Decide which type you actually care about. NAB (Numenta) explicitly notes AD is hard to benchmark precisely for exactly this reason (Source: NAB, arXiv 1510.03336).
• Catastrophic forgetting / drift: pure online learning without replay can forget older regimes. Literature handles this with incremental coresets / experience replay (e.g., ONER, CADIC). For point-anomaly detection a robust online estimator + adaptive threshold is fairly resilient, but if you must detect drift/regime change you'll want a drift detector or a small buffered normal profile. (Review of incremental/continual AD methods — arXiv preprints 2511.08634, 2412.03907, 2201.06763.)
• Key subtlety worth flagging: the phrase "the model should evolve whenever new data is available" + "don't store data" is internally coherent only if the evolution is online state update, not offline rebatching over stored history. All the recommended options above are online-state-update.

5. Scaling to 150M series — feasibility math

• State budget is the first design decision. With hundreds of bytes to a few KB per series you can hold all 150M detectors in ~100 GB–600 GB of distributed memory; anything larger (e.g., a per-series deep network) becomes infeasible at this count.
• Prefer O(1) closed-form updates (EWMA, seasonal coefficients, EVT quantile updates) so per-point cost is essentially constant and CPU-throughput-bound, not learning-bound.
• Two-stage filtering to keep compute bounded: run the cheap statistical detector on every point of every series; escalate to the heavier shared model (Tier 1) only on a window flagged as suspicious. This decouples the 150M-scale "always-on" cost from the expensive-model cost.
• Tuning at scale is not manual. 150M series can't be individually tuned by hand; the Tier-1 global model should generate per-series settings (meta-learning / fast model recommendation), with online updates refining them (Source: Meta-Learning for Fast Model Recommendation).
• Where it runs with InfluxDB: InfluxDB's stack is batch/stream-oriented; Kapacitor is InfluxData's streaming engine with an alert node and User-Defined Functions (UDFs) for custom anomaly algorithms applied to incoming points — a natural host for the online per-series scorer if it must stay inside the Influx ecosystem. (Source: Kapacitor docs.) Alternatively run a stream processor (Flink/RisingWave/etc.) that reads new points from InfluxDB, updates per-series state, and writes alerts back.

6. Options matrix (complexity → expressiveness)

(Adapters/PEFT context: TimesFM ships a LoRA fine-tuning example — timesfm finetuning dir; PEFT discussion for TSFMs — tsfm.ai blog.)

7. Recommendations (pragmatic path)

1. Reframe the goal as reconstruction/forecast residual + adaptive threshold, not "a foundation model that detects anomalies." Accuracy comes from the residual stream and a good dynamic threshold.
2. Use the kick-start to train one shared representation / meta-prior over the 150M series (a TSFM or autoencoder as an encoder, or a meta-learner). This is the only place full data is needed.
3. Deploy a tiny online per-series detector (online stats + EVT adaptive threshold) with bounded in-memory state only — no raw-series retention. This satisfies "evolves with new data" and "no data stored."
4. Buy down scale risk with the tiered/filter design so only suspicious windows hit the expensive shared model.
5. Validate on a representative stratified sample of tens–hundreds of series — you can never hand-inspect 150M — and use that sample to set Tier-1 hyperparameters and the meta-tuning objective.
6. Run it as a stream update (Kapacitor UDF or a stream processor) integrated with InfluxDB, rather than offline batch retraining.

8. Caveats on evidence

• The strongest claims (One-Liners; arXiv 2412.19286) are preprint / in-review sources; the vendor-documented workflows (MOMENT tutorial, BigQuery/TimesFM) present usage examples, not independent benchmark wins. Treat "foundation models beat baselines at AD" as not established; treat "foundation models are useful as feature extractors + after adaptation" as the safer claim.
• Most comparison studies focus on zero-shot; fine-tuned/adapted TSFMs are consistently the more favorable regime — consistent with the two-tier recommendation.
• Exact end-to-end systems at 150M series, online, with no retention were not found in the literature as a single turnkey product; the design above is assembled from the best-supported building blocks rather than one proven system.

References

• One-Liners (ICLR 2026): https://openreview.net/pdf?id=H27kvyG4qf · bench https://github.com/necdetduruk/tsfm-anomaly-bench
• Critical TSFM-AD evaluation: https://arxiv.org/html/2412.19286v1
• MOMENT: https://github.com/moment-timeseries-foundation-model/moment / https://arxiv.org/abs/2402.03885
• TimesFM: https://github.com/google-research/timesfm · Chronos: https://github.com/amazon-science/chronos-forecasting / https://doi.org/10.48550/arxiv.2403.07815
• ChronosAD (TSFMs as feature extractors): https://arxiv.org/html/2606.01300v1
• iADCPS (incremental meta-learning, online update, dynamic threshold): https://arxiv.org/abs/2504.04374
• Meta-Learning for Fast Model Recommendation: https://proceedings.mlr.press/v224/navarro23a.html
• SPOT / Extreme Value Theory streaming AD: https://asiffer.github.io/libspot/ · KDD17: https://www.kdd.org/kdd2017/papers/view/anomaly-detection-in-streams-with-extreme-value-theory
• AnEWMA: https://www.scitepress.org/Papers/2025/134378/134378.pdf
• Numenta HTM (continuous online learning): https://www.numenta.com/resources/research-publications/papers/unsupervised-real-time-anomaly-detection-for-streaming-data/ · NAB: https://github.com/numenta/nab / https://arxiv.org/pdf/1510.03336
• Streaming AD libraries: ABERRANT https://github.com/OliverHennhoefer/aberrant · StreamAD https://github.com/Fengrui-Liu/StreamAD
• Continual/incremental AD (forgetting/drift): https://arxiv.org/abs/2201.06763 · https://arxiv.org/html/2412.03907v2 · https://doi.org/10.48550/arxiv.2511.08634
• InfluxDB/Kapacitor (InfluxData streaming engine, alert + UDFs): https://docs.influxdata.com/kapacitor/v1/

Adapted from Getting Started with Ralph (AI Hero). The original uses Claude Code + Docker Desktop on Linux. This version swaps in two open-source agents — opencode and pi coding agent — drops Docker entirely, and covers Linux and Windows. Shared concepts appear once; each OS/agent combo gets its own section with only what's different.

What is Ralph?

Ralph is a technique for running AI coding agents in a loop: you run the same prompt repeatedly, the AI picks its own tasks from a PRD, commits after each feature, and you come back later to working code. The loop stays the same no matter which agent or OS you use:

For tips on task sizing, prioritization, and feedback loops, see the 11 tips for AI coding with Ralph.

What changes without Docker

Docker Desktop sandboxes gave you three things you now own yourself:

1. No sandbox. docker sandbox run claude isolated the agent. opencode and pi run directly with your user's permissions — they can install packages, run servers, and touch any file you can. Start in a disposable VM or a scratch folder, run the once-script a few times first, and review commits before going fully AFK.
2. Git identity is on you. Sandboxes auto-injected git config for commit attribution. Without Docker, set it once:

   git config --global user.name "You"
   git config --global user.email "you@example.com"

3. Credentials are yours to manage. No Docker volume — each agent stores auth in its own config directory. For scripted (headless) runs, prefer exporting a provider API key in your shell.

Step 1 — Write the PRD (identical for all four combos)

Ask the agent for a plan, iterate until you're happy, then save it as PRD.md:

• opencode: the TUI has a plan mode — press Tab to switch modes, iterate, then tell it to save the plan to PRD.md.
• pi: no built-in plan mode (by design). Ask pi interactively to draft a plan, iterate in chat, then have it write PRD.md.

Create the empty progress file:

touch progress.txt

The PRD can be any format (markdown checklist, JSON, prose). What matters: clear scope, and tasks the agent can pull out one at a time.

Step 2 — Install & authenticate (pick your combo)

Auth done in the TUI persists for later headless runs; an exported API key works everywhere including scripts.

A. Linux + opencode (no Docker)

Human-in-the-loop run — ralph-once.sh (watch it, run it again):

#!/bin/bash
opencode run -p --auto "@PRD.md @progress.txt
1. Read the PRD and progress file.
2. Find the next incomplete task and implement it.
3. Commit your changes.
4. Update progress.txt with what you did.
ONLY DO ONE TASK AT A TIME."

chmod +x ralph-once.sh
./ralph-once.sh

• --auto auto-approves permission requests so the loop never stalls (omit it for a stricter run).
• -p/--print formats output cleanly; @file attaches files to the prompt.

AFK loop — afk-ralph.sh (run with a cap: ./afk-ralph.sh 20):

#!/bin/bash
set -e
if [ -z "$1" ]; then
  echo "Usage: $0 <iterations>"
  exit 1
fi
for ((i=1; i<=$1; i++)); do
  result=$(opencode run -p --auto "@PRD.md @progress.txt
1. Find the highest-priority task and implement it.
2. Run your tests and type checks.
3. Update the PRD with what was done.
4. Append your progress to progress.txt.
5. Commit your changes.
ONLY WORK ON A SINGLE TASK.
If the PRD is complete, output <promise>COMPLETE</promise>.")
  echo "$result"
  if [[ "$result" == *"<promise>COMPLETE</promise>"* ]]; then
    echo "PRD complete after $i iterations."
    exit 0
  fi
done

set -e exits on errors, $1 caps iterations (runaway-cost protection), and the sigil check stops the loop early when the PRD is done.

B. Linux + pi coding agent (no Docker)

Human-in-the-loop run — ralph-once.sh:

#!/bin/bash
pi -p "@PRD.md @progress.txt
1. Read the PRD and progress file.
2. Find the next incomplete task and implement it.
3. Commit your changes.
4. Update progress.txt with what you did.
ONLY DO ONE TASK AT A TIME."

chmod +x ralph-once.sh
./ralph-once.sh

• pi has no permission popups by design — tool calls just execute. -p is print (headless) mode; @file attaches files. Use --tools read,write,edit,bash (the default set) or a narrower allowlist if you want extra safety without a sandbox.

AFK loop — afk-ralph.sh:

#!/bin/bash
set -e
if [ -z "$1" ]; then
  echo "Usage: $0 <iterations>"
  exit 1
fi
for ((i=1; i<=$1; i++)); do
  result=$(pi -p "@PRD.md @progress.txt
1. Find the highest-priority task and implement it.
2. Run your tests and type checks.
3. Update the PRD with what was done.
4. Append your progress to progress.txt.
5. Commit your changes.
ONLY WORK ON A SINGLE TASK.
If the PRD is complete, output <promise>COMPLETE</promise>.")
  echo "$result"
  if [[ "$result" == *"<promise>COMPLETE</promise>"* ]]; then
    echo "PRD complete after $i iterations."
    exit 0
  fi
done

C. Windows + opencode (no Docker)

Runs fine from PowerShell (scripts below) or cmd. Install with scoop install opencode or choco install opencode, run opencode once, /connect, and pick a provider.

Human-in-the-loop run — ralph-once.ps1:

# Run once, watch it, run again.
opencode run -p --auto "@PRD.md @progress.txt
1. Read the PRD and progress file.
2. Find the next incomplete task and implement it.
3. Commit your changes.
4. Update progress.txt with what you did.
ONLY DO ONE TASK AT A TIME."

Set-ExecutionPolicy -Scope Process Bypass   # if scripts are blocked
.\ralph-once.ps1

AFK loop — afk-ralph.ps1 (run with .\afk-ralph.ps1 20):

param([int]$Iterations)
if (-not $Iterations) { Write-Host "Usage: .\afk-ralph.ps1 <iterations>"; exit 1 }

for ($i = 1; $i -le $Iterations; $i++) {
  $result = opencode run -p --auto "@PRD.md @progress.txt
1. Find the highest-priority task and implement it.
2. Run your tests and type checks.
3. Update the PRD with what was done.
4. Append your progress to progress.txt.
5. Commit your changes.
ONLY WORK ON A SINGLE TASK.
If the PRD is complete, output <promise>COMPLETE</promise>."
  Write-Output $result
  if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
  if ($result -match "<promise>COMPLETE</promise>") {
    Write-Host "PRD complete after $i iterations."
    exit 0
  }
}

$result collects opencode's stdout, $LASTEXITCODE propagates failures, and -match detects the completion sigil.

D. Windows + pi coding agent (no Docker)

pi requires a bash shell on Windows — install Git for Windows (pi auto-detects C:\Program Files\Git\bin\bash.exe). Everything below is bash run from a Git Bash terminal. Install pi, then authenticate (pi → /login, or export ANTHROPIC_API_KEY in the Git Bash session).

Human-in-the-loop run — ralph-once.sh (in Git Bash):

#!/bin/bash
pi -p "@PRD.md @progress.txt
1. Read the PRD and progress file.
2. Find the next incomplete task and implement it.
3. Commit your changes.
4. Update progress.txt with what you did.
ONLY DO ONE TASK AT A TIME."

chmod +x ralph-once.sh
./ralph-once.sh

AFK loop — afk-ralph.sh:

#!/bin/bash
set -e
if [ -z "$1" ]; then
  echo "Usage: $0 <iterations>"
  exit 1
fi
for ((i=1; i<=$1; i++)); do
  result=$(pi -p "@PRD.md @progress.txt
1. Find the highest-priority task and implement it.
2. Run your tests and type checks.
3. Update the PRD with what was done.
4. Append your progress to progress.txt.
5. Commit your changes.
ONLY WORK ON A SINGLE TASK.
If the PRD is complete, output <promise>COMPLETE</promise>.")
  echo "$result"
  if [[ "$result" == *"<promise>COMPLETE</promise>"* ]]; then
    echo "PRD complete after $i iterations."
    exit 0
  fi
done

Same as the Linux+pi variant — the only difference is where the shell comes from (Git Bash), so these files work verbatim on both. If pi ever says it can't find a shell, point it at Git Bash via shellPath in ~/.pi/agent/settings.json:

{ "shellPath": "C:\\Program Files\\Git\\bin\\bash.exe" }

Customizing the loop

Ralph is just a loop, so it's endlessly swappable:

• Swap the task source. Instead of a local PRD, pull from GitHub Issues, Linear, or beads — the agent still picks what to work on.
• Change the output. Instead of committing to main, create a branch + PR per iteration to triage a backlog.
• Different loop types: test coverage (write tests until coverage hits target), linting (fix errors one by one), duplication (refactor jscpd clones), or entropy (clean up code smells).
• Per-agent extras: pi can load skills/extensions that gate tools or checkpoint git commits; opencode supports custom agent modes and permissions config. Both work fine inside the same loop.

Original article: Getting Started with Ralph · Tips: 11 tips for AI coding with Ralph

Observability is the property of a system whose internal state can be inferred from its external outputs. In modern distributed/microservice systems these outputs are commonly grouped into the "three pillars" — metrics (numeric time series such as CPU, latency,request rate), logs (semi-structured text events), and traces (causal request paths across services) — with events (deployments, config changes, alerts) as afourth, cross-cutting signal. The OpenTelemetry specification defines these signal typesand their data models directly (OpenTelemetry docs).

AIOps, a term popularized by Gartner (industry analyst, not a primary technical source — flagged), refers to platforms that apply analytics and ML to operations data to automate detection, correlation, and remediation. The practical driver for applying ML at scale is volume and velocity: cloud-native systems emit telemetry at cardinalities and rates that exceed human triage capacity, and static thresholds generalize poorly across seasonal, multi-tenant, and rapidly-deploying workloads. The AIOps problem is therefore dominated by unsupervised / weakly-supervised learning, because ground-truth incident labels are scarce, delayed, and noisy.

2. Anomaly Detection Methods

2.1 Statistical / classical

The oldest and still widely deployed family models the "normal" signal explicitly and flags deviations.

• Moving averages / EWMA: an exponentially weighted mean \(\hat{x}_t = \alpha x_t + (1-\alpha)\hat{x}_{t-1}\) with residual thresholding.
• Holt-Winters triple exponential smoothing captures level, trend, and seasonality; it underpins many production forecasters (e.g., historically in RRDtool/Graphite).
• ARIMA (\(ARIMA(p,d,q)\)) models autocorrelation of a differenced series; anomalies are points with large forecast residuals.
• Seasonal decomposition (classical and STL, Cleveland et al., 1990, J. Official Statistics) separates \(x_t = T_t + S_t + R_t\); anomalies are outliers in the remainder \(R_t\). Twitter's AnomalyDetection package built on STL + generalized ESD is a well-known open-source instance (repo, not peer-reviewed — flagged).
• 3-sigma / MAD: flag points beyond \(\mu \pm 3\sigma\), or robustly beyond a Median Absolute Deviation multiple.

These methods are cheap, interpretable, and strong baselines; they struggle with multivariate dependence and non-stationary regime changes.

2.2 Traditional ML

• Isolation Forest (Liu, Ting, Zhou, ICDM 2008, DOI:10.1109/ICDM.2008.17) isolates anomalies via random partitioning; anomalies require fewer splits (shorter path length).
• One-Class SVM (Schölkopf et al., 2001, Neural Computation, DOI:10.1162/089976601750264965) learns a boundary enclosing normal data in a kernel feature space.
• Clustering-based outlier detection with k-means or DBSCAN (Ester et al., KDD 1996) flags points far from any dense cluster; LOF (Breunig et al., SIGMOD 2000, DOI:10.1145/342009.335388) scores local density deviation.

2.3 Deep learning

• LSTM/GRU forecasting + residual thresholding for sequences; the LSTM approach to system-log anomaly detection is exemplified by DeepLog (see §2.5).
• Autoencoders / VAE: reconstruction error as anomaly score. OmniAnomaly (Su et al., KDD 2019, DOI:10.1145/3292500.3330672) uses a stochastic recurrent VAE for multivariate metrics and popularized the SMD/SMAP/MSL benchmarks.
• CNN for time series (dilated/temporal convolutions) and Seq2Seq encoder–decoder models for forecasting-based detection.
• LSTM-VAE and GAN-based detectors (e.g., MAD-GAN, Li et al., 2019) extend this reconstruction paradigm.

2.4 Modern / SOTA (2021–2025)

Transformers dominate long-horizon multivariate forecasting and, adapted, anomaly detection:

• Informer (Zhou et al., AAAI 2021, arXiv:2012.07436) — ProbSparse attention for long-sequence forecasting. (verified)
• Autoformer (Wu et al., NeurIPS 2021, arXiv:2106.13008) — decomposition + Auto-Correlation. (verified)
• PatchTST (Nie et al., ICLR 2023, arXiv:2211.14730) — patching + channel independence ("a time series is worth 64 words"). (verified)
• TimesNet (Wu et al., ICLR 2023, arXiv:2210.02186) — reshapes 1D series into 2D by periodicity for general analysis incl. anomaly detection. (verified)
• Anomaly Transformer (Xu et al., ICLR 2022, arXiv:2110.02642) — association-discrepancy criterion. (verified)
• TranAD (Tuli et al., VLDB 2022, arXiv:2201.07284) — deep transformer with adversarial training for multivariate anomaly detection. (verified)
• Diffusion / LLM-based: diffusion models and reprogrammed LLMs are emerging for forecasting/imputation (see §5 foundation models).
• Graph neural networks for multivariate metrics model inter-series dependency; GDN (Deng & Hooi, AAAI 2021, arXiv:2106.06947) learns a graph over sensors and flags edge-deviation. MTAD-GAT (Zhao et al., ICDM 2020) uses dual graph-attention.

Caveat on transformer AD benchmarks. Several works (e.g., "revisiting time-series anomaly detection" critiques, 2022–2023) argue that popular point-adjustment evaluation inflates scores; comparative rankings should be read with this in mind. I have not re-verified individual leaderboard numbers here.

2.5 Unsupervised log anomaly detection

• Drain (He et al., ICWS 2017, DOI:10.1109/ICWS.2017.13) — fixed-depth parse tree for online log-template extraction. (Venue ICWS 2017 verified via Crossref.)
• DeepLog (Du et al., CCS 2017, DOI:10.1145/3133956.3134015) — LSTM over log-key sequences; predicts next key and flags surprises.
• LogAnomaly (Meng et al., IJCAI 2019, DOI:10.24963/ijcai.2019/658) — template2vec semantic embeddings + sequential + quantitative modeling.
• LogRobust (Zhang et al., ESEC/FSE 2019, DOI:10.1145/3338906.3338931) — attention-BiLSTM robust to log instability/evolution.
• Log-template clustering (LogCluster, LKE) groups templates to reduce dimensionality before detection.

Comparison table.

3. Clustering and Signal Correlation

Clustering serves two operational goals: compression (turn millions of log lines into a few hundred templates) and noise reduction (collapse redundant alerts).

• Log clustering: Drain (parse-tree), LogCluster (Lin et al., ICSE 2016, DOI:10.1145/2889160.2889232) clusters logs to identify problems; Spell (Du & Li, ICDM 2016) uses longest-common-subsequence parsing.
• Metric clustering: grouping correlated time series (via correlation/DTW distance) identifies redundant signals and shared failure modes.
• Event correlation & alert aggregation: temporal + topological correlation groups alerts into incidents, reducing "alert storms." Much of this is vendor-implemented; primary technical treatments appear in AIOps-challenge literature rather than product docs.
• Trace/span clustering: clustering trace structures/latency profiles surfaces anomalous request paths in microservice systems.
• Incident clustering — grouping alerts believed to share a root cause — is a bridge to RCA: a well-formed cluster localizes where to look before why is determined.

4. Root Cause Analysis and Fault Localization

• Spectrum-based fault localization (SBFL) ranks components by suspiciousness from pass/fail execution spectra (e.g., Ochiai, Tarantula) — mature in software testing, adapted to services.
• Causal inference / causal graphs: learn a causal DAG over metrics/services (e.g., PC/Granger-based) and trace anomaly propagation. CloudRanger (Wang et al., CCGrid 2018, DOI:10.1109/CCGRID.2018.00076) builds an impact graph for cloud-native RCA.
• Microservice dependency graphs: MicroRCA (Wu et al., NOMS 2020, DOI:10.1109/NOMS47738.2020.9110353) localizes anomalous services on an attributed service graph.
• GNN-based RCA: graph neural nets over service/topology graphs propagate anomaly evidence to rank likely culprits.
• LLM-assisted RCA: recent work uses LLMs to summarize evidence, hypothesize causes, and draft mitigations. RCACopilot (Chen et al., 2023, arXiv:2305.15778 — ID verified: "Automatic Root Cause Analysis via Large Language Models for Cloud Incidents") and Microsoft studies on LLMs for incident RCA are representative. Treat efficacy claims cautiously: reported gains depend heavily on retrieval quality and evaluation setup.

5. Recent Trends (2023–2025)

• LLM / LLM-agent observability. Domain LLMs and agents parse logs, answer natural-language queries over telemetry, and drive investigation loops. OWL: A Large Language Model for IT Operations (Guo et al., 2023, arXiv:2309.09298) (verified) is an explicit ops-domain LLM with an Ops benchmark. "Observability copilots" are largely commercial (Datadog Bits AI, Grafana/Elastic AI assistants) — product docs, not peer-reviewed.
• Foundation / self-supervised models for time series. Zero-shot forecasters: TimesFM (decoder-only foundation model, Das et al., 2023, arXiv:2310.10688) (verified), TimeGPT-1 (Garza et al., 2023, arXiv:2310.03589) (verified), and LLM-reprogramming via Time-LLM (Jin et al., ICLR 2024, arXiv:2310.01728) (verified). These promise few/zero-shot deployment across heterogeneous metrics.
• Generative AI for synthetic telemetry: using generative models to synthesize logs/ traces for augmentation and rare-fault simulation — active but still emerging; strong peer-reviewed baselines are sparse (flagged as low-verification).
• eBPF-based observability: kernel-level, low-overhead instrumentation (Cilium/Hubble, Pixie, Parca) enables auto-generated metrics/traces without code changes. Primary sources are the projects' docs/kernel docs rather than papers.
• OpenTelemetry standardization: OTel (CNCF) is now the de facto vendor-neutral instrumentation and semantic-conventions standard, unifying the pillars (opentelemetry.io) — a primary spec source.
• Causality-driven AIOps: shift from correlation to intervention/causal models for RCA.
• Benchmarks & datasets:
• LogHub (He et al., collection of system-log datasets; arXiv:2008.06448).
• NAB — Numenta Anomaly Benchmark (Lavin & Ahmad, 2015, arXiv:1510.03336).
• TSB-AD — recent reproducible TS-anomaly benchmark (VLDB 2024/2025; associated reproducibility paper indexed on arXiv — exact ID not cleanly verified in this pass, flagged).
• GAIA dataset and AIOps Challenge datasets (KPI/multi-source) from the AIOps community — dataset repos, not peer-reviewed papers.

6. Production Tooling and Ecosystem

Open-source AIOps frameworks include Merlion (Salesforce, arXiv:2109.09265) and PyOD/TODS for anomaly-detection pipelines. Vendor "AI" features (Datadog Watchdog, Elastic ML jobs, Splunk MLTK) are documented in product docs but their internal algorithms are only partially disclosed — treat capability claims as vendor-sourced.

7. Open Challenges and Future Directions

• Evaluation validity: point-adjustment and leaky benchmarks inflate AD scores; the field is moving toward stricter protocols (TSB-AD, revisiting-AD critiques).
• Label scarcity & concept drift: production telemetry is unlabeled and non-stationary; online/continual learning remains hard.
• Multimodal fusion: jointly reasoning over metrics + logs + traces + topology is still immature; most methods handle one pillar.
• Causality vs. correlation: reliable causal RCA at scale is unsolved; causal graphs are sensitive to hidden confounders and sampling.
• LLM reliability: hallucination, cost, latency, and reproducibility limit LLM-agent RCA in production; grounding via retrieval and tool use is the current mitigation.
• Foundation-model transfer: zero-shot TS models are promising but under-validated on operational anomaly tasks specifically.

References

Verified against arXiv metadata in this pass are marked ✔.

1. ✔ Zhou et al. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, AAAI 2021. arXiv:2012.07436
2. ✔ Wu et al. Autoformer: Decomposition Transformers with Auto-Correlation, NeurIPS 2021. arXiv:2106.13008
3. ✔ Nie et al. A Time Series is Worth 64 Words (PatchTST), ICLR 2023. arXiv:2211.14730
4. ✔ Wu et al. TimesNet: Temporal 2D-Variation Modeling, ICLR 2023. arXiv:2210.02186
5. ✔ Xu et al. Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy, ICLR 2022. arXiv:2110.02642
6. ✔ Tuli et al. TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series, VLDB 2022. arXiv:2201.07284
7. ✔ Guo et al. OWL: A Large Language Model for IT Operations, 2023. arXiv:2309.09298
8. ✔ Das et al. A decoder-only foundation model for time-series forecasting (TimesFM), 2023. arXiv:2310.10688
9. ✔ Garza et al. TimeGPT-1, 2023. arXiv:2310.03589
10. ✔ Jin et al. Time-LLM: Time Series Forecasting by Reprogramming Large Language Models, ICLR 2024. arXiv:2310.01728
11. Liu, Ting, Zhou. Isolation Forest, ICDM 2008. DOI:10.1109/ICDM.2008.17
12. Schölkopf et al. Estimating the Support of a High-Dimensional Distribution (One-Class SVM), Neural Computation 2001. DOI:10.1162/089976601750264965
13. Breunig et al. LOF: Identifying Density-Based Local Outliers, SIGMOD 2000. DOI:10.1145/342009.335388
14. Ester et al. A Density-Based Algorithm for Discovering Clusters (DBSCAN), KDD 1996.
15. Cleveland et al. STL: A Seasonal-Trend Decomposition Procedure Based on Loess, J. Official Statistics 1990.
16. Su et al. OmniAnomaly: Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Networks, KDD 2019. DOI:10.1145/3292500.3330672
17. Deng & Hooi. Graph Neural Network-Based Anomaly Detection in Multivariate Time Series (GDN), AAAI 2021. arXiv:2106.06947
18. He et al. Drain: An Online Log Parsing Approach with Fixed Depth Tree, ICWS 2017. DOI:10.1109/ICWS.2017.13
19. Du et al. DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning, CCS 2017. DOI:10.1145/3133956.3134015
20. Meng et al. LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs, IJCAI 2019. DOI:10.24963/ijcai.2019/658
21. Zhang et al. Robust Log-Based Anomaly Detection on Unstable Log Data (LogRobust), ESEC/FSE 2019. DOI:10.1145/3338906.3338931
22. Lin et al. Log Clustering Based Problem Identification for Online Service Systems (LogCluster), ICSE 2016. DOI:10.1145/2889160.2889232
23. Wang et al. CloudRanger: Root Cause Identification for Cloud Native Systems, CCGrid 2018. DOI:10.1109/CCGRID.2018.00076
24. Wu et al. MicroRCA: Root Cause Localization of Performance Issues in Microservices, NOMS 2020. DOI:10.1109/NOMS47738.2020.9110353
25. He et al. LogHub: A Large Collection of System Log Datasets, 2020. arXiv:2008.06448
26. Lavin & Ahmad. Evaluating Real-Time Anomaly Detection Algorithms — the Numenta Anomaly Benchmark (NAB), 2015. arXiv:1510.03336
27. Bhatnagar et al. Merlion: A Machine Learning Library for Time Series, 2021. arXiv:2109.09265
28. OpenTelemetry Documentation. opentelemetry.io/docs (primary spec)
29. Prometheus Documentation. prometheus.io/docs (primary)
30. Grafana Documentation. grafana.com/docs (primary)

Unverified / flagged in text: TSB-AD exact arXiv ID (VLDB reproducibility paper —not cleanly resolved in this pass); GAIA and AIOps-Challenge dataset provenance (communitydataset repos, not peer-reviewed papers); generative-synthetic-telemetry peer-reviewedbaselines (sparse — emerging area); all vendor "AI" capability claims (Datadog Watchdog/BitsAI, Elastic ML, Splunk MLTK) are product-doc sourced, not peer-reviewed; "AIOps" definitionattributed to Gartner (analyst, not a primary technical source). (Drain ICWS 2017 venue andRCACopilot arXiv:2305.15778 were both verified post-hoc via Crossref/arXiv metadata and areno longer flagged.)

We operate a distributed system at scale. When it breaks, we know what broke and where — the affected service, the degraded operation, the time it started. This takes minutes.

We do not know why.

The answer exists somewhere in our telemetry — buried in a log line, encoded in a metric curve, hinted at in a trace fragment, or recorded in a change ticket. But these artifacts live in separate systems, speak different languages, and carry no common thread. Bridging them is a human act: an engineer opens half a dozen consoles, triangulates timestamps, forms a mental model of dependencies, and manually reasons backward from symptom to cause. This takes hours. Sometimes days. Every single time.

This is not a tooling gap. It is a reasoning gap. And we intend to close it.

I. The Aegis Thesis

Correlation is not causation, but it is the raw material from which causation can be inferred. If we can fuse change records with telemetry signals across all three pillars — logs, metrics, traces — and if we can construct even a partial map of how our services depend on one another and on their infrastructure, then we can automate the hypothesis-generation step of root cause analysis. We can give the operator a ranked, evidence-backed causal timeline in under five minutes, instead of a blank search bar and a prayer.

Aegis does not aim to replace the engineer. It aims to compress the mechanical, multi-system search-and-correlate phase — the part that burns time without burning insight — so that the engineer starts their investigation already holding a set of plausible explanations, each backed by evidence drawn from the systems they would otherwise query by hand.

II. The Signals We Have

We possess three families of telemetry, each with different character:

Logs

The richest and messiest signal. Some structured, some semi-structured, some nearly free-form text. They carry error messages, latency measurements, status codes, and — where maintained — ownership attribution and caller identity. They are the ground truth of what the application experienced.

Metrics

The cleanest signal. System metrics, application metrics, infrastructure metrics, trace-derived metrics. Well-typed, regularly sampled, pre-aggregated. They tell us that something changed and when, with precision. An existing anomaly pipeline already computes per-metric confidence intervals continuously.

Traces

The connective tissue — where they reach. Distributed tracing gives us span-level visibility into request flow, service-to-service call graphs, and latency propagation. But instrumentation is incomplete: critical infrastructure components, particularly database systems, are excluded from tracing due to agent stability concerns. The dependency graph has holes where it matters most.

Beyond Telemetry

We also possess a rich change record: every database modification, software deployment, and configuration alteration is recorded with precise timestamps through a formal approval pipeline. We know who changed what and when. What we lack is the connection from change to consequence — the blast radius, the dependent services, the expected behavioral signature.

III. The Gaps We Must Bridge

Aegis is defined as much by what is missing as by what is present:

1. No universal correlation identifier. Some signals carry trace IDs, request IDs, or transaction IDs. Most do not. Cross-signal correlation today relies on temporal proximity and manual search.

2. Traces amputated at the infrastructure boundary. The most critical dependency — the persistence layer — is invisible to our distributed tracing. When a database degrades, we see the symptoms in the services that call it but cannot see inside.

3. A fragmented dependency graph. No single source maps which services depend on which other services, databases, or infrastructure. Tracing provides partial edges. Logs encode caller-callee relationships in fields that are being deprecated. The configuration management database aspires to centrality but is not yet complete.

4. Change events carry no blast radius. A database administrator alters a parameter. The change is recorded. But nothing in that record says "this database serves these twelve services." That knowledge lives in human minds, scattered across teams.

5. Ownership decays. Service ownership fields, where they exist, are populated at creation time and rarely maintained. A field written five years ago may point to a team that no longer exists.

Aegis must work despite these gaps. It cannot wait for all data to be clean, all instrumentation to be complete, or all dependencies to be declared. It must make defensible inferences from partial information and express its uncertainty honestly.

IV. What Aegis Does

Given a degraded service and a time window, Aegis returns:

A ranked causal timeline. A chronologically ordered sequence connecting change events to anomaly onset to symptom emergence, with each candidate root cause scored by confidence and accompanied by supporting evidence drawn from logs, metrics, and traces.

The operator receives this as a report — pushed to a communication channel and persisted in our log aggregation system for visualization — within five minutes of trigger.

Aegis is triggered in two ways:

• Automatically, when the anomaly pipeline detects a metric exceeding a severity threshold.
• Manually, when a platform operator initiates an investigation during an incident or post-incident review.

In its first version, Aegis is non-interactive. It generates a complete report. Drill-down, hypothesis challenging, and iterative refinement are deferred to later iterations.

V. How Aegis Reasons

The Engine: Heuristics First

Aegis v1 uses explicit, auditable rules. Each rule encodes a causal pattern: a type of change event, a temporal window, an expected signature in telemetry, and a scoring function. Rules compete against each other for explanatory power. Aegis ranks candidates; it does not pronounce a single verdict.

We begin with rule-based reasoning because it is transparent, debuggable, and requires no training data. It can be deployed immediately. Statistical and machine learning approaches are natural extensions when we have accumulated a labeled corpus of resolved incidents.

Candidate Rule Families

Change-Induced Degradation. A change occurs on an entity. Within a bounded temporal window, dependent services exhibit anomaly onset, error rate increases, and novel error patterns in logs that were absent before the change.

No-Change Degradation. An anomaly with no proximate change event. Suggests gradual resource exhaustion, external dependency failure, or traffic anomaly. Evidence is drawn from metric saturation curves and upstream correlation.

Cascading Failure. Service A degrades. Service B, which depends on A, degrades with a temporal lag consistent with propagation. Service C follows. Evidence is drawn from trace spans and temporal ordering across the dependency graph.

Infrastructure Event. Multiple unrelated services degrade simultaneously. Correlated with a host-level or network-level anomaly. Evidence is drawn from infrastructure metrics and network flow data.

The Dependency Graph: Synthesized, Not Discovered

Since no single source provides a complete dependency graph, Aegis constructs one from fragments:

• Trace spans from distributed tracing systems, where instrumented.
• Caller-callee relationships inferred from log fields.
• Application-to-host mappings from the configuration management database.
• Network flow data, mapped to services.
• Operator-declared known dependencies, weighted as high-confidence edges.

The graph is constructed lazily, scoped to the investigation at hand. It is never expected to be complete — only useful. Every edge carries a confidence weight, and Aegis propagates uncertainty through its inferences.

Correlation vs. Causation

Aegis employs multiple strategies to avoid conflating coincidence with cause:

• Rank, do not decide. All plausible candidates are presented. The operator retains judgment.
• Intervention analysis. If a changed entity serves N dependent services and all N degrade, the causal link is strong. If only one degrades, the link is weak.
• Temporal precedence. Changes that occurred after symptom onset are excluded as causes.
• Counterfactual weakening. If the same or similar change occurred previously without degradation, the hypothesis is discounted.
• Signal convergence. Hypotheses supported by multiple independent telemetry pillars gain confidence; those supported by only one are penalized.

VI. What Aegis Does Not Do

Clarity about scope is as important as clarity about function.

Aegis does not:

• Replace the on-call engineer. It gives them a head start, not a final answer.
• Automate remediation. It identifies probable causes; it does not trigger rollbacks or configuration reversions.
• Require perfect data. It is designed to degrade gracefully with partial information, expressing lower confidence when signals are sparse.
• Operate continuously in v1. Investigations are triggered on-demand; continuous pre-computation of causal indices is a future optimization.
• Provide interactive drill-down in v1. The initial report is a complete artifact; iterative questioning comes later.

VII. The Measure of Success

Aegis v1 is successful if:

1. Platform operators receive a ranked causal hypothesis list in under five minutes from trigger.
2. The true root cause appears in the top three candidates in the majority of investigations, as measured against post-incident review outcomes.
3. Incidents resolved during the event increase, because root cause is identified while the incident is still active — reducing the volume of formal post-incident reviews.
4. Operators adopt Aegis as their first investigation step, displacing manual multi-system querying as the default workflow.

VIII. The Path Forward

This manifesto describes intent, not implementation. The next steps are concrete:

1. Ratify the scope. Confirm the heuristic-first approach, trigger modes, output format, and latency target.
2. Design the initial rule catalog. Convene domain experts — service owners, database administrators, incident commanders — to articulate the causal patterns they recognize today.
3. Prototype the dependency graph construction. Build a throwaway integration against real data sources and measure coverage against a sample of known service dependencies.
4. Define data contracts. Formalize the query interfaces and response schemas for each source system.
5. Establish an evaluation baseline. Instrument current manual investigation workflows to capture time-to-root-cause for a representative set of historical incidents, so we can measure Aegis against reality.

Aegis is not a dashboard. It is not an alerting system. It is a reasoning prosthesis — a machine that does the mechanical correlation work so that the human can do the thinking work. Our operators deserve to start their investigations with a hypothesis, not a blank page.

Picnic Decision Dataset:

A simple dataset.

The above toy dataset contains 14 samples of wind measurements and their respected picnic/no-picnic records. The first thing to do is to find the how much information does Picnic column have. There is 9 Evet and 5 Hayır records, total of 14 measurements. Thus, the entropy of Picnic variable is following.

Entropy of picnic.

Now, if we split the Picnic variable by the wind situation yes/none, the average reduction in the entropy would be the information gain. To do that, let's put the yes and no measurements next to each other in a row, and split it by the wind measurements.

graph TD
    Root[YYYYYYYYNNNNN] -->|Yes| Left[YYYNNNN]
    Root -->|None| Right[YYYYYYYYNN]

Split by wind.

Here, we should find the entropy in each leaf node. Then, we are going to calculate the weighted average of entropy by taking account of the E/H count in each node.

In the left most leaf node, the entropy is 1 since picking a letter results with the probability of 0.5 of Y and 0.5 of N. For the right node, we can calculate the entropy as follows.

Entropy of the right node.

Now, we can calculate the weighted average of entropy. There is 6 samples in the left, and 8 samples in the right. The following equation calculates the weighted average.

Weighted average of the entropies.

The last part is the calculating the reduction in the entropy. We just need to extract the average entropy from the first one.

Reduction in entropy.

This result tells us if we split the Picnic variable by the wind situations, there will be a 0.0561 reduction in the entropy. In another words, there will be a 0.0561 of information gain in this split.

In a decision tree, we do that for all of the independent variables. We calculate the each individual information gain for all of the independent variables. Then, we choose the feature that causes the largest reduction in entropy and split the tree into branches. We continue to the process until there is no split to make (entropy is 0) or to meet a stop condition (maximum split count or maximum depth of the tree). There are multiple ways to calculate the information gain for different purposes. Information gain is only one of them. Maybe, in another post I will write about some of them. Until then, see you in another day. Bye...

In this first story of mine, I will try my best to clear out the concepts of entropy and some other related concepts such as information gain and information gain ratio. I also want to suggest a video about entropy by Luis Serrano. You can find the video here. While this is an excellent video, it is not enough to fully understand the entropy. You need to make lots of reading sessions and maybe implement it with a programming language. Now, let's get back to the subject.

I am not a physicist but as far as I know, entropy in physics corresponds to the energy of particles in a media. If we give an example, the four phases of matter would be a good candidate.

The leftmost image (ice) has the lowest entropy since the particles are not moving as fast as the other two. The middle (water) has more energy than ice. So the entropy is higher than ice. The rightmost (water vapor) has the highest entropy since the particles move fastest.

This idea applies to the information theory in the sense of probability. Claude Shannon is the man who brings the idea in information theory. Suppose you have a computer. This computer can only output a sequence of characters. But some characters are permitted to repeat. The list of all possible characters that can present in the output is the dictionary of the computer. The question is here how much information do you have about the output of the computer?

A computer with a dictionary consists of the only letter "A" outputs a string of "AAAAAAAAAAAA".

The above computer has a dictionary \(D = \{ \text{"A"} \}\). So, we will have a guaranteed output of "A", right? This means that we know a lot about the output. We will get into how we will compute entropy, but at this point let's say that this output has an entropy value of 0. Because we are certain about the output. It will consist of nothing else but the letter "A" only.

What if our dictionary consists of more than one letter or even numbers? For example, dictionary \(D = \{ \text{"A"}, \text{"B"} \}\). Then the output of the computer might be as follows.

A computer with a dictionary consists of letter "A" and "B", outputs a string of "AAAAAABBBBBB".

Let's say from this output we need to pick a letter. Remember that each pick is independent of each other. So, what is the probability of producing an output that has the same order as in the original output sequence? The answer to this question is comes from the probability, right?

The probability of picking "A" or "B" independently is 0.5. There are 12 letters in the output. So we have multiplied 12 independent probability of picking different letters.

As you can see, the result is very small. This means that producing the same output as the computer is very difficult. We have little information about the output. Our chance to pick a letter and knowing what it is is the same as tossing a coin. This result suggests us we have a higher entropy value here.

But why do we need the entropy? We have the probability and it tells us something about the information amount of the data. Here is the reason. Actually, you can see it yourself, too. If we have fewer letters in the sequence, let's say 4 instead of 12; two for "A" and two for "B", the result will be 0.0625. But the entropy will still be high. So, what's change? You see that the amount of letters in the data affects the output of the probability exponentially. This is a problem for our computers, right? Smaller the floating point number, smaller the precision. You may be heard of the vanishing gradient problem in Recurrent Neural Networks (RNN). This is the reason for it and this is the starting point of entropy.

We need a way to not reducing the output that much but at the same time take care of all the letters in the sequence. What if we try to get rid of the exponential function (the repeated multiplication) by converting it to a linear function. How do we do that? Maybe the answer is in the summation. Because repeated summation is just a linear function, unlike multiplication. So, the conversion can be easily made by logarithms. Then let's take the logarithm of the above equation. There is only a little problem here. Which number should be used as the base of the logarithm? You can pick whatever number you want. However, we are from the area of computation. We love to express numbers and information as bits. A bit is a 1 or 0. So, a bit can be expressed in base 2. This means that we want to express the data with two symbols. Therefore, it is natural that we choose 2 as the base for the logarithm. Now, let's take the logarithm at base 2.

Logarithm base 2 of the above equation.

What a beautiful number here. But, hmm…, you see there are two little problems here. First, we have 12 letters in the sequence and the result is 12. Will this number increase as the letter count increases? Yes, it will increase. So we can normalize it with the letter count by dividing it to 12 in this case.

The result is normalized with the letter count.

This is much better. But, why the number is negative? There is no point to proceed with a negative number while trying to measure the information in a sequence. So, let's multiply the expression with -1.

The previous expression is multiplied by -1 to get rid of the redundant negative sign.

This is it. We made to the end. Almost. We still haven't converted the multiplication into the summation.

The multiplication is eliminated by taking advantage of the property of logarithms.

The first six expressions of the sequence are for the letter "A" and the remaining six is for the letter "B". Now, let's make this more obvious and distribute the multiplication over the whole expression.

The coefficient at the beginning of the whole expression is distributed. In the meantime, the summations are grouped together each for a letter (6 for A and 6 for B).

This is the entropy. For this sequence, as we change the number of letters, the probability of producing the exact same output will change. However, the entropy will never change as the proportion of letter counts are constant regardless of the total number of letters in the sequence. If we generalize the entropy, the following expression can be obtained.

The general formula for entropy.

\(P\) is the probability of picking ith distinct element from the sequence. The knowledge and entropy are opposites. If we have high knowledge about a sequence, the entropy is lower, and vice versa. Let's make another example with a different sequence.

An example sequence with a dictionary of D = {"A", "B", "C"}.

Here we have 3 distinct letters in a 12 letter sequence. Let's calculate the entropy with the above formula.

The entropy of the previous sequence.

Ok, we have calculated the entropy. But, what's the result even means? Actually, the result is the average number of yes/no questions to guess the next letter in the sequence. Be careful about that. You need to ask the questions in the smartest way possible.

Entropy is used in many ML methods. One of the popular ML methods is Decision Tree. In the decision tree, we try to find the independent variable that splits another dependent variable in the best way possible. How to find that variable? The answer is we split the target (dependent) variable with each of the other independent variables and check the reduction in entropy after each of the splits. The variable that causes the most reduction is the winner. This reduction is called information gain.

There are other types of entropy reduction metrics. For example, information gain ratio. It is a very close concept to information gain. I am planning to write about each of them later in separate stories. I hope this post is helpful. If I made a mistake in the post, please let me know. See you soon. Bye…