I expected a Hidden Markov Model to find hidden market regimes I couldn’t see by hand. It found one thing very clearly — volatility — which was not the thing I asked for, and in reaching for it, it threw away the only real signal I had. This is a post about a model that confidently learned the wrong lesson.
This is a companion to my post on order-flow imbalance, where I reproduced a famous microstructure paper and then found it couldn’t be traded at a profit. The HMM was supposed to be the clever part — the thing that turned a weak signal into a real one. It’s the most interesting result in the whole project, so it gets its own writeup: what an HMM is, how I used it, what I expected, and what actually came out.
Why reach for an HMM at all
A Hidden Markov Model assumes the thing you observe (order flow, returns) is generated by a small number of hidden states you can’t directly see, and that the system hops between those states over time according to fixed probabilities. You never observe the state; you infer it from the observations. The appeal for trading is obvious and seductive: markets feel like they’re in regimes — trending, choppy, calm, panicked — and if a model could tell you which regime you’re in right now, you could act differently in each.
The real power of an HMM isn’t on one variable, either. It’s supposed to find latent structure across many correlated features at once — to notice that when order flow, volatility, spread, and book imbalance line up a certain way, you’re in a distinct state with distinct behavior. That was the thesis: feed it a rich set of microstructure features and let it discover regimes I’d never find by eyeballing.
How I used it
The setup, concretely:
- States: three (later four), each emitting its observations from its own Gaussian.
- Training: fit with Baum-Welch (the EM algorithm for HMMs). This is unsupervised — the model discovers states purely from the shape of the data, with no labels, no targets.
- Naming the states: the model only knows “state 0, 1, 2” — statistically meaningful, semantically empty. So after fitting, I assign each training bar to its most-likely state and look at the average forward price move that followed bars in each state. Highest average → “UP,” lowest → “DOWN,” middle → “FLAT.” That same average doubles as the predicted magnitude.
- Split by time, never shuffled. First 70% of the timeline trains, last 30% tests. Shuffling would leak the future into the past.
flowchart LR
X["OFI features<br/>(one or many)"] -->|Baum-Welch / EM| S["3 hidden states<br/>(unlabelled)"]
S -->|avg forward move<br/>on TRAIN| N["UP / FLAT / DOWN<br/>+ magnitude + duration"]
N -->|causal filter<br/>on TEST| P["per-bar prediction"]
P --> V["validate:<br/>hit-rate vs 50%"]
The part almost everyone gets wrong: don’t leak the future
This is the single most important line of code in the project, and it’s a trap I’ve watched smart people fall into.
The naive way to get state probabilities on your test set is model.predict_proba(X_test) over the whole block. It looks innocent. It is lookahead. Under the hood that runs the forward-backward algorithm, which conditions each bar’s state on the entire sequence — including bars in the future. So the “prediction” for the state at 10:00:00 secretly depends on what happened at 10:00:05. You never put the future price into the model’s inputs, and it still leaks in indirectly, through the state inference. Live, you could never reproduce it, and your beautiful backtest is a fiction.
The honest version uses the filtering distribution: at each bar i, infer the state from observations 0…i only — the past and present, nothing else. That’s the forward algorithm, stopped at i, taking the last value. I wrote it two ways: a dead-simple O(n²) version that re-runs on each growing prefix, and a fast O(n) scaled-forward filter that carries one vector forward a bar at a time. Then — and this is the point — every single run asserts the two agree to numerical precision on the first few hundred bars. If the fast path ever drifts from the honest-but-slow path, the run aborts and I don’t trust the output. Equivalence checked, not assumed.
If you take one thing from this post and you’re building HMMs on time series: your test-time inference must be causal, and you should prove it is.
The data
Identical to the order-flow project: Alpaca SIP tick quotes on SOXL (a 3× leveraged semiconductor ETF), order-flow imbalance computed on the event stream and summed into one-second bars, regular hours only, each day independent. The single-feature runs used a 15-day, 319k-bar window (223k training bars); the multi-feature runs used the full four-year set. The forward target — the move over the next 10 bars — is the answer key: it names states and scores predictions, and it never enters the model’s input.
What I expected
Sticky, directional regimes. An “UP” state the market would sit in for a while, that actually preceded up moves; a “DOWN” state that preceded down moves; and — the real hope — that adding volatility, spread, and book-imbalance features would let the HMM carve out regimes a one-line rule couldn’t, and lift the weak order-flow signal into something tradeable.
Result 1 — one feature: a fancy way to smooth the sign of order flow
Three states on one-bar OFI, scored at the 10-second horizon: 50.0% directional hit-rate. A coin flip. Look at what the states actually were:
| State | OFI it emits | avg forward move | label | stays put (p) | lasts |
|---|---|---|---|---|---|
| high OFI | +85.8 | +0.0173 | UP | 0.46 | ~2 bars |
| near-zero OFI | +0.5 | +0.0028 | DOWN | 0.67 | ~3 bars |
| negative OFI | −10.6 | +0.0067 | FLAT | 0.67 | ~3 bars |
Two things jump out. First, every state’s forward move is positive — the state I labelled “DOWN” was followed by +0.0028 on average. There is no down state. “DOWN” just means “went up the least.” The model cannot tell you the price is about to fall because, in this data, it essentially never learned a falling regime; it learned “a lot up,” “a little up,” and “medium up.” Second, the states “last” about two to three seconds. A regime that survives three seconds isn’t really a regime; it’s just short-lived noise. With a single feature, a three-state HMM is barely more than three bins of OFI magnitude plus a little persistence. It’s really just a smoothed version of sign(OFI).
Result 2 — many features: it learned volatility, and broke the signal
So I gave it the rich feature set — trailing 60-second OFI (the actual directional signal), 60-second realized volatility, and book imbalance — expecting the multi-feature magic. Here’s what the model built (features shown as z-scores):
| State | trail-60 (flow) | rvol-60 (vol) | forward move | stays put | lasts |
|---|---|---|---|---|---|
| 0 | +0.04 | +0.91 | +0.000094 | 0.006 | ~1 bar |
| 1 | +0.04 | +0.90 | +0.000095 | 0.024 | ~1 bar |
| 2 | −0.03 | −0.56 | +0.000019 | 0.997 | ~320 bars |
Read the columns. The flow feature — the thing I actually wanted it to use — is essentially zero in every state (±0.04). The feature doing all the separating is volatility: two “loud” states and one “calm” state. Given a direction signal and a volatility signal side by side, the HMM went straight for volatility and ignored direction entirely.
It gets worse. States 0 and 1 are nearly identical and flip-flop every single bar (they “stay put” 0.6% and 2.4% of the time) — that’s the EM algorithm using two Gaussians to fit the fat tails of the high-volatility cluster, then presenting the alternation between them as if it were real dynamics. Meanwhile the labels were assigned by ranking forward moves of +0.000094 vs +0.000095 vs +0.000019 — differences far below noise, so out-of-sample the “UP” and “DOWN” labels meant nothing and their ordering inverted.
The number that says it all: on the exact same test bars, the raw trailing-OFI feature had a rank correlation with the forward return of +0.106. The HMM’s posterior-weighted prediction, built from that same feature, had a correlation of −0.003. The model took a genuinely predictive continuous variable, compressed it into three volatility bins, and destroyed the directional information in the process. A four-state version did no better — 52% across all bars, 55% in its best state, still losing to a one-line selective rule that just trades the extreme tail of the raw signal.
Why it happened
An unsupervised model’s only job is to describe the shape of the data — to place its states where the variance is. And in intraday microstructure, the overwhelming majority of the variance is volatility: it’s huge, it’s persistent, it’s visually obvious in the distribution. The directional signal is real but tiny — a 0.1 correlation explains about 1% of the variance of future returns. So the HMM spends all of its states carving up the 99% and never even notices the 1%. It’s not being dumb; it’s doing exactly what it was built to do. I asked it to describe the data and it described the loudest thing in the data, which wasn’t the thing I could trade.
The verdict
The HMM told me the same thing twice, in two different ways: the dominant structure in this data is loud versus calm. That’s a real and useful fact — it’s a decent volatility-regime detector. It is a terrible direction caller, and worse, when handed a small directional edge it overwrites it with volatility. The saved model file ended up as the one artifact I deliberately left behind when I moved the project on — the disproven part.
The broader lesson matched everything else in this research: model complexity doesn’t earn its keep when the constraint is information, not extraction. The signal was weak and the cost of trading it was high; no amount of latent-state machinery changes either of those. If I wanted the HMM to do a real job, it wouldn’t be calling direction — it would be a gate, flagging the high-volatility regime where the raw flow signal is strongest, and letting a simple rule do the trading. The right role for the fancy model was to flag the volatility regime, not to pick the trade. Figuring out which of your tools is actually load-bearing — and being willing to demote the impressive one to a supporting part — turned out to be most of the skill.