Building a Trading Bot That Survives Its Own Failures

A trading bot is a distributed system that runs unattended against a live brokerage. A crash, a message delivered twice, or stale local state about what you hold each has a direct financial cost.

I built an automated execution bot that traded SOXL — a 3× leveraged semiconductor ETF that moves 5–10% on a normal day and 30% on a high-volatility day — against a live brokerage account, unattended, all day. Every bug in that setting has a financial cost, so I designed the bot around five reliability guarantees and spent most of the project verifying that they hold under failure: crashes, duplicate messages, network gaps, and disagreement between the broker’s record and the bot’s local state.

This post covers that reliability work. Whether the strategy had an edge is a separate question, measured in another post; it did not. The executor had to be correct regardless of what it traded.

The guarantees

I set out to make five things true no matter what failed:

1. Never double a position. A message delivered twice must not become two trades.
2. Never trust my own memory of what I hold. The only source of truth is the broker.
3. Never place an unbounded-cost order. The worst possible fill has to be a number I chose in advance.
4. Always keep a protective exit — even if the server dies.
5. Know within minutes if it has silently broken.

Everything below exists to enforce one of those five.

Architecture

flowchart TB
    TV["TradingView<br/>strategy fires an alert"]
    TV -->|webhook| NG[nginx]
    NG --> LIVE["Flask · LIVE"]
    NG --> PAPER["Flask · PAPER<br/>mirror"]
    NG --> TEST["Flask · TEST<br/>canary"]
    LIVE -->|marketable-limit orders| AL[("Alpaca<br/>broker = source of truth")]
    PAPER --> AL
    TEST -->|"tiny round-trip every 30 min"| AL
    AL -.->|"position state, checked on every decision"| LIVE
    LIVE -.->|"fills · errors · /panic"| TG([Telegram])
    TEST -.->|"buzzes my phone if the round-trip doesn't complete"| TG

Three isolated stacks — live, paper, and a test canary — behind nginx, each reconciling against the broker as the single source of truth and reporting to Telegram.

Guarantees 1 & 2 — idempotency and a single source of truth

The bot is driven by webhook alerts, and webhooks are unreliable by construction — they arrive late, twice, or not at all. Two design decisions handle that:

Answer immediately, trade in the background. A webhook sender that doesn’t get a fast 200 assumes failure and retries — so if the server did the trading before responding, one intent could become two or three orders. Instead it acknowledges the request instantly and does the actual work asynchronously.

The broker is the only source of truth; local state is a cache. Before the bot acts, it asks the broker what it actually holds — the one record that survives a process crash — and recognizes a duplicate alert instead of acting on it. If in-process memory is treated as authoritative about external state, a restart or a dropped message corrupts it.

Guarantee 3 — bounded-cost orders

On a 3× fund, a market order pays whatever the book asks, and during a large price move that cost can be substantial. The bot therefore never sends a market order. Every order is a limit placed just past the current price; if it doesn’t fill, the bot cancels and re-places it more aggressively in stages — 0.025% → 0.1% → 0.25% → 0.5% → 1% — and that last number is a hard ceiling. The worst price I can pay is 1% past where I aimed, because the order types make anything worse impossible.

Guarantee 4 — a protective exit that survives the server

Every position is guarded by a trailing stop, and the stop adapts to the clock so the protection never depends on my server being alive at the wrong moment:

• During regular hours, it rests a native stop order on the exchange. If my server dies, the exchange still exits the position.
• In extended hours, where the broker’s native stops don’t trigger, it switches to a hidden software trail that tracks the high-water mark internally and fires its own exit.

It ratchets up, never down.

Guarantee 5 — know within minutes if it’s broken

The failure mode this guards against is silent: the process is running, the logs show nothing unusual, and no work is being done. The system therefore tests itself continuously:

• A canary instance placing real trades. A third instance places and closes a small real round-trip every 30 minutes, all day. If that round-trip stops completing, an alert goes to my phone. It is an end-to-end health check covering the whole path — webhook, order placement, fill, reconciliation — using real money, because a mock cannot detect that the real broker has started rejecting orders.
• A dead-man heartbeat detects a process that is still running but has stopped doing work.
• Every fill and error lands in a Telegram channel labeled by environment, and a /panic command sells everything from my phone.
• Margin stays disabled unless I deliberately flip a flag.
• Every deploy is a git pull into a symlink and nothing else — one atomic, reversible operation.
• 54 unit tests against a fake broker. The real broker’s sandbox never produces the partial fills, rejections, and race conditions that matter. I wrote a fake broker that returns those conditions on demand and tested against it.

Correctness under partial failure

The guarantees above came from bugs. Each one happened in the gaps between components:

• Canceling open orders before checking the position. The entry code canceled all open orders before checking whether it already held a position. So a duplicate alert was correctly rejected — but not before it had already deleted the stop protecting my open position.
• The stop order blocked my emergency sell. My emergency sell-everything got refused: the protective stop resting on the exchange had reserved all my shares, so the broker rejected the market-wide sell.
• End-of-day close logic scheduled when SOXL doesn’t trade. My end-of-day close logic was scheduled for 8pm–4am. But SOXL doesn’t trade then, and the alerting engine only runs when a new bar arrives — so my safety rail was code that could never execute. I found out by holding an overnight position I should never have had.
• The fill at a price the chart never showed. I got filled at a price the chart said never occurred. Charts show the last trade; your buy pays the ask; at 1am the gap between them was over 1%. That one fill is why the system now runs a live order-book dashboard that tracks which venue is currently trading — because a quote from a venue that isn’t active is stale.

The dashboard I built after that bug: a live tape that timestamps and tags every print and tracks which venue is actually trading. Account figures blacked out — the prices and the tape are real.

How I knew it worked

Three things gave me confidence:

• The canary exercised the whole path with real money every 30 minutes. It ran against the live system, not a mock of it.
• The fake broker let me force the failures that matter — partial fills, rejections, duplicate messages — and assert the guarantees held.
• The simulator and the live results agreed. I rebuilt the strategy as a Python simulator with costs that weren’t guesses (measured alert latency, real execution costs from my own fills, observed spreads) and checked it against the broker’s own behavior until they matched — 277 simulated trades versus 279 real ones.

What lasted

The direction-predicting strategy is finished. It had no edge once trading costs were accounted for, which I documented separately. The machinery around it is still running: the idempotent webhook layer, the reprice ladder, the clock-aware failover stop, the broker reconciliation, the canary, and the order-book tape. Together they form the trade executor that remains in use.

At short timescales, trading is an execution-and-reliability problem, not a prediction problem. The transferable results were about running an unattended system correctly against an unreliable network and an external source of truth: idempotency, reconciliation, bounded failure, self-monitoring, and testing the failure modes rather than the success path.

---

Stack: Python, Flask, nginx, systemd, Alpaca API, TradingView Pine. Reliability: three isolated environments, a synthetic canary trade every 30 minutes, a dead-man heartbeat, /panic kill switch, margin off by default, atomic symlink deploys, 54 unit tests against a fault-injecting fake broker. Verification: a cost-accurate simulator reconciled against live broker behavior (277 vs 279 trades).