NASDAQ ITCH Replay Engine
NASDAQ publishes its full order flow as TotalView-ITCH 5.0, a stream of binary messages that add, execute, cancel, delete and replace orders. This engine plays a recorded day back over UDP multicast as if it were live, and a separate participant program receives it, checks the sequence numbers, and rebuilds the order book for every security. It never matches orders. It applies what the feed reports.
- Role
- Independent - design & implementation
- Stack
- C++23, CMake, GoogleTest, Linux UDP multicast, network namespaces
- Status
- Personal project · public on GitHub
How it works
An exchange feed and a participant
The replay server streams the ITCH file from disk through a fixed 512-byte buffer, so a 12.9 GB day never has to fit in memory. It sends one message per datagram behind a 10-byte big-endian header: a 64-bit sequence number and a 16-bit message count. It can send as fast as possible, or at a multiple of the original speed, using the messages’ own timestamps.
The participant runs two threads. The feed thread receives each datagram, checks its sequence number, and decodes the ITCH bytes into a fixed-size event. The book thread applies those events to the books. A lock-free single-producer, single-consumer ring of 131,072 events is the only state they share.
A script sets up two Linux network namespaces joined by a virtual Ethernet pair, so the feed can run between two isolated network stacks instead of over loopback.
A terminal book viewer listens to the same feed with the same two-thread design. It lists every security with its best bid and offer, and opens a depth ladder for any one of them, drawn with plain ANSI escapes.
The recording at the top is the viewer during a real-time replay of the first 30 seconds of the 27 March 2019 open. The replay starts at 09:30 with no resting orders, so the first spreads are wide until the book fills in.
Architecture
- 1Sequence check. Each datagram is classified as in order, a gap, a duplicate or late. Duplicates are dropped; gaps are counted, not recovered.
- 2The queue. Producer and consumer state sit on separate cache lines, and each side reads the other’s index only when the ring looks full or empty.
- 3The books. Price levels are FIFOs linked through the orders themselves, and orders come from a fixed pool, so time priority is arrival order.
- 4The oracle. book_checker runs the same parser and book code straight from the file. Identical reports mean the network and threads changed nothing.
Verification
Checking a replay with diff
A separate program, book_checker, builds the same books straight from the file, with no network and no queue, and writes the same deterministic report. A replay is correct when the two reports are identical.
Replaying the first 30 seconds of the 27 March 2019 open, 1.82 million messages, over loopback multicast gave 0 gaps, 0 duplicates, and books byte-identical to the offline build, both at full speed and in real time. Offline, book_checker rebuilds the full day, 417 million events across 8,693 securities, in 3 minutes 41 seconds on one thread with about 218 MiB of memory.
60 GoogleTest tests cover the queue, the object pool, ITCH encoding and decoding for seven message types, sequence validation, and the order book. They pass in plain, ThreadSanitizer, and AddressSanitizer with UBSan builds.
Decisions
Three choices in the hot path
A queue that rarely touches the other side’s index
Producer and consumer state sit on separate cache lines. Each side keeps its own index as a plain integer, plus a cached copy of the other side’s. It reads the other side’s atomic, with acquire, only when the cached copy says the ring is full or empty, and publishes with release. There is no read-modify-write and no sequentially consistent ordering, and all slots are usable because the counters only grow.
Orders from a pool, levels as intrusive lists
Each price level is a FIFO of orders linked through the orders themselves, so arrival order is time priority and no timestamps are compared. Orders come from a fixed-capacity pool whose free list is threaded through the unused slots, so adding or removing an order never calls the allocator for the order itself.
Reconstruct, never match
A replace is a delete plus an add, so the order goes to the back of its new level and loses time priority, as it does on the exchange. A crossed book is left crossed. The participant shows the state NASDAQ published, not a corrected one.
The tradeoff
Price levels live in std::map and order lookup in std::unordered_map, so a new order or a new price level still allocates. Gaps are detected and counted but not recovered. The publisher makes one blocking sendto per message, which holds an untuned loopback replay to between about 100,000 and 168,000 messages a second, while the offline path runs at about 1.9 million events a second.
Technical deep dive
Sequence numbers and gaps
Each datagram is classified as in order, a gap, a duplicate, or late. Duplicates are dropped so they can’t be applied twice, and processing continues across a gap.
The end of the stream is a frame with a message count of zero, sent three times 10 ms apart so a single lost datagram can’t leave the participant waiting.
Reading ITCH fields
ITCH integers are big-endian and some are odd widths, like the 6-byte timestamp. Each field is copied into a zero-initialized integer, byte-swapped, and shifted, which avoids misaligned access and packed structs.
Prices are 32-bit fixed point with four implied decimals. Both add messages become one ADD event and both execute messages one EXECUTE event, and the event type stays trivially copyable so it moves through the queue by value.
Throughput, with context
Measured on an AMD Ryzen AI 9 365 laptop, Linux, GCC 14, RelWithDebInfo, with no thread pinning or tuning. The networked range comes from five replays of the 30-second slice over loopback. The participant’s queue never filled, which suggests the publisher is the limit.
There are no per-message latency measurements yet, so the site doesn’t quote any.
Build and tooling
CMake presets build with Ninja: a development build, a test build, ThreadSanitizer on clang-18, AddressSanitizer with UBSan, and an -O3 -march=native benchmark build. GoogleTest is fetched and pinned by hash.
Technical details
The queue’s push and pop
The real code, lightly trimmed. The producer only loads the consumer’s index when its cached copy says the ring is full, and the consumer does the same when it looks empty.
template <typename... Args>
[[nodiscard]] bool try_emplace(Args&&... args)
{
auto const write = producer_.next_;
if (write - producer_.cached_tail_ >= capacity_)
{
producer_.cached_tail_ =
consumer_.published_.load(std::memory_order_acquire);
if (write - producer_.cached_tail_ >= capacity_) {
return false;
}
}
alloc_traits::construct(allocator_,
storage_ + (write & mask_),
std::forward<Args>(args)...);
++producer_.next_;
producer_.published_.store(producer_.next_, std::memory_order_release);
return true;
}
bool try_pop(reference val)
{
auto const read = consumer_.next_;
if (read == consumer_.cached_head_)
{
consumer_.cached_head_ =
producer_.published_.load(std::memory_order_acquire);
if (read == consumer_.cached_head_) {
return false;
}
}
auto* const element = storage_ + (read & mask_);
val = std::move(*element);
alloc_traits::destroy(allocator_, element);
++consumer_.next_;
consumer_.published_.store(consumer_.next_,
std::memory_order_release);
return true;
}