dxFeed Order Book Reconstruction

Introduction

dxFeed market data feeds (real-time, delayed or historical) allow clients to reconstruct order books, price level aggregations, and aggregations by Market Maker or a data provider. In this document, we describe an order book reconstruction. For reconstruction, you can:

  1. Use OrderBookModel class in dxFeed API.

  2. Check out Order event attributes and implement the reconstruction algorithm.

We recommend using dxFeed API, because it correctly implements a reconstruction protocol.

dxFeed API

To get the best and smoothest solution for the order book reconstruction, we recommend using our dxFeed Java API. It provides an OrderBookModel class which builds a high-quality order book from a set of orders. This class keeps track of transactions and snapshots, filters data, etc. Learn more about events and classes in dxFeed API.

Reconstructing an order book without the OrderBookModel class of dxFeed API

You can reconstruct an order book without using the OrderBookModel class in dxFeed API. For this, do the following:

  1. Learn about Indexes and EventFlags fields to know how to track snapshots, updates and transactions.

  2. Follow the algorithm.

Indexes

An order book consists of an undefined number of orders. Each exchange specifies ID for each order. Since each exchange uses different IDs, we assign our indexes to each order (we put orders in special slots that have indexes assigned).

The index of an order distinguishes different entries in the order book.

Info

An order may move to another index, so this identifier is not permanent. Every event can be a new order, an order modification, or an order cancellation.

Index reusing

Every forthcoming Order event can reuse the index after its clearing. Index reusing is working the following way:

  1. Receive a snapshot of all levels. Snapshots start with the SNAPSHOT_BEGIN (EventFlag.SnapshotBegin = 0x04) flag and end with the SNAPSHOT_END (EventFlag.SnapshotEnd = 0x08) flag.

  2. Receive updates in form of Order events. Each event has an Index value for referencing some order's address.

  3. Add a new order if the received Index is unique.

  4. Receive the REMOVE_EVENT (EventFlag.RemoveEvent = 0x02) flag if an order is cancelled. It has a single meaningful identifier inside – the Index value. Ignore other fields such as Size in this case.

Modification and partial execution have similar operations:

  1. Receive the REMOVE_EVENT flag.

  2. Receive a new Order event.

Note

You may also conflate these two actions into a single Order event with an updated Size.

There should be no REMOVE_EVENT with an unknown Index, but in case that happens, such an event should be ignored.

Regional events

Since every regional event indicates exchange (non-zero source ID or exchange code in QD-model), you need to reconstruct an order book for each source ID separately. For example, you need to create two order books, if you have both order#bzx and order#byx events in your market data feed.

Event flags applicable to Order event

Event flags define the borders of snapshots and transactions retransmissions:

  • TX_PENDING indicates a pending transaction update. When TX_PENDING is true, it means that an ongoing transaction update, that spans multiple events, is in process

  • REMOVE_EVENT indicates that the event with the corresponding index has to be removed

  • SNAPSHOT_BEGIN indicates when the loading of a snapshot starts. Snapshot load starts with a new subscription and the first indexed event that arrives for each non-zero source ID on a new subscription may have SNAPSHOT_BEGIN set to true. It means that an ongoing snapshot consisting of multiple events is incoming

  • SNAPSHOT_END indicates the end of a snapshot. It indicates that the data source has sent all the data about subscription for the corresponding indexed event

Snapshots

Order event stream consists of snapshots and updates for these snapshots. Snapshots start with the SNAPSHOT_BEGIN flag and end with the SNAPSHOT_END flag.

Info

Due to possible reconnections and retransmissions, the snapshots can overlap each other, so in the event stream the flags can intersect. It is possible to find SNAPSHOT_END before SNAPSHOT_BEGIN, or it is possible to find SNAPSHOT_BEGIN after SNAPSHOT_BEGIN, and so on. Therefore, the reconstruction algorithm should extract a complete snapshot ignoring the wrong residues of other snapshots.

If you catch the SNAPSHOT_BEGIN flag, you have to dismiss the previous order book.

Transaction model

Updates of order books happen as a sequence of transactions. If a transaction consists of several events, then all these events except the last one has a TX_PENDING flag. If an event does not have a TX_PENDING flag, it means that this event finishes the current transaction or, if there is no current transaction, then this event is operated as a single-event transaction.

Snapshots can overlap with order book modifications and transactions. While the snapshot is being transmitted, updates of the order book can occur. There are two consequences:

  1. The snapshot can have modifications of the order book.

  2. A snapshot update during the transfer can create a transaction until the end of the snapshot (we will add the TX_PENDING flag until the end of the snapshot). It means that it will create an artificial transaction until the end of this snapshot.

Examples

One transaction:

Order#NTV,AAPL,20180926-100107.858-0400,0,1219,20180926-100107-0400,854:1,223.31,17,1307,NSDQ,EventFlags=TX_PENDING
Order#NTV,AAPL,20180926-100107.858-0400,0,1623,20180926-100107-0400,854:2,223.29,59,1303,NSDQ

Single-event transaction:

Order#NTV,AAPL,20180926-100107.826-0400,0,723,20180926-100107-0400,822:0,223.07,100,1303,NSDQ

Two single-event transactions:

Order#NTV,AAPL,20180926-100107.441-0400,0,2449,20180926-100107-0400,437:0,223.3,100,1307,NSDQ
Order#NTV,AAPL,20180926-100107.441-0400,0,525,20180926-100107-0400,437:1,223.26,100,1303,NSDQ

Algorithm

The proper order book reconstruction algorithm implies the use of a pending queue, which accumulates events and is applied transactionally to the reconstructed integral model of an order book. Order book reconstruction algorithm tracks incoming event flags and the current accumulation status of the pending queue to determine the next action.

Info

In the case of non-zero source IDs, you need to reconstruct an order book for each source id separately. For example, if you have both order#bzx and order#byx events in your market data feed, you need to distinguish these events and create separate pending queues and consistent order books.

The algorithm is as follows:

  1. Use a pending queue.

  2. Order a consistent order book that we reconstruct.

  3. Create isSnapshot, txPending, hasSnapshot flags to control their snapshots.

  4. If the event has SNAPSHOT_BEGIN flag, then clear the pending queue, set isSnapshot = true, hasSnapshot = false.

  5. If the event has SNAPSHOT_END flag and isSnapshot is true, then set isSnapshot = false and hasSnapshot = true.

  6. If the event has TX_PENDING, then set txPending = true.

  7. If the event does not have TX_PENDING, then set txPending = false.

  8. Add the event to the pending queue.

  9. If isSnapshot is false and txPending is false, then apply (see below) pending queue to the consistent order book:

    • If hasSnapshot is true, then clear (remove all orders) the consistent order book and set hasSnapshot = false

    • Go through the pending queue and apply each order to the consistent order book:

      • If this order has REMOVE_EVENT flag, then remove the event with this index from the consistent order book

      • Else add or update events with this index to the order book

    • Clear pending queue

  10. Wait for the next Order event.

This algorithm also works in case you need to reconstruct price levels or aggregations by Market Maker.

Note

If an event has more than one flag, then they are separated by a comma. For example, EventFlags=SNAPSHOT_BEGIN,TX_PENDING.

How to order incoming order book?

We send snapshots at the beginning of your subscription. After that, you start to get updates - a sequence of transactions or single-event orders. The order book sequence keeps an order of simultaneously coming trades.

Note

You can receive snapshots again in case of reconnection.

The following example illustrates the result for requesting the order book:

PS C:\workspace\qds> java -jar qds-tools.jar connect mddqa.in.devexperts.com:7400 Order#BATE BREm:BATE -c history
I 200921 141633.645 [main] QD - Using QDS-3.291-20200619-1026, (C) Devexperts
I 200921 141633.667 [main] QD - Using address mddqa.in.devexperts.com:7400
I 200921 141633.724 [main] QD - Using scheme com.dxfeed.api.impl.DXFeedScheme
I 200921 141634.082 [main] MARS - Started time synchronization tracker using multicast 239.192.51.45:5145 with ZXuXj
I 200921 141634.097 [main] MARS - Started JVM self-monitoring
I 200921 141634.105 [main] QD - connect with collectors [History]
I 200921 141634.191 [main] ClientSocket-Distributor - Starting ClientSocketConnector to mddqa.in.devexperts.com:7400
I 200921 141634.195 [mddqa.in.devexperts.com:7400-Reader] ClientSocketConnector - Resolving IPs for mddqa.in.devexperts.com
I 200921 141634.201 [mddqa.in.devexperts.com:7400-Reader] ClientSocketConnector - Connecting to 89.113.131.96:7400
I 200921 141634.233 [mddqa.in.devexperts.com:7400-Reader] ClientSocketConnector - Connected to 89.113.131.96:7400
D 200921 141634.262 [mddqa.in.devexperts.com:7400-Reader] QD - Distributor received protocol descriptor multiplexor@IYrNN [type=qtp, version=QDS-3.291, opt=hs, mars.root=mdd.qa-local.multiplexor-preview, filter=![&]*&nwc] sending [TICKER, STREAM, HISTORY, DATA] from 89.113.131.96
==HISTORY_DATA
=Order#BATE EventSymbol Index Time Sequence Price Size Flags MarketMaker
Order#BATE BREm:BATE 21 0 0 NaN 0 3 \NULL EventFlags=SNAPSHOT_BEGIN
Order#BATE BREm:BATE 20 0 0 NaN 0 3 \NULL
Order#BATE BREm:BATE 19 20200921-141550+0300 114:2 7.935 120 1063 \NULL
Order#BATE BREm:BATE 18 20200921-141415+0300 286:0 8 1313 1067 \NULL
Order#BATE BREm:BATE 17 20200921-140351+0300 155:1 8.04 300 1067 \NULL
Order#BATE BREm:BATE 16 20200921-141415+0300 75:0 8 377 1067 \NULL
Order#BATE BREm:BATE 15 20200921-141513+0300 615:0 7.935 377 1063 \NULL
Order#BATE BREm:BATE 14 0 0 NaN 0 3 \NULL
Order#BATE BREm:BATE 13 20200921-140349+0300 846:0 8.045 241 1067 \NULL
Order#BATE BREm:BATE 12 20200921-141201+0300 926:0 7.91 241 1063 \NULL
Order#BATE BREm:BATE 11 20200921-141415+0300 247:0 7.93 1325 1063 \NULL
Order#BATE BREm:BATE 10 20200921-141627+0300 954:1 7.94 240 1063 \NULL
Order#BATE BREm:BATE 9 20200921-141629+0300 714:0 7.995 198 1067 \NULL
Order#BATE BREm:BATE 8 20200921-140008+0300 478:0 7.915 291 1063 \NULL
Order#BATE BREm:BATE 7 20200921-141627+0300 954:0 7.94 191 1063 \NULL
Order#BATE BREm:BATE 6 20200921-141425+0300 646:0 8.1 2000 1067 \NULL
Order#BATE BREm:BATE 5 20200921-141415+0300 140:0 8.04 2000 1067 \NULL
Order#BATE BREm:BATE 4 20200921-140335+0300 642:0 7.825 333 1063 \NULL
Order#BATE BREm:BATE 3 20200921-141202+0300 596:0 8.12 321 1067 \NULL
Order#BATE BREm:BATE 2 20200921-141417+0300 286:0 7.835 2000 1063 \NULL
Order#BATE BREm:BATE 1 20200921-141512+0300 363:0 7.89 2000 1063 \NULL
Order#BATE BREm:BATE 0 0 0 NaN 0 0 \NULL EventFlags=REMOVE_EVENT,SNAPSHOT_END
Order#BATE BREm:BATE 7 0 0 NaN 0 3 \NULL
Order#BATE BREm:BATE 10 0 0 NaN 0 3 \NULL
Order#BATE BREm:BATE 7 20200921-141638+0300 514:0 7.94 191 1063 \NULL
Order#BATE BREm:BATE 10 20200921-141638+0300 514:1 7.94 300 1063 \NULL
Order#BATE BREm:BATE 9 0 0 NaN 0 3 \NULL
Order#BATE BREm:BATE 9 20200921-141640+0300 714:0 7.995 198 1067 \NULL
Order#BATE BREm:BATE 7 0 0 NaN 0 3 \NULL
Order#BATE BREm:BATE 10 0 0 NaN 0 3 \NULL
Order#BATE BREm:BATE 9 0 0 NaN 0 3 \NULL
Order#BATE BREm:BATE 7 20200921-141647+0300 754:0 7.94 191 1063 \NULL
Order#BATE BREm:BATE 9 20200921-141647+0300 754:1 7.94 330 1063 \NULL
Order#BATE BREm:BATE 10 20200921-141648+0300 194:0 7.995 198 1067 \NULL

To receive the order book for the required time field:

1. Take the earliest snapshot.

2. Update all orders sequentally until the required time field.

Note

The whole transaction may be excluded if the required time field is in the middle of the TX_PENDING transaction.