Part 1 built a single capture stack: one tickerplant, one writer, one reader, with each instrument written into its own directory and served back through virtual tables. There is no RDB, and end of day does not move data between processes.

In Part 2, we run several capture stacks in parallel. Each stack writes to its own database root, while the readers serve all of the roots as a single table. This allows capture capacity to scale by adding stacks rather than putting more load through a single stack.

Why More Than One Stack

A standard tick capture puts one tickerplant in the middle of the capture path. Feeds publish into it and subscribers read from it, so all of the day’s traffic goes through the same process. The tickerplant is fast, but it is still a single process. Once it becomes the bottleneck, adding more capacity to the rest of the stack does not solve the problem.

Running several tickerplants and splitting the feeds between them removes that bottleneck, but introduces a different problem. Anything that needs the full universe now has to read from several independent streams and present them as one table. An RDB is not a good fit for this because it keeps the current day in memory. Combining several streams means merging that data in memory, and at end of day the RDB then has to produce one consistent view from those separate sources.

This pack does not use an RDB. Each writer writes every instrument to disk in its own directory, and readers expose those directories as virtual tables. A reader can attach to all of the roots and query across them directly, without first merging the streams into one in-memory dataset.

This lets capture capacity grow by adding stacks. Each stack has its own tickerplant, feed and writer, so the individual tickerplants do not have to handle the traffic from the other feeds. The WDBs can therefore scale out by adding more stacks.

Several Stacks, Each With Its Own Root

Each stack writes to its own database root. Readers are separate from the stacks and attach to every root, so any reader can answer queries across the whole estate.

There are two settings that define the shape of a deployment. VTSTACKS controls the number of capture stacks and VTIDBS controls the number of readers:

VTSTACKS=2 ./deploy/bin/torq.sh start all            # two stacks, one reader
VTSTACKS=3 VTIDBS=2 ./deploy/bin/torq.sh start all   # three stacks, two readers

The rest of the configuration is derived from these values. Each stack gets its own root, enumeration domain and tickerplant, while each reader gets the full list of roots. The process file is generated at startup, so there is no separate process configuration to maintain. Per-writer values are passed as launch parameters as well.

With multiple writers, a few parts of the single-writer setup no longer work as they did before. The processes continue running and the logs can still look normal, so the problems only show up when the resulting data is checked.

Where Multiple Writers Cause Problems

These problems are not caused by giving each writer its own root. They come from having more than one writer.

1. One Stack Rolls While Another Is Still Writing

The reader keeps a current partition as its live boundary. Dates before that are treated as immutable and are not rescanned, which keeps catalogue rebuilds cheap as history grows. With one writer, that works because there is only one process that can still add directories to the current date.

With several writers, one writer rolling to the next day does not mean the others have finished with the old one. The reader has a single boundary across all the roots it attaches. If wdb1 rolls first, it can move current forward while wdb2 is still writing to the previous date. Any directories wdb2 creates after that fall outside the range the reader considers mutable, so they are not discovered.

The boundary therefore needs to come from all writers, rather than just the newest date on disk. current stays at the earliest partition that any writer still has open, and only moves forward once all writers have moved past it.

2. A Reader Is Registered With Only One Writer

Existing directories do not need to be announced because their paths are resolved when a query runs. The notification is only needed when a new directory appears.

A writer notifies every reader it knows about. With one stack, that covers the whole estate, regardless of how many readers are running. The single writer can notify all of them.

With several stacks, each writer only knows about the readers it is registered with. A reader finds its writer by process type and takes the first result, so it registers with one writer and the others are not notified. If another writer creates a new directory, that reader will only find it during the periodic sweep, which can take up to 30 seconds.

Every writer is now given the full list of readers in the generated process file, allowing each one to notify all readers. A new directory is then picked up by every reader within about two seconds.

KDB-X Community Edition Licence Limits

A reader connects to every writer and to discovery, so its connection count grows as stacks are added. On licences with a concurrent connection limit, this puts a practical cap on the number of stacks a reader can support.

Once the limit is exceeded, the reader continues capturing but refuses client connections with ‘conn. This can make the reader look like it is down even though the process itself is still running.

On the KDB-X community licence, the connection limit is 16, which makes four stacks the practical limit. The startup script reads the limit from the licence and warns when the selected topology would exceed it. If the licence reports 0W, no warning is shown and the topology is allowed to grow until another resource, such as disk capacity, becomes the limiting factor.

Changing The Shape Of A Running Estate

Changing the number of stacks changes how instruments are divided between them. Existing data should therefore be cleared before changing VTSTACKS, otherwise the same instrument can end up in multiple roots or data can be left in a root that is no longer part of the estate.

Stop the estate, remove the database and tickerplant logs, then start it again with the new shape:

./deploy/bin/torq.sh stop all
rm -rf deploy/data/db* deploy/data/tplogs
VTSTACKS=3 VTIDBS=2 ./deploy/bin/torq.sh start all

Keeping the same stack count does not require the data to be cleared. Each writer replays its own tickerplant log when it starts again.

Readers are different. A new reader can be added without restarting the existing stacks because it starts with the current root list and attaches to all of them:

VTIDBS=3 ./deploy/bin/torq.sh start all    # starts idb3 alone, existing stacks unchanged

What A Root Per Writer Costs

Giving every writer its own root means giving every root its own enumeration domain. Symbol values are therefore not shared between roots.

This shows up when a query groups by a symbol column across multiple roots. Each enumeration domain is treated separately, so the same symbol can appear as more than one group:

select rows:count i by side from trade           / 4 groups - one per domain
side| rows
----| ----
buy | 101
sell| 96
buy | 150
sell| 122

select rows:count i by value side from trade     / 2 - correct
side| rows
----| ----
buy | 251
sell| 218

Filtering and grouping on the partition column continue to work as before.

This can be mitigated by using a single enumeration file. The writers can either point .Q.ens to a shared location or use one physical file symlinked into each root. The symbol values then remain consistent across roots and the grouping behaves as expected.

The trade-off is that the stacks now depend on the same enumeration file. They need to share a filesystem, and a problem with that file affects all of them. This is still manageable and is the same arrangement used by a standard partitioned database, where all tables use one sym file.

The other cost is on the reader’s catalogue. Each root adds another directory tree to scan, so building the catalogue does one more pass per date and per table. The extra scan happens when a date is first seen. The reader then reuses the dates it already holds and only rescans the live one.

Why Not One Shared Root

Putting all stacks into one tree would be simpler in terms of directory layout, but it also puts the writers back into competition for the same storage.

The first issue is IO. Several capture points sharing one root also share the same IO resources. Splitting the roots means adding another stack can add another independent path for capture rather than making all writers compete for the same tree.

The other issue is recovery. When a writer restarts, the normal recovery path deletes the partition it is about to rebuild and then replays its own tickerplant log. With one writer owning the root, that is safe and repeatable. With multiple writers sharing the same root, the partition also contains data belonging to the other writers. One writer can therefore delete data that it cannot restore from its own log. Preventing this on a shared root means limiting each writer’s delete to the directories it owns. That requires keeping track of which directories belong to which writer.

Takeaway

The underlying capture model stays the same. Instruments are still stored as directories, readers still serve them through virtual tables, and end of day does not move data between processes.

The main change is that each writer now has its own root and tickerplant, which removes the shared-root recovery problem and lets capture capacity grow by adding stacks. The trade-off is a separate enumeration domain for each root, although this can be avoided by sharing a single enumeration file. The reader also has an additional directory tree to scan when rebuilding its catalogue.

The default is still one stack and one root.