TorQ and KDB-X Community Edition

The arrival of KX’s KDB-X Community Edition license certainly opens up new possibilities for both new and experienced kdb+ users. It provides a free option for commercial projects, supports open data formats and includes a module framework that promises to expedite the journey to that first production system (no more writing log/timer/tz.q libraries).

However, the community edition embeds resource limits, and developers must take them into account when building a new application or migrating from kdb+. The documentation describes these limits, and you can also check them in the REPL using a new built-in function:

The release notes also show the error messages that each limit triggers. All four limits affect a system differently, but when we ensured that the TorQ Finance Starter Pack (FSP) complied with the license, only the conns limit caused issues—that is, the number of simultaneous connections a process may open.

Here we describe the changes we made in both the core TorQ code and the Finance Starter Pack, as these may prove instructive for the wider TorQ community (note: all TorQ processes mentioned in this blog are documented here).

Discovery Process Registration

The Discovery Service was our first port of call. Each process uses Discovery to register their own availability, find other processes and subscribe to receive updates for new process availability. The connection limit renders this service useless, so instead we configured every component to use the static process.csv file as their source of information about other services.

We set the following variables to false if a connection limit existed in the process:

The conditional check reverts TorQ to using Discovery if no limit exists (either in kdb+ or fully-licensed KDB-X), which is preferred for resiliency.
It’s also worth mentioning that the core gateway init was not using these variables, so a small code change was required there.

Tidy Up Connections

Before we deactivated any other processes, we removed unnecessary connections:

• A duplicate entry in .servers.CONNECTIONS created an extra connection from the Segmented Chained Tickerplant (SCTP) to the segmented tickerplant (STP), so we deleted the duplicate entry
• The Write Database (WDB) application config was overriding the sortworker core .servers.CONNECTIONS setting, so we added a sortworker app config file to restore the correct value
• The default .servers.CONNECTIONS config for all processes included the RDB and HDB, which caused the SCTP to connect to them, so we updated the default config to remove those connections
• Fixed a bug where RDB was connecting to the gateway, in addition to the gateway already connecting to the RDB

Remove Optional Processes

Unsurprisingly the RDB, monitor and STP had the most connections. Each process exceeded the limit, so we needed to deactivate some components.

Because the FSP is a demo system, we could switch off processes that sat outside the main data path, such as the reporter, monitor, file alerter, and data quality service.

We disabled these processes by setting the startwithall flag in process.csv to 0 for each one. TorQ must still function fully on a limit‑free license; in that case, simply revert the flag to 1 (see the FSP README).

We deactivated the following processes:

• reporter1 – used to run periodic reports on specific processes
• monitor1 – monitors the health of the other processes in the system
• filealerter1 – periodically scans a set of directories for user-specified files
• dqc1 – runs checks on other TorQ processes to check the quality of data in the system
• dqcdb1 – database process for dqc1
• dqe1 – receives daily statistics of other TorQ processes
• dqedb1 – database process for dqe1

We also removed the IEX feed as part of this work because the API it relied on had been switched off.

Conclusion

For the TorQ Finance Starter Pack, getting the data capture system running under the KDB-X Community Edition license was a relatively straightforward task.

However, re-enabling full functionality under this license will demand a significant architecture redesign, likely adding more complexity to avoid the connection limit. The performance of key components are also set to become constrained by the embedded cores, threads and memory limits, so users should take time to estimate their resource requirements before adopting.