Jonny Press
kdb+ and KDB-X are very flexible. Over the last few months we’ve been running a series of blogs which are focussed on “non-standard” ways of capturing and accessing data in kdb+ and KDB-X. The aim of these is to simplify the process architecture by making all data available via a single process without a gateway, and to reduce the overall memory footprint of the solution. The first three are possible in both kdb+ and KDB-X, the latter three are only possible in KDB-X.
Each of these blogs is backed by a set of open source code (either a TorQ extension pack or a KDB-X module) to allow you to experiment and analyse yourself. Code is linked from each blog to our Github.
Who Needs An RDB?
An initial article focussed on the relative merits of a simplified architecture, leaning heavily on the improvement of storage performance.
TorQ: No Stinking RDBs
A TorQ package built on a the proposed simplified architecture.
The Last Good Reason for an RDB: The Return of the Attribute
Extending the previously outlined approach with a modification to the kdb+ internals to automatically support an in-memory attribute column alongside data stored on disk.
Virtual Table Starter Pack
Virtual Tables are a KDB-X concept that allow construction of bespoke partitioning schemes across data stored in different formats. This blog focuses on using that structure to natively build an equivalent”No RDB” approach into a TorQ pack, whilst allowing for compression and avoiding the need for internal modification.
Virtual Tables Scaling Capture
Extending the Virtual Tables approach to scale horizontally, whilst still making all data available in a single process.
Parquet Storage and Virtual Tables in KDB-X
A bonus! Looking at how Parquet can be directly accessed via Virtual Tables, and introducing a KDB-X module to allow for easy, configurable data export to Parquet.
Share this:
