ARK 1: kdb+ Framework

Blog ARK 29 Feb 2024

Jonathon McMurray

The ARK voyage begins

Last month, Gary posted a blog offering us a timely reminder that it’s always worthwhile reviewing our kdb+ architecture, and promising a series of monthly installments about different aspects of maturity for kdb+ applications. Here, we’ll look at the first of those maturity characteristics – the kdb+ framework.

We’ll discuss what a kdb+ framework is (and isn’t), the benefits of using one, typical features, some examples and we’ll finish up with some red flags – what to look out for to keep your kdb+ journey plain sailing. Along the way we’ll also discuss the merits and pitfalls of incorporating non-kdb+ technology into a kdb+ framework.

What is a framework?

In general terms, a Software Framework is software that provides generic functionality to allow developers to more rapidly develop applications by only writting additional application-specific code. Naturally, a kdb+ framework applies this same principle to kdb+, providing generic functionality that can be used by developers to build kdb+ applications more rapidly. The framework by itself doesn’t “do” anything, it is the application built on top of the framework that serves a business purpose.

There are two main components in a kdb+ framework; default or template processes, and library code. Additionally, a framework will have it’s own paradigms and standards for concepts such as code organization/layout and configuration.

Default Processes

A default process is a ready-built component within a kdb+ framework, that implements a common process type found in kdb+ systems. A very common example of this would be a tickerplant, which will be at the heart of many kdb+ systems. Generally speaking, a framework is likely to have a number of such default processes which can be used “out of the box” in an application built using that framework – different frameworks will have different ways of configuring which processes should be included in your application.

The framework should also provide convenient ways to customise the default processes at the application level – both in terms of modifying configurations, and by modifying or extending behaviour through the loading of additional code. The mechanisms for doing this will vary by framework.

Of course, in the vast majority of applications, additional custom processes will be needed as well as the default processes.


The other key component in a kdb+ framework will be library code. This is code that can be loaded and used in any process within the framework – often some will be loaded by default in every process of an application using the framework, while other libraries may only be loaded on-demand. Common examples of library code that frameworks will provide include logging, timers (extending kdb+’s single timer function), connections (allowing processes within the application to connect to each other without hardcoded hosts & ports etc.) and numerous others.

Often there will be dependencies between included libraries, and frameworks should provide a mechanism for specifying dependencies/load order to account for this.

Where is the line between framework and application?

As a general rule, any generic functionality (useful to many apps) will likely be in the framework and anything specific to the business application will be in the application.

Where the team developing the framework is also developing an app using the framework, this may make things less clear as to where a particular piece of code should be placed – a good rule of thumb is to ask “Would this be useful to other apps?” and if so, place it in the framework.

Remember that a framework by itself won’t “do” anything – it just provides the building blocks that an application will put together and expand upon.

Why use a framework?

There are a number of advantages to using a kdb+ framework to build an application:

  1. Ease & speed of application development
    In a mature kdb+ framework, all core functionality should be present, allowing developers to focus on business logic and bespoke application-specific functionality, rather than wasting dev time implementing generic functionality.
  2. Battle-tested components
    When using a mature framework, it will have been tested in real-world use cases and many bugs found and squashed over time – this should result in a more stable base upon which to build your application.
  3. Improved interoperability with other teams
    Where the same framework is used by multiple teams within an organisation, this can ease interoperability as (in a well designed framework) there are likely built-in utilities for connecting to other apps, and those apps will use many of the same libraries .etc. so many of the same funtions devs are familiar with will be available.
  4. Better mobility to other teams
    In cases were a developer moves from one team to another within the organisation (or even between organisations, if a public framework is being used), the learning curve for the new application is greatly reduced if same framework is being used.

Key features of a kdb+ framework

Below is a list of what could be considered key features of any mature, robust kdb+ framework

  1. Default processes include all core kdb+tick processes, and then some
    There should be present, at very minimum, tickerplant, RDB and HDB. There should also be an intraday writedown database (in TorQ we call this WDB, some frameworks call it IDB), and a load-balancing gateway to manage user-queries. Another useful default process is a chained tickerplant (CTP), for preventing slow subscribers on main TP.
  2. Resilience & recovery
    It is critical that kdb+ applications be fault-tolerant, so framework must provide functionality for processes to recover intraday after a failure and restart. This will mean at a minimum maintaining tickerplant logs & having automated replay functionality. For more rapid recovery in case of partial failures, there may be functionality to recover intraday data from RDB instead of replaying logs.
  3. Discovery mechanism
    In any application, q processes are going to need to communicate with each other. Therefore, they will need to be able to make connections – this should be possible via a discovery mechanism provided by the framework, to avoid having hardcoded ports in application code. This may be an additional q process, or some library code to interact with an external discovery service. For this to function, there will need to be some concept of naming or labelling processes, usually by type.
  4. Strong suite of provided libraries
    There are many pieces of functionality that are eventually required by pretty much every kdb+ application, and these should all be included with a mature kdb+ framework. An opinionated list of these include

    1. logging – should be able to be logged to disk or console, should be flexible & extensible (e.g. adjusting formatting, extend to push to log monitoring tools etc.)
    2. timers – needs to allow multiple timers (vs q’s single timer) and should provide functionality both to schedule a task to run once at a set time, or to repeat on a set period
    3. usage logging – log queries received from users
    4. pub/sub – at basic level this can just be Kx’s u.q, but there are useful extensions that can be made e.g. making use of broadcast functionality for publish
    5. email – very often useful to send email alerts, reports etc.
    6. timezone utils – most apps will need to convert some timestamps from one timezone to another
  5. User access controls
    The framework should provide tools and functionality to control which users have access to which processes, and ideally also what variables and functions they can access within those processes. Usually there will be users who should only be able to access a small subset of the data in the system, and there should be tools to configure this. Generally speaking users should have no direct access to processes such as TP, RDB, HDB etc. and should only be able to send queries to gateway, or if necessary, subscribe to chained TP.
  6. Flexible and extensible
    A good framework needs to be able to support multiple different applications with varying business requirements. This means that it may need to support different architectures, it needs to work when processes are distributed in different ways (e.g. multiple hosts, on prem/in cloud etc.), and processes need to be customisable. Application devs shouldn’t have to modify the framework in order to build their apps (and if they do, that will make it more difficult to update the framework over time). There should be sufficient config options and functionality to extend or overwrite code in default processes.

What frameworks are available?

Some frameworks are publicly available, and can be used by anyone (some are free of charge, some are for paying customers only). Others are specific to a particular organisation, and kept privately within that organisation.

There are a few main public frameworks available:

Beyond this, many of our clients have their own, internal frameworks used across various apps within those companies.

Incorporating other technologies

Some frameworks are purely (or mostly) kdb+, with all default processes implemented in q. Other frameworks will make use of other existing technologies for various components.

For example, a common component of kdb+ frameworks will be a discovery service, allowing processes to connect to each other without using hardcoded hosts and ports. This can be implemented entirely as a q process, with that q process keeping track of all running processes within the application & providing this information to other processes as needed. Alternatively, a non-kdb+ specific discovery service (for example, ) could be used, generally with a q library to interact with the service. This could simplify q processes finding non-q processes within the application and vice versa, if that is applicable.

There are some pros to each approach, shown below:

kdb+ only Incorporating other tech
Simplify tech stack – knowing kdb+ allows inspection of full framework

Simplify deployment – only need to deploy q & the application, no external libraries, apps, runtimes etc.

Tighter integration with framework

Can use well-known, battle-tested components, rather than re-implementing functionality in q

May allow better interoperability with non-kdb+ applications

Quicker framework development if only a simple wrapper library required

Red flags of kdb+ frameworks

There are a number of potential red flags when it comes to kdb+ frameworks, a few of these are listed below:

  1. Inconsistent coding style
    When the framework has an inconsistent coding style (e.g. a mixture of lowercase, camelCase, PascalCase and snake_case in naming variables and functions), this can make learning the framework more difficult and slow application development.
  2. Multiple ways to do the same thing
    When a particular task can be achieved in numerous ways in a framework (with no clear advantage to one or the other), it can be confusing for developers to know which to select, and can lead to inconsistency within and across applications. This in turn increases the effort to maintain as future devs have to understand each of the methods used. So ideally, there should be one obvious way to do a particular thing. Of course, in some cases there will be good reason to have multiple ways with clear logic for deciding which to use – this should be well documented. Which leads us on to…
  3. Poor documentation
    The bane of many software development projects, and kdb+ frameworks are no exception. Documentation is difficult to do well, but it is vitally important to a strong framework. Documentation should be clear and concise, but also thorough. It must be kept up to date with any and all changes, and should be written from the perspective of an application developer (note that it is the app developers responsibility to document their app for end users).
  4. Excessive external dependencies
    This one may be more of an amber flag – it’s not necessarily a problem in itself, but it may complicate deployments if there are many external libraries, apps, runtimes etc. that need to be installed & configured in order for applications built with the framework to be deployed. In some cases, this may only complicate first deployment, but often the external dependencies will also need updated regularly for security reasons, so these complications may continue. These dependencies may also impose limits on, for example, server operating system (e.g. if a dependency only works on RHEL7 but organisation wants to upgrade all servers to RHEL8).

Bon voyage

Hopefully this blog has been helpful to you and will be a valuable tool as you evaluate kdb+ frameworks for future applications, or consider “framework-izing” code already in use in your application, for the benefit of other teams within your organisation. If we can be of any assistance to you in this area, please get in touch – we’d love to work with you and give more tailored advice for your specific use case.

The next step on our journey through kdb+ application maturity will be released next month – keep your eyes peeled!

Share this: