ARK 1: kdb+ Framework

Last month, Gary posted a blog offering us a timely reminder that it’s always worthwhile reviewing our kdb+ architecture, and promising a series of monthly installments about different aspects of maturity for kdb+ applications. Here, we’ll look at the first of those maturity characteristics – the kdb+ framework.

We’ll discuss what a kdb+ framework is (and isn’t), the benefits of using one, typical features, some examples and we’ll finish up with some red flags – what to look out for to keep your kdb+ journey plain sailing. Along the way we’ll also discuss the merits and pitfalls of incorporating non-kdb+ technology into a kdb+ framework.

As a general rule, any generic functionality (useful to many apps) will likely be in the framework and anything specific to the business application will be in the application.

Where the team developing the framework is also developing an app using the framework, this may make things less clear as to where a particular piece of code should be placed – a good rule of thumb is to ask “Would this be useful to other apps?” and if so, place it in the framework.

Remember that a framework by itself won’t “do” anything – it just provides the building blocks that an application will put together and expand upon.

There are a number of advantages to using a kdb+ framework to build an application:

1. Ease & speed of application development
   In a mature kdb+ framework, all core functionality should be present, allowing developers to focus on business logic and bespoke application-specific functionality, rather than wasting dev time implementing generic functionality.
2. Battle-tested components
   When using a mature framework, it will have been tested in real-world use cases and many bugs found and squashed over time – this should result in a more stable base upon which to build your application.
3. Improved interoperability with other teams
   Where the same framework is used by multiple teams within an organisation, this can ease interoperability as (in a well designed framework) there are likely built-in utilities for connecting to other apps, and those apps will use many of the same libraries .etc. so many of the same funtions devs are familiar with will be available.
4. Better mobility to other teams
   In cases were a developer moves from one team to another within the organisation (or even between organisations, if a public framework is being used), the learning curve for the new application is greatly reduced if same framework is being used.

Below is a list of what could be considered key features of any mature, robust kdb+ framework

1. Default processes include all core kdb+tick processes, and then some
   There should be present, at very minimum, tickerplant, RDB and HDB. There should also be an intraday writedown database (in TorQ we call this WDB, some frameworks call it IDB), and a load-balancing gateway to manage user-queries. Another useful default process is a chained tickerplant (CTP), for preventing slow subscribers on main TP.
2. Resilience & recovery
   It is critical that kdb+ applications be fault-tolerant, so framework must provide functionality for processes to recover intraday after a failure and restart. This will mean at a minimum maintaining tickerplant logs & having automated replay functionality. For more rapid recovery in case of partial failures, there may be functionality to recover intraday data from RDB instead of replaying logs.
3. Discovery mechanism
   In any application, q processes are going to need to communicate with each other. Therefore, they will need to be able to make connections – this should be possible via a discovery mechanism provided by the framework, to avoid having hardcoded ports in application code. This may be an additional q process, or some library code to interact with an external discovery service. For this to function, there will need to be some concept of naming or labelling processes, usually by type.
4. Strong suite of provided libraries
   There are many pieces of functionality that are eventually required by pretty much every kdb+ application, and these should all be included with a mature kdb+ framework. An opinionated list of these include
   1. logging – should be able to be logged to disk or console, should be flexible & extensible (e.g. adjusting formatting, extend to push to log monitoring tools etc.)
   2. timers – needs to allow multiple timers (vs q’s single timer) and should provide functionality both to schedule a task to run once at a set time, or to repeat on a set period
   3. usage logging – log queries received from users
   4. pub/sub – at basic level this can just be Kx’s u.q, but there are useful extensions that can be made e.g. making use of broadcast functionality for publish
   5. email – very often useful to send email alerts, reports etc.
   6. timezone utils – most apps will need to convert some timestamps from one timezone to another
5. User access controls
   The framework should provide tools and functionality to control which users have access to which processes, and ideally also what variables and functions they can access within those processes. Usually there will be users who should only be able to access a small subset of the data in the system, and there should be tools to configure this. Generally speaking users should have no direct access to processes such as TP, RDB, HDB etc. and should only be able to send queries to gateway, or if necessary, subscribe to chained TP.
6. Flexible and extensible
   A good framework needs to be able to support multiple different applications with varying business requirements. This means that it may need to support different architectures, it needs to work when processes are distributed in different ways (e.g. multiple hosts, on prem/in cloud etc.), and processes need to be customisable. Application devs shouldn’t have to modify the framework in order to build their apps (and if they do, that will make it more difficult to update the framework over time). There should be sufficient config options and functionality to extend or overwrite code in default processes.

Some frameworks are publicly available, and can be used by anyone (some are free of charge, some are for paying customers only). Others are specific to a particular organisation, and kept privately within that organisation.

There are a few main public frameworks available:

• https://github.com/DataIntellectTech/TorQ – this is our favourite, although we might be a little biased
• https://github.com/exxeleron/enterprise-components – another free, open-source framework
• https://code.kx.com/q/devtools/#kx-platform – Kx Platform, available from Kx at an additional cost
• https://code.kx.com/insights/1.8/microservices/index.html – kdb Insights, an additional product also available from Kx

Beyond this, many of our clients have their own, internal frameworks used across various apps within those companies.

Some frameworks are purely (or mostly) kdb+, with all default processes implemented in q. Other frameworks will make use of other existing technologies for various components.

For example, a common component of kdb+ frameworks will be a discovery service, allowing processes to connect to each other without using hardcoded hosts and ports. This can be implemented entirely as a q process, with that q process keeping track of all running processes within the application & providing this information to other processes as needed. Alternatively, a non-kdb+ specific discovery service (for example, https://www.consul.io/ ) could be used, generally with a q library to interact with the service. This could simplify q processes finding non-q processes within the application and vice versa, if that is applicable.

There are some pros to each approach, shown below:

There are a number of potential red flags when it comes to kdb+ frameworks, a few of these are listed below:

1. Inconsistent coding style
   When the framework has an inconsistent coding style (e.g. a mixture of lowercase, camelCase, PascalCase and snake_case in naming variables and functions), this can make learning the framework more difficult and slow application development.
2. Multiple ways to do the same thing
   When a particular task can be achieved in numerous ways in a framework (with no clear advantage to one or the other), it can be confusing for developers to know which to select, and can lead to inconsistency within and across applications. This in turn increases the effort to maintain as future devs have to understand each of the methods used. So ideally, there should be one obvious way to do a particular thing. Of course, in some cases there will be good reason to have multiple ways with clear logic for deciding which to use – this should be well documented. Which leads us on to…
3. Poor documentation
   The bane of many software development projects, and kdb+ frameworks are no exception. Documentation is difficult to do well, but it is vitally important to a strong framework. Documentation should be clear and concise, but also thorough. It must be kept up to date with any and all changes, and should be written from the perspective of an application developer (note that it is the app developers responsibility to document their app for end users).
4. Excessive external dependencies
   This one may be more of an amber flag – it’s not necessarily a problem in itself, but it may complicate deployments if there are many external libraries, apps, runtimes etc. that need to be installed & configured in order for applications built with the framework to be deployed. In some cases, this may only complicate first deployment, but often the external dependencies will also need updated regularly for security reasons, so these complications may continue. These dependencies may also impose limits on, for example, server operating system (e.g. if a dependency only works on RHEL7 but organisation wants to upgrade all servers to RHEL8).

Hopefully this blog has been helpful to you and will be a valuable tool as you evaluate kdb+ frameworks for future applications, or consider “framework-izing” code already in use in your application, for the benefit of other teams within your organisation. If we can be of any assistance to you in this area, please get in touch – we’d love to work with you and give more tailored advice for your specific use case.

The next step on our journey through kdb+ application maturity will be released next month – keep your eyes peeled!