Accelerated Migration to Managed kdb Insights

Porting to Amazon Finspace with Managed kdb Insights has been streamlined with Data Intellect’s Migration service’s new tool, which automates a key part of the process.

Before we get into what this tool does, let’s first discuss what Managed kdb Insights is and what needs to be considered when porting to it.

Introducing Amazon FinSpace with Managed kdb Insights

Amazon FinSpace with Managed kdb Insights revolutionises AWS data processing and analytics for capital markets. By offering kdb applications as a fully managed AWS service, it delivers the unparalleled scalability and availability associated with cloud computing. Here’s why it’s a game-changer versus on-premises kdb systems:

• Scalability: Scaling up on-premises requires prior investment in hardware and planning. Leveraging a cloud-based solution removes these constraints, providing more adaptability in changing workloads.
• Reliability: By utilising AWS’s multi-Availability Zone (AZ) architecture, Managed kdb Insights ensures high fault tolerance and availability. Distributing kdb processes across multiple AZs not only enhances availability but also mitigates the risk of downtime or data loss, resulting in a much more robust application.
• Security: Integration with AWS’s strong security features, AWS Identity and Access Management (IAM), simplifies managing a kdb system and bolsters data protection.

Key Considerations When Porting

When transitioning a kdb system to Managed kdb Insights, there are certain constraints to consider, including:

• System Command Restrictions: Some kdb system commands are blacklisted, so alternative approaches have to be taken where they are used. Managed kdb Insights blocks certain system commands for security reasons such as ensuring underlying logic is not affected, i.e. it is not possible to change a kdb service’s port (unless setting the port to 443).
• .z Function Limitations: Overwriting .z functions cannot be done directly, instead this is handled via wrapper functions. This protects critical business logic in .z functions from being overwritten while also allowing extra functionality to be attached to them.

All in all, blacklisted system commands and .z function overwrites need to be rewritten for compatibility with Managed kdb Insights. However, before diving into this rewriting process, we first need to identify what code needs changes. This step has two main considerations:

1. Time: It may take a few minutes to manually find all cases in a single script, or hours if reading many. This can drag out the porting process for longer than it needs to be.
2. Multiplicity: Due to kdb being a versatile language, there are many ways to write the same functionality. This makes spotting every case in code more difficult and thus slower and more error-prone.

The Functionality

Our new tool is specially designed to handle the complexities of the many ways kdb system commands and .z overwrites can be written. It is simple to use, it gives users the power to scan specific directories and individual files, with the option to narrow down the targets with regular expressions.

The program systematically examines each targeted script and methodically categorises any detected issues. Upon completion, the user is provided with a CSV file listing out the category and precise location of each problematic line of code, allowing for incompatibilities with Managed kdb Insights to be resolved more swiftly.

Benefits

Four key benefits that this tool brings are:

1. Speed: The program scans hundreds of scripts in tens of seconds using a standard Lenovo ThinkPad T460. Therefore saving potentially hours of manual work in the porting process.
2. Exhaustivity: It catches every single case it has been told to hunt for. By contrast, a developer will be nowhere near as infallible. Extra efficiency in spotting problematic code leads to reduced errors and thus less time rechecking for those missed.
3. Modularity: Expanding the scanner to check for more cases is a breeze. Easy expansion minimises the bugs that can be introduced and also allows for the tool’s usefulness to grow over time.
4. Configurability: It is possible to scan a directory, a specific list of files or it is also possible to include/exclude files matching a particular pattern. This enhances the tool’s flexibility to suit more tasks and increases efficiency by allowing users to ignore irrelevant files.

Context

As mentioned before, this tool is thorough at spotting the many ways the same functionality can be written. This section will highlight that exhaustivity by showing a few examples of what it can catch.

For this example, a short (nonsensical) kdb script has been made, containing code that would cause issues if used in Managed kdb Insights. There are also a few lines in the script that are fine to be used in Managed kdb Insights that we do not want the scanner to pick up. In the script below we have:

• Blacklisted Port Changes: Lines 1-5 contain multiple ways to change a listening port in kdb
• .z Overwrites: Lines 7-12 are different variations on how the .z.pi function can be overwritten
• Red Herrings: Lines 14-16 do not contain blacklisted functionality and should not be picked up by the scanner

Results

As per the goals of this example, the scanner successfully caught every port change and .z overwrite without picking up any of the red herrings. For those interested in how this information is presented in the resulting CSV, see below:

This tool’s comprehensive scanning capabilities ensure that it accurately identifies incompatibilities while avoiding false positives. This level of thoroughness is key to the reliability and overall usefulness of the tool.

Overall, porting to Managed kdb Insights has been made a lot faster and easier with the addition of this tool to Data Intellect’s Migration services. Its speed and accuracy at spotting incompatibilities in a codebase allows for a much smoother porting process. Its configurability allowing users to finely select the files they want to scan lets it fit in seamlessly with both simpler and more complex workflows. Additionally, the tool’s modular structure allows for its inevitable expansions to be added with ease, growing the tool’s usefulness over time.

Contact us if you are interested in Data Intellect’s Migration services: info@dataintellect.com