WebSockets, HTML5 and kdb+

The aim of this post is to introduce the idea of using HTML5 and kdb+ together. HTML is a markup language used to build web pages and HTML5 is the latest version. It comes with many new features but one we will focus on is WebSockets which allows us to easily send and receive data from kdb+.

WebSockets allows web applications to maintain bidirectional communications with server side processes over one TCP socket. The method that was widely used before WebSockets is AJAX. AJAX consisted of continually polling a connection to receive new data, opening a new connection, downloading data and then closing the connection. With WebSockets, once the connection between the client and server is established it stays open and data can be sent down to the client in real time. kdb+ has been supporting WebSockets since v3.0.

What does this mean for kdb+?

By allowing access to database through a web browser we are able to use HTML and JavaScript to do things that the kdb+ console cannot, such as create front end user interfaces or data visualizations. It makes the database accessible beyond the console and anyone with a device that has a modern web browser can access the data.

Let’s get started

This is a brief introduction on using WebSockets and JavaScript, if you would like to learn more about HTML and JavaScript check out the courses on Codecademy. Before we start be sure you have your development environment set up:

1. Download this q script
2. Download this index.html file and open it in your favourite text editor for inspection
3. Start up a q server on a port of your choice and load the q script
4. Open the index.html file in either Chrome or Firefox
5. Open the developer console by pressing F12, this is used to see the console log messages and for debugging

Data handling

We need to establish a method for sending and receiving data between kdb+ and the browser. There are different methods and it depends on what you want to be able to do from the web browser. The easiest method would be to build a string and evaluate it on the kdb+ side but the best option is to separate the components of the function into the function name and arguments. This is a great option to build from and allows for more functionality such as checking the function name against a dictionary of allowed functions before executing it. The data sent from the browser will be formatted in JSON and so we will use Arthur Whitney’s json.k library for encoding and decoding JSON. As the binary data will be serialized in kdb+ we need the ability to deserialize and serialize in the browser, this is achieved by using Kx’s c.js library.

For the example below we will use a basic version of the separated function components option. The evaluate function is run on the received JSON data, it evaluates the func property and then applies the arguments to it. The diagram below shows the keys chema of the JSON data being sent over WebSockets from the Browser to kdb+.

Data handling in kdb+

For this basic example, handing data that is sent over WebSockets is easy. The WebSocket handler .z.ws deserializes and decodes the JSON data into a dictionary. The dictionary values are used to evaluate the function with its arguments, then encoded into JSON, serialized and then sent back down the connection handle.

\l json.k

dataformat:{`name`data!(x;y)}

evaluate:{dataformat[x`func;(value x`func) @ value x _ `func]}

.z.ws:{neg[.z.w] -8!.j.j @[evaluate;.j.k -9!x;{'"error: ",x}]}

Data handling in the browser

A WebSocket connection to the URL argument is created and assigned to the ws variable, this enables us to change the methods and properties of the WebSocket connection. The binaryType property is set to force an ArrayBuffer conversion when a binary message is received. The event handlers allow us to execute a function when an event occurs. In the example below once the WebSocket opens, the onopen event handler executes and data is sent. Once a message is received the onmessage event handler executes and the message is shown in the console window. The ws.sendcmd function formats the message correctly before it is sent using the ws.send function.

var ws = new WebSocket('ws://localhost:1234');
// Required by c.js 
ws.binaryType = 'arraybuffer';
// WebSocket event handlers
ws.onopen = function () {
     console.log("Open");
     // Sends a JSON encoded message
     ws.sendcmd("sum",2,3);
};
ws.onclose = function () {
     console.log("Closed")
};
ws.onmessage = function (event) {
     var data = JSON.parse(deserialize(event.data));
     console.log(data.data);
};
ws.onerror = function (error) {
     console.log("Error " + error.data);
};
// Used to send function calls to kdb+. e.g. ("sum",1,2,3)
ws.sendcmd = function (args) {
  if(arguments.length<=1) return false;
  var len = arguments.length,data = {},i;
  // First argument used as func property, rest are used as arg properties  
  data['func'] = arguments[0];
  for(i = 1;i<len;i++){
    data['arg'+i] = arguments[i];
  }
  ws.send(serialize(JSON.stringify(data)));
};

Conclusion

• This tutorial has taught you how to send and receive data to kdb+ via WebSockets
• There is so much potential for building web applications that interact with kdb+
• For a more thorough explanation of using HTML5 and kdb+ together check out our document