Jonathon McMurray
In the modern web, APIs (Application Programming Interfaces) form a large part of programmatic interfacing with web services. Many of the services you likely use everyday provide APIs, allowing third party developers to build applications around these services, whether that means retrieving data, receiving notifications of certain events, or sending information to the service for processing. Numerous services used frequently by developers offer public APIs, such as GitHub and Slack. Often these APIs will be “RESTful” (or almost RESTful), also known as “REST APIs”.
Over the last few months, we have built a simple example of a Slack bot in kdb+, kdbslack, and we thought this would be a good opportunity to discuss the flexibility of kdb+ when it comes to interacting with web APIs, using some interesting examples from the Slack API (and touching on a few other examples towards the end of the post). The Slack API isn’t a REST API, but it’s similar, and the techniques are transferable.
kdbbot has improved our lives greatly at AquaQ, from giving us inspirational quotes & jokes…
…to giving us daily status reports on our dev server…
Maybe you have your own ideas for how you could integrate Slack with your kdb+ applications – for example, perhaps you currently get email alerts for errors, but your team is communicating on Slack, and it might allow for quicker response and more immediate discussion if the alert happened there. Or perhaps another web service would be more useful, for example automatically creating a Jira ticket or GitHub issue to investigate why a process failed this morning. All these will be covered by the end of this blog post!
So how does kdbbot manage all this? kdb+ has two built in functions for performing HTTP requests – .Q.hg (GET) and .Q.hp (POST). Typically, most web APIs will use one or both of the GET and POST methods – generally speaking, GET is used for data retrieval, while POST is used to send information to the relevant service. Both can accept parameters, although these are usually passed in a slightly different manner; for GET requests, parameters are added to the URL, while for POST requests the parameters are in the body of the HTTP request.
For example, one way of passing parameters in an HTTP request is as a URL encoded string. In this type of string, key-value pairs are passed, with an “=” separating the key & value, and an “&” separating pairs. For example, the following q dictionary:
q)`abc`def`ghi!(`example;123;5.6) abc| `example def| 123 ghi| 5.6
would become the URL encoded string: "abc=example&def=123&ghi=5.6"
For a GET request, this would be appended on the URL following a “?” e.g. http://www.example.com/?abc=example&def=123&ghi=5.6
For a POST request, this string would be sent in the request body, and the “Content-Type” in the HTTP header would be defined as “application/x-www-form-urlencoded”
In q, these two requests would be performed with .Q.hg and .Q.hp respectively, like so:
q).Q.hg`$":http://www.example.com/?abc=example&def=123&ghi=5.6" q).Q.hp[`:http://www.example.com;"application/x-www-form-urlencoded"]"abc=example&def=123&ghi=5.6"
Note that in kdb+ 3.6 .Q.hg and .Q.hp will take strings for the URLs instead of hsyms – if writing code that should run on either, you will need to take account of this.
So how about Slack? Slack provides a large number of API methods, detailed here. Our kdbslack code implements some of these (in util/slack.q). In general, these API methods require at least one parameter (an authentication token, generated by creating an “app” on the Slack website), and most methods can take either an URL encoded string (as described above) or JSON. Either way, these are sent as POST requests (even though many methods involve retrieving data), and so .Q.hp is used. However, to send JSON, the authentication token must be sent as a custom HTTP header…more on that in a little bit! For URL encoded string, the authentication token is sent within the request body.
A simple example of performing an API request with .Q.hp:
q)tok:first read0`:config/slack_token
q).Q.hp[`:https://slack.com/api/users.list;"application/x-www-form-urlencoded";"token=",tok]
"{\"ok\":true,\"members\":[{\"id\":\"U8XFNXXXX\",\"team_id\":\"T02FKXXXX\",\"name\":\"fake.username\",\"deleted\":false,\"color\":\"bc3663\",\"real_name\":\"Fake Username\",\"tz\":\"Europe\\/London\",\"tz_label\":\"British Summer Ti..
Of course, the return is a JSON object, meaning it can be parsed in q using .j.k.
Within the kdbslack repo, there exist a few “helper” functions to ease use of the API. For example, when it comes to performing the URL encoding, there is a function to convert a KDB dictionary into a URL encoded string:
q).post.urlencode `abc`def`ghi!(`example;123;5.6) "abc=example&def=123&ghi=5.6"
There is also a slightly extended version of .h.ty called .post.ty to include the Content-Type string for URL encoded strings under the key `form. This makes it a little easier to work with the API requests. For example:
q).Q.hp[`:https://slack.com/api/users.info;.post.ty`form;.post.urlencode `token`user!(tok;"U8VJ54DMK")]
"{\"ok\":true,\"user\":{\"id\":\"U8VJ5XXXX\",\"team_id\":\"T02FKXXXX\",\"name\":\"jonathon.mcmurray\",\"deleted\":false,\"color\":\"5a4592\",\"real_name\":\"Jonathon McMurray\",\"tz\":\"Europe\\/London\",\"tz_label\":\"British Summer ..
And of course, there are wrapper functions for some common tasks, like posting into a specific channel, with a custom username & emoji icon:
q).slack.postase["Critical bug encountered in data loader! ```'type```";.slack.chanlist"loaderteam";"kdbbot";":warning:"]
Now one final point; we mentioned earlier that in order to send JSON to the slack API one must supply the authentication token as an HTTP header. There are also numerous other APIs (e.g. GitHub) where the authentication token must always be in a header. Unfortunately, this isn’t possible with .Q.hp. However, we have also begun work on a simple HTTP requests library that has a number of features, the primary being the ability to easily add custom HTTP headers. This library is available on GitHub also: https://github.com/jonathonmcmurray/reQ. Using this library, we can use JSON with Slack (supported methods listed here: https://api.slack.com/web#methods_supporting_json) and make requests to other services such as GitHub e.g.
q)ghtok:first read0`:gh_token q).req.get["https://api.github.com/repos/jonathonmcmurray/kdbslack";enlist[`Authorization]!enlist"token ",ghtok] id | 1.187481e+08 name | "kdbslack" full_name | "jonathonmcmurray/kdbslack" owner | `login`id`avatar_url`gravatar_id`url`html_url`followers_ur.. private | 0b html_url | "https://github.com/jonathonmcmurray/kdbslack" description | "A framework for a KDB back end to a Slack bot" fork | 0b url | "https://api.github.com/repos/jonathonmcmurray/kdbslack" forks_url | "https://api.github.com/repos/jonathonmcmurray/kdbslack/fo.. keys_url | "https://api.github.com/repos/jonathonmcmurray/kdbslack/ke..
Note that the reQ library automatically detects a Content-Type of JSON and parses the response to a q object – the intention is to extend this to other Content-Types where appropriate and parse these also into q objects.
The reQ library also handles HTTP redirects, automatically redirecting if the server requests the client to do so. In addition, it will accept URLs as strings, symbols or hsyms, and should already work fine on kdb+ 3.6 as well as earlier versions.
The reQ library also contains some examples for using the Jira & GitHub APIs, for example:
q)\l examples\github.q q)\l examples\jira.q q).jira.createissue["EP";"Make a blog post";"It's time to write a blog post about REST APIs with kdb+\n\nGet it done ASAP!";`Task;`jonathonmcmurray;`task`blog] "http://localhost:8080/rest/api/2/issue/10021" q).gh.createissue["jonathonmcmurray";"training";"Make a blog post";"It's time to write a blog post about REST APIs with kdb+\n\nGet it done ASAP!";`task] "https://github.com/jonathonmcmurray/training/issues/1"
Currently the reQ library is not used within kdbslack, but several parts of it are implemented there separately (e.g. the urlencode function, the ty dictionary). Our intention is to update kdbslack to make use of these functions from reQ wherever possible. We’ll also be adding more examples of web request based scripts to the reQ repo (there are a couple there already), so be sure to watch it for updates!
Share this:
