Prompt Engineering – kdb+ and ChatGPT

This post was not actually written by ChatGPT

ChatGPT is an artificial intelligence model that can engage in conversation with users and give intellectual and technical responses, with coding/programming being a key feature that it is particularly good at.  It can be seen to have the ability to generate code from scratch, adjust existing code to a better format, add comments to code, along with explaining and debugging code to a high standard.  Having seen various posts and videos being shared online illustrating how impressive the AI model’s capabilities have been using several popular coding languages, we decided to test these functionalities with its use and knowledge of q/kdb+.

We thought that q would be a real test for the chatbot and hopefully show how powerful this tool can be. As a learning task for AI, q/kdb+ presents challenges due to the lower number of learning resources online when compared to more popular programming languages such as Javascript, Python, Java or C++. Looking at the positives though, having a smaller proportion of knowledge online for the AI model to use will be a good test for ChatGPT and its true capabilities and may enable us to guide the AI to better responses more quickly.

At first, I wasn’t that impressed with ChatGPT’s knowledge of the q language.  From the beginning it was clear to see that it struggled to create code with the correct syntax.  Having never used ChatGPT before this was very frustrating as it was so close to the answer and to be honest, I was rooting for it to get it correct first time. But as I continued working with it, it was clear that this was a very powerful, versatile tool and something that will strongly develop over time. The more I interacted with the AI model, the more I was surprised at how adaptable its responses were.  Its ability to take in observations from the user and integrate them into its next batch of answers was highly skilful. It can be seen to have some sort of semantic understanding involved, having the capability to process literal written key words/functionality related to the coding language it is working with and then being able to put this correctly in place.  It also has the capability to interpret and explain q/kdb+ code to a high level but on some occasions it did struggle with this, giving incorrect information back to a user that might not have the q knowledge to realise that it is wrong.  It may not have the ability to replace kdb+ developers (just yet), but for the casual user that has some previous knowledge of using the programming language, it is definitely a useful resource to have access to. 

The Basics

One thing that I was very interested in was to see ChatGPT’s ability to create code from scratch. I began at the basics, asking it to create some example tables that would be typical of any kdb+ stack.  When asked to create a basic trades table, the schema was correct, but it struggled to create code that inserted the correct data.

There was no previous mention from the user of what columns or data types were needed for a ‘sample trades table’, the basis of the table was created through the systems knowledge, which was quite impressive.  But when populating the table, it made a few mistakes.  Firstly, adding in duplicate entries (above) and then creating a number of ‘rank’ errors (below) with the next batch of responses.

ChatGPT depends on user input and the feedback that is relayed by the user, for the results in which it creates.  I was highly impressed at the ability of the chatbot to break down the feedback it was given and then being able to put these changes into place.  From the example below, the knowledge that the AI model has to interpret what is incorrect with the code and adjust accordingly is brilliant.  Having the ability to acknowledge what element of the code snippet is the second argument and also having the ability to create the correct syntax to complete the code is quite powerful.  This outlines the semantic understanding that is involved with this AI model, acknowledging what has been given back from the user through written communication, and then interpreting this into code to give a more sophisticated answer.

With the AI model having the ability to remember previous interactions with the user to adapt its future answers, its knowledge is forever growing every time you use it.  Using the same conversation feed every time you interact with the program resulted in stronger answers being created.  This was seen as a significant advantage when using ChatGPT with q/kdb+ related questions.  With the first batch of questions having a range of ‘OK’ answers, the more I used it and corrected on some of the solutions given, I could see that it was able to apply these corrections to its answers.  After giving the chatbot some feedback from its creation of the ‘trades’ table, I asked it to create a sample quotes table and it did so correctly on the first try.

Interpreting kdb+ Code

ChatGPT also had a reasonably good ability to interpret q code that was input by the user. Yes, on one or more occasions it was not able to fully understand the whole snippet of code but there were elements that would be beneficial for a user of any ability, whether they are just starting with the q language or a senior developer. With the more limited nature availability of online q and kdb+ resources compared to other coding languages, it can be understandable to have some mistakes within its solutions. Below is a great example of the potential that ChatGPT has and how strong its answers can be. Asking the AI model to explain a q function that could be used to create a rack from a given table, it was able to establish the significance of each parameter, how it was used within the function and also explain each aspect of the function in great detail. Having the ability to break down snippets of code, realise what each element does and then to relay the information in written form to the user, shows how advanced this tool is and how powerful its capabilities are. I believe that as more people use this AI model in regard to the q language, that each answer will eventually be very precise and accurate.

Along with giving a good written description of the code, the online tool was able to give the user a good visualisation of an output that might come from the use of the given code.  To have an Artificial Intelligence tool that has this ability to interpret code and show a visual example of the code being executed is a great asset to have. Even though the AI model is unable to execute live code, stating in a response “As an AI language model, I am not able to create visualizations directly. However, you can use q’s built-in visualization libraries or other third-party libraries such as matplotlib or ggplot in combination with q to create visualizations from the data returned by …“, it still gives a good visual representation of results. There does seem to be a limit to this though as the more complicated code will only result in written descriptions. 

qSQL Queries

I could really see a huge difference in ChatGPT’s ability to create ’better’ code that was precise and to the point, the more I used this online resource.   There were a number of attempts while using ChatGPT to create qSQL queries, that did fail and needed some feedback from myself to adjust the query so that it would work and give the correct result. But after working/interacting with the chatbot for a while it did seem to have the capability to create well rounded qSQL queries, example seen below.   I did find it impressive that it was able to create this query when prompted.  The response also included a good written description of how each aspect of the query works.  One thing that was incorrect was the schema given for the trades table, as it did not define the data types correctly (which it had completed successfully on several other occasions).  Even though this was quite a good answer, it did highlight one flaw that ChatGPT could have for a novice kdb+ developer that is just beginning to explore the language. As a beginner developer would not be able to correctly give as much feedback back to the AI model so that it can start to return more advanced queries.

“select TWAP:(next time – time) wavg price, minprice:min price, maxprice:max price, VWAP:size wavg price by sym from trades”

Back To Basics

As mentioned, the ability that ChatGPT has to adapt and change its future answers is incredible. Using the same conversation feed every time you interact with the program resulted in stronger answers being created. After interacting with the AI model for a while and going back and asking it to complete the first task that I set out for the chatbot (asking it to create a sample trades table), its answer had changed greatly. It was able to use the knowledge from any previous code that I asked it to evaluate, to enhance its answer. I was highly impressed that without any indication, it used the ‘til’ functionality to now populate the table. The argument could be put forward by anyone that looks at the code below, that this will create a length error as it only creates 10 sym entries. But taking a step back from that, a tool that has the ability to adapt itself, learn from mistakes and build more sophisticated answers (without asking or any indication) is definitely a resource that has no limits to its capabilities as time progresses.