TorQ Permission Framework

Blog Data Analytics & Engineering 10 Jan 2017

Ryan McCarron

Our latest update to TorQ adds our new permissions system. Using this system any user connecting over IPC to a permissioned process can have selectively tiered access to any table or variable. This extends to limiting access to functions, through a fully customisable set of user privileges, while still allowing a super-user role with access to all functions, variables, and (importantly) user control tables. This framework has been designed with restricting access to sensitive data in mind – by default, variable and function access is automatically restricted for any user until access is granted.

The key features of this framework are:

  • Process access control with user/password tables.
  • Variable access control with user groups.
  • Function and argument restriction with user roles.
  • Partial table access through virtual tables.
  • Gateway-only permissioning.
  • Return value size restriction.
  • Public user access control.

The permissions framework itself is contained in the new .pm namespace, and is loaded as a handler into each process.

When a user attempts to connect to a process running permission control, their username and password are checked against the internal username/password table, and the user can connect when they supply the appropriate data. This access control can be extended to any external password management system such as LDAP, kerberos, oauth, etc.

TorQ Permissions FlowchartAfter the user has connected, they can start attempting queries. The framework checks all incoming queries against the user control tables (listed, along with example setup functions, in the updated TorQ documentation) using the key function .pm.requ[user;query]. For example, in the case of a query on a table, the framework checks if a user:

  • Can access the table being queried, if the table is directly named (e.g “trades”).
  • Has permission to run “select” queries.
  • Has read access on the table.
  • Is querying a virtual table, and if so should the query be amended.
  • Has requested an amount of data under the size limit.

If the query passes these checks, the user has the result of the query returned. If any checks fail, an appropriate error is passed back instead. If a user attempts to access a variable which isn’t listed in the access table, they are implicitly not granted access.

In addition to full table restriction, partial restriction is enforced as above through virtual tables, which enforce a where clause specified in the virtual table definition to subset the data on any incoming query. Granting access to only this virtual table for a set of users effectively screens the full data set from that userbase.

Similarly, when a user attempts to run a named function on the system, a different set of checks is performed, to ensure the user:

  • Is permitted to run the named function.
  • Has supplied arguments they are permitted to use to this function.

As before, any failure in these checks results in an error message, while success on all checks returns the function output.

As this may not be suited to some active-development systems, we’ve also included a permissive mode, in which a variable not listed in the access control table is given implicit access for all users, while still restricting access to any variables already in the control table. This allows for fluid development while still ensuring data security.

Access control through the gateway has dual functionality in-built into the system – by default, when permissions are enabled on the gateway and any target process, all gateway target processes must contain their distinct table permissions, and a synchronous or asynchronous query will behave as above. However, the permissions can instead be enforced directly on the gateway using the function control API.

Since all gateway-routed queries come in using either .gw.syncexec[query;target] or .gw.asyncexec[query;target], the parameter restrictions on these functions can be set to run a permission check on the query against gateway-stored access tables (using the central query check function .pm.requ). This method of permission control further allows restriction of the target process type through checking of the second gateway call function parameter.

The system finally supports public login – when enabled with the “-public 1” flag on process start, a user who connects without a password will not be rejected, but instead added to a table of public users, and the appropriate public groups and roles, until they disconnect. These groups and roles can be configured as any other.

The admin role is established through granting all functions and all parameters, and exists by default as the “admin:admin” user:password combination. A user with all functions by default has access to all variables.

The updated TorQ package comes with the additional default systemuser role, assigned to the processes themselves to allow smooth operation. While the permissions framework is designed to be performant, a check on every query comes with a minor overhead. Because of this, the tickerplant is by default unpermissioned, to prioritise low-latency operation.

Below is a short example of a permissioned process with four users, two roles, two groups, and a virtual table “newtrades”, which restricts queries to the last hour from table “trades”. The function “getdata[`syms;`buckettype;`bucketsize]” performs an xbar aggregation on the specified sym for the last hour, and returns the last result in each bucket.

In this example all users can connect, but only those in group “top” can free-form select, and the only table they can access is a virtual table. The getdata function can be run by both quants and public users, however, public users can only access data for the `GOOG sym, with the other parameters restricted.

Further examples can be found in the updated TorQ documentation.


.pm.addtogroup[;`top] each `John`Paul
.pm.addtogroup[;`basic] each `George`Ringo


.pm.assignrole[;`quant] each `John`Paul
.pm.assignrole[;`public] each `George`Ringo

.pm.grantfunction[`getdata;`public;{x[`syms] in `GOOG}]

Below is an example of the responses returned for each user connected to the process over handle “h”. Click a user’s group to change from top to basic and back, or role to change from quant to public, then click one of the query buttons below the table to attempt that query.

[iframe src=”” width=100{e673f69332cd905c29729b47ae3366d39dce868d0ab3fb1859a79a424737f2bd} height=470px]

Here, basic users fail the access check on the virtual table while top users get the last hour of trades, and quants can successfully run getdata on `AAPL syms, while public users can’t run the query with these parameters.

Using this framework provides a robust mechanism for data security within our TorQ data capture system, while still maintaining both flexibility for user access and development environments, and customisability for specific implementations of either permission control or TorQ itself. The framework is now available as part of TorQ on github. If you want to know more, feel free to get in touch.

Share this: