Incapture Technologies

Inside the Cloud

Incapture Technologies Blog

 

Reflex Scripting

Published:
October 30, 2014
Author:

Many tasks run in or against a Rapture environment can involve some relatively simple data manipulation – but manipulation that could require some reasonably sophisticated logic. You can execute these tasks by writing code in a more traditional language and compile and deploy that code to the Rapture platform as a fully fledged server application or as a step in a workflow. (Both of these techniques will be discussed in a future blog post). Rapture also has a scripting language – called Reflex – which can be used as another tool to achieve the same ends. This post gives a brief overview to Reflex and how it is used.

Reflex was intended to be a simple language that was very specific to the task in hand – manipulating data in Rapture. As such it tends to be more procedural than “object oriented” and has a very loose type system, using coercion wherever possible to “do the right thing”. Some more modern constructs have been created when they help with the specific task – for example there are some functional implementations such as map/filter/fold as they are good techniques for manipulating data.

A simple example

In this example we imagine we have a document repository “test” with a document in it called “//test/input/one”. The contents of this document are reproduced below:

{
   "value" : 21,
   "name" : "Test",
   "inScope" : true,
   "description" : "A piece of test data"
}

In our example we want to write a script that will take this document, multiply the value field by 2 and then write out the changed document to another location. We will only do this for documents that have a true value in the field “inScope”.

In our first use of Reflex we will use native Rapture API calls:


def workWith(inputUri)
   content = #doc.getContent(inputUri);
   contentAsMap = fromjson(content);
   if contentAsMap['inScope'] do
      newUri = replace(inputUri, 'input','output');
      contentAsMap['value'] = contentAsMap['value'] * 2;
      println("Would write ${contentAsMap} to ${newUri}");
      #doc.putContent(newUri, json(contentAsMap));
   end
end

workWith('//test/input/one');

Going through this example line by line we see that we first define a function “workWith” that we call in line 12 with our input URI. This gives us the ability to call workWith with many different URIs if we see fit. Within the function we first (line 2) retrieve the content from Rapture given the document’s URI. The content returned by the getContent call is a simple text string, which we convert to a Reflex “map” with the built-in “fromjson” call. Once we have a map we can use simple Reflex indexing to retrieve values. In line 4 we test for whether “inScope” is set. If it is we compute a new place to save our new document (by simply replacing the text “input” with “output” and multiply the value field by 2.

In line 7 we print out what we are about to do – using Reflex’s substitution ability for strings (so that the value of contentAsMap and newUri are printed out) and then finally in line 8 we put the content back.

The output from running this Reflex script would be:

Would write {value=42, inScope=true, name=hello} to //test/output/one

In fact Reflex has a simpler technique to retrieving and saving content when it is formatted as json. The operators <-- and --> can be used as replacements to the getContent/json and fromjson/putContent calls. So the more idiomatic Reflex script for performing this task would be:

def workWith(inputUri)
   content <-- inputUri;
   if content['inScope'] do
      newUri = replace(inputUri, 'input','output');
      content['value'] = content['value'] * 2;
      println("Would write ${content} to ${newUri}");
      content --> newUri;
   end
end

workWith('//test/input/one');

Reflex Containers

The above Reflex script was actually run using a command line tool “ReflexRunner” which can be invoked from a desktop. We could have also run the script on a Rapture web server or via an API call (either with the script stored in Rapture or passed as a parameter). Reflex runs inside the concept of a container and this section will describe what we mean by that.

When a Reflex environment is used for the execution of a script it runs in the concept of a container that has a series of hooks to underlying implementation of that Reflex functionality. For example, in the script above we were loading and saving data from Rapture – one of the hooks we have to wire up in a container is the mapping between making an API call in Reflex and how that API call is implemented. In a desktop based container the “wiring” uses the Rapture client side API to connect to a remote Rapture environment. When running on a Rapture instance directly the API is wired direct.

The diagram below shows the hooks that form part of this container architecture for Reflex:

ReflexContainer

In this diagram we have a script running in a container with the links to the wired interfaces. The interfaces in sky blue are usually always implemented in a container.

Each of the hooks is described briefly below with some examples of how they are wired up depending on the container location:

API Handler

The API handler is the link to the connected Rapture API (the use of #doc.putContent in the above example). In a desktop this will use the client Rapture API, within Rapture itself this will be bound to the Rapture kernel.

Data Handler

This handler is used to implement the push and pull operators (–> and <--). The implementation is usually simply using the API handler, though this hook allows a more efficient implementation to be used if available.

Debug Handler

This handler (when implemented) gives the hook the ability to pause and inspect the state of a Reflex script. The handler is only usually implemented when user interaction is available but because the debug hook is called for each statement (and partial statement) executed a “debug handler” is also available that records performance statistics for the execution of a Reflex script.

IO Handler

Certain Reflex built-in functions involve manipulating “files” and “folders”. The binding of these concepts to an environment is the responsibility of this handler. On the server this is usually a null implementation (you cannot access files on a Rapture server). For desktop execution this is bound to the underlying operation system file structure.

Script Handler

Reflex scripts can “include” or “import” other Reflex scripts. This handler is used to work out how to retrieve those referenced scripts and often delegates to the API handler.

Output Handler

Built-in Reflex functions such as “print” have the default behavior of printing out the arguments onto the stdout stream. On a server this may not be the best approach – particularly if you want a calling application to see the output. The output handler implementation can determine the best way of handling printed information.

Input Handler

The input handler is the opposite of the output handler – it determines how the built in functions of Reflex that are responsible for getting data from a “user” are implemented.

Cache Handler

The cache functions of Reflex can help performance in that data can be stored in a cache for later retrieval – with the assumption that the cache entry may not necessarily exist if a follow on script is run on a completely different (and disconnected) Rapture server or desktop. Usually the cache handler is implemented as a simple time and space limited mapping store.

Specific Reflex Containers

Within the Rapture platform there are a number of pre-built Reflex containers – which can be extended by a developer by implementing or extending the handlers described above.

Desktop execution

There are two desktop Reflex containers – the “ReflexRunner” application which takes as a parameter a file containing a Reflex script and the connection parameters to a Rapture environment, and a widget in the Rapture “Vienna” environment (a thick client graphical UI environment which will be the subject of a future post).

Server execution

On a server there are a number of different containers, depending on need.

The first is the container used when the API call #script.runScript is executed. The API is bound to the kernel, there is no debugging or input or file io and any output is captured and stored in the system log of the server.

Next is a variant of that container when #script.runScriptExtended is executed. RunScriptExtended returns the output from the script after execution and so the output handler is bound to a different implementation.

Finally there is the container that is used when executing a script as part of a workflow (workflows are the subject of a future post). A workflow has the concept of a workflow “context” and the script has access natively to this context as variables pre-injected into the Reflex container. Also the return value of the script is captured and use to drive the future execution or direction of the workflow.

Language Overview

We’ve seen how a simple script is created and we’ve also seen the types of container a Reflex script can run in. We’ll now have a quick look at some of the features of the Reflex language. The post cannot describe every feature but there is a Reflex manual that can be requested from Incapture that contains that information.

General structure

Reflex has variables – capable of storing a number of different types. The type of a variable is inferred by its content. Reflex understands booleans, strings, numbers, lists, maps, sparse matrices, “files” (or streams) and native Java objects. So all of the following is valid syntax in Reflex:

x = true; // boolean
y = 1;  // number
z = 'a string'; // string
a = "another string"; //string
m = {};  // map
m['value'] = 4;
l = [ 1, 2, 3, 4 ];  // a list
l2 = [ x, y, z, a, m, l]; // a list containing different types

Reflex has simple operators on these values. The “index” operator is shown above for maps, the same structure with a number can be used to access members of a list. Other operators are the usual standard set with the usual precedence rules:

notx = !x;
twoY = y * 2;
letterA = z[0];
theFirstElement = l[0];
y /= 2;
y *= 2;
y = y + 10;
newY = (y * y) + (twoY * twoY);

Reflex has simple logic and loop constructs for controlling flow. The usual “if” “while” “for”. Reflex can enclose blocks with either “do” / “end” pairs or “{” “}”.

for x = 0 to 10 do
    if x % 2 == 0 do
        println("{x} is even");
    end
end

Procedures

Procedures can be defined using the syntax shown in the example at the start of this post. The name of a procedure can be used as a first class object to some of the more functional methods.

def twoTimes(x)
   return x * 2;
end

println(twoTimes(2));

Pull and Push

Reflex can retrieve and place data into a Rapture environment using a simple set of operators:

data = {};
data['one'] = 1;
data --> '//myrepo/data/1';
dataBack <-- '//myrepo/data/1';
assert(data['one'] == 1);

Files and IO

When supported by the container Reflex can access the underlying IO/File system of the environment:

x = file('myfile.csv');
for y in x do
   println("Line ${y}");
end

The Rapture API

The Rapture API is exposed to Reflex by using the # symbol followed by the API area and then the function within that API. Parameters to the API are cooerced to the types needed by the implementation of the API. If the return value is a complex object it can either be inspected directly (using a dotted method notation) or converted to a map using the “json” built-in.

x = #doc.getContent("//myrepo/data/1");

Built in functions

Reflex has many built-in functions that are fully defined in the reference manual. In this post I’ll highlight a couple to give a sense for the type and range of functions:

date
The date built-in returns a Reflex “date” object. If passed zero parameters the date will be “now”, otherwise the date will be the result of parsing the argument passed. Dates can be manipulated using simple arithmetic and can also respect calendars such as a business week.
keys
When working with maps it’s often useful to enumerate over the keys of the map, or at least check to see if a given value is in the keys. This built-in returns the keys of a map.
size
The size of a variable depends on its type. For a list it’s the number of elements in the list. For a string it’s the length. For a map it’s the size of its keys.
unique
Unique returns the elements of a list that are unique – removing duplicates.
cast
The cast built-in can be used to coerce the type of a variable manually. E.g.

x = 1;
y = cast(x, 'string');
z = cast(y, 'number');
// y will be '1', z will be 1

split
The split command takes a string and splits on a certain character (e.g. a comma). An additional parameter helps with splitting content that could contain quoted strings.

A working example

We’ve seen a simple Reflex script, some ideas about the logic and some examples of built-in functions. Finally we’ll look at a real world Reflex script that is used in a working system to parse a file passed by a remote system into more structured documents in Rapture. Some of the code has been modified to protect confidentiality.

// Create positions and asset information and pricing from a blob
//

require '//common/util' as common;

const DATE = common.getToday();
//const BLOB = "blob://idp.external.report/client/${DATE}/eod";
const BLOB = '/Users/amkimian/Mock_SPOS_INCAPTURE_D_20141028.csv';

posDate = DATE;
fund = 'ALPHACAPTURE';

def normalizeSymbol(val) 
  if val == null do
     return val;
  end
  return urlencode(val);
end

def maybeNumber(val)
  try do 
     return cast(val, 'number');
  end
  catch e do
  end
  return val;
end

f = file(BLOB);
first = true;
for line in f do
  vals = split(line, ',', true);
  if first do
    ks = vals;
    first = false;
  else do
    pos = 0;
    doc = {};
    for x in vals do
      if size(x) > 0 do
        doc[ks[pos]] = maybeNumber(x);
      end
      pos = pos+1;
    end
    strategy = doc['Standard Strategy'];
    substrat = doc['Strategy'];
    asset = normalizeSymbol(doc['Symbol']);
    path = "//idp.eod.client.pos/${posDate}/${fund}/${strategy}/${substrat}/${asset}";
    added = {};
    added['fund'] = fund;
    added['date'] = posDate;
    added['strategy'] = strategy;
    added['substrat'] = substrat;
    added['asset'] = asset;
    doc['enrich'] = added;
    println(path);
    doc --> path;
  end
end

In this real world example we’re taking a file (actually stored as a blob in Rapture) which is structured as a CSV file. Each row of the file is converted into a document that is stored in Rapture – with the first line used to define the keys of each document.

Most of this example should be pretty readable with a number of notable additions we haven’t covered in this post:

The “require” directive is loading another script from Rapture and putting it in the namespace “common”. This is then used to call the “getToday” function in that common code (it returns the current business date).

The “normalizeSymbol” procedure calls the built-in “urlencode”. This helper function ensures that the string used for the symbol contains only characters that are valid for a URI in Rapture.

The “maybeNumber” procedure uses exceptions as a lazy method of seeing whether a field in the CSV file is a number or not. It tries to “cast” the value to a number. If it works that is returned. Reflex has an exception construct (try/catch/finally).

Summary

The Reflex scripting language in Rapture is simple but has reasonable depth for advanced users. This blog post has scratched the surface with some examples of use and some context around how Reflex is used and deployed. Within Rapture, Reflex scripts can be stored and executed using the Rapture API (running on a container in the server) or executed locally with a long-wired connection to a Rapture environment. Reflex scripts tend to be used for small tasks in workflows, for housekeeping tasks or as the back-end of a web server based application. For more detailed information about Reflex feel free to contact us.


Subscribe for updates