Incapture Technologies

Blog

Incapture Technologies Blog

 

Blog

Docker & Rapture Revisited, Etienne

In an earlier post (Rapture and Docker ) I talked about how we quickly put together some Docker http://www.docker.com images for running Rapture, and the means by which we linked various containers together to form a standalone Rapture environment. In this post I’ll talk about the next steps in this journey – where we’ve created a tool to manage our Docker deployments so that it is trivially easy to start and stop Rapture environments for demo purposes. We’ll be using this tool “behind the scenes” for clients and prospective clients so that they can quickly get an isolated environment together to play with.

The tool/application we created has been internally branded “Etienne”. It’s written in go-lang (http://golang.org and presents a web page for an operator to interact with. It also has an API that is driven by an internal administration Rapture environment we use for managing and supporting clients.

On start up with nothing actively running, the web page looks something like this:

Screen Shot 2015-01-13 at 11.11.09 AM

As part of the configuration of the instance we’ve passed it three main things – information about the Docker environments that are available to the app (along with security credentials to connect to them), information about topologies that can be deployed (collections of images and how to bind them together) and finally Etienne also acts as a pass through between the outside world and Rapture so the domain name that Etienne is bound to is provided. In this way if we start an environment called “alan” the domain name “alan.test.incapture.net” might be bound to the Rapture environment started. (Note that we don’t use test.incapture.net for this purpose so clicking around there won’t do anything!).

When we start a topology, Etienne picks an appropriate set of Docker environments (based on load and with a preference for services to be started on the same environment), starts the images (creating docker containers) and then binds them all together. After starting an environment — I called this “alan” — the Etienne screen looks something like this:

Screen Shot 2015-01-13 at 11.24.27 AM

In this environment I have four different images in play. The first two provide the underlying fabric for this environment – MongoDB and RabbitMQ in this example. The third is a Rapture environment itself. The final one is a special container that has a prepackaged Rapture feature within. On startup this container connects to Rapture and installs the feature on to it – it’s a nice component based way of taking a base environment and enhancing it with functionality.

Once started the Etienne UI has a number of features that are simply passed down to the underlying Docker environment. For example, you can view the logs of a given Docker container:

Screen Shot 2015-01-13 at 11.26.50 AM

And the link on the screen also allows someone internal to our network to log into Rapture. (External users will go through the external domain name which will get proxied through Etienne). Clicking on that URL will present a log in screen, and once logged in the UI is based on whatever was in the base platform and any additional features.

Screen Shot 2015-01-13 at 11.30.18 AM

With Etienne we have a very useful tool for building and deploying Rapture environments. The power really comes from a combination of Docker (the containerization) and the modular approach of Rapture which allows us to compose an environment from a number of parts – underlying “fabric” of a database and messaging, a base core code base and a suite of features that enhances that environment. With these together we will be able to present a wide variety of custom demonstrations of Rapture for many different audiences.

That comes next! Watch this space… If you would like a demonstration of Rapture before we have this automated please don’t hesitate to contact me at Extending Rapture – Reflex

In this article and the next I’ll show two different ways in which we can extend Rapture. They are not the only ways to extend Rapture but they will show the general approach that was taken in the architectural design.

Here I’ll show how we can extend the Reflex language with new functionality. As Reflex can be used in numerous places within a Rapture centric ecosystem an extension to the language can greatly increase the power and flexibility of the Rapture platform and the availability of any systems that the extension links to.

General interface

For this example we’ll choose Twilio http://www.twilio.com as a service that we want to interact with through Reflex. Twilio is a good choice as it has a well defined and structured api that can be easily integrated with a Java based application.

Rapture is written in the Java language and Reflex extensions must be written in the Java language (or target the Java Virtual Machine). Reflex extensions must implement the following interface:

package reflex.importer;

public interface Module {
    void configure(List<ReflexValue> parameters);
    void setReflexHandler(IReflexHandler handler);
    void setReflexDebugger(IReflexDebugger debugger);

    boolean canUseReflection();
    boolean handlesKeyhole();

    ReflexValue keyholeCall(String name, List<ReflexValue> parameters);
}

Configuration

This interface has three configuration style methods – the first passes configuration parameters from the Reflex script to the module and the other two pass in two important interfaces that the module can use to interact with Rapture (the ReflexHandler) and a debugger. Modules typically save this information for use in actual implemented calls.

Invocation approach

The next set of calls are really probes by Reflex to determine how calls in the script should be passed to the module. The simplest approach is to use reflection – where a method in Reflex called “hello” would correspond to a method on the implementing class that looks like the following:

...
   public ReflexValue hello(List<ReflexValue> params);
...

In this way the function can be passed parameters from the script and return a value back to the script. I’ll talk about the ReflexValue class a little later.

The second method is to use a keyhole mechanism. Using this technique the “hello” call would invoke the keyholeCall method with the first parameter being “hello” and the second parameter containing the parameters passed from the script.

Twilio Example

For our Twilio Reflex add-in we’ll use Twilio’s Java API library and in particular attempt to send a SMS message using the system. The pseudo code to send an SMS is reproduced below:

        TwilioRestClient client = new TwilioRestClient(ACCOUNT_SID, AUTH_TOKEN);
        Account account = client.getAccount();
        SmsFactory smsFactory = account.getSmsFactory();
        Map<String, String> smsParams = new HashMap<String, String>();
        smsParams.put("To", params.get(0).asString());
        smsParams.put("From", params.get(1).asString());
        smsParams.put("Body", params.get(2).asString());
        Sms sms;
        sms = smsFactory.create(smsParams);

Here we create an Account object from a TwilioRestClient (noting that we have to pass in an ACCOUNT and an AUTH TOKEN to make that happen) and then there are three extra pieces of information we need to send the sms – who it’s from, who it’s to and the body of the message.

We could pass in the ACCOUNT and AUTH TOKEN in every single call but this is the purpose of the interface’s configure call. We can start by implementing that:

package reflex.module;

...
public class Twilio implements Module {
    private String ACCOUNT_SID = "";
    private String AUTH_TOKEN = "";

    @Override
    public void configure(List<ReflexValue> arg0) {
        if (arg0.size() == 2) {
            ACCOUNT_SID = arg0.get(0).asString();
            AUTH_TOKEN = arg0.get(1).asString();
        }
    }
}

Here we assume the configuration is passed in the form of a string to the module, something like the following:

import Twilio as tw with ("ZZZZZZ6a624d40443a09039a0256e78a","827ea98ac8dbf6ac51c77eZZZZZZZc");

The AUTH_TOKEN and ACCOUNT_SID are part of our contract with Twilio.

The implementation of our sendSms function is as follows:

   public ReflexValue sendSms(List<ReflexValue> params) {
        TwilioRestClient client = new TwilioRestClient(ACCOUNT_SID, AUTH_TOKEN);
        Account account = client.getAccount();
        SmsFactory smsFactory = account.getSmsFactory();
        Map<String, String> smsParams = new HashMap<String, String>();
        smsParams.put("To", params.get(0).asString());
        smsParams.put("From", params.get(1).asString());
        smsParams.put("Body", params.get(2).asString());
        Sms sms;
        try {
            sms = smsFactory.create(smsParams);
            return new ReflexValue(sms.getStatus());
        } catch (TwilioRestException e) {
            throw new ReflexException(-1, String.format("Could not send sms - %s",e.getMessage()), e);
        }
    }

It’s simply adapted from the example code above with some exception handling thrown in.

The final pieces of code in our module satisfies the interface and lets Reflex know that it should use reflection:

public class Twilio implements Module {

    ...

    @Override
    public ReflexValue keyholeCall(String name, List<ReflexValue> parameters) {
        return null;
    }

    @Override
    public boolean handlesKeyhole() {
        return false;
    }

    @Override
    public boolean canUseReflection() {
        return true;
    }

    @Override
    public void setReflexHandler(IReflexHandler handler) {
       // We do not use Reflex Handler
    }

    @Override
    public void setReflexDebugger(IReflexDebugger debugger) {
       // We do not use the Reflex Debugger
    }

   ...
}

That’s about it – now, if this jar file is on the classpath of a Rapture application that is invoking a Reflex script the following code (with appropriate account information!) will send an SMS:

import Twilio as tw with ("ZZZZZZ6a624d40443a09039a0256e78a","827ea98ac8dbf6ac51c77eZZZZZZZc");
res = $tw.sendSms("+15121112222","(888) 111-2222","Hello from Rapture!");
println(res);

In some of our application environments we use this exact technique to send alert messages in case of certain failures.

Samples

We’ve created some other Reflex addins as samples – some of them more for fun and some very relevant:

  • A charting library that creates SVG charts
  • A library to allow searching of an ElasticSearch database
  • A Microsoft Excel Spreadsheet reader and writer
  • A PDF file creator
  • An interface into Wolfram Alpha
  • An interface into Yahoo Financials FX api

As before – if you’d like more information about Incapture or Rapture please drop me a line personally or to our general email address info@incapturetechnologies.com and we will get back to you for a more in depth discussion.


Rapture and Docker

Docker is a container virtualization platform that “lets you quickly assemble applications from components and eliminates the friction that can come when shipping code” (see www.docker.com).

I thought it would be interesting to see what it would take to completely deploy a sample Rapture environment using Docker. It was straightforward to pull together a number of containers for this purpose and have a complete Rapture environment running on Docker in about 30 minutes – with the boot up time of the environment measured in seconds.

host ~> docker ps
CONTAINER ID        IMAGE                               COMMAND                CREATED             STATUS              PORTS                                               NAMES
c13338c5c5c8        incapture/raptureapiserver:latest   "/bin/bash /rapture/   About an hour ago   Up About an hour    0.0.0.0:49181->8665/tcp                             lonely_ardinghelli
96a4817c7da6        incapture/rabbitmq:latest           "rabbitmq-start"       20 hours ago        Up 20 hours         0.0.0.0:49175->15672/tcp, 0.0.0.0:49176->5672/tcp   rabbitmq
bc7fe0f0ed87        incapture/mongo:latest              "/bin/sh -c usr/bin/   21 hours ago        Up 21 hours         0.0.0.0:49154->27017/tcp                            mongo
host s004 ~>

Defining the lower level images

Docker introduces the concept of an image. We create containers from images. For my sample Rapture environment I used MongoDB as the database system and RabbitMQ as the messaging system. A quick internet search gave me some good recipes for those images – here expressed in their Dockerfiles (the configuration file used to create an image).

For mongodb:

FROM ubuntu:latest
MAINTAINER Alan Moore alan.moore@incapturetechnologies.com
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
RUN echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | tee /etc/apt/sources.list.d/10gen.list
RUN apt-get update
RUN apt-get install -y mongodb-org
RUN mkdir -p /data/db
EXPOSE 27017
ENTRYPOINT usr/bin/mongod

And for rabbitmq (a little bit more tricky due to the key signing of the install)

FROM dockerfile/ubuntu
MAINTAINER Alan Moore alan.moore@incapturetechnologies.com
ADD bin/rabbitmq-start /usr/local/bin/

# Install RabbitMQ.
RUN \
  wget -qO - https://www.rabbitmq.com/rabbitmq-signing-key-public.asc | apt-key add - && \
  echo "deb http://www.rabbitmq.com/debian/ testing main" > /etc/apt/sources.list.d/rabbitmq.list && \
  apt-get update && \
  DEBIAN_FRONTEND=noninteractive apt-get install -y rabbitmq-server && \
  rm -rf /var/lib/apt/lists/* && \
  rabbitmq-plugins enable rabbitmq_management && \
  echo "[{rabbit, [{loopback_users, []}]}]." > /etc/rabbitmq/rabbitmq.config && \
  chmod +x /usr/local/bin/rabbitmq-start

# Define environment variables.
ENV RABBITMQ_LOG_BASE /data/log
ENV RABBITMQ_MNESIA_BASE /data/mnesia

# Define mount points.
VOLUME ["/data/log", "/data/mnesia"]

# Define working directory.
WORKDIR /data

# Define default command.
CMD ["rabbitmq-start"]

# Expose ports.
EXPOSE 5672
EXPOSE 15672

(This was taken almost directly from https://github.com/dockerfile/rabbitmq)

With those files in place I could create the images using these commands (run in the folder that contained the appropriate Dockerfile)

docker built -t incapture/mongo .
docker build -t incapture/rabbitmq .

After these commands completed I had a set of local images that I could use to start MongoDB and RabbitMQ.

Starting the lower level containers

With those images created I could quickly start a container for the two applications. I gave them very specific names in my example rather than using docker’s automatic naming facility:

docker run  -d incapture/mongo --name=mongo
docker run  -d incapture/rabbitmq --name=rabbitmq

After running these commands I have two local containers running on my host – one for MongoDB and one for RabbitMQ.

Linking

Before I created my Rapture image I had to consider how Rapture was going to “find” the two services I had started previously. The connection information for MongoDB and RabbitMQ is stored in a set of configuration files that are either on the resource (class) path of Rapture or in a very specific location on the filesystem – the location driven by an environment variable. Normally this configuration file has the name or ip address of the underlying service – but in this case I didn’t want to put a specific name in the configuration file – I would rather have a more general name and use docker’s linking (on add-host) facility to bind the service I had started to that general name. In this case I used the general names “rabbit” and “mongo”. We’ll see how that is referenced in a moment.

The RaptureAPIServer image

Rapture has a pre-packaged application available that wraps the core platform along with a servlet based binding for the API and a web front end for general operational management. It’s called “RaptureAPIServer”. For this example I am going to take that codebase and create an image from it.

To do this I copied a built (and installable) version of RaptureAPIServer. Once built a RaptureAPIServer has three subfolders – a bin folder for the start scripts and the website, a lib folder for the java libraries and an etc folder for the configuration. I created a local folder for my image creation and copied in the built application. Structurally it looks like this:

Screen Shot 2014-12-16 at 1.22.15 PM

Next I added two additional configuration files for the bindings to the services. I placed these in the etc/rapture/config folder:

RaptureMONGODB.cfg

default=mongodb://rapture:rapture@mongo/RaptureMongoDB

RaptureRABBITMQ.cfg

default=amqp://guest:guest@rabbit:5672/%2f

Note that these refer to a general server “mongo” for MongoDB and “rabbit” for RabbitMQ.

Finally I created the DockerFile for RaptureAPIServer. It is reproduced below:

FROM ubuntu:latest
MAINTAINER Alan Moore <alan.moore@incapturetechnologies.com>
RUN apt-get update && apt-get install -y default-jdk
COPY app/RaptureAPIServer /rapture
ENV RAPTURE_CONFIG_HOME /rapture/etc/rapture/config
WORKDIR /rapture/bin
EXPOSE 8665
ENTRYPOINT [ "/bin/bash", "/rapture/bin/RaptureAPIServer" ]

The docker file basically installs the java jdk (as Rapture needs java to run) and copies the contents of the Rapture application into a folder called “/rapture”. It sets up the environment variable RAPTURE_CONFIG_HOME (where Rapture can optionally look for configuration files), sets the working directory appropriately, exposes the port used by the API and web site and invokes the start script for Rapture.

We build the image in the standard way:

docker build -t incapture/raptureapiserver .

Running Rapture

Finally we can start a Rapture container. The only special thing to do in this case was to link the new container with the previously started services:

docker run -d --link mongo:mongo --link rabbitmq:rabbit -P incapture/raptureapiserver

Rapture starts perfectly well – initializing the MongoDB environment for first use.

21:29:31,716  INFO [main] (ConfigFileReader.java:72) - Found default for RaptureLOGGER.cfg in classpath. URL: jar:file:/rapture/lib/RaptureAppConfig-1.1.29.99999999999999.jar!/rapture/config/defaults/RaptureLOGGER.cfg
21:29:31,720  INFO [main] (ConfigFileReader.java:80) - RAPTURE_CONFIG_HOME is set to /rapture/etc/rapture/config, retrieve config from there for RaptureLOGGER.cfg
21:29:31,723  INFO [main] (ConfigFileReader.java:84) - No global config found for RaptureLOGGER.cfg.
21:29:31,723  INFO [main] (ConfigFileReader.java:86) - No app-specific config found for RaptureLOGGER.cfg.
21:29:31,798  INFO [main] (RaptureAppService.java:22) <> [] - Starting RaptureAPIServer
21:29:31,819  INFO [main] (RaptureAppService.java:23) <> [] - ==================================
21:29:31,821  WARN [main] (RaptureMBeanServer.java:72) <> [] - No JMX port defined
21:29:31,827  INFO [main] (AddinLoader.java:56) <> [] - Loading addins
21:29:31,828  INFO [main] (ConfigFileReader.java:72) <> [] - No default found for RaptureLOCAL.cfg.
21:29:31,829  INFO [main] (ConfigFileReader.java:80) <> [] - RAPTURE_CONFIG_HOME is set to /rapture/etc/rapture/config, retrieve config from there for RaptureLOCAL.cfg
21:29:31,829  INFO [main] (ConfigFileReader.java:84) <> [] - Found global config for RaptureLOCAL.cfg in RAPTURE_CONFIG_HOME. URL: /rapture/etc/rapture/config/RaptureLOCAL.cfg
21:29:31,829  INFO [main] (ConfigFileReader.java:86) <> [] - No app-specific config found for RaptureLOCAL.cfg.
21:29:31,834  INFO [main] (AddinLoader.java:48) <> [] - Using /rapture/lib/RaptureAPI-1.1.29.99999999999999.jar from path of class library as the base folder
21:29:31,835  INFO [main] (AddinLoader.java:64) <> [] - Addins will be loaded from /rapture/addins
21:29:31,835  INFO [main] (RaptureWebServer.java:41) <> [] - Configured as a web server
21:29:31,864  INFO [main] (ConfigLoader.java:38) <> [] - Loading Rapture Config
21:29:31,870  INFO [main] (ConfigFileReader.java:72) <> [] - Found default for rapture.cfg in classpath. URL: jar:file:/rapture/lib/RaptureAppConfig-1.1.29.99999999999999.jar!/rapture/config/defaults/rapture.cfg
21:29:31,870  INFO [main] (ConfigFileReader.java:80) <> [] - RAPTURE_CONFIG_HOME is set to /rapture/etc/rapture/config, retrieve config from there for rapture.cfg
21:29:31,871  INFO [main] (ConfigFileReader.java:84) <> [] - Found global config for rapture.cfg in RAPTURE_CONFIG_HOME. URL: /rapture/etc/rapture/config/rapture.cfg
21:29:31,871  INFO [main] (ConfigFileReader.java:86) <> [] - No app-specific config found for rapture.cfg.
21:29:31,975  INFO [main] (ConfigLoader.java:84) <> [] - Successfully loaded config file
21:29:31,976  INFO [main] (SysRepoCache.java:31) <> [] - Bootstrap config is REP {} USING MONGODB { prefix="rapture.bootstrap" }
21:29:32,076  INFO [main] (ConfigFileReader.java:72) <> [] - Found default for RaptureMONGODB.cfg in classpath. URL: jar:file:/rapture/lib/MongoDb-1.1.29.99999999999999.jar!/rapture/config/defaults/RaptureMONGODB.cfg
21:29:32,076  INFO [main] (ConfigFileReader.java:80) <> [] - RAPTURE_CONFIG_HOME is set to /rapture/etc/rapture/config, retrieve config from there for RaptureMONGODB.cfg
21:29:32,077  INFO [main] (ConfigFileReader.java:84) <> [] - Found global config for RaptureMONGODB.cfg in RAPTURE_CONFIG_HOME. URL: /rapture/etc/rapture/config/RaptureMONGODB.cfg
21:29:32,078  INFO [main] (ConfigFileReader.java:86) <> [] - No app-specific config found for RaptureMONGODB.cfg.
21:29:32,080  INFO [main] (MongoDBFactory.java:58) <> [] - Host is mongodb://rapture:rapture@mongo/RaptureMongoDB
21:29:32,095  INFO [main] (MongoDBFactory.java:62) <> [] - Username is rapture
21:29:32,095  INFO [main] (MongoDBFactory.java:63) <> [] - Host is [mongo]
21:29:32,096  INFO [main] (MongoDBFactory.java:64) <> [] - DBName is RaptureMongoDB
21:29:32,096  INFO [main] (MongoDBFactory.java:65) <> [] - Collection is null
21:29:32,877  INFO [main] (AuditLogCache.java:87) <> [] - Creating audit log provider for kernel with config LOG {} using LOG4J {}
21:29:32,880  INFO [main] (AuditLogFactory.java:52) <> [] - Creating audit log from config - LOG {} using LOG4J {}
21:29:32,892  INFO [main] (ConfigFileReader.java:72) <> [] - No default found for RaptureRUNNER.cfg.
21:29:32,892  INFO [main] (ConfigFileReader.java:80) <> [] - RAPTURE_CONFIG_HOME is set to /rapture/etc/rapture/config, retrieve config from there for RaptureRUNNER.cfg
21:29:32,893  INFO [main] (ConfigFileReader.java:84) <> [] - No global config found for RaptureRUNNER.cfg.
21:29:32,893  INFO [main] (ConfigFileReader.java:86) <> [] - No app-specific config found for RaptureRUNNER.cfg.
21:29:32,894  WARN [main] (ConfigFileReader.java:103) <> [] - No config files found for RaptureRUNNER.cfg
21:29:32,894  INFO [main] (ConfigFileReader.java:112) <> [] - Unable to find overlay file for RaptureRUNNER.cfg
21:29:32,901  INFO [main] (Log4jAudit.java:127) <> [] - Tue Dec 16 21:29:32 UTC 2014 (user=raptureApi)  [kernel] kernel: Instance started
21:29:32,901  INFO [main] (Kernel.java:865) <> [] - AppStyle is webapp
21:29:32,901  INFO [main] (RaptureWebServer.java:51) <> [] - Configuring server on port 8665
21:29:32,907  INFO [main] (RaptureWebServer.java:55) <> [] - Starting WebServer for Rapture

And the web site (available through the host port mapping) is available also:

Screen Shot 2014-12-16 at 1.30.58 PM

Next steps

This was just a quick demonstration on how easy it is to create a Rapture environment within a set of Docker containers. In the real world more work would be involved in automating the bindings and handling how to bind containers that are running on different hosts – but there is nothing inherently difficult about how to do that. In fact a higher level Rapture environment and workflows could manage the topology of these satellite environments!

As before – if you’d like more information about Incapture or Rapture please drop me a line personally or to our general email address info@incapturetechnologies.com and we will get back to you for a more in depth discussion.


A Reflex Sandbox

If you’ve been reading this blog for a while you may be interested in having a little play with Rapture and perhaps Reflex. We’re in the process of setting up a sandbox environment for this and wanted to create some simple web pages for interacting with the environment. This post shows how I created a simple page that can be used to edit and run Reflex scripts. The real page will have a number of additions to help guide the novice user and will probably look a little different but the core aspects of the work will be the same.

In a previous post I talked about an architectural approach we’ve used have a Web page (Javascript) talk to a Rapture back end — we used Reflex scripts as the “processing” on the server, returning json formatted documents back to the client. The code on the page ends up being simple Ajax calls to Rapture.

Let’s start with a simple screen shot of the page in action:

Simple Reflex View

In this page we have a Reflex script being displayed on the left, the output on the right, some buttons and parameter information at the bottom left and that’s about it. The real “function” of the web page could therefore be broken down into:

  • Read a Reflex script from Rapture and display it.
  • Edit that script
  • Save that script
  • Run that script on Rapture, capturing the output
  • Display that output
  • Alter parameters passed to script execution

We also did not want to write everything from scratch so we leaned on the large body of open source and free software available in this area:

The code for the page consists of the html for the general layout and some javascript for the main interaction. A truncated form of the page is reproduced below – it’s a simple bootstrap container:

</pre>
<div class="container-fluid">
<div class="row">
<div class="col-md-6">
<div class="panel panel-default">
<div class="panel-heading">Reflex Script</div>
<div class="panel-body"></div>
<div class="panel-footer">
<div></div>
<div><button class="btn btn-info" id="saveScript">Save</button>
 <button class="btn btn-error" id="runScript">Run</button></div>
</div>
</div>
</div>
<div class="col-md-6">
<div class="panel panel-default">
<div class="panel-heading">Output</div>
<div class="panel-body"></div>
<div class="panel-footer">Return value from execution:
<div class="alert alert-info" id="execOutput"></div>
</div>
</div>
</div>
</div>
</div>
<pre>

It’s worth pointing out at this point that I’m not a UI or UX expert and Incapture Technologies is hiring. After you’ve finished critiquing my little hack I’d love to hear from you if you want to join our team.

The layout above contains simple placeholders for the main functionality. The main work is done in the associated javascript code. I’ll split that up and explain each in turn.

The first snippet of code is about parsing the parameters passed to the web page. In this example the script to edit/run and parameters are passed in the url string. We of course rely on entitlements to protect the environment from callers manipulating those parameters – if you recall from an earlier post every api call in Rapture is protected by entitlements.

var QueryString = function() {
    var query_string = {};
    var query = window.location.search.substring(1);
    var vars = query.split("&");
    for ( var i = 0; i < vars.length; i++) {
        var pair = vars[i].split("=");
        if (typeof query_string[pair[0]] === "undefined") {
            query_string[pair[0]] = pair[1];
        } else if (typeof query_string[pair[0]] === "string") {
            var arr = [ query_string[pair[0]], pair[1] ];
            query_string[pair[0]] = arr;
        } else {
            query_string[pair[0]].push(pair[1]);
        }
    }
    return query_string;
}();

I found this code on the internet a while back. There are now better ways of doing this but for my little page this does the trick – after execution the variable QueryString will be a dictionary with the parameter names and values.

The first real work I needed to do was to load the Reflex script associated with the parameter “id” passed to the page. This snippet does that job:

function startup() {
    hider();
    setupParameters();

    var id = QueryString.id;
    $.ajax({
        url : "../web/getScript.rrfx?id=" + id,
        dataType : 'json',
        success : function(data) {
            editor.setValue(data.content, -1);
            editor.resize();
            editor.setTheme("ace/theme/monokai");
            editor.getSession().setMode("ace/mode/reflex");
            window.scrollTo(0, 0);
        }
    });

}

The variable editor is setup earlier on – it’s an instance of an “ACE” editor:

var editor = ace.edit("editor");
editor.setTheme("ace/theme/monokai");

Basically the loading code makes an Ajax call to the endpoint “web/getScript.rrfx” with the parameter “id” being the script to load. Rapture interprets this as “run the Reflex script web/getScript setting the variable web.id to the passed parameter”. I’d previously installed this Reflex script in the Rapture environment, it looks like this:

response = {};
script = #script.getScript(web.id);
response.content = script.script;
println(json(response));

The simple Reflex script loads the Reflex script (which contains information about the script as well as the script itself, and then returns just the script as the content of the response. The script prints out the json format of the map which is what is returned to the JavaScript page. If you look at the Javascript code above you see it setting the editor content to being “data.content”.

So after this process we will have a nice editor showing our Reflex script, which we can edit and save. The saving code is attached to the save button and it looks like this:

$("#saveScript").on("click", function() {
    var d = editor.getValue();
    var id = QueryString.id;
    var vals = {};

    vals['id'] = id
    vals['contents'] = encodeURIComponent(d);

    $.ajax({
        url : "../web/putScript.rrfx",
        dataType : 'json',
        type : 'POST',
        data : vals,

        success : function(data, textStatus, jqXHR) {
            $("#message").text("Saved document");
        }
    });
    return false;
});

This code should feel similar to the earlier one – it’s taking the content of the editor and the script to save – it’s packaging them up as a set of parameters to be sent to a Reflex “web/putScript” call. The contents of that Reflex script is reproduced below:

id = web.id;
contents = web.contents;
if #script.doesScriptExist(id) do
   rscript = #script.getScript(id);
   rscript.script = contents;
   #script.putScript(id, rscript);
else do
   #script.createScript(id, "REFLEX", "PROGRAM", contents);
end
println({ "status" : "ok"});

This code basically checks to see if the script already exists – if it does it loads the script, updates the contents and then saves the script back. If it doesn’t it calls a specific Rapture API call to create a new script. Again note that the script above is run in the security context of the logged in user and every API call made has its entitlements checked.

So we’ve loaded and saved a script, the final real part of our page is running the script and seeing the output. The first part (running the script) follows our usual pattern:

$('#runScript').click(function() {
    hider();
    var scriptURI = $('#contentName').text();
    $('#runningProgress').show();
    $.ajax({
          url: "../web/runScriptView.rrfx",
          dataType: 'json',
          data: QueryString,
          success: function(data) {
                $('#runningProgress').hide();
                $('#scriptOut').show();
                $('#execOutput').text(data.returnValue);
                $('#execOutput').show();
                processOutput(data.output);
          },
          error: function (error) {
               $('#runningProgress').hide();
               $('#scriptOut').show();
               $('#execOutput').text(error.responseText);
               $('#execOutput').show();
          }
    });
});

Here there’s a little complexity processing the return value (we’ll look at that in a moment) but the execution of the script is done by executing the web url “/web/runScriptView” and that is itself a Reflex script that looks like the following:

scriptURI = web['id'];
res = #script.runScriptExtended(scriptURI, web);
println(json(res));

The API call runScriptExtended basically runs the script and captures all of the output that script does (println calls for example) and the return value from the execution of the script. This structure is then returned to the caller – in this case the JavaScript ajax success or error functions.

In our little page I wanted to do very trivial display parsing – if the return was json then I could display that in an editor type context – if it was html I could display it directly, otherwise I’d just treat it as raw text. So the complicated looking code in the success/error functions of our execution code is there to show the output in the right format and to display the return value. In this example I did the rather fragile technique of looking at the first character returned to determine the format and called different display code based on the format:

function processOutput(output) {
    if (output[0][0] == '{') {
        processJson(output);
    } else if (output[0][0] == '<') {
        processHtml(output);
    } else {
        processText(output);
    }
}

function processJson(output) {
    $('#outputJson').show();
    outputJ.setValue(output[0], -1);
    outputJ.resize();
    outputJ.setTheme("ace/theme/monokai");
    outputJ.getSession().setMode("ace/mode/javascript");
}

function processHtml(output) {
        var div = document.getElementById('htmlOut');
        // data is raw html
        var data = "";
        for(var i=0; i< output.length; i++) {
            data = data + output[i];
        }
        div.innerHTML = data;
        $('#outputHtml').show();
}

function processText(output) {
    $('#outputText').show();
    var oldTable = document.getElementById('execOutputList'),
    newTable = oldTable.cloneNode(false),tr,td;
    var tbody = document.createElement('tbody');
    for(var i=0; i< output.length; i++) {
       tr = document.createElement('tr');
      addCell(tr,output[i]);
      tbody.appendChild(tr);
    }
    newTable.appendChild(tbody);
    oldTable.parentNode.replaceChild(newTable, oldTable);
}

function addCell(tr, element) {
    var td = document.createElement('td');
    if (element == undefined) {
       td.appendChild(document.createTextNode(''));
    } else {
     td.appendChild(document.createTextNode(element));
    }
    tr.appendChild(td);
}

For html output the picture looks something like:

Screen Shot 2014-12-10 at 8.25.16 AM

And for simple text we have:

Screen Shot 2014-12-10 at 8.26.43 AM

And that is pretty much it! The code for our simple page is split across three different layers – the html layout, the Javascript client side plumbing and the server side Reflex processing sitting on top of Rapture. Together they can make rich user experiences with a short development lifecycle. Look out for the sandbox Reflex explorer in the wild soon!

Layers

As before – if you’d like more information about Incapture or Rapture please drop me a line personally or to our general email address info@incapturetechnologies.com and we will get back to you for a more in depth discussion.


Data Provenance

When developing portfolio management and attribution applications at other firms a requirement that was never explicitly called out but absolutely required was some measure of information provenance. The conversation started usually with the end result of a long process – “why was this aspect of the portfolio down (or up)?” or more usually “All things being equal this is different, why?”. The inputs driving that performance spanned multiple systems, people and processes. It could have been driven by simple market moves, a manual intervention somewhere in a process (particularly in a quantitive based investment process) or a poorly implemented hedge. It could also be caused by friction in the process – a delay between decision and execution. In those systems it was difficult to get a real handle on answering these questions easily – each question was more a project for an analyst and something that was difficult to automate.

With Rapture we wanted to move closer to this idea of capturing data about the process and the relationships between activities and information created and managed by these processes. Once the data is captured the idea is to present that data through a Rapture API so that applications or tools can be built to automate the generation of answers to these “why?” questions.

At its core Rapture defines the idea of a “relationship repository”. The concept follows the same model as other repository types (such as documents) in that applications use a standard API to interact with a repository and Rapture provides the underlying bindings to a lower level implementation, usually in some external data store. A relationship repository manages the storage of directed graphs of information. For those not familiar with the concept of a directed graph the following diagram will help to explain:

SimpleDG

Here we have two “concepts”, A and B. They are linked together by a relationship R and the relationship goes from A to B (hence the direction). Real directed graphs are much more complicated. The following diagram could perhaps document the relationship between trading orders generating trade executions which form a set of positions which with a set of prices and curves produce some measurement of risk. (Some labels have been abbreviated for clarity).

ComplexDG

In Rapture’s case we want to have a method to capture these relationships – the api gives an application developer the ability to register and manage the links between entities which can be arbitrary and application specific. An important distinction is a special system relationship repository that Rapture can maintain automatically. The idea behind this repository is the realization that the URL within Rapture is a unique reference to an entity in the system – the idea of the URL:

doc://idp.order/ABC/ONE/ORD1123345@1

explicitly references the first version of a trading order in a system. The other factor is that Rapture maintains a context as the Rapture API is used – whether the API is used in a complex workflow/decision process or by an application manually invoking the API. So that if Rapture detects that something has just made the following pseudo api calls:

get content doc://idp.order/ABC/ONE/ORD1234455
put content doc://idp.execution/ABC/TRD1223

Then there could be a relationship between these two entities and Rapture can (if so configured) record that relationship automatically. Furthermore there are other Rapture entities that could be added to this relationship map – the user entity that made these calls could be bound to each of these entities, the application (if known) or script or workflow that is in play could also be recorded. In fact the simple act of loading and saving some content could generate a reasonably large amount of relationship data – with this feature enabled (it is optional) Rapture becomes a “big data” generator on the relationships between people, processes and data in the environment. The scalability of such a repository is dictated by the underlying implementation attached to the feature and the usefulness of the data is dictated by the sophistication of an application written to interrogate and navigate the relationship repository.

The API for manual definition and retrieval of these concepts is known as the “relationship” api in Rapture. A selection of calls is reproduced below:

   // Create a repository in which to store relationship information
   Boolean createRelationshipRepo(String relationshipRepoURI, String config);

   // Store a relationship link and return its URI
   String createRelationship(String relationshipAuthorityURI, String fromURI, String toURI, String label, Map(String,String) properties);

   // Retrieve a relationship link
   RaptureRelationship getRelationship(String relationshipURI);

   // Delete a relationship link
   Boolean deleteRelationship(String relationshipURI);
   
   // Get all the relationship links with the specified rapture resource as the "from" half of the link.
   List(RaptureRelationship) getOutboundRelationships(String relationshipRepoURI, String fromURI);
   
   // Get all the relationship links with the specified rapture resource as the "to" half of the link.
   List(RaptureRelationship) getInboundRelationships(String relationshipRepoURI, String toURI);
   
   // Get all the relationship links with the specified label.
   List(RaptureRelationship) getLabledRelationships(String relationshipRepoURI, String relationshipLabel);   
   
   // Get relationships from a given node
   RaptureRelationshipRegion getRelationshipCenteredOn(String relationshipNodeURI, Map(String, String) options);   

In many cases in the calls above the options parameter is used to fine tune queries (depth of search and filters).

To circle back to the first paragraph of this post – the answer to some of these questions could be answered in a technical way by an application that could visualize the idea that we have captured the facts that:

This position was generated from these trades and these trades came from these orders that were created by this application process. Order 15 of this set was manually changed by this user with the comment “manual override due to liquidity constraints”. The landscape of relationships for “today” versus “yesterday” differed only by this manual change and on a 30 day history when a manual change has been made this user has improved performance on 80% of the changes.

An admirable goal – not Big Brother, more effective use of data to improve processes and inform decision making – one of our reasons for building Rapture in the first place.

As before – if you’d like more information about Incapture or Rapture please drop me a line personally or to our general email address info@incapturetechnologies.com and we will get back to you for a more in depth discussion.


Document Meta data and versions

In a previous post we talked about Rapture Document repositories and mentioned briefly the idea that documents had attached “metadata”. This post talks about that metadata and also a related feature – document versioning.

All documents in repositories can have additional information stored with them. Some of this information is generated automatically by Rapture but a user can use the Rapture API to add their own custom meta data associated with the document. When retrieving data from Rapture you can ask for just the document contents, just the meta data or both.

The automatic system meta data generated contains the following information:

  • The user name associated with the api context that saved the document
  • The date and time the document was saved (in UTC)
  • The current version of the document – even if the repository does not store previous versions a understanding of how many times the document was saved can give an indication of the “version”.

API calls

There are a number of API calls that directly manipulate or retrieve meta data. They are all in the “doc” api section:

DocumentWithMeta getMetaContent(String docURI);
DocumentMetadata getMetaData(String docURI);

Custom meta data is also known as “attributes”. The following api calls manage attributes:

Boolean addDocumentAttribute(String attributeURI, String value);
List<Object> addDocumentAttributes(String attributeURI, List<String> keys, List<String> values);
XferDocumentAttribute getDocumentAttribute(String attributeURI);
List(XferDocumentAttribute) getDocumentAttributes(String attributeURI);
Boolean removeDocumentAttribute(String attributeURI);

It is worth talking about the “attribute URI” parameter in the above calls. Within Rapture, attributes on documents use the “bookmark” uri convention to define the attribute. So, given a document called:

//idp.data/test/one

Setting an attribute “a1” to the value “42” would be a call similar to the one below (given in Reflex)

#doc.addDocumentAttribute(“//idp.data/test/one#a1”,”42”);

Retrieval of a single attribute follows the same convention.

Versioning

A versioned document repository is one where previous versions of documents can be retained and retrieved at a later date. As you add new documents to a repository any existing documents are preserved in this version history. An api call can be used to trim out old versions from a repository.

Internally a document repository maintains a concept of “the current version”. This is the version that is retrieved if you pass in an unadorned URI to an api call (e.g. //idp.data/test/one above). Using a getMetaData call you can retrieve the version of that latest document and then use a slightly adjusted URI to retrieve previous versions. The script below illustrates this concept:

docMeta = #doc.getMetaContent(“//idp.data/test/one”);
version = docMeta.metaData.version;
previous = version - 1;
previousDocument <— (“//idp.data/test/one@“ + previous);

VersionHistory

You could loop back until you get to version 1 (the first document created) although depending on archiving status earlier versions may already have been expunged from the system.

There are a couple of API calls that rely on version information:

DocumentWithMeta revertDocument(String docURI);
Boolean archiveVersions(String repoURI, int versionLimit, long timeLimit, Boolean ensureVersionLimit);
Boolean putContentWithVersion(String docURI, String content, int currentVersion);

Revert document takes the previous version of a document and makes that the latest version of a document (making the existing “latest version” the previous version)

Archive versions removes old versions of documents given limits on how many versions or time should be preserved (e.g. keep all versions written in the last month, make sure that you keep at least ten versions).

Put content with version attempts to save a new document over an existing one. It assumes that the code saving the document has previously loaded the document and therefore has knowledge of the version of the document it has loaded. If that version does not match the call will fail. This is a form of “optimistic locking” – the assumption is that no-one else should be modifying this document but if someone does the call will fail and will have to be retried (reloaded and re-edited perhaps). Rapture also has a locking system for more formal procedures around modification of data.

This short post gave a quick overview of additional information that can be stored with a document (meta data and attributes) and for some repositories how versioning can be used to maintain a complete history of the changes to a document over time.

As before – if you’d like more information about Incapture or Rapture please drop me a line personally or to our general email address info@incapturetechnologies.com and we will get back to you for a more in depth discussion.


Entitlements

A very important part of any platform environment is the means by which users of the platform are given privileges to perform actions on the platform. Rapture has an entitlements system that provides this function.

Within Rapture everything that takes place is coordinated through an API call – even internally larger “meta” api calls in turn call lower API calls to make their activity happen – nothing in the system bypasses this mechanism. As an example, creating a Reflex script in Rapture actually involves (lower down) saving content to a special internal document repository – and the means by which that takes place is by calling the “putContent” API call for document repositories.

Also every user in Rapture, once logged in, has a concept of a “context” – this is server side state (shared with all Rapture instances) that identify the user associated with a given request in a secure way. This context is always passed (internally) through every API call made in Rapture.

Rapture’s entitlements system works at the API level and by the concept that users can be part of functional entitlement “groups”. The concept of “an entitlement” is a function of the API call being made and the parameters to that API call. It’s best illustrated with an example:

In an earlier post we talked about the API to Rapture and the fact that the API was template based and a process autogenerates both server side and client code. Here is an example of a single call in this template language in all of its glory:

[Store a document in the Rapture system. The new URI name is returned; 
  this allows parts of the new name to be automatically generated]
    @entitle=/data/write/$f(docURI)
    @public String putContent(String docURI, String content);

In this example we are defining the function “putContent” in the document (doc) api. The important line for this discussion is

    @entitle=/data/write/$f(docURI)

This instructs Rapture to check for the “entitlement” that is made up of the prefix /data/write combined with the document URI parameter. So, if we were calling the function like this:

    #doc.putContent("//test/london/one", ...);

The entitlement to check would be:

    /data/write/test/london/one

Rapture now looks for the most specific entitlement that matches that string. By that it checks entitlements in the following order:

/data/write/test/london/one
/data/write/test/london
/data/write/test
/data/write
/data

If it finds a match for that entitlement it looks at which entitlement groups are associated with that entitlement and then sees whether the calling user is a member of any of those groups. If they are the API call can proceed as normal. So in this way the naming of entities and their structure can be important in a Rapture environment if you wish to use this structure to assist in proper partitioning of roles. In the example above the document was named “//test/london/one” with an implication that perhaps an entitlement group called “London” could be part of an entitlement “/data/write/test/london” and only members of that group could write documents below that part of the naming hierarchy.

The final piece of entitlements is how they are created and managed. Rapture provides some User Interfaces to manage these entities but these are simply using the “entitlements” api. Of course the entitlements api is controlled through entitlements as well and there are specific checks in some of these api calls to ensure that a call to an entitlements api doesn’t revoke all access and produce an irreversible outcome! The api is really a “CRUD” api across entitlements and entitlement groups:

[Entitlements are a very important part of the security of Rapture, and the Entitlement api is the way in which information about this entitlements is updated. The api is of course protected by the same entitlements system, so care must be taken to not remove your own entitlement to this api through the \emph{use} of this api.

Entitlements work like this. Users can be members of entitlement groups, and entitlement groups are members of entitlements. Each api call within Rapture is associated with an entitlement path, and when a user wishes to execute that api call they are checked to see if they are a member of that entitlement (by seeing which groups they are members of). Some api calls have dynamic entitlements, where the full name of the entitlement is derived from fundamental concepts such as typename, displayname, queuename etc. If an entitlement with the specific name exists that is used, otherwise the full entitlement path is truncated one part at a time until an entitlement is found.]

api(Entitlement) {
    @entitle=/admin/ent
    @public List(RaptureEntitlement) getEntitlements();

    @entitle=/admin/ent
    @public RaptureEntitlement getEntitlement(String entitlementName);

    @entitle=/admin/ent
    @public RaptureEntitlement getEntitlementByAddress(String entitlementURI);

    @entitle=/admin/ent
    @public RaptureEntitlementGroup getEntitlementGroup(String groupName);

    @entitle=/admin/ent
    @public RaptureEntitlementGroup getEntitlementGroupByAddress(String groupURI);

    @entitle=/admin/ent
    @public List(RaptureEntitlementGroup) getEntitlementGroups();

    @entitle=/admin/ent
    @public RaptureEntitlement addEntitlement(String entitlementName, String initialGroup);

    @entitle=/admin/ent
    @public RaptureEntitlement addGroupToEntitlement(String entitlementName, String groupName);

    @entitle=/admin/ent
    @public RaptureEntitlement removeGroupFromEntitlement(String entitlementName, String groupName);

    @entitle=/admin/ent
    @public Boolean deleteEntitlement(String entitlementName);

    @entitle=/admin/ent
    @public Boolean deleteEntitlementGroup(String groupName);

    @entitle=/admin/ent
    @public RaptureEntitlementGroup addEntitlementGroup(String groupName);

    @entitle=/admin/ent
    @public RaptureEntitlementGroup addUserToEntitlementGroup(String groupName, String userName);

    @entitle=/admin/ent
    @public RaptureEntitlementGroup removeUserFromEntitlementGroup(String groupName, String userName);
}

Rapture also has the concept of dynamic user membership of groups. In this case the membership of groups can be delegated to a user (well, developer!) defined class that can potentially reach out to an external system to determine that. An example of where that is used with our system company Incapture Investments is where we want to be explicitly sure that the user making a request for data that originated from Bloomberg is made by a user that is actually logged into Bloomberg at that point. Other uses could include using an external LDAP directory for group membership.

A simple real world example will help close up on entitlements. The screenshot below shows a number of entitlement groups used in a particular Rapture deployment – with functional groups of users being assigned read rights to parts of the data environment :

Entitlements

As before – if you’d like more information about Incapture or Rapture please drop me a line personally or to our general email address info@incapturetechnologies.com and we will get back to you for a more in depth discussion.


Web Development and Rapture

At Incapture we’ve been developing quite a number of web front ends to support both Rapture from a Developer Operations perspective and from a product perspective when creating applications for our sister company Incapture Investments. We’ve settled on a good design pattern and this blog post will describe how we create interactive web pages that use Rapture as a back end.

For these pages the standard model has been to use Javascript for the “active” side of the web page and in particular jQuery as a library that helps make a lot of the calls to Rapture more boilerplate. Furthermore in the applications / uses of Rapture that we are talking about here (internal dev ops and specific product work) we have chosen to use JSON as a format for storing information in Rapture repositories. JSON adapts well to being manipulated from within Javascript running within a browser.

There are usually two activities that are going on inside one of our pages – requests for Rapture data that needs to be formatted / displayed appropriately and requests to change things on the server, usually based on some user based action. In all cases we are using the ajax calls from within jQuery to perform these steps.

Before going into detail on the browser/javascript side it’s worth talking about the end point that the browser will be interacting with. Rapture’s API does have a Javascript implementation and that could certainly be used to interact with Rapture. However in this case it does make sense to let the server perform any prefiltering/precomputing/formatting work instead of passing raw data to the browser. This of course saves transport time (we are only sending data we need) and it can be good for security – it’s the server that contains the logic for the processing, not the client.

The technique we use is to use a special servlet that is installed in any Rapture API server. This servlet gives the caller the ability to invoke a Reflex script on the server and have that script return back some content for the client. Usually the return content is json formatted and the client has the ability to pass parameters to the Reflex script being executed. The script is executed in the entitlements context of the user that is logged in to the servlet.

As an example, consider the following screen image of browsing one of our internal test environments:

Reflex Web Servlet Screen shot

Here we were browsing to:

/rapture/web/oper/statusList.rrfx

and the response (printed prettily in this browser) was something like:

[ {
  "serverName" : "testaim001",
  "instance" : {
    "serverGroup" : "BloombergAIM",
    "appInstance" : "BloombergAIM",
    "appName" : "BloombergAIM",
    "status" : "STOPPED",
    "lastSeen" : "2014-11-17T12:58:07",
    "needsRestart" : false
  }
}, {
  "serverName" : "testdcp001",
  "instance" : {
    "serverGroup" : "DataCaptureServer",
    "appInstance" : "DataCaptureServer",
    "appName" : "DataCaptureServer",
    "status" : "RUNNING",
    "lastSeen" : "2014-11-17T15:03:19",
    "needsRestart" : false
  }
}, {
  "serverName" : "testdcp002",
  "instance" : {
    "serverGroup" : "DataCaptureServer",
    "appInstance" : "DataCaptureServer",
    "appName" : "DataCaptureServer",
    "status" : "RUNNING",
    "lastSeen" : "2014-11-17T15:03:16",
    "needsRestart" : false
  }
}, {
  "serverName" : "testidp001",
  "instance" : {
    "serverGroup" : "IDPWebServer",
    "appInstance" : "IDPWebServer",
    "appName" : "IDPWebServer",
    "status" : "RUNNING",
    "lastSeen" : "2014-11-17T15:03:16",
    "needsRestart" : false
  }
}
]

So given a simple URL call we get back some interesting JSON which we could use to update a web page. In fact the following screen shot (the main web front end to RaptureRunner, described in an earlier blog post) is showing this same information:

Rapture Runner processed output

The given URI above “means” to Rapture that it should execute the script

script://web/oper/statusList

and anything printed out from that script will be returned to the browser. In this case the content of that script is :

// Returns the list of servers and their status

servers = #runner.getRunnerServers();
ret = [];
for server in servers do
    statusInfo = #runner.getRunnerStatus(server);
    for instance in keys(statusInfo['statusByInstanceName']) do
       entry = {};
       value = statusInfo['statusByInstanceName'][instance];
       entry['serverName'] = statusInfo['serverName'];
       entry['instance'] = value;
       ret = ret + entry;
    end
end
println(json(ret));

And as you can see it makes some API calls to Rapture (the runner API gives information about the state of servers running in a Rapture environment) and then it creates a summary array of status information in the variable called “ret”. The final line converts the ret array into a json string and prints that out – which means that the code that is invoking that can simply work with the json content directly.

At the browser end the page that uses this call and displays the table shown earlier has JavaScript that looks something like this:

     $.ajax({
           url: "../web/oper/statusList.rrfx",
           dataType: 'json',
           success: function(data) {
          		 var oldTable = document.getElementById('cloudServers'),
          		 newTable = oldTable.cloneNode(false),tr,td;
          		 var thead = document.createElement('thead');
          		 addHeader(thead, 'Server');
          		 addHeader(thead, 'App');
          		 addHeader(thead, 'Status');
          		 addHeader(thead, 'Last Seen');
          		 newTable.appendChild(thead);
          		 var tbody = document.createElement('tbody');
          		 for(var i=0; i< data.length; i++) {
          		    tr = document.createElement('tr');
          		    addCell(tr, data[i].serverName);
          		    addCell(tr, data[i].instance.appInstance);
          		    addCell(tr, data[i].instance.status);
          		    var ls = new Date(data[i].instance.lastSeen);
          		    addCell(tr, ls);
          		    tbody.appendChild(tr);
          		 }
  		    newTable.appendChild(tbody);
  		    oldTable.parentNode.replaceChild(newTable, oldTable);
           }
           });

with functions such as addCell and addHeader doing appropriate things to create tr/td/th entries as part of the table. The ajax call in this case does not need to pass in any parameters but these can be done in the normal way – automatically creating a “web” variable in the Reflex script so that they can be accessed.

So for our web development we have a very simple pattern – in the browser we use JavaScript and jQuery to send a request to a Rapture server which then executes a Reflex script. That Reflex script returns back some json (or can throw errors of course) which can then be parsed in the success function attached to the ajax call. Code is nicely layered between formatting and user interaction (browser side code) and server side processing. Reflex is a reasonable (and simple) scripting language that helps with this as it also understands json content and provides many methods to manipulate such content.

Also because Reflex scripts in Rapture are simply documents that can be installed it is possible to use them for a business level product UI. The code below is a script we use to return “today’s trades” for a web front end. The web side of it is really a widget that can display trades given “an array of trade documents”. The fact that we use that widget with the return “value” from this script is what makes that instance of the widget show “today’s trades” in a consistent way:

require '//common/commonOrder' as order;
day = web['d'];
if day == 'now' do
// Get today's orders
  bd <-- '//idp.businessdate/official';
  day = bd['today'];
end
query = 'SELECT rowId WHERE executionDate="' + day + '"';
results = #index.queryIndex('//idp.execution',query);
response = [];
for x in results['rows'] do
   row = {};
   id = x[0];
   tradeId = "//idp.execution/${id}";
   tradeDoc <-- tradeId;
   orderDoc = order.locateOrder(tradeDoc['id']);
   row['id'] = tradeId;
   row['execution'] = tradeDoc;
   row['order'] = orderDoc;
   response += row;
end
println(json(response));

There’s a lot going on in that script but it is self contained and has server side logic that would be much more complex to embed within a JavaScript source on the client.

As before – if you’d like more information about Incapture or Rapture please drop me a line personally or to our general email address info@incapturetechnologies.com and we will get back to you for a more in depth discussion.


Rapture Runner

In a cloud/distributed environment with a large number of standard processes running we wanted a standard mechanism for ensuring that all processes have a consistent configuration and that there is a central place that operational managers can view the “state of the system”. We create the Rapture Runner process to handle this – it is not an absolute requirement that Rapture processes are coordinated through Rapture Runner but for some deployments it can be a very useful tool to manage what could be a complex environment.

Rapture Runner is an embedded Rapture application – it contains a complete copy of the Rapture kernel and can therefore access all of the APIs of Rapture internally. In an environment using Rapture Runner it is intended that this is the master process for all Rapture processes in an environment – each server contains one instance of Rapture Runner. As the process starts up it connects to the Rapture data and messaging environment and spawn any other Rapture processes that are configured to run on a given server. These processes could be binaries already installed on the server (through something such as Puppet or Chef perhaps) or stored within Rapture as blobs. The configuration for what processes are run on what servers is stored within Rapture and managed through a specific “runner” api.

The general architecture is shown in the diagram below:

RaptureRunner

Runner Configuration

Rapture Runner manages three main concepts:

Applications and Libraries

An application in Rapture Runner is a process that can be run. The definition in Rapture Runner (showing the web front end view below) gives an application a name, description and version. This information is used by Rapture Runner to locate the binaries for the application along with the “APPSOURCE” configuration parameter. A similar technique is used to define optional (java) libraries that should be embedded within the downloaded application.

Rapture Runner applications

Schedule

A schedule defines how and when an application should be run. Using a cron style definition, Rapture Runner can choose to ensure a process is running on certain days and between certain times. The schedule also controls which “server group” the process should be running on:

Rapture Runner Schedule

Server group

Finally Rapture Runner manages the concept of a “server group” – containing an explicit or implicit definition of what physical/virtual servers are members of a given group:

Rapture Server Groups

Server startup

With this configuration in mind we can now describe how Rapture Runner performs its work. On startup, an instance determines which server groups it belongs to. Membership of a server group implies that, using the schedule entries, certain processes should be running. For those processes that should be running, Rapture Runner downloads the application and library binaries and starts the application.

From that point on Rapture Runner is actively checking 3 things. If the schedule indicates that a new process needs to be started or an existing process should be stopped (perhaps a maintenance window) that activity is performed. If an application managed by Rapture Runner terminates unexpectedly it is restarted – up to a maximum restart count. Finally through the API (and through the web interface) an application can be manually restarted or reconfigured.

Status

The Rapture Runner API also manages the state of active servers running in this context – particularly their status and capabilities. This can be queried through the API and the web front end to Rapture Runner uses this to display the current status:

Rapture Runner Status

Summary

Rapture Runner is an ideal tool for those environments where operations wishes to manage the state of Rapture applications through Rapture. Because the runner API is available through the Rapture API the configuration management of the environment can be delegated to other tools or managed through the web interface of Rapture Runner. As always with Rapture there is a great deal of optionality to allow for many different deployment and management approaches.


Execution Server

When designing a system on the Rapture platform there’s a lot that can be done using Reflex scripts, workflows containing in built feed logic or even client side applications that connect to Rapture “on demand” as they see fit. There is also another class of application which (a) needs to usually run as part of some higher level “process” or workflow and (b) cannot be directly embedded into the Rapture platform – i.e. it is not a Reflex script or a piece of Java code. The Rapture Execution Server and associated workflow plugins is a technique that can be used to bring these external “batch style” application fragments into a more controlled execution environment.

The Execution Server is a standalone Rapture application that reacts to specific workflow step initiation messages. These messages tell the Execution Server to kick off a process with a particular (and custom) command line, and if necessary the application can download Rapture data to a local set of folders so that the spawned application need not use the Rapture API at all to connect back to the system. Instead the application can load the data from files and folders and write output data to other files and folders. Once the process finishes the Execution Server can take that generated output and write it back to Rapture. In this way Execution Server can handle both applications that can simply use the Rapture API to interact with the system and applications that have no knowledge of the Rapture API and can simply interact with files.

Furthermore, the Execution Server can have restrictions on the paths that can be executed from it (to control rogue process execution) and can, if needed, download the “executable” from Rapture before the process is kicked off. This is especially useful for script based applications such as Python.

An example workflow

To show how a workflow is constructed that invokes Execution Server we will show a fragment of a workflow that calls a Python script. The workflow is normally constructed using the Rapture API calls or embodied in a feature (the subject of a future blog post), what is reproduced below is the lower level configuration that is the fundamental definition of this workflow:

{
  "workflowURI" : "workflow://idpResearch/workflows/TestModel",
  "semaphoreType" : "WORKFLOW_BASED",
  "semaphoreConfig" : "{\"maxAllowed\":1}",
  "steps" : [ {
    "name" : "setRunId",
    "executable" : "dp_java_invocable://execmgr.RunIdStep",
    "view" : {
      "MODE" : "#DATE"
    },
    "transitions" : [ {
      "name" : "",
      "targetStep" : "Step1_Configuration"
    } ]
  }, {
    "name" : "Step1_Configuration",
    "executable" : "dp_java_invocable://execmgr.ExecProcessStep",
    "view" : {
      "wrapperInfo" : "%{\n  \"appArg\" : \"blob://model.test/python/Test_Configuration_AM.py\",\n  \"appPath\" : \"/usr/bin/python\",\n  \"inputConfig\" : {\n  },\n  \"outputConfig\" : {\n    \"dir_output_pkl\" : \"*blob://model.test/output/${runId}/pkl/$$n$$e\"\n  }\n}"
    },
    "transitions" : [ {
      "name" : "",
      "targetStep" : "$RETURN:ok"
    }, {
      "name" : "error",
      "targetStep" : "$FAIL"
    } ]
  } ],
  "startStep" : "setRunId",
  "category" : "execServer",
  "view" : {
  },
  "defaultAppStatusNamePattern" : "%idpModel/${$__date_string}/test"
}

In this example we have two real workflow steps and both use the “java invocable” technique of defining the implementation of the step. This technique means that java code embedded in the application processing the workflow will be called. Right at the bottom of the definition, the “category” of the workflow is “execServer” and this instructs the workflow system that step messages by default will only be received by the Execution Server – and it is the Execution Server that contains the implementation of these steps.

RaptureExecutionServer

The first step calls the invocable method “setRunId”. This is simply a convenience method used in this workflow to define a common “context” for an instance of a workflow. It means that output information (such as status of this run) is segregated from other runs of the same workflow.

If the first step completes successfully it moves to “Step1_Configuration”. In the real workflow this example is taken from there are 9 other steps – each invoking applications in slightly different ways. The key to interaction with the Execution Server is with the view parameters. In a workflow the view parameters are passed to the implementation of the step. In most cases what is passed and the format of that information is quite specific to the step and in this case it is no different. The format of the “wrapperInfo” parameter defines:

(a) the application to execute – in this case a Python script stored in a Rapture blob repository
(b) the way to invoke this application – in this case use the python interpreter
(c) any input and output that Execution Server needs to either download before the process executes or after it has finished. In this case the contents of the folder “dir_output_pkl” will be uploaded back to Rapture as blobs in the location given.

And so when this step is executed by Execution Server it will perform the following tasks:

1. Setup an area in the local filesystem to store this temporary “run”.
2. Download the python script from Rapture to this area.
3. Download any input documents (there is none in this configuration)
4. Create a configuration file that defines where the input and output files are, including defining where “dir_output_pkl” actually is. The configuration file also contains information about how the process should connect back to Rapture.
5. Spawn the python process to kick of this script, passing the configuration file location as a parameter.
6. The python script runs. It can use the configuration file to determine where to place any output. In this case it will write a bunch of Python “pickle” files to the folder referenced by “dir_output_pkl”.
7. After the process finishes, Execution Server will look at every file written to dir_output_pkl and upload those files as blobs back to Rapture – into the location

blob://model.test/output/${runId}/pkl/$$n$$e

where $$n$$e is replaced by the name of the file (and its extension) and ${runId} is replaced with the unique runId of this workflow.
8. If all has worked correctly the step will return a successful result, which in this case means that the workflow engine will finish the workflow. (In a larger workflow it will usually move to the next step).

Summary

The Rapture Execution Server is a tool to help a workflow reach out to spawn a remote process to do work as part of a workflow. It provides a control framework around this process and runs it in a consistent way. It is one technique that can be used to integrate applications into the Rapture platform environment.



Subscribe for updates