Incapture Technologies

Inside the Cloud

Incapture Technologies Blog

 

Rapture Sheet Repositories

Published:
October 27, 2014
Author:

This is part of the Rapture Series of posts – providing a general overview of the features of Rapture. In this article we describe the sheet repository type of Rapture which is used to store spreadsheet styled data indexed by a unique text key (a uri).

What is a repository?
In Rapture a repository is a place to store information. A repository has a name (unique in a given Rapture instance) and an underlying implementation. The idea is that application developers interact with a repository using a consistent API and Rapture takes care of the details of how to manage the information in the underlying implementation. The implementation in this case refers to a database system and the systems supported currently cover a wide range of technologies from traditional relational databases to the newer “distributed key-value stores”. A list of the technologies currently supported is provided later in this post.

A quick example
Before diving into the details it is worth giving a preview to help set the stage. Although Rapture is a platform it does have an operations web interface that can be used (amongst other things) to browse the data stored in the environment.

In the screen capture below you see a typical view of some of the sheet repositories in a Rapture environment.

Screen Shot 2014-10-27 at 12.11.42 PM

A Rapture environment can have many sheet repositories and the data is usually divided by purpose.

A sheet repository holds “sheets” and in the web ui if we select one of the sheets we can view its contents:

Screen Shot 2014-10-27 at 12.13.54 PM

In this case the sheet is used as a structured configuration document to help with the loading of data from Bloomberg.

Using the API
The Rapture API is the consistent way of interacting with the platform. The API is split into different sections and the section for document repositories is called “sheet”. Using the API in a general sense is the subject of a later post but it is worth giving some general observations about how we designed the API for use.

The Rapture API is available in a number of different places. For client applications (programs that are not hosted within Rapture) the client API is used. This API is connected to a Rapture environment through the login section of the API and once connected the application can use the other API sections to interact with the system. The client API is available for Java, Javascript (for browsers), .NET, Python, Ruby, Go and VBA. Although the syntax varies slightly the meaning and use of each of the API calls is consistent.

As an example, the code in Java to retrieve the data in the sheet in the screen above would be something like:

            HttpLoginApi loginApi = new HttpLoginApi("http://rapture", new SimpleCredentialsProvider(user, password));
            loginApi.login();
            ScriptClient sc = new ScriptClient(loginApi);
            RaptureSheetStatus content = sc.getSheet().getAllCells("//datacapture/input/strategies/COMLS.sheet",0,0);

Repository Implementation
Before putting data into a repository it needs to be created or registered within the environment. There is of course an API call to do this – it takes the name of the repository and its configuration:

#sheet.createSheetRepo("//test.sheet", [a config string]);

The configuration string defines two things – the underlying implementation technology and the general feature set supported (whether the repository is versioned for instance).

The format of the configuration string is as follows:

SHEET { [configuration] } USING [implementation] { [configuration] }

Describing all of the options available is beyond the scope of this post but some examples will help clarify the syntax:

SHEET {} USING MONGODB {}     // A sheet repository using MONGODB
SHEET {} USING MONGODB { prefix="testdoc" }  // A sheet repository using MONGODB on a specific collection
SHEET {} USING CASSANDRA { prefix="test" } // A sheet repository using Cassandra as a backing store

Sheet structure
A sheet in Rapture is mainly used to store row/column style data. The API is specifically tuned for “poking” data into a cell given a row and column coordinate. The content stored is actually an aspect of the third “dimension” of the data in the sheet and this dimension is used throughout the API (zero being the “content”). A value of one in the dimension field is about storing the format that can be used to display the data. Typically this is used when a sheet is to be displayed in a much richer display environment or when exporting a sheet to a report such as a PDF.

For efficient UI representation a sheet also maintains the concept of a “current sheet version” or “epoch”. Each change to a sheet (adding, modifying and removing cells usually) causes an epoch number to increment and each modification in a sheet is tied to its epoch number. Whenever a caller retrieves data from a sheet they also receive the latest epoch number – and they can pass that number in to each call. In this way a “long polling” technique can be used to retrieve data from a sheet. In the first call the UI retrieves all of the data and the latest version number. In subsequent calls the request is for “all changes since the version I’ve already received” – in most cases this will be a set of cells that have changed, and an efficient UI can simply update those elements that have changed, with optional transient highlighting or flashing of the change.

Finally the sheet API also allows the developer to attach Reflex scripts to a sheet – when executing a script attached to a sheet the “current sheet” is automatically injected into the script environment as the variable “s” so that manipulation of data on the sheet is easier syntactically. This is currently a little used feature – it is often used in demonstrations rather in production code.

A tour of the API
The Sheet API section of Rapture is used to interact with sheet repositories. We’ve seen a getAllCells call and a create repository call. The API is rounded out with calls to (a) manage sheet repositories (create, destroy, modify), (b) manage sheets in a repository (create, destroy, update formatting and scripts) and (c) manage the sheet data (put cells and retrieve cells). Each API call is controlled by entitlements so an administrator can configure who can perform each of these tasks.

Although a blog post isn’t the place to describe these calls in general it is worth listing them out so the breadth of coverage implied in the sheet api set can be appreciated. The goal in Rapture is to have a very open API that can be used to manipulate all aspects of the system and this is reflected in number of calls available. Entitlements are used to ensure unintended consequences or unapproved calls being made. A typical user level application would use only a small subset of these calls.

   Boolean createSheetRepo(String sheetURI, String config);
   // Get sheet repository config metadata
   SheetRepoConfig getSheetRepoConfig(String sheetURI);
   // Get repository config metadata for all sheets]
   List(SheetRepoConfig) getAllSheetRepoConfigs();
   // Get the hierarchy of name of sheets and sheet repositories
   List(RaptureFolderInfo) getChildren(String sheetURI);
   // This method removes a Sheet Repository and its data from the Rapture system. There is no undo.
   Boolean deleteSheetRepo(String repoURI);
   // This api call can be used to determine whether a given type exists in a given authority.
   Boolean doesSheetRepoExist(String repoURI);

For managing sheets within a repository:

   // Create a sheet, initially empty. If the sheet exists it is unaffected.
   RaptureSheet createSheet(String sheetURI);
   // Remove a sheet
   RaptureSheet deleteSheet(String sheetURI);
   // Does a sheet already exist
   Boolean doesSheetExist(String sheetURI);
   // Render this sheet to this blob (as a PDF)
   Boolean renderSheet(String sheetURI, String blobURI);
   // Get all of the formatting styles in this sheet
   List(RaptureSheetStyle) getAllStyles(String sheetURI);
   // Remove a named style
   Boolean removeStyle(String sheetURI, String styleName);
   // Create a named style
   RaptureSheetStyle createStyle(String sheetURI, String styleName, RaptureSheetStyle style);
   // Get a list of all of the scripts on the sheet
   List(RaptureSheetScript) getAllScripts(String sheetURI);
   // Remove a script
   Boolean removeScript(String sheetURI, String scriptName);
   // Create a script
   RaptureSheetScript createScript(String sheetURI, String scriptName, RaptureSheetScript script);
   // Run a script on a sheet
   Boolean runScriptOnSheet(String sheetURI, String scriptName);
   // Get the script associated with a sheet
   RaptureSheetScript getSheetScript(String sheetURI, String scriptName);
   // Get all of the ranges in this sheet
   List(RaptureSheetRange) getAllRanges(String sheetURI);
   // Remove a range from a sheet
   Boolean removeRange(String sheetURI, String rangeName);
   // Create a range in this sheet
   RaptureSheetRange createRange(String sheetURI, String rangeName, RaptureSheetRange range);
   // Get all of the notes (comments)in this sheet
   List(RaptureSheetNote) getAllNotes(String sheetURI);
   // Remove a note from a sheet
   Boolean removeNote(String sheetURI, String noteId);
   // Create a note on this sheet
   RaptureSheetNote createNote(String sheetURI, RaptureSheetNote note);
   // Make a copy of a sheet (usually the source is a "template")
   Boolean cloneSheet(String sheetURI, String newSheetURI);

For managing data in a sheet:

   // Set the data for a single cell
   String setSheetCell(String sheetURI, int row, int column, String value, int dimension);
   // Set a group of cells all in one go
   Boolean setBulkSheetCell(String sheetURI, int startRow, int startColumn, List(List(String)) values, int dimension);
   // Set a rectangular block of data
   Boolean setBlock(String sheetURI, int startRow, int startColumn, List(String) values, int height, int width, int dimension);
   // Get the value of data at a given cell
   String getSheetCell(String sheetURI, int row, int column, int dimension);
   // Set data given the name of a cell instead of its coordinates
   String setNamedSheetCell(String sheetURI, String rangeName, String value, int dimension);
   // Retrieve a named data cell
   String getNamedSheetCell(String sheetURI, String rangeName, int dimension);
   // Get all cells that have changed since a modification version (set epoch to 0 to get all data)
   RaptureSheetStatus getAllCells(String sheetURI, int dimension, Long epoch);
   // Get a sheet in a non-sparse format
   RaptureSheetDisplayForm getSheetAsDisplay(String sheetURI);
   // Get a subset of the sheet
   List(RaptureSheetRow) getSheetRangeByName(String sheetURI, String rangeName);
   // Get a subset of the sheet by coordinates
   List(RaptureSheetRow) getSheetRangeByCoords(String sheetURI, int startRow, int startColumn, int endRow, int endColumn);
   // Delete a whole column
   Boolean deleteColumn(String sheetURI, int column);
   // Delete a whole row
   Boolean deleteRow(String sheetURI, int row);
   // Delete a cell
   Boolean deleteCell(String sheetURI, int row, int column, int dimension);

Uses of Sheet Repositories
Sheet Repositories in Rapture are mainly used for configuration documents or as a staging point before rendering a report.

Behind the scenes Rapture for Financial Services uses sheet repositories to configure features such as data capture.

Summary
In summary Rapture sheet repositories give a separation of responsibility between the application developer (putting and getting content) and the underlying operational concerns (which database vendor to use, how to configure and manage that). This separation allows changes to be made beneath an application without changing that application in any way. A sheet is also a good metaphor for some data sets and that makes it easier to build configuration and management screens for some applications.

In the next post we will talk about Reflex Scripting in much more detail.


Subscribe for updates