Deployment
Deployment of Repository Data (Transformations, Connections and Directories)
The deployment of worksheets, connections, and directories is not done from the GUI but via shell script or web service call. This allows the deployment to be automated and integrated into existing deployment processes.
The smallest deployment unit for transformations is a worksheet.
During deployment, the validity of a worksheet is checked and it is rejected in case of error.
Deployment via Shell Script
The Linux datasqill user provides the functionality for deploying repository data in its bin directory with deploySheet.sh.
This script has the following call syntax:
deploySheet.sh --url URL --test --verbose --option OPTIONS --file FILE
--url URL: Specify the URL to the service of the datasqill server. By default
the connection datasqill_server is used via getkey.
This is usually http://localhost:17491/datasqill-server/service
--test: Only test the deployment. Same effect as --option {"dryRun" : "true"}
--verbose: shows which parameters are effectively used
--file FILE: Specify the filename of the object to be deployed. By default stdin is used.
--option OPTIONS: Specify a JSON with the options
OPTIONS is a JSON with enclosing curly braces which can contain the following attribute pairs:
{
"validate" : "true",
"dryRun" : "false",
"intoFolder" : null
}
The values specified in the options represent the default value of the option.
If validate is set to "false", the file is deployed without validation.
Caution: Non-validated worksheets can lead to missing source table specifications causing incorrect execution order.
With dryRun set to "true", the file is only validated (and validated if specified) but not effectively added to the repository.
If you specify an existing ID of a transformation directory for intoFolder, the worksheet is copied as a new worksheet into the specified directory. All existing IDs of objects, transformations, worksheet, etc. are reassigned.
Caution: This option is only possible on the repository of the development environment.
All objects in dsr (datasqill repository) format can be used as source files. These files are based on HJSON.
Thus, worksheets, connections, and directories can be deployed with the same command.
An example call could look like this
$ deploySheet.sh --option '{"validate":true,"dryRun":false,"intoFolder":20791}' --file Sheet.dsr
Deployment without errors
Deployment via Web Service
Alternatively, deployment can be done via web service.
The payload of the service has the following format:
{
"command" : "DeployRepository",
"payload" : {
"content" : "{\nversion: 1\n transformationData:\n {\n sheets:\n [\n {\n id: 2851\n attributes:\n {\n documentation: Sonderzeichen \" : { } []\n name: Deployment via Service\n order: DEPLOYMENT VIA SERVICE\n folder_id: 1158\n folder: /Manual\n }\n nodes:\n [\n {\n id: 2853\n type: 0\n layout:\n {\n x: 340\n y: 230\n width: 160\n height: 30\n }\n attributes:\n {\n module: Virtual\n documentation: null\n name: Übertrage\n action: null\n }\n }\n {\n id: 2852\n type: 1\n layout:\n {\n x: 340\n y: 180\n width: 160\n height: 30\n }\n attributes:\n {\n field1: Quelle\n field2: null\n field3: null\n objectTypeId: 3\n connectionId: 0\n dependency: Y\n }\n }\n {\n id: 2854\n type: 1\n layout:\n {\n x: 340\n y: 280\n width: 160\n height: 30\n }\n attributes:\n {\n field1: Ziel\n field2: null\n field3: null\n objectTypeId: 3\n connectionId: 0\n dependency: Y\n }\n }\n ]\n edges:\n [\n {\n id: 2855\n from: 2852\n to: 2853\n }\n {\n id: 2856\n from: 2853\n to: 2854\n }\n ]\n }\n ]\n }\n}",
"deployOptions" : {
"validate" : "true",
"dryRun" : "false",
"intoFolder" : null
}
}
}
The options in deployOptions are to be used the same as described in the previous section.
The content field contains the entire object to be deployed (e.g. a worksheet) in HJSON format as a JSON string.
A simple worksheet looks like this in HJSON:
{
version: 1
transformationData:
{
sheets:
[
{
id: 2857
attributes:
{
documentation: null
name: Small Sheet
order: SMALL SHEET
folder_id: 1158
folder: /Manual
}
nodes:
[
{
id: 2859
type: 0
layout:
{
x: 320
y: 120
width: 160
height: 30
}
attributes:
{
module: Virtual
documentation: null
name: dummy
action: dummy
}
}
]
}
]
}
}
To deploy it, it must be converted to a JSON string.
In Java this is done e.g. with the JsonStringEncoder class from the Jackson library and the quoteAsString method:
static String sheetToString(String sheetFileName) throws Exception {
JsonStringEncoder e = JsonStringEncoder.getInstance();
try (FileInputStream fis = new FileInputStream(sheetFileName)) {
String unquoted = streamToString(fis);
return new String(e.quoteAsString(unquoted));
}
}
To send it via curl on Linux, for example, the dsr object must be converted first.
Here is a script to convert the file and deploy via curl:
#!/bin/bash
head='{"command" : "DeployRepository","payload" : {"content" : "'
tail='","deployOptions" : {"validate" : "true","dryRun" : "false","intoFolder" : null}}}'
content=$(cat "$1" | sed 's,\r,,g;s,\\,\\\\,g;s,",\\",g' | awk -v ORS='\\n' '1')
echo "$head $content $tail" | curl -v --request POST --header "Content-Type: application/json" --data-binary @- $(getkey datasqill_server)
- head is the first part of the JSON object,
- content is the HJSON object wrapped in a JSON string
- tail is the last part of the JSON
In content, Windows CR is removed, double quotes and backslashes are escaped. Line breaks are also escaped with backslash.
getkey is used here to get the web address of the datasqill service. It can also be specified directly.
Deployment of Data Structures
Data structures and static content in the datasqill repository are deployed with the shell script deploySchema.sh. This functionality is used during installation or update of datasqill.
With each datasqill update, a file is delivered that updates the datasqill repository from any previous version to the current version using deploySchema.sh.