Module Gate

DsModGate stages data from production. The goal is to spare source systems: the same data should not be imported again for every environment such as DEV, TEST, or QA. The module is a switch that datasqill inserts in front of the source import. When configured, it loads the data from the productive DWH database instead of from the original source (SAP, web service, remote database).

You do not select DsModGate in the GUI. runJavaModule.sh starts it as the Java entry point and passes the real import module with -DmoduleClass. On Validate, the request is forwarded unchanged. On Run, the switch can replace the original load with a load from the productive DWH table (stageFromDB or Snowflake cloneFromDB). If that replacement is not possible, the original import runs.

Name Meaning
Module (not selectable in the GUI)
Module Class DsModGate
Type Java
Purpose Switch for source imports: stage from the productive DWH instead of loading the source again
Transformation Code Taken from the delegated module
Sources / Targets Taken from the delegated module

When the gate is active

All of the following must be true. Otherwise the original module runs.

  1. Command is Run (not Validate).
  2. -DmoduleClass is set (required for every start; without it the gate fails with error -101).
  3. A source object type is known: JVM property -DsourceObjectTypeId, or a built-in mapping for the module class.
  4. The action has exactly one table target.
  5. The sources of that object type belong to a single source database (the target database may appear as an extra source).
  6. The source connection defines stageFromDB and/or cloneFromDB.
  7. The structure of the target table has not changed: column names and data types in the productive copy and in the target match. Otherwise the gate does not apply and the original import runs.

Built-in source object types:

Module class Object type ID Typical source
DsModRemote 1 Database table
DsModSAP 10 SAP table
DsModWebServiceClient 8 Web service

Other modules (Insert, Check, Call procedure, Send E-Mail, …) always go through to moduleClass. They are not rewritten.

Connection parameters

Set these on the source connection (SQTS_DB_<id>), not on the target.

Name Meaning
stageFromDB Datasqill connection ID of an alternative database that already holds a copy of the target table (same schema and table name).
cloneFromDB Name of a Snowflake database from which the target table can be CLONEd. Used only when the target is Snowflake. If both are set on Snowflake, cloneFromDB wins and stageFromDB is ignored.

stageFromDB

The gate opens the staging connection and the target connection and compares metadata of <schema>.<table> (the target). It uses the staging copy only if:

  • the staging connection can be opened and is not a dummy connection
  • the table exists in staging and is not empty
  • column names and types match the target (columns SQDV_* and, for table SAP_CE10058, ZEITSTEMPEL are ignored in the generated SELECT)

Then it does not run the original module. It builds a Load from Database action (DsModRemote) that selects from the staging table and writes into the real target:

  • Truncate Before = N if the target name starts with VV_, otherwise Y
  • source = staging connection / same schema and table name as the target
  • SQL = SELECT <matching columns> FROM <schema>.<table>

The run log prints stageFromDB <id> will be used or a reason why it was not used (dummy connection, empty table, schema mismatch, …).

Anonymization

With JVM property -DdoAnonymize=Y, each selected column can be wrapped in an expression from dws_konfig.vv_anonymisierung (schema_name, tabellen_name, spalten_name → ausdruck). Default is N (no anonymization). Anonymization applies to stageFromDB only, not to Snowflake clone.

cloneFromDB (Snowflake)

If the target database is Snowflake and cloneFromDB is set, the gate drops the target table and recreates it with:

CREATE TABLE <schema>.<table> CLONE <cloneFromDB>.<schema>.<table>

On clone failure it attempts UNDROP TABLE to restore the previous table. rows_processed is the row count of the cloned table. Clone and anonymization are mutually exclusive.

JVM properties

These are set by the module runner, not in the GUI.

Name Meaning
-DmoduleClass Java class of the real module (required), e.g. DsModRemote.
-DsourceObjectTypeId Overrides the built-in object type for the source used to look up stageFromDB / cloneFromDB.
-DdoAnonymize Y or N (default N). Column expressions from dws_konfig.vv_anonymisierung when using stageFromDB.

runJavaModule.sh starts:

java ... -DmoduleClass=${MODULE_CLASS} ... DsModGate

Fallback

If the productive copy is missing, empty, or structurally different, datasqill falls back to the original import without failing the gate itself.