Release Notes

datasqill 4.0.0

February 13, 2024
  • All:
    • upgrade to java 17
  • backend:
    • datasqill server is now based on SpringBoot. TOMEE is no longer required and can deinstalled
    • Error message can now be longer than 500 characters
    • Log configuration moved from application.properties to datasqill.env
    • deployment:
      • Fixed deletion of empty sheet during deployment
      • New option for the deployment to save the deployed Version no to the control table for versions
      • Use the newest version of deployed entity when updating the version table
      • Added functionality to use the deployment for own objects
  • modules:
    • New module for call Salesforce Web Services (Sales cloud)
    • DsModWriteExcel
      • limit the column size to 100 characters to avoid that the excel limit for column size is outside its bounds
    • DsModSAP
      • added support for RFC_READ_TABLE without compression
    • DsModTemplateSQL
      • History module supports now SQL Server
      • All templates of DsModTemplateSQL use now the quoted table and schema names
  • GUI:
    • New Editor Monaco for action content and documentation
      • Syntax highlighting
      • Documentation in markdown
      • New Editor functions like mini map, zoom / font size, see more details on pages about the Monaco editor
    • Documentation editor
      • Visualisation of rendered markdown
      • Live-Rendering of markdown during editing
      • Insert pictures into documentation

datasqill 3.5.8

February 5, 2023
  • modules: DsModRemote: bug fix for a close logic of dbWriter
  • GUI: bug fix for missing or wrong formatted Json config file as program argument

datasqill 3.5.7

January 16, 2023
  • modules: DsModWriteExcel: added new functionality to generate Excel without a template
  • GUI: bug fix for Windows path in logging properties file
  • GUI: additional logging properties introduced in local settings

datasqill 3.5.6

November 23, 2022
  • backend: for prepared statement use execute instead of executeUpdate
  • backend: bug fix of 3.5.5. Missing CAST added

datasqill 3.5.5

September 18, 2022
  • backend: added email notification(ERROR event), if a batch cannot be started
  • backend: preparation for h2 release 2.x
  • GUI: fixed the problem with enter key in text area

datasqill 3.5.4

July 31, 2022
  • backend: added action_id, action_name, module_name for default trigger variables
  • GUI: added copy Path to Sheet context menu
  • GUI: added copy path in action and copy fullname, copy select in object
  • GUI: NEW Batch overview Tab
  • GUI: changed Batch editor Tab layout
  • GUI: added edit information for Connection Tab

datasqill 3.5.3

March 09, 2022
  • GUI: Preparation for using a batch as a batch element

datasqill 3.5.2

Februar 10, 2022
  • backend / GUI: Preparation for h2 Version 2.1
  • backend / GUI: Preparation for java 17
  • modules: DsLoadFlatfile:
    • archive processed files only
    • added variable connection_id and target_connection_id for trigger
  • modules: DsModExecProcedure added variable connection_id and target_connection_id for trigger
  • modules: DsModWebServiceClient: added variable connection_id and target_connection_id for trigger
  • modules: DsModWriteExcel: added new functionality to replace / exchange a sheet in existing Excel
  • modules: DsModSendMail: corrected the senders domain address
  • modules: DsModRunShell: Added loop query to select parameters from database und run the script per returned row
  • scheduler: CPPS(Critical path based scheduling) algorithm added. Thanks to Sergey with his master thesis
  • backend: deploySheet.sh new option to deploy a sheet as new (duplicate the sheet and the content with get new IDs) 
    The used ids will be renumbered with numbers of the sequence of the target database.
--> Target must be the development database / environment, because the sequence for object-ids is needed
deploySheet.sh --option '{"validate":"true", "intoFolder":target-folder-id}' < sheet-file
  • GUI: Enable batches as batch element. Capturing not enabled
  • GUI: cancel confirmation dialogs: make them all ask the same question and act the same way
  • GUI: Settings/Connections completly reworked:
    • new Statement set editor
    • new Trigger editor for Database and Database Types
    • edit items via popup window
  • GUI: Action editor added Trigger editor
  • GUI: In batch element editor set default priority to NULL
  • GUI: On sheet Tab added right mouse funtionality "Go to path". Moves the focus in the navigation tree to the sheet
  • GUI: Added right click on database object to find usage of this object

datasqill 3.5.1

September 30, 2021
  • module: Trigger implemented in DsModExecProcedure, DsModSendMail and DsModWebServiceClient
  • module: DsModWriteExcel: add a validation for target and template directory
  • backend: simplified upgrade of datasqill versions

datasqill 3.5

June 16, 2021
  • backend: major rework of file system structure:
    • directory structure modified. ~/apps removed
    • all jar-files are placed in ~/lib
    • all jdbc files in ~/lib/jdbc
    • central setting of environment settings in bin/datasqill.env
    • files like datasqill-server/conf/server.xml datasqill-server/bin/setenv.sh etc. are no longer customized
  • backend: New tables for modul trigger. All database specific commands are now in this trigger tables (Konzepte > Trigger)
  • scheduler:
    • added partition dependency
    • added functionality to have dependency of virtual objects between sheets. Added an attribute in sqts_object_type to enable this
    • object type 1 (database table / view) and object type 2 (global virtual object) set to enable intersheet dependency
  • datasqill-api: renamed to datasqill-api.jar and includes required libraries in correct version
  • module: Java Modul Interface simplified. All Java modules now use bin/runJavaModule.sh
  • module: Java Moduls fire module trigger. Currently implemented in DsModTemplate and DsModRemote
  • module: DsModSendMail
    • added attribute "Mail Body Mime Type"
  • module: DsModTemplate
    • History module can now handle "ignore" and "inplace" columns
    • History module now works with Oracle

datasqill 3.4.7

February 12, 2021
  • backend: Create unique typeNames for Oracle by adding current time in milliseconds
  • backend: Improved Logging on modulelibrary
  • backend: Bug Fix for write to SQTS_REQUST_RESUL of long String(>4k)
  • backend: datasqillDBConnection:
    • If supportsIdentifierCase is set to false in the connection all identifiers are used in lower case
    • If the database does not support transactions, all transaction statements are ignored (commit, rollback, setAutoCommit)
    • Error message if a key cannot be found with mappertool / getkey
  • backend: Singleton handling in ConnectionReader for testing purposes improved.
  • backend: Exception thrown instead of "false" return value on reading connection data by key.
  • backend: datasqillDBValidate use more often try with resource, do a rollback in exasol dependencies and use unique view names
  • backend: draft version for testing a connection are integrated into datasqill library
  • backend: mappertool:
    • Command keyfile simplified and usage of GLOBAL_KEYFILE removed
    • New order for specifying a keyfile. Environment Variable KEYFILE added for this purpose
  • module: DsModMetaData:
    • Logging can now be set on module level for this module
    • Improved exception handling for wrong DB
  • module: DsModWriteExcel:
    • Bug Fix for suppressEmptySheet condition
    • Bug Fix with templateRow (it was not possible to insert data into excel on first cell)
  • module: DsModTemplate:
    • Make use of global variable to do some activity after writing into a table
    • Bug Fix with time format of module_start-dates
    • Logging improved in loops
  • module: DsModRemote: use several schemata from keyfile
  • module: DsModWebServiceClient:
    • added set for custom RequestProperty
    • Logging improved
  • module: First beta of history module for postgres and exasol

datasqill 3.4.6

December 06, 2020
  • backend: Resolve synonym columns in Oracle
  • backend: Detect nullable columns of target tables in Exasol
  • backend: Removed Column next_start_dt from tv_sqts_batch table
  • backend: Null pointer during deployment fixed
  • backend: Catch error when setting autocommit to false. Some lazy drivers do not support the setting of autocommit, because the driver has no autocommit implementation
  • backend: New logging format for all services
  • backend: Repository connection pool is now using the keyfile to retrieve user and password
  • backend: Moved Connection pool from tomee.xml to resources.xml (included in datasqill war file)
  • backend: The scheduluer process is closing every hour the repository connection and purges all idle connection of the connection pool
  • backend: The scheduluer does not use prevIsScheduled and prevIsMissing if the dependency is to the same batch
  • backend: Avoid Merge statement, because of memory leak in h2
  • modules: DsModExecProcedure, DsModRemote, DsModTemplateSQL: Make sure that all database connections are closed
  • module: DsModTemplateSQL:
    • When comparing nullable columns with Exasol use (a = b OR A IS NULL AND b IS NULL) instead of DECODE(a, b, 1, 0) = 1. According to Exasol the Joins will be faster
    • Prepared to generate several statements
    • Use freemarker when validating Loop and Main Query to allow usage of variables
  • module: DsModLoadFlatFile:
    • Logging can now be set on module level for this module
    • Additional Attribute for moving the file after successful load to an archive directory
    • Allow to switch between regular expressions and file wildcard syntax, when selecting files
    • Make use of global Variable home_directory (set in vv_sqts_config or vv_sqts_config_global). If set all source paths are relative to home_directory
  • module: DsModWebServiceClient:
    • Enabled to set https proxy host and port. In keyfile the variables https.proxyHost and https.proxyPort can be set. If both is given the https proxy will be used
    • Logging can now be set on module level for this module
    • New functionality to use basic authentication if a password is given. The password must be encoded
  • GUI: Changes to trans db connections via setting dialog get effective without restart

datasqill 3.4.5

September 09, 2020
  • backend: Bug Fix for batch dependency conditions
  • backend: Bug Fix for load all dependencies views in ActionDependency
  • backend: Use additional action attribute to get the newest change in action
  • module: Bug Fix for the library to get metadata of a query
  • module: Bug Fix for the library added setNull if the object is null. Required when writing in batches
  • module: DsModMetadata: Improvement when opening the source database
  • GUI: Bug Fix for array index exception in NumberTextField
  • GUI: TimeTextField in batch tab replaced by 2 NumberTextFields for hours and minutes
  • GUI: Improvement to calculation of timeOffsetProperty in Batch and handling of user input
  • GUI: new Icon

datasqill 3.4.4

July 24, 2020
  • backend: New Index to speed up insert of batch elements into the Queue
  • backend: New Index to speed up GUI when opening a batch run
  • backend: Bug fix: stay in loop if signal has been received
  • module: DsModRemote: Delete rows from target database using a selected rowset from source database
  • module: DsModRemote: Log amount of deleted rows
  • module: DsModRemote: Nullpointer exception fix with dummy databases
  • module: DsModTemplate: Use columns in insert order
  • module: DsModVirtual: Bugfix with path
  • module: New library function for database writer

datasqill 3.4.3

June 22, 2020
  • backend: Solved Problem with locking of batches by adding new table for next run date
  • GUI: Do no longer show several batch instances in one run view
  • GUI: Show logging after validate

datasqill 3.4.2

April 17, 2020
  • backend: validate now provides same variables as for execution
  • backend: improvements to install scripts
  • backend: validation during deployment now obeys sqdv_delete_flag
  • GUI: Sorting for monitoring of actions improved
  • GUI: Disabled viewing of all batches of a day / recent batches to avoid performance problems
  • module: DsModCompare: Use correct connection in case no reference query
  • module: DsModCheck: Commit disabled for hive
  • datasqill: New Member Alexander in datasqill Team

datasqill 3.4.1

April 7, 2020
  • backend: Fix for problem with deployment.
  • backend: Fix for problem when executing an Oracle procedure.
  • module: Fix in DsModWriteExcel for loop-query validation.
  • GUI: Fix for a security problem with workbench logging.
  • H2: improved resource cleanup; jar file changed

datasqill 3.4

March 27, 2020
  • GUI: Performance improvement for monitoring of runs
  • GUI: Channel output from "stdout", "stderr" and "log" added to validate action result.
  • GUI: Show path added to sheet tab as context menu item
  • GUI: New status "unknown" for objects in validation added
  • GUI: Double click on Recent Runs does no longer open all entries
  • GUI: Folder and sheet items in transformation sub tree now displayed with id
  • GUI: Bug fix for missing hot keys (Select All, Copy, Paste) on sheet after click on grid
  • GUI: Bug fix for reload sheet with no changes after save
  • GUI: Performance improvement on first load of Object dialog
  • module: DsModExecProcedure can now run Exasol scripts
  • module: DsModSendMail debugging information added
  • module: DsModTemplateSQL enabled for looping queries
  • backend: DatasqillMail debugging information added. Add in datasqill-server/conf/logging.properties a line containing "de.softquadrat.datasqill.mail.DatasqillMail.level = FINE" to enable debug output
  • backend: Allow several schema-variables per connection, i.e. schemaList={ds=DATASQILL,sq=SOFTQUADRAT}
  • backend: Fixed problem with open cursors in SheetBuilder
  • backend: Switched to use connection pool for services and scheduler. Tomee no longer uses new connection with getkey
  • backend: Problem "validate sheet locks other service activities" solved
  • services: Timeout for long webservice calls extended to one hour
  • scheduler: Made scheduler more robust and retry with reconnect to database

datasqill 3.3.2

February 17, 2020
  • modules: DsModTemplate use variables
  • modules: logging improved
  • backend: Implementation of variable manager enhanced
  • backend: deploySchema now via webservice
  • backend: First steps to remove nextstartdate from batch table
  • services: ValidateSheet now with given sheet content
  • validate: Use current schema in Exasol

datasqill 3.3.1

December 31, 2019
  • GUI: Added gitCommitDate and gitRevision
  • GUI: New logo added to splash and about dialogs
  • GUI: Save Folders with and without content
  • backend: DeploySheet reworked

datasqill 3.3.0

December 16, 2019
  • GUI: Active keepalive to database
  • GUI: Edit connections and schema
  • GUI: Tooltip now yellow with line breaks
  • modules: msserver dependencies and validate fixed. Position of error can no longer be delivered
  • modules: postgres dependencies enhanced
  • modules: First templates for insert, merge, update, time series
  • modules: central function to prepare a call to a database procedure
  • modules: Freemarker introduced to DsModTemplate
  • modules: Allow SAP access in loops
  • backend: Views reduced to minimum requirements of GUI. Tempspace and statistical versioning columns removed from queue view
  • services: Validate bug fix
  • services: New variable manager in experimental state
  • services: bug fix of calculation if frequency for batch interval was set to 0
  • scheduler: dont restart a batch if one job was killed manually

datasqill 3.2.2

September 09, 2019
  • GUI: Cut between Config and Transformation DB improved
  • GUI: Set user name and ID in database (used by trigger)
  • modules: DsModSendMail, NPE in case value for placeholder is NULL
  • modules: New Module to execute a stored procedure in the database. Currently released for sqlserver
  • modules: Release allocated CLOBs in Module DsModRemote. Required if the loop contains a lot rows
  • modules: further DSRT functions for Bulk write to oracle db
  • backend: Generalized runtimeProcedure to use all customized function in pg_dsrt_interface
  • backend: hive enabled
  • services: Enable Restart Batch if batch instance is in status interrupted
  • services: Validate reworked

datasqill 3.2.0

July 15, 2019
  • GUI: New locking for H2
  • GUI: Show Version in Title removed
  • modules: snowflake enabled
  • modules: DsModRunShell now with parameter
  • modules: mysql enabled
  • modules: mssql enabled
  • modules: DsModTemplate can handle quoted table names
  • backend: new deployment format for database objects in hjson
  • backend: Use everywhere new table sqts_batch_instance_dependency
  • backend: First releases for deploySchema and deploySheet
  • backend: performance improvement for monitoring batches
  • H2: Trigger for versioned datasqill tables
  • H2: start/stop scripts
  • H2: Full support as repository database

datasqill 3.1.0

May 17, 2019
  • GUI: Show Version in Title
  • GUI: Try one reconnect after broken connection
  • GUI: Memory leaks in SQL Area fixed
  • GUI: New menu with developer tools (Get Heap Dump, Show Memory Usage, Run Garbage Collection
  • GUI: About dialog enhanced
  • GUI: Bug fix for double click on tabs and splitpane
  • modules: New module app, which allows to generate in Database Mappings from SQL-Templates with first module for insert from select
  • backend: git backend Folder structure reworked
  • backend: New backend install
  • backend: Modules selectable per customer
  • server: Nano HTTP removed
  • server: Revision: Adjust tomme server port to datasqill port 17491
  • services: Capture system exit calls
  • services: Persist dependencies to table sqts_batch_instance_dependency
  • services: Process control of child processes improved (kill command)

datasqill 3.0

January 31, 2019
  • First complete Version with java FX and H2 experimental