DIRECT v2.0 is a comprehensive overhaul of the widely adopted framework for data logistics process orchestration, serving as the 'control framework' of the data solution. Significant improvements have been made, including the removal of many legacy SSIS artifacts and replacing these with more generic components. The framework has been tested on on-premise SQL deployments as well as Azure SQL Family databases and Fabric.
The primary driver for this new version was to optimize it for Fabric deployment, which has also enhanced its interoperability, making it compatible with other platforms.
New features
- Standardised documentation and procedures: Although DIRECT retains its core functionality, a full standardisation has been applied across all components.
- Batch enhancements: Batches can now be linked to parameters and can also trigger other batches via the new 'Batch Hierarchy' object. The RunBatch function now supports executing child batches.
- Improved DACPAC deployment: A more robust DACPAC deployment process now includes pre- and post-scripts.
- Enhanced logging and debugging: More comprehensive debugging tools have been introduced, including formatted debug logs for easier troubleshooting.
- Regression testing: Automated regression tests ensure DIRECT functions as expected even after changes are applied.
- Success indicators: DIRECT events now return a 'Success Indicator' code, useful for platforms with limited capabilities to handle stored procedure execution. This allows platforms to interpret procedure results and determine the next steps, ensuring process and logic failures can be detected.
- Metadata support: A framework metadata table has been added to facilitate future upgrades.
- Full documentation overhaul
Breaking changes
- Field name updates: DATETIME fields have been renamed to TIMESTAMP (e.g., in omd.SOURCE_CONTROL).
- Indicator terminology: The 'Inactive Indicator' has been renamed to 'Active Indicator' across the system. Now, Active Indicator = 'Y' means the process is allowed to run.
- Status codes: Status codes now use descriptive words instead of single letters. For example, 'E' is now 'Executing', and 'S' is 'Successful'.
- Schema separation: Runtime and static metadata have been separated by schema. Reference values are now housed in the omd_metadata schema.
- Code storage update: Executed code at the module level is now stored in a separate table (omd.MODULE_INSTANCE_EXECUTED_CODE) using a hashed version of the code as a foreign key.
Bug fixes
- A syntax error no longer occurs when setting a Batch/Module relationship to inactive.
Known issues
- Rollback support: Rollback is still only supported in-engine. Future releases will enable the output of Module Instance IDs to allow rollback management externally.
- Event log improvements: Additional work is needed on omd.AddMessageLogToEventLog to fully connect debug and other messages to the event log. A placeholder is in place at the time of writing.