D# contains a method and supporting tools for agile data warehouse development. The method is based on needs-oriented modeling of operational concepts. At the same time, the concept model is a common language for all parties, an important defining document and a starting point for the technical solution of the data warehouse.

The central tool of D# is the D# Engine, which is used to automate a significant part of the data warehouse development work. In practice, the D# Engine generates the standard SQL code needed for the database structures, loading mechanisms and interfaces based on the conceptual model completely automatically.

Overview

Business-oriented Conceptual modeling is a fundamental part of D♯. The model elements are complemented with appropriate metadata (including source system mappings), after which the D♯ Engine tool can generate the entire technical implementation of the model, that is, the Data Warehouse. Not quite magic, but rather fully transparent deterministic automation based on a carefully selected minimal set of complementary information superimposed on the model. For all practical purposes, it’s about achieving more by doing less, in as many situations as possible.

Conceptual Modeling as part of D# Toolbox

Modeling is done with a dedicated modeling tool such as Visual Paradigm (UML Object Model modeling style) or Ellie (Entity-Relationship modeling style). As in the screencaps, a Conceptual model has been created using Ellie and then imported to D♯ Engine. The modeling tools also support the additional metadata needed to automatically generate the Data Vault.

D# Engine Generates all Data Vault Structures

Fed with the Conceptual model and source data mappings, D♯ Engine generates a Data Vault 2.0 structure that can be populated by the source data. Stored procedures are generated that perform hashing, loading and other operations that are needed to keep the data in order. Additionally, the (possibly multisource) data is automatically collected, merged and presented through a set of views built on top of the Data Vault structure. Each view perfectly represents a class/entity in the Conceptual model, so the developer can navigate the view structure using the original Conceptual model as a map. So, for those who don’t specifically need it, the Data Vault structure is conveniently hidden behind the view structure.

D# Engine Deploys the DW to Production

All changes and additions are safely and incrementally installed. No manual editing is needed. After installation, data can be loaded into the DW. All code will run natively on the target platform, so the presence of D♯ Engine is not needed for this. A set of default source system specific orchestration procedures that execute the individual hash and load procedures are automatically provided by D♯ Engine, but you may also build your own using whatever technology suits your situation.

D# Engine Excels in Everyday DW Maintenance

While the deployed solution will operate standalone within its target platform, D♯ Engine can be used for daily maintenance and debugging tasks with its ad hoc SQL-generating features. The Conceptual model operates as the “user interface” to the underlying Data Vault table structure, so you can work with the original classes instead of the individual tables. In fact, you don’t even need to know about the actual table structure in order to get to the data.

D# Engine is Customizable

Nobody likes black boxes. The main principle of D♯ is maximal openness. The entire user interface as well as user commands are scriptable (and are indeed scripted) using the C♯ programming language, and all of the functionality can be tailored by the user to replace default functionality. SQL code generation is based on editable templates, and the templates are populated by editable C♯ code. It doesn’t get more flexible than this.

DSharp Engine

by DSharp Oy