The didmos module ETL Flow (Extract, Transform, Load, Workflow) extracts data from different sources – for example ERP, SAP, XML or SQL databases – and synchronises them in a central metadirectory. Crucial processes are the identification of data based on weighted attributes (duplicate detection), and the integration of them to one dataset (data harmonisation).
didmos ETL Flow
ETL Flow
Workflow and Functions
The workflow of didmos ETL Flow is configured via LDAP objects, enabling a parallel processing of tasks. Their parameters and values can be assigned among each other, be kept in the memory and/or be filed in a dataset. Even in the case of an error, conditional jumps to other tasks are possible.
A typical workflow could be as follows:
- reading the data sources (e.g. via an SQL query)
- converting into DSMLv1 as intern standard format
- preparing via XSLT
- identifying the data in the Identity & Access Management system
- if necessary, combining redundant data in one data source
- combining multiple data sources (detecting redundant values after prioritizing the data sources)
- reading the Identity Management core
- calculating the difference
- importing the changes
Structure
ETL Flow consists of individual submodules for recurring tasks, i.a. connectors (conversion modules, various system tools), identifiers, Xsltransform, merger/intramerger, Dsmldiff and workflow.
