The didmos module ETL Flow (Extract, Transform, Load, Workflow) extracts data from different sources – for example ERP, SAP, XML or SQL databases – and synchronizes them in a central metadirectory. Crucial processes are the identification of data based on weighted attributes (duplicate detection), and the integration of them to one dataset (data harmonization).
Workflow and functions
The workflow of didmos ETL Flow is configured via LDAP objects, enabling a parallel processing of tasks. Their parameters and values can be assigned among each other, be kept in the memory and/or be filed in a dataset. Even in case of an error conditional jumps to other tasks are possible.
A typical workflow can be as follows:
- reading the data sources (e.g. via an SQL query)
- converting into DSMLv1 as intern standard format
- preparing via XSLT
- identifying the data in the Identity & Access Management system
- if necessary, combining redundant data in one data source
- combining multiple data sources (detecting redundant values after prioritizing the data sources)
- reading the Identity Management core
- calculating the difference
- importing the changes
ETL Flow consists of individual submodules for recurring tasks, i.a. connectors (conversion modules, various system tools), identifiers, Xsltransform, merger/intramerger, Dsmldiff and workflow.