During the ERP transformation at Handel Schweiz it became clear that data quality is a critical success factor for correct invoicing, reporting and efficient operational processes. Duplicates in company addresses and contacts were a particular challenge, as they do not always appear as exact one-to-one duplicates but often contain small deviations, different spellings or supplementary information.
CONSENSO TECH developed a concept and a prototype for intelligent duplicate detection. Addresses and contacts are converted into a mathematical form and compared pairwise using methods such as TF-IDF and cosine similarity. This makes it possible to identify similar records where classic duplicate checks reach their limits.
The approach enables a more flexible and more precise data quality check than plain standard matching. Potential duplicates can be scored by similarity and then processed further through workflows, UI-supported cleansing or reporting.