Critical. Authoritative. Strategic.


CBR is proudly produced & published
by Technews
Issue Date: November 2004 (es)

Data profiling delivers rapid reporting times and high ROI

November 2004

Data profiling, a necessary first step in every data integration project, can relieve IT organisations of what is otherwise a tedious, time-consuming and error-prone manual process, and deliver a huge return on investment, says Estelle De Beer, BI practice manager at Sybase SA.

Demand for business intelligence is growing. Regulatory requirements are increasing the need for reporting and accountability, cost and risk reduction are driving closer scrutiny of non-compliance or fraud, and risk mitigation is a top priority for data management specialists.
It is estimated that, in the typical business application environment, data is growing at a rate of 125% a year. By the end of 2004, Gartner predicts companies will be managing 30 times more data than in 1999. In a typical enterprise application environment, you will find anywhere from 6 to 15 copies of production data. According to the Standish Group, 72% of data migration, integration and consolidation projects significantly overrun or fail.
Traditional, manual methods are falling short of demands. Skilled IT consultants spend valuable time on manually verifying data quality. The manual analysis of a single attribute typically requires between three and five hours. Using automated data profiling on the other hand, an attribute can be profiled in as little as 5 to 10 minutes by an experienced user, or in 20 to 30 minutes by a novice user.
Using manual data profiling, at an employee cost per hour of R110 and five hours/attribute, the cost per attribute is R550. If there are 2500 attributes to be profiled, the total cost is R1 375 000. With automated data profiling, the time taken is reduced to 30 minutes/attribute, so the actual cost is R137 500.
This results in a return on investment, after investing R650 000 in a data profiling tool, of R587 500 for the first project. The ROI for every additional project of the same size is R1 237 500.
A data profiling solution can help companies gain a complete and accurate understanding of source data and minimise risk in projects dependent upon that data.
It can also:
* Accelerate project completion.

* Dramatically reduce both expected and unexpected (overrun) project costs.

* Increase the quality of project results.

* Help gain a thorough understanding of corporate data quality, content, relationships and structure prior to engaging in data migration, integration or consolidation projects.

* Correct inconsistencies, redundancies and inaccuracies in corporate databases - the essential first step in any successful data-driven IT initiative.
Many products help companies standardise data, move it, handle transaction data, or migrate it from databases and systems to data warehouses. If any of this work is based on a weak foundation of incomplete or inaccurate information about source systems, the project will likely suffer delays, cost overruns, headaches and failure.
One of the challenges in managing data quality is that valuable business data is trapped in disparate, proprietary systems; corporate databases and systems are often inaccurate; and documentation regarding data and business rules is outdated and inaccurate.
IT can spend months or even years trying to integrate or use legacy data without having a thorough knowledge of the data content, structure and quality.
Understanding data sources is difficult because manual data profiling is a tedious, slow, labour-intensive and error-prone process; metadata documentation may be missing, incomplete or badly out of date; source code for legacy systems may no longer be available; relationships between data elements are not always obvious; assumed relationships and dependencies may be wrong; databases are not static; and managing data successfully from these disparate sources requires having a thorough source system knowledge.
A data profiling solution:
* Delivers accurate source system knowledge.

* Improves corporate data quality and accuracy.

* Enables accurate data migration, integration and consolidation.

* Helps integrate multiple, disparate data sources.

* Mitigates risk in data management projects.

* Minimises overruns in enterprise applications projects.

* Improves productivity of data management projects.

* Improves merger and acquisition data integration success.

* Reduces costs.

* Allows data warehouse consolidation to be completed on time at a fraction of the cost that would have otherwise been incurred.

* Powers a constant process of automated data profiling and quality management that assures consistent, reliable data within any system, like a massive enterprise data warehouse.

* Gives data managers a continuous stream of standard reports on column profiling, redundancy profiling and metadata to help pinpoint potential problems and prevent data quality errors before they become an issue.

* Allows business units across the enterprise to view data quality profiling and assessment reports before they employ data warehouse content to make business decisions or execute business activities based on the data.

* Gives immeasurable ROI - millions can be saved on the cost of manual profiling and data assessment, and a substantial reduction in business risks achieved.
A data profiling solution typically most benefits larger companies that have disparate systems. It also helps those organisations with large volumes of data that has been in a system for a long time and needs to be updated quickly, especially in terms of complying with legislation.
It is especially useful in medium to large companies in highly competitive industries such as financial services, telecommunications and healthcare. Data migration and consolidation, enterprise application integration and e-commerce, and CRM and customer loyalty projects can all benefit from data profiling.
Estelle De Beer, BI practice manager at Sybase SA
Estelle De Beer, BI practice manager at Sybase SA
For more information contact Estelle de Beer, Sybase SA, 011 804 3740,

Others who read this also read these articles

Search Site

Search Directory

  • Search for:


Previous Issues