In the first evening of my 6-week course on Data Quality Improvement I try to communicate the high cost of poor data quality (PDQ) to my students. Their eyes are usually starting to close while I cover the course administrative details as for most of them it has already been a hard week of working their full-time jobs and caring for their family. And now they are expected to stay awake during this course that is required for their certificate in Data Analytics or Database Administration.
I can usually catch their attention when I ask them to guess what percentage of the average organization’s annual operating revenue is wasted due to poor data quality. A couple of them will mumble out a guess, which I encourage, and as the guesses come in they range from a conservative 1-2% to the wild-and-crazy extremes of 50-75%, which usually draws a giggle from the rest of the class.
Before I share the actual percentage to the class, I explain more about how the source for the PDQ cost is the Larry English ’s book “Information Quality Applied ” that had just come out in 2010 when I took some Total Information Quality Management (TIQM) courses from him in Tennessee. Larry was the creator of TIQM, and a recognized expert and international evangelist for improving data quality, so I was inspired to learn first hand from him …
Towards the end of his compilation of research on the costs of poor data quality to organizations, Larry’s book states :
“The costs of Poor Information Quality – based on updated research and measurement – are from 20 to 35 percent of an organization’s operating revenue wasted in recovery from process failure and information scrap and rework.”
So, 20-35% is just a number, and some students jot it down in their notebooks – but most are not surprised by that percentage. In fact, I have had students say they think it should be higher based on their experience! The reason these students think that the percentages might be even higher is because of what they see in their organizations from their low to middle level positions, where they are often hands on with the front line collection of data and the operational decision making based on that data. My students are taking night classes to improve their careers, but will they take their real world understanding of the quality of the organization’s data with them as they ascend the corporate ladder. Will their perception change of that stark statistic of 20-35% of their organization’s operating revenue being wasted every year due to poor data quality? I must imagine what the executives might think of this because in my 5 years of teaching Data Quality Improvement there has never been a C-level executive take my course so they don’t get the chance to share their perspective with me and the rest of the students.
To better estimate the costs of poor data quality in your organization, you can follow these 5 simple steps.
From my 30+ years experience in the BI/DW world I have observed that the executives do not always understand how difficult it is for their staff to prepare the pristine reports for them every month. These perfect reports – where there are no glaring glitches because the regular data quality problems have been massaged and filtered out manually yet again. Perfect reports with no differences because those differences were ironed out by those everyday reconciliations continuously being completed in the background, with the inefficiently wrangled results incorporated into the pristine reports. Perfect reports that are ‘messy or difficult’ to drill down to the source data, which is nearly always a signal of the inefficiencies that waste their employees time every single month and will continue to be an invisible drag on their organization for the foreseeable future unless something is done.
Larry suggests that one of the causes for this disconnect is that “Western management tends not to go to the Gemba (the Japanese term for the ‘Real Place where the Value Work takes place‘) to observe the information processes and find and fix the Root Causes of the broken processes.” The executive cannot be expected to understand the full costs of providing the information to their organization’s decision-making processes unless they have measured those costs.
Organizations that do measure data quality understand that reducing those costs gives them a competitive advantage over rivals who do not continuously improve their provisioning of low-cost, timely and accurate information.
Would you agree? Do the executives in your organization underestimate the full costs of data quality?
