Why do businesses engage in data quality programs in the first place? Often, efforts revolve around getting data to a state of fitness that is aligned with operational and transaction processing needs but these may also be tied into a more importance compliance or industry mandated requirement that is tied into regulation or risk reduction and penalty avoidance.
The financial services sector as a whole, for example, has a whole set of challenges tied into the requirements around financial regulated conduct, but they also have all these other compliance hoops that they need to periodically leap through that relate to privacy and data protection regulations. In addition, in the financial services sector, you’re often dealing with someone else’s assets or liabilities and so there is a particular level or accuracy, precision, confidentiality and control that is implicitly expected to be in place to minimize loss, manage risk and provide brand and product assurances.
GDPR, of course, is top of mind in Europe for almost every organization, but there is a simmering set of privacy and compliance challenges in the US where it is expected the exact same kinds of exacting regulation will kick in at the State and potentially at the federal level.
Accountability vs responsibility
With any of these requirements, whether it be accounting policies, financial conduct or citizen privacy issues, the question that has to be asked, is who is responsible? The leaders of the organization are accountable but who typically sponsors, mandates and ensures adherence?
It could be operations, accounting, risk, IT, or even a dedicated compliance team. The specifics of who mandates the introduction of a DQ program is perhaps less important than the overarching sponsorship, the dedication to the program and the commitment of resources. While IT may execute, the highest level of corporate sponsorship and endorsement is critical for the program to have the necessary commitment within the organization.
A good program goes a little further, in as much as it empowers and enables good DQ practices, measures and goal setting. IT may own the infrastructure, the technology, the assurance pieces of availability etc but they cannot realistically own the data or the process that is applied to gathering and maintaining the data.
This is a pretty tough situation to be in, in IT, particularly if it is impossible or at least very difficult to get anyone in the business to recognize and resolve data quality issues or own any of the issues.
The flow of new records and data change will inevitably be continuous, depending on the velocity of the business. The challenge is setting up measures and controls to identify and capture badly entered or captured records and defining , who is ultimately responsible for remediation?
Not an IT problem
Data quality is not necessarily an IT problem even though they often run the tools for collection. If the technology is weak or deficient in some controls, the problems may initially lie with IT.
There is a balance needed here, more controls mean slower or potentially blocked data entry when controls are over indexed. When I worked for one of the large mobile carriers in the US, the insistence on the web sales side, was that every attempted shopping cart checkout needed to end in a success message for the customer, even if they failed contact verification, credit approval or had a misshapen or faulty shopping cart.
The rationale, at the time, was that a captured lead was better than an abandoned cart. I cannot say whether this was a good or a bad approach, all I can say is that when we had high stakes promotions and deals that were incredibly attractive, the number of broken checkouts that went into the triage bucket was incredible, and hard to wade through and resolve. It straddled an IT and business problem but in the end the business had to make the call.
When integrations fail
Businesses need a degree of dynamism in the data quality dial-up. Something that allow flexibility even when the integration is tight, between systems.
There has to be an implicit expectation that specific data attributes be correct and fully provided up front when integrations ae built. If anything is amiss, you’ll introduce problematic data that will need identification, triage and remediation or resolution.
Often IT will suggest that they cannot get the business side to take ownership and so they have to implement processes of saving or discard with manual intervention. If inbound records have insufficient information to support an investigation to resolve, they are simply discarded without logging or history. This may be a good approach for some industries, but for say a financial services customer this may be a very bad decision.
Some industries, like charity organization often aren’t taking any money until there is an explicit commitment, the same is also true of shopping carts, if records are not attached to pledge or financial undertakings (like a credit card charge) then discard for incomplete items seems the most pragmatic and appropriate course of action. Increased frequency of discard may however point to something else going on in the business or the technology stack that needs attention though, so assuming that the data itself is where the problem start and ends may not be exactly the right approach. There may be a need to do some sort of root cause analysis of anomalous results especially if they are seen to be bucking a trend.
Measuring what matters
The same cannot be said to be true for a business that holds investment funds or deposits. The same is likely also not true for a business that has debts or is owed money by debtors. Appropriate attribution and collection are fundamental.
The escalation of data quality issues to designated data ownership at the organizational level, needs to happen fast and in a regularized way and that can only happen if you have a suitable DQ program in place.
Data quality and data governance cannot, Irrespective of the size of the organization the amount of the data and the characteristics of the business be the responsibility of a few select elites, , data quality is a company or organization-wide responsibility.
The governance and determinants of what constitutes good or acceptable data must be a “fitness for use” based decision that should be taken by the users of the data within the business. This cannot be IT going it alone.
IT doesn’t use the data for anything except creating relationships between data and data sets. The management of the technologies that assist with data governance and data management can be the responsibility of IT but the underlying data itself must be managed, owned and governed by the business.
About the author
Clinton Jones has experience in international enterprise technology and business process on four continents and has a focus on integrated enterprise business technologies, business change and business transformation. Clinton also serves as a technical consultant on technology and quality management as it relates to data and process management and governance. In past roles, Clinton has worked for Fortune 500 companies and non-profits across the globe.