Gable Blog | 7 Common Data Quality Issues

.avif)

The 7 most common data quality issues

Note: In the real world of data engineering, fixing common data quality issues is much more important than ranking them.

1. Human error

18th-century poet Alexander Pope must have spent some time working at an IT helpdesk, because his famous “To err is human” is prescient as it relates to the hands-down most common source of data quality issues.

• Examples: Incorrectly coded data, miskeyed info during manual data entry, and mistakes made during data transformation processes.
• Impact: In 2021, Gartner survey data showed that the average organization lost $12.9 million annually to poor data quality. Some estimate that the figure is now closer to $15 million. Human error is attributed to 1-4% of that total, meaning these too-human errs may cost organizations $150,000-$600,000 every single year.

2. Duplicate data

Duplicate data occurs when the same piece of data gets entered multiple times. When this happens in blog writing, it drives Grammarly crazy. However, in a database, duplicate data creates duplicate records.

• Examples: A single transaction recorded multiple times, a customer forgetting they’d signed up for a service and signing up again, or two datasets merged from different sources without being properly de-duplicated.
• Impact: Supply-chain and inventory inefficiencies, increased storage costs, misleading data trends, inflated performance metrics.

3. Incomplete data

As opposed to too much of the same thing (i.e., duplication), incomplete data is another common data quality issue where gaps occur in a dataset due to missing data.

• Examples: Incomplete transaction details like the last four digits of a credit card, incomplete product descriptions, or missing customer contact information.
• Impact: Increased risks of fraud, difficulties in marketing personalization, ineffective data analysis, and flawed data-driven decision-making.

4. Inconsistent data

Inconsistent data commonly occurs when discrepancies in data formats, entries, or standards occur. Often due to varying data sources, data inconsistencies can arise across an array of points as organizational data gets recorded, stored, and interpreted.

• Examples: Date formatting, currency symbol variations, and inconsistencies regarding units of measurement.
• Impact: Complicates data integration, hinders data processing, and invites potential errors in data analysis.

5. Inaccurate data

Regardless of how consistent and complete it is, inaccurate data still impacts overall data quality.

• Examples: Incorrect addresses, incorrect pricing information, or outdated inventory levels.
• Impact: Degradations in customer satisfaction, operational inefficiencies, poor decision-making, and diminished trust in data over time.

6. Ambiguous data

Like some expensive TV series currently available on popular streaming services, ambiguous data lacks clarity, precise definitions, or much-needed context.

• Examples: Data fields labeled only as “value” or “score,” data entries noted to be “Pending” with no additional timing information, and acronyms used without being defined.
• Impact: Cross-departmental misunderstandings, flayed analysis, inaccurate or shallow reporting, inconsistencies or errors within integrated systems.

7. Hidden data and dark data

• Examples: Dormant customer data, unused log files, archived emails, historical transaction data, and sensor data from the Internet of Things (IoT).
• Impact: Hidden data can lead to increased costs, inefficient data utilization, data siloing, and compliance issues. Comparatively, dark data results in wasted resources, missed insights, security issues, and data management complexities.

Prevention > cure: How data contracts counteract data quality issues

Many traditional methods and data quality tools are available to data teams looking to address data quality issues. But it’s hard to argue with the logic that the best way to address an issue is to keep it from occurring in the first place. Data contracts, drafted and enforced upstream, are the preeminent solution in this regard, evidenced by how they can eliminate or counteract every single issue on our list:

Human error: Standard data contract drafting involves working with stakeholders and data consumers to standardize data entry procedures.

Duplicate data: Data contracts can enforce deduplication processes.

Incomplete data: Contracts can specify mandatory fields and establish protocols for handling missing data.

Inconsistent data: Data contracts mandate data format consistency across all data sources.

Inaccurate data: Contracts that include accuracy checks can verify data correctness.

Ambiguous data: Data contracts crystallize definitions and context for data fields.

Hidden and dark data: Well-drafted data contracts outline data management and usage protocols, ensuring that all data collected is easily discoverable and usable.