Fix, validate and deduplicate — with a change log.
Cleaning makes a dataset usable without hiding what changed. Every fix is a logged, reversible operation tied to the rule that triggered it.
How a cleaning engagement runs
- 1
Profile
Measure completeness, types, ranges, duplicates and encoding problems per field. The profile is delivered to you before any rule is written, so you see the problems in numbers rather than anecdotes.
- 2
Agree rules
Write validation and repair rules with you. Nothing is changed by default. Each rule has an id, a plain-language description, a severity and an action: flag, repair, or exclude from delivery.
- 3
Apply and log
Run the rules; each change is recorded with the rule id, before and after values, the record id and the rule version that applied.
- 4
Deliver and monitor
Deliver the cleaned set with its change log, and re-run on each update. A new source version produces a new change log and a comparison against the last run.
Typical cleaning operations
Schema validation
Types, required fields, allowed values and referential checks.
Deduplication
Exact and near-duplicate detection, with merge decisions logged rather than silent.
Encoding and OCR repair
Character-set fixes and common OCR artifacts, flagged where confidence is low.
Outlier flags
Values outside agreed ranges marked for review, not deleted.
Date and format sanity
Impossible dates, future dates and mixed formats caught before they reach a filter or a chart.
Cross-field consistency
Checks that related fields agree, such as a court belonging to the stated jurisdiction.
What each logged change records
The change log is a dataset of its own. You can query it, audit it, and reverse any change by replaying the before value.
| Field | Contents |
|---|---|
| record_id | The source record that was flagged or changed. |
| field | The field affected, or the whole record for exclusions and duplicate decisions. |
| rule_id + rule_version | Which rule fired, and which version of it. |
| action | flag, repair, exclude, or cluster (for duplicates). |
| before / after | The original value and the repaired value. Flags have no after value. |
| confidence | For heuristic repairs such as OCR fixes, the confidence that triggered the change or the review. |
| run_id | The cleaning run, tied to a dataset version and a timestamp. |
Duplicates in a legal corpus
LAW.co's corpus contains some cases imported under more than one id. A cleaning pass would cluster those duplicates and choose a canonical id, but keep every original id resolvable, so citations into the old ids still work.
The decision itself is data: which ids were clustered, on what evidence (matching case name, court, date and citation), and which id was chosen as canonical and why. If a cluster turns out to be wrong — two different cases with similar names — it is split by reversing one logged decision.
That is the general pattern: cleaning adds a decision layer on top of the source rather than destroying information.
What you receive
- A data profile per field, before and after cleaning
- The rule set, versioned and in plain language
- The cleaned dataset with source values recoverable
- A queryable change log for every flag and repair
- A review queue for low-confidence repairs
- A re-run on every source update, with a diff
Questions buyers ask
Do you delete bad records?
Not by default. Records failing validation are flagged and can be excluded from delivery, but the originals are retained unless you ask otherwise.
Can cleaning rules run on every update?
Yes. Rules are versioned and re-applied when the source updates, and the change log is regenerated.
Can you clean data that a provider licenses to us?
Only within that provider's terms. Cleaning a licensed dataset produces a derived layer, and some licenses restrict derivatives or redistribution. We check before starting.
Do you fix the provider's copy?
No. Cleaning is applied to your delivered copy or as a layer on top of it. Systemic problems we find are reported back to the provider so they can fix them at the source.
How do you handle ambiguous repairs?
They go to a review queue instead of being applied. You or a reviewer you designate accepts or rejects each one, and the decision is logged.
What sample do you need to scope the work?
A representative extract and a description of the problems you see. We profile it and propose rules before quoting.
Can cleaning feed a RAG index or an agent?
Yes. Cleaned records keep their source ids and provenance, so an index built from them can still cite the original record, and excluded records simply never reach the index.
Have a messy dataset?
Send us a sample and the problems you see. We will profile it and propose rules before changing anything.