Breast Cancer Wisconsin Deduplication Analysis

699 raw rows compared across exact-row, features-only, and ID-only duplicate definitions

Raw Dataset Size
699
Reference baseline
Rows read from the source CSV before any deduplication.
Cleaned Dataset Rows
691
8 exact duplicates removed
This matches the full-row deduplication result in the cleaned export.
Features-Only Survivors
463
236 rows would be dropped
Ignoring ID creates the most aggressive reduction because many records share the same measured features.
Repeated IDs
46
54 duplicate-ID rows
Most repeated IDs appear twice, but a few recur up to six times.
What the comparison shows

The cleaned dataset removes only exact duplicate rows, reducing the raw file from 699 rows to 691. That approach is conservative: it preserves records when the ID or any clinical feature differs.

Two alternative duplicate definitions behave very differently. If duplicates are defined by identical clinical features while ignoring ID, 236 rows become redundant and only 463 rows survive. If duplicates are defined by ID alone, 54 rows are redundant and 645 survive.

Repeated IDs are not automatically interchangeable records: 39 of the 46 repeated IDs have more than one distinct feature profile, and 4 repeated IDs span both class labels. That means the ID-only rule can collapse clinically distinct entries.

Exact-row drop rate: 1.1%
Features-only drop rate: 33.8%
ID-only drop rate: 7.7%
Key takeaways
  • Full-row matching reproduces the cleaned file exactly and removes only 8 rows.
  • Features-only matching removes nearly one-third of the dataset, so it is best treated as a strong normalization assumption rather than a routine cleanup step.
  • ID-only matching is less aggressive overall, but it can still merge rows with different feature values and, in a few cases, different class labels.
Scenario comparison: surviving rows vs dropped rows

Each bar uses the same 699-row baseline so the effect of each deduplication rule is directly comparable.

How often IDs repeat in the raw dataset

Bars count repeated IDs by frequency; the overlaid line shows how many extra rows those repeats contribute under the ID-only rule.

Top repeated IDs: occurrences vs distinct feature profiles

Bars show unique feature profiles per ID, while the line shows total rows for that ID. When the bar equals the line, every reuse introduces a different feature pattern.

Most repeated IDs in the raw data
ID Rows Distinct Profiles Class Values
Exact duplicate groups removed in the cleaned dataset
ID Rows in Group Class