Cross Validated
2026-08-16 20:52 UTC
By Jak Potvin
AI-113-20260816-social-media-14253098
Is a deterministic linkage gate the honest choice when no labelled data exists, and can its false negative rate be bounded at all?
I am linking two public datasets of vehicle incidents published by the same regulator (NHTSA) through entirely separate channels. One is incident filings submitted by manufacturers under a standing order. The other is consumer complaints filed by vehicle owners. Neither source publishes a key identifying an individual vehicle or an individual event, and no external record states which filing corresponds to which complaint. There is no labelled data and no way to obtain any. The shared fields Reporting entity name. Free text, formatted differently in each source, so it needs normalization and fuzzy comparison. An 11 character VIN prefix. WMI, VDS, check digit, model year, plant. Characters 12 to 17, the serial identifying one specific vehicle, are published by neither source. A shared prefix means "same configuration, model year and plant", which thousands of vehicles share. Incident date. Filings publish month only (for example "APR-2026"). Complaints carry a full date. The finest shared resolution is the month. What I built A deterministic gate. Two records link only if all three hold: same reporting entity after normalization, same 11 character VIN prefix, incident months within 1 of each other. Entity name similarity is banded at 90 and above for an automatic merge, 80 to 89 for a flagged candidate never merged unattended, below 80 rejected. No similarity score overrides any of the three conditions. The design deliberately biases toward non-linkage, because a false link f…
I am linking two public datasets of vehicle incidents published by the same regulator (NHTSA) through entirely separate channels. One is incident filings submitted by manufacturers under a standing order. The other is consumer complaints filed by vehicle owners. Neither source publishes a key identifying an individual vehicle or an individual event, and no external record states which filing corresponds to which complaint. There is no labelled data and no way to obtain any. The shared fields Reporting entity name. Free text, formatted differently in each source, so it needs normalization and fuzzy comparison. An 11 character VIN prefix. WMI, VDS, check digit, model year, plant. Characters 12 to 17, the serial identifying one specific vehicle, are published by neither source. A shared prefix means "same configuration, model year and plant", which thousands of vehicles share. Incident date. Filings publish month only (for example "APR-2026"). Complaints carry a full date. The finest shared resolution is the month. What I built A deterministic gate. Two records link only if all three hold: same reporting entity after normalization, same 11 character VIN prefix, incident months within 1 of each other. Entity name similarity is banded at 90 and above for an automatic merge, 80 to 89 for a flagged candidate never merged unattended, below 80 rejected. No similarity score overrides any of the three conditions. The design deliberately biases toward non-linkage, because a false link f…
Full article content could not be extracted automatically. Read the original below.
Source:
Cross Validated
· stats.stackexchange.com