Take a wind farm operator, for instance. When sensor data about the turbines is generated, it comes from a particular turbine at a particular position in a specific wind farm at a known location, running at a specific speed in specific weather conditions, at a particular time. “If you have other turbines also working in the field, the performance of your turbine will go down, but the field performance will go up,” says Aronchick. “The performance of your turbine going down isn’t a negative, but unless you have the context at the point of data collection, you’re going to make your life much harder later on, when someone asks about the efficiency.”

Metadata needs to be much richer, and it needs to be added as early in your data pipeline as possible when you have the most detail available to make sense of the structure and complexity of the data, Aronchick adds. “You want to capture as much about the data you’re collecting as possible, where it doesn’t require insane activity to do so.”

But not all the metadata you need will be generated with the data, he says. You almost certainly need to augment and annotate your data, and provide extra structure, especially for something like a point of sale system with very light metadata. “Data comes off these things in poor structure,” he says. “It’s not OpenLineage, it’s often a CSV or a text record, and you have to reconstruct them into a full structured log. So do smart things where you’re creating data. That might be compressing, sampling, converting, appending metadata to it, and enforcing schema and lineage all before you start moving anything.”