In general, we won't know: it is an undecidable problem.

For reference, Bashir et al. and Sehra et al. proved that for a given dataset, (even if given unlimited training time) it is undecidable whether an encodable learning algorithm will always overfit or underfit respectively. So we can never know if we have got the 'right' model (in terms of generalization error).

FYI, This problem is related to the halting problem and minimum description length (MDL).