One thing you definitely want to do is to setup your validation/testing splits to mimick exactly the situation you are interested in. I.e. are you trying to predict for a new group based on seeing other groups OR will you have some data for a group (in the past) and other groups (it may also matter to ensure temporal ordering there), then you want to predict the future for that group? You'd want to split in the same way that you'd have data available in practice to ensure you evaluation is realistic.

Not doing so can be very misleading: e.g. imagine trying to predict tomorrow's temperature in a town, but I tell you tomorrow's temperature in the two towns to the left and the right (obviously those are great features, especially if you historically see how they differ, but you wouldn't know that information today). Similarly, it is sometimes a lot easier to predict something once you've seen some data for a group, but if in practice you wouldn't have that, you don't want to answer the "hypothetically if I had this data which I won't have" performance question.

That's especially important, because many modern machine learning models (e.g. gradient boosted decision trees aka GBDTs or the less performant older random forrest) don't have really good buildt-in ways of dealing with hierachical data. E.g. in GBDTs you'd often want to sub-sample groups rather then records for each tree similarly in RFs when you do the bagging. Still, very often they still perform perfectly well. Other methods have ways of dealing with groups, e.g. in neural networks you could use embeddings for groups, in traditional regression models you can use random effects for groups. Alternatively, things like target encoding methods can be applied in any setting (but is very finicky to get right without overfitting, while e.g. CatBoost has an in-buildt version with a good reputation).

The split model thing you describe doesn't sound very sensible to me. But, hey, try it with a proper validaiton set up. I.e. if you train a second model somehow on a subset of the data etc. (which sounds like it likely won't be helpful, but who knows), you don't do any such splitting on the data on which you validate, there you apply your two models exactly like you would on unseen data where you don't know any outcomes (so presumably, you'd se some threshold on the predictions of the first model to either decide you know the answer or need the second model, too, and then take the predictions from that). You definitely must not subset / change the validaiton or test data based on the true outcome. If you do this properly, then you'll see how well this split-model thing works.