Is my complaint classification task even learnable? How can I prove it’s not?

I'm building a model to classify Arabic complaints as either correctly or incorrectly routed. But here's the catch: The label is assigned after a human specialist reviews internal systems, contacts companies, and uses knowledge not available in the text. So even if the text sounds reasonable, only the specialist knows if it's misrouted — based on coverage rules, communication history, etc. I trained a model with text + metadata (company, product, etc.) and got good metrics (F1 ≈ 88%), but I can’t trust them, because I can’t verify if the model is really learning something meaningful. My question: 💡 What are the steps or experiments I can do to figure out if this task is even learnable from the available data? I don’t want to keep improving a model that’s fundamentally flawed. Thanks in advance 🙏