How to evaluate performance of classification model for different subsets of classes?

Consider a classification problem where there are N classes. While this may seem strange, I have a model that processes features, and essentially, evaluate which classes are impossible (or near impossible) to classify correctly based on the features. Therefore, we can exclude these classes when making predictions because we already know we fail in such cases (in other words quantify when model is useless). I want to evaluate how well this, but simply evaluating accuracy isn't a fair comparison. Consider I am unable to classify all but 2, then I could compare the accuracy for the classes I "can classify" and "cannot classify". However, I will have 50% for the classes I "can classify" by just randomly guessing in this case. Maybe, it appears as a good improvement, but a random guesser has high accruacy relative to the number of classes. What metrics can be used to evaluate such cases? There must be a metric for this kind of scenario.