I'm working on an image retrieval system (not classification). I have 5,000 images as the data set. 500 images of this dataset are the query images used for retrieval evaluation. these 500 images represent 10 different landmarks. the retrieval evaluation requires to evaluate each landmark using the average precision. and then mean average precision is measured to evaluate the 10 landmarks. However, I have a different number of query images for each landmark. some landmarks have 200 (out of 500) images as a query image and some have only 10.

I'm required to divide the 500 query images into 5 folds.

My question is how to perform the k-fold cross validation when the query images for each landmark varies from 10 to 200.

in other words, how to deal with k-fold cross-validation in multiple classes and the sizes of the classes are different.

my work is similar to the evalaution of this paper.

EDIT

as an example: I have 5000 images represents 10 landmarks. I have 500 query images (out of the 5000 images). The query images are as follows:

landmark 1: 50 images (out of the 500).

landmark 2: 10 images (out of the 500).

landmark 3: 70 images

:

landmark 10: 200 images.

I need to measure the retrieval performance for each landmark. The required number of folds is 5. Which means the 500 are supposed to be divided into 5 folds with 100 each.

My question is: how to deal with the query landmarks of different sizes when distributing them across the folds?