I am attempting to compare a few methods for multi-class classification using caret: 'multinom' (logistic regression), 'nnet' (neural net), and 'svmPoly' and 'svmLinear' (two types of support vector machines). As expected, when I randomly generate data, 'multinom' and 'svmLinear' give me roughly chance 10-fold-validation accuracies (1/6 = .1666...). But 'nnet' and 'svmPoly' give above-chance cross-validation accuracies.

The code below generates random datasets 100 times, each time training a a 6-class classifier and collecting 10-fold cross-validation accuracies. I have set caret's train function to perform PCA (including centering and scaling) on the data, keeping only the top 36 of PCs (i.e., 10% of the number of observations/rows - a number low enough to keep the model full rank).

Also below is a series of histograms plotting the frequency of the model's cross-validation accuracies across 500 simulations for each of the 4 methods. Because the data are randomly generated, accuracies should form normal distributions around chance (i.e., 1/6 = .1666...). Only the multinom method does this, with mean accuracy = .1667. 'nnet' gives mean accuracy = .1923 (which is significantly higher than multinom: t(1577)=30.3, p<.0001); 'svmPoly' gives mean accuracy = (also higher than multinom, p<.0001); and 'svmLinear' gives mean accuracy =.1652 (not significantly different from multinom p=.23).

Needless to say, this shouldn't be possible. Can anyone help me understand what's going on? Thanks!!

### Packages
library('caret') # for machine learning functions
library('nnet')

### Hard code metadata
n.classes = 6 # 6 classes to decode
n.observations.per.class = 60 # number of rows per class
n.cols = 118 # number of features/columns
n.rows = n.observations.per.class * n.classes # total number of rows
n.pcs = floor(n.rows / 10) # keep this number of PCs
n.sims = 100 # number of simulations to run per method

### Classification methods to try:
methods = c('multinom','svmPoly','nnet','svmLinear')
### Initialize storage
accuracies <- data.frame(matrix(NA, nrow=n.sims, ncol=length(methods)))
colnames(accuracies) <- methods

### Classify
# 100 simulations (or whatever n.sims is set to)
for(i in 1:n.sims){
  for(method.loop in methods){
    # Create simulated data
    sim.data <- data.frame(matrix(data = rnorm(n.cols * n.rows),
                                  nrow = n.rows, ncol = n.cols))
    # Add Y labels (6 classes: A thru F)
    sim.data$Y <- rep(c("A","B","C","D","E","F"),
                      each = n.observations.per.class)
    
    # Classify
    current.classifier <- train(Y ~ .,
                                data = sim.data,
                                method = method.loop,
                                preProcess = "pca",
                                trControl = trainControl(preProcOptions = list(pcaComp = n.pcs),
                                                         method='cv', number=10,
                                                         allowParallel = FALSE),
                                verbose=FALSE)
    
    # Get accuracy
    best.result <- current.classifier$results[which(rownames(current.classifier$results) == rownames(current.classifier$bestTune)),]
accuracies[i, method.loop] <- best.result$Accuracy
    
    # Clean up workspace
    rm(sim.data, current.classifier, best.result)
  }
  # Status update
  message("Loop ",i," of ",n.sims," complete.")
}

# Print mean accuracies:
colMeans(accuracies)

# Statistically different?
t.test(accuracies$nnet, accuracies$multinom)

Histograms of cross-validation accuracies for each method; black dashed line = 1/6 (chance) and solid green line = mean accuracy over the 1000 simulations. Results of simulations

Edit: The same problem occurs for binary classification and without centering, scaling, and PCA. This means that the problem is not related to pre-processing or the fact that the classification problem is multi-class. (To create the image below I only minorly changed the code above: I created only 2 classes -- A and B, with 180 observations/rows each; I generated only 36 features/columns (rather than 118); and I got rid of the preprocessing options in the train() and trControl() functions.) I have also tried setting linout to TRUE for 'nnet' but the results are the same. Binary classification