How to analyse results of classification for time series + sliding windows

Here is my context: I have a time series composed of only 1 features. I want to be able to classify between two classes. To get more information out of these data, I am using a sliding time windows. For example, if my time series contains 1000 observations with, I use a sliding window of 20 observations, and I iterate through all 1000 observations. I associate iteration t of my time series with the first value of the window (it t=1: window have t in [1,21]; if t=150, window have t in [150,170]) For each window of 20 samples, I extract additional features (let's say mean and variance for simplicity). Doing so, I have a new data set of size 1000*2. Then, I train a classifier on this data set. I do not use time as a feature. In order to evaluate the model, I have a test set (a time series of 200 observations). I do the same process with the time window to get a data set of 200*2. From there, I compare the true class from predicted class and create my confusion matrix. To summarise quickly the whole process: Transform a time series x of N observations and f features into a training set X of N observations and F features with sliding time window Train classifier with X Transform another time series y into a test set Y (same process as 1) ) Predict class of Y with trained model Create confusion matrix Now, here is my problem: When I get my confusion matrix, I get a lot of false positive. The reason is that if the class are extremely different in the time series, the statistical va…