I am doing a simulation study that involves estimating the parameter $\theta$ under a specific experimental design. $\theta$ is the parameter that take on the value 1 if algorithm A is better than algorithm B, and 0 otherwise. I am repeating my simulation study for $N$ times so that I can count the number of times where the algorithm A performed better than the algorithm B. If the number of times that algorithm A performed better than algorithm B is greater than $\frac{N}{2}$, then I can conclude that $\theta$ is likely to be equal to 1.

But I am not sure how to pick the "right" value for $N$, and someone advise me that I can analyze the Monte Carlo standard error of the probability to determine the value for $N$. However, I am not very familiar with doing this kind of thing, and I am seeking a guidance on this.

The only idea that I can come up with is that if we let $\hat{\theta}$ to be an indicator variable that is equal to 1 if the event occurs, then $\sqrt{\frac{1}{N(N-1)}\sum^{N}_{i=1}(\hat{\theta}_i-\bar{\theta})^2}$ can be used as the Monte Carlo SE of estimate of bias for $\theta$, where $\bar{\theta}$ is the sample average of $\hat{\theta}$'s. However, I am not exactly sure how to use this information to determine the optimal value for $N$. Should I just try varying the value of $N$ and collect $\theta_1 \cdots \theta_N$ from each value of $N$, calculate the Monte Carlo SE for each case, and see if which value of $N$ would minimize Monte Carlo SE?

How can I determine the ideal value of $N$ by doing Monte Carlo? Any guidance will be greatly appreciated.