First, let’s clarify terminology: stochastic gradient descent means doing update one sample at a time, if you use batches it’s batch gradient descent, if you train on all data at once, it’s just gradient descent.
- How does this batching work? Do I randomly pick one data point to train from each batch? Meaning, after the end of first run, I will have one estimate.
You make an update using all the data in a batch, same as in gradient descent you would use all your data.
- Is it possible that I will have 4 different estimates after having 4 different runs? Should I choose whichever gives the smallest error?
You are splitting the data randomly to batches, so yes, results may differ between trainings. If you train long enough they should converge. Moreover, with gradient descent you usually initialize the data randomly, so for this reason alone you could get different results, even without batches.