Say I have a binary response variable, Y, that I model using a logistic model with four predictors, A, B, C and D. To make matters concrete, imagine that Y = 1 designates a respondent registering support for something, and 0 an absence of support.
Having estimated the relevant parameters on some sample, S, I then want to see what proportion of 1s (i.e., support) I likely would have seen, had all observations in S taken on a particular value on A. Assume conditions for causal inference are satisfied for A, so that changing its value can be thought of as a (hypothetical) intervention.
So I create a "new" sample, S*, identical, to S, save for each observation taking on the desired value on A. I then use the fitted model to “predict” the probability of Y = 1 for each observation in that sample. Taking the mean of those predictions I get an estimated proportion of support under the relevant intervention.
My question is: how should I quantify the uncertainty of that estimate? I can think of three ways, but am not sure which one (if any) makes sense:
- Resample from the predicted probabilities of the model and bootstrap a confidence interval for the relevant mean that way.
- Calculate a confidence interval for the prediction made on each observation (the probability that respondent 1 registers support, etc.) like here, and then create a confidence interval for mean support by taking the means of the upr and lwr values of the individual predictions.
- Resample from S* to fit a large number of models, generate predictions on each model, and then bootstrap a confidence interval for the relevant mean from these predictions.
Any advice here would be greatly appreciated.