Which model should I apply on sequential data?

I need to predict a binary vector given a sequential dataset meaning the current datapoint depends on its predecessors as well as (known) successors. So, it looks something like this: Given the sequence: X = [x_1, x_2, x_3, ..., x_N] I want to predict: Y = [y_1, y_2, y_3, ..., y_N] with y_i \in {0, 1} a binary label Now, the sequence X is fully known, that is: future observations in the sequence are completely known at any time. Therefore, in contrast to normal time series data, I can also harness any x_i+1, x_i+2,... from the full X sequence for predicting y_i at any time and not only x_i-2, x_i-1, x_i etc. Also the data X is a sequence of R^dxN vectors, i.e. N d-dimensional datapoints containing real numbers. In my case the dimensionality of the data is d=140 . Now, what I want to predict is the following: What is y_i given X , e.g. what is y_3 given the observations x_1, x_2, x_3, x_4, x_5 ? So, eventually I need something like this: for i in range(N): predict y_i given X = [x_1, x_2, x_3, ..., x_N] This actually is a many-to-one prediction task. Now you could use a RNN, but the problem imho is that "future" observations are not taken into account when applying an RNN. Maybe I am wrong about this assumption. But therefore I am asking: Which model would you suggest to use for this problem?