My real data is very complicated, but briefly it can be formalized as this problem below: currently I have many groups of datapoints, each consisting of 5 datapoints, I want to design an algorithm to label each of the 5 points in a group from 0~4 (Input is a group of 5 points, output is their labels 0~4). On a 2d scatter plot, the problem is like:

Train Set
| name | feature1 | feature2 |  group   |   label  |
| ---- | -------- | -------- | -------- | -------- |
| G1L0 |    1     |    1     |   1      |    0     |
| G1L1 |    4     |    3     |   1      |    1     |
| G1L2 |    4     |    5     |   1      |    2     |
| G1L3 |    2     |    4.5   |   1      |    3     |
| G1L4 |    6     |    5     |   1      |    4     |
| ---- | -------- | -------- | -------- | -------- |
| G2L0 |    2.6   |    0.8   |   2      |    0     |
| G2L1 |    3.9   |    3.5   |   2      |    1     |
| G2L2 |    3.4   |    4.5   |   2      |    2     |
| G2L3 |    2.3   |    5.5   |   2      |    3     |
| G2L4 |    5.6   |    4.5   |   2      |    4     |
Test Set
| ---- | -------- | -------- | -------- | -------- |
| G3L0 |    2.1   |    1.1   |   3      |    0     |
| G3L1 |    4.3   |    3.7   |   3      |    1     |
| G3L2 |    4.9   |    5.4   |   3      |    2     |
| G3L3 |    3.6   |    5.3   |   3      |    3     |
| G3L4 |    6.5   |    5.5   |   3      |    4     |

The key issue is that the classification of each of the 5 datapoints depends on the information of other points within the group, globally the data suffers greatly from variation of the data, specifically, if I train a traditional classifier on my train set it may end up like: But if I simply use this classifier to predict the test set there will be problem:

This is because globally the variation across different groups is huge, but within a single group I can easily classify them manually. Experience is something like "Three points at top, from left to right they are label 3, label 2, label 4, then for two points at below they are label 0 (left), label 1 (right). Also, the classifier may label two points in a same group as 3, but this is impossible to happen (We must have label 0~4 in a group).

I know this problem can be easily solved by normalization within the group, however, my real problem is 10-dimensional with more than 15 datapoints, and there are different types of variations and normalization by group is basically not feasible as I tried for a week.

Therefore, is there any way I can utilize the idea of "considering the position of other points in a group"? That is, encode the relative position information into a feature?