Predicting Rider Conversion in Sparse Data Environments with Bayesian Trees
At Lyft, understanding how riders go through our user experience is fundamental to operating a healthy marketplace. Specifically, it is important to have a robust model determining if a rider will actually request a ride after entering a destination and viewing a price and ETA. Accurately predicting this decision, that we call conversion , informs countless decisions across our platform. Whether it is to better balance supply and demand, improve user experiences, optimize recommendations and advertisement, understand long-term engagement, decide how to distribute coupons… rider conversion prediction is a central challenge for the Lyft business. However, predicting human behavior at scale is incredibly complex, and the exact same person might well open the app just to check current availability or actually to request a ride after viewing our prices. The contexts under which riders make their conversion decisions are extremely diverse and almost unique to each session. A user’s intent changes based on where they are and where they want to go, what time it is, their previous interactions with the platform, current supply-demand market conditions, to cite a few. When we try to model this using standard machine learning approaches, we run into a significant challenge: data sparsity . The Challenge of High Cardinality and Sparsity To accurately predict conversion, we need to slice our data very thinly across many categorical features. Imagine trying to predict the conversion proba…
