Weekly Digest #75

Articles

Using Gamma Distribution to Improve Long-Tail Event Predictions

Using weights obtained from gamma distribution to better model long-tail data

1. First, we find the best-fit gamma distribution for the delivery duration data. In our case, we found α=4 and β =1 to have the best fit.
2. Next, we set the weight function inversely proportional to the PDF of the gamma distribution to allow the model to learn from the rare long-tail events.
3. We split the training data samples into two groups at the peak density (the Mode). The group on the left side of the peak density has a shorter delivery time, and the right side group has a longer delivery time.
4. The sample weights of the data points are tuned differently for samples on the left and right sides of the peak according to business constraints. This helps tune for earliness as well as lateness.

How Spotify Uses Semantic Search for Podcasts

1. Create embedding for query-episode pairs
2. Retrieve episode through Approximate Nearest Neighbors (ANN) search
3. Evaluated through Mean Reciprocal Rank (MRR), the average reciprocal rank of a correct answer.