How does Yelp decide which relevant business or service to show you as an ad within 100 milliseconds of your visit? What are the criteria and metrics by which we measure success of our ad serving system? In this talk, the audience will learn about how Yelp figures out the best ad to show a user during his visit to Yelp: via a 2nd price auction amongst all the matching advertisers. Powering this 2nd price auction is a Machine Learning based system that predicts Click Through Rates (CTR) for all ads and an Auto-Bidding system that determines the optimal bid price for each ad per user request. Yelp's local advertising presents challenges that are unique compared to display, social or mobile advertising. I'll motivate this via some trends and data observations. One of the interesting aspects is business categories and geolocation: How far are people willing to travel to visit a restaurant? What about professional services like plumbers: are users less or more sensitive to how far those are compared to restaurants? I'll provide examples of how we use our open-sourced Map Reduce package (MRJob) to scale ML feature engineering and performance metric computation. I'll also provide details on our Machine Learning pipeline built using the popular python packages: numpy, scipy and sklearn. This talk would give you an in-depth overview of advertising systems, and why with increasingly sophisticated ad systems, in future we will wonder why we ever hated ads!