Time filter

Source Type

New York City, NY, United States

Perlich C.,M6D Research | Dalessandro B.,M6D Research | Raeder T.,M6D Research | Stitelman O.,M6D Research | And 2 more authors.
Machine Learning

This paper presents the design of a fully deployed multistage transfer learning system for targeted display advertising, highlighting the important role of problem formulation and the sampling of data from distributions different from that of the target environment. Notably, the machine learning system itself is deployed and has been in continual use for years for thousands of advertising campaigns - in contrast to the more common case where predictive models are built outside the system, curated, and then deployed. In this domain, acquiring sufficient data for training from the ideal sampling distribution is prohibitively expensive. Instead, data are drawn from surrogate distributions and learning tasks, and then transferred to the target task. We present the design of the transfer learning system We then present a detailed experimental evaluation, showing that the different transfer stages indeed each add value. We also present production results across a variety of advertising clients from a variety of industries, illustrating the performance of the system in use. We close the paper with a collection of lessons learned from over half a decade of research and development on this complex, deployed, and intensely used machine learning system. © 2013 The Author(s). Source

Raeder T.,M6D Research | Stitelman O.,M6D Research | Dalessandro B.,M6D Research | Perlich C.,M6D Research | Provost F.,New York University
Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Most data mining research is concerned with building high-quality classification models in isolation. In massive production systems, however, the ability to monitor and maintain performance over time while growing in size and scope is equally important. Many external factors may degrade classification performance including changes in data distribution, noise or bias in the source data, and the evolution of the system itself. A well-functioning system must gracefully handle all of these. This paper lays out a set of design principles for large-scale autonomous data mining systems and then demonstrates our application of these principles within the m6d automated ad targeting system. We demonstrate a comprehensive set of quality control processes that allow us monitor and maintain thousands of distinct classification models automatically, and to add new models, take on new data, and correct poorly-performing models without manual intervention or system disruption. © 2012 ACM. Source

Dalessandro B.,M6D Research | Stitelman O.,M6D Research | Perlich C.,M6D Research | Provost F.,NYU and M6D Research
Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

In many online advertising campaigns, multiple vendors, publishers or search engines (herein called channels) are contracted to serve advertisements to internet users on behalf of a client seeking specific types of conversion. In such campaigns, individual users are often served advertisements by more than one channel. The process of assigning conversion credit to the various channels is called \attribution," and is a subject of intense interest in the industry. This paper presents a causally motivated methodology for conversion attribution in online advertising campaigns. We discuss the need for the standardization of attribution measurement and ofier three guiding principles to contribute to this standardization. Stemming from these principles, we position attribution as a causal estimation problem and then propose two approximation methods as alternatives for when the full causal estimation can not be done. These approximate methods derive from our causal approach and incorporate prior attribution work in cooperative game theory. We argue that in cases where causal assumptions are violated, these approximate methods can be interpreted as variable importance measures. Finally, we show examples of attribution measurement on several online advertising campaign data sets. Copyright 2012 ACM. Source

Discover hidden collaborations