A lot of progress has been made over the past decade on process & tooling managing large-scale, multi-tier, multi-cloud apps & API's. In contrast, there is far less common knowledge on best practices for managing machine learned models (classifiers, forecasters, etc.) - especially beyond the modeling / optimization / deployment process, once they are in production.

This talk summarizes best practices & lessons learned across the entire life cycle of such systems, based on nearly a decade of experience building & operating such systems at Fortune 500 companies across several industries. The talk is intended for engineering leaders, architects & ops managers, and covers seven areas:

• Development process for fast, scalable & reproducible experimentation
• Iterative development process for unsupervised & semi-supervised learning projects
• Online versus offline accuracy metrics, and why both are needed
• Common mistakes in measuring the accuracy of machine learned models
• Handling feedback for non-stationary and adversarial learning projects
• Staffing models, roles & responsibilities during the development of data science projects
• DevOps, maintenance & support for models and systems in production