Validating and Safeguarding Production AI

This long course focuses on the operational lifecycle of agentic AI systems: robust partitioning and dataset management, automated retraining pipelines, continuous monitoring for drift and anomalies, testing and secure deployment, and performance optimization of code and pipelines. You will practice partitioning strategies (time-series and stratified), monitoring and drift detection metrics (PSI and KS), and build CI/CD notebooks and automated workflows for model retraining and re-deployment using tools like MLflow and GitHub Actions. The course addresses software-engineering best practices—clean code, profiling, unit and integration testing—and dependency risk assessment to maintain secure, reliable production systems. Practical assignments include building monitoring alerting rules, implementing retraining triggers, diagnosing runtime bottlenecks, and integrating human-in-the-loop feedback systems to continuously improve models in production while ensuring high code quality and security hygiene

• Build automated CI/CD pipelines for model retraining and redeployment
• Write clean, performant Python code applying testing and profiling
• Implement anomaly detection using statistical methods and human feedback
• Create unbiased datasets and analyze model performance
• Apply software engineering best practices for secure AI production systems

• Basic familiarity with AI concepts and terminology
• Readiness for hands-on practice via exercises or case studies