To address continuous large-scale evaluation of diagnostic systems, we proposed a novel patient vignette simulation algorithm within an iterative clinician-in-the-loop methodology for semi-automatically evaluating and deploying medical diagnostic systems in production.