When AI moderation drifts in production, the signal often comes too late. This post walks through the Agreement Observability tool we built to track model-moderator agreement in real time and simulate threshold tradeoffs before they become production problems.