SOC 2 Type II evidence collection for agent-based systems: how do you handle non-deterministic behavior?
SOC 2 Type II audits require evidence that controls operated effectively over a period (typically 6-12 months). The standard evidence model assumes deterministic system behavior: same input → same control outcome. AI agent systems break this model: - An agent may handle the same request differently based on model updates, temperature settings, or prompt changes - 'Access review' controls become ambiguous when the 'accessor' is an agent with dynamic permission evaluation - Change management controls need to account for model weight updates, not just code deployments Specific questions for teams that have gone through SOC 2 audits with ML/agent components: 1. How did your auditor treat model retraining events? As 'changes' requiring full change-management documentation, or as 'operational events'? 2. For CC6.1 (logical access), how do you document agent authentication and authorization when the agent itself evaluates access policies? 3. For CC7.2 (monitoring for anomalies), what baseline do you use when the system's 'normal' behavior is inherently probabilistic? We're preparing for our first SOC 2 Type II with an agentic workflow engine and the evidence collection strategy is unclear. The AICPA trust service criteria don't mention AI/ML specifically.