Testing Framework for Conditional Access Policies: Avoiding Production Lockouts

Introduction

Misconfigured conditional access policies can cost enterprises enormously. A Fortune 500 company had 10,000 employees locked out for four hours due to a single misconfigured policy. Another organization spent eight days and over 60 support calls to recover admin access after a phishing-resistant MFA policy malfunction.

With business downtime costs ranging from $300,000 to $1 million per hour, deploying untested conditional access policies represents an unacceptable risk. The solution lies in implementing a PowerShell-based testing framework that simulates all sign-in scenarios before a policy reaches production.

The What If API Revolutionizes Policy Testing

The old What If tool in the Entra portal was useful but manual. The Graph API version (POST /v1.0/identity/conditionalAccess/evaluate) transforms policy testing into an automatable and scriptable operation. The Test-MgIdentityConditionalAccess PowerShell cmdlet from the Microsoft.Graph.Identity.SignIns module encapsulates this API elegantly.

Here's a production-ready test that simulates a high-risk Android sign-in to SharePoint:

1# Requires: Microsoft.Graph.Identity.SignIns v2.6+
2# Permission: Policy.Read.All (application or delegated)
3Connect-MgGraph -Scopes 'Policy.Read.All'
4 
5$params = @{
6    signInIdentity  = @{
7        "@odata.type" = "#microsoft.graph.userSignIn"
8        userId        = "15dc174b-f34c-4588-ac45-61d6e05dce93"
9    }
10    signInContext    = @{
11        "@odata.type"       = "#microsoft.graph.applicationContext"
12        includeApplications = @("00000003-0000-0ff1-ce00-000000000000")
13    }
14    signInConditions = @{
15        devicePlatform = "android"
16        clientAppType  = "browser"
17        signInRiskLevel = "high"
18        country         = "US"
19        ipAddress       = "40.77.182.32"
20        deviceInfo      = @{ isCompliant = $false }
21    }
22    appliedPoliciesOnly = $false  # Returns ALL policies with evaluation status
23}
24 
25$results = Test-MgIdentityConditionalAccess -BodyParameter $params
26$results | Where-Object { $_.policyApplies -eq $true } |
27    Format-Table displayName, state, @{
28        N='Controls'; E={$_.grantControls.builtInControls -join ', '}
29    }

Key API Points

The API evaluates each enabled and report-only policy in your tenant. Each result includes a policyApplies boolean and analysisReasons explaining precisely why a policy matches or not, with values like users, application, platform, or notEnoughInformation.

Creating Regression Tests with Maester

Raw API calls verify individual scenarios. The open-source Maester framework (maester.dev) transforms these scenarios into repeatable Pester-based regression tests that run in CI/CD. Developed by Merill Fernando (Microsoft PM) and security MVPs Fabian Bader and Thomas Naunheim, Maester includes 20+ out-of-the-box conditional access tests.

1Install-Module Maester
2 
3Describe "Contoso CA Regression Suite" {
4    It "SharePoint access requires MFA for all users" {
5        $userId = (Get-MgUser -UserId 'user@contoso.com').Id
6        $result = Test-MtConditionalAccessWhatIf -UserId $userId `
7            -IncludeApplications '67ad5377-2d78-4ac2-a867-6300cda00e85'
8        $result.grantControls.builtInControls | Should -Contain "mfa"
9    }
10 
11    It "Azure Portal blocked for non-admin users" {
12        $userId = (Get-MgUser -UserId 'standard@contoso.com').Id
13        $result = Test-MtConditionalAccessWhatIf -UserId $userId `
14            -IncludeApplications 'c44b4083-3bb0-49c1-b47d-974e53cbdf3c'
15        $result.grantControls.builtInControls | Should -Contain "block"
16    }
17}

Anti-Lockout Deployment Pipeline

Organizations that get locked out share a common pattern: they ignore report-only mode, forget break-glass exclusions, or deploy directly in the portal without version control.

Recommended Pipeline Architecture

1SigninLogs
2| mv-expand ConditionalAccessPolicies
3| where ConditionalAccessPolicies["result"] == "reportOnlyFailure"
4| project TimeGenerated, UserPrincipalName, AppDisplayName,
5    PolicyName = ConditionalAccessPolicies["displayName"]
6| summarize BlockCount = count() by UserPrincipalName, PolicyName
7| sort by BlockCount desc

Essential Enterprise Safeguards

Mandatory Break-Glass Accounts

Maintain at least two cloud-only emergency access accounts on your onmicrosoft.com domain, excluded from all CA policies, with FIDO2 keys stored in a fireproof vault. Test them quarterly.

Alerts on Policy Changes

Set up alerts on every policy change with this KQL query in Azure Monitor:

1AuditLogs
2| where OperationName in (
3    "Add conditional access policy",
4    "Update conditional access policy",
5    "Delete conditional access policy")
6| extend Actor = tostring(parse_json(
7    tostring(InitiatedBy.user)).userPrincipalName)
8| extend PolicyName = tostring(TargetResources[0].displayName)

Regulatory Compliance

Don't overlook compliance. SOC 2 CC8.1 requires documented change management with testing before deployment. ISO 27001:2022 A.8.9 explicitly mandates configuration management policies for security-affecting changes.

Immediate Action Plan

Start small but effective. Export your current policies to JSON and commit them to a private repo - this already gives you backup and audit trail. Add What If API calls for your three highest-risk scenarios:

• Admin portal access
• Legacy authentication attempts
• Non-compliant device sign-ins

Integrate Maester in a scheduled GitHub Action. In a few weeks, you'll have a testing framework that transforms CA policy changes from "hope it works" to "prove it works".

Useful Links

• Microsoft Graph What If API Reference – Official documentation for the conditionalAccess/evaluate endpoint with request/response examples
• Maester Framework Documentation – Open-source PowerShell framework for testing Microsoft 365 security with built-in conditional access tests
• Plan Your Conditional Access Deployment – Comprehensive Microsoft planning guide including report-only mode, break-glass accounts, and best practices
• Emergency Access Account Management – Official guidance on configuring and securing break-glass accounts
• Microsoft.Graph.Identity.SignIns Module – PowerShell module documentation for managing conditional access policies via Graph API
• Conditional Access Report-Only Mode – Microsoft documentation on testing policies without enforcement
• Azure Monitor KQL Query Language – Reference documentation for writing conditional access event monitoring queries
• ISO 27001:2022 Control A.8.9 – Configuration management requirements for critical security changes

Glossary of Terms

What If API: Microsoft programming interface allowing simulation of conditional access policy evaluation without actually applying them.

Break-glass accounts: Emergency access accounts excluded from security policies, used only in case of critical system failure.

FIDO2: Passwordless strong authentication standard using public-key cryptography for enhanced security.

Maester: Open-source automated testing framework for the Microsoft 365 ecosystem, specialized in validating security configurations.

Report-only mode: State of a conditional access policy that evaluates conditions without blocking access, generating only audit logs.

Pester: PowerShell unit testing framework for validating script and system configuration and behavior.

Workload Identity Federation: Azure authentication method enabling external workloads to access Azure resources without managing secrets.