Maester - writing a custom test

Maester.dev is an open-source project which provides a framework to help maintain the security configuration of Microsoft cloud tenants (see https://maester.dev/) - and it's pretty great!

While the standard tests will get you to a very good tenant configuration, especially with regards to Conditional Access Policies, it is easy to better match the tests to your own needs.

Warning - this article assumes a basic understanding of Microsoft licensing with regards to conditional access!

For example both tests MT.1012 and MT.1013, which deal with risk based conditional access policies (that need an entra P2 license) assume, that you have them in place for all of your users. Should you have a mixed environment with users with both an entra id P2 and an entra id P1 license, the test will fail, as you probably have only enabled this conditional access policy for your licensed users.

Short detour (we'll be back with Maester in a minute - I promise):

Build a dynamic group, that includes all users with a specific service plan (here Entra ID P2). You can find the names and identifiers here: Product names and service plan identifiers for licensing - Microsoft Entra ID | Microsoft Learn

Use that identifier to build a dynamic group for your licensed Entra ID P2 users with the following rule: "user.assignedPlans -any (assignedPlan.servicePlanId -eq "eec0eb4f-6444-4f95-aba0-50c24d67f998" -and assignedPlan.capabilityStatus -eq "Enabled")"

Now use that group to correctly scope your risk based conditional access policy.

And now back to maester!

As both tests incorrectly assume, that the risk based conditional access policies are not in place - both look for "all users" I decided to rewrite them as custom tests. Let's have a look at both tests before writing the custom tests. Both are started from "Test-ConditionalAccessBaseline.Tests.ps1" from the standard test folder (tests\Maester\Entra).

Both tests check if entra ID P2 licenses are in place and skip the whole test, if only entra ID P1 licenses are found. And both invoke a Powershell function from the Maester library. So, lets head off to find out what is happening within the library.

(Get-Module -ListAvailable Mae*).Path should point you to your module directory. There in the public folder are both Test-MtCaMFaForRiskySignIn and Test-MTCaRequirePasswordChangeForHighUserRisk (or if you don't find them on your local machine - just look them up on github).

Anyway, here we find that both tests check for all users:

if ( $policy.grantcontrols.builtincontrols -contains 'passwordChange' -and $policy.conditions.users.includeUsers -eq "All"
-and $policy.conditions.applications.includeApplications -eq "All" -and "high" -in $policy.conditions.userRiskLevels
)

Now that we finally have found out what these tests do, let's write a custom test that does not include the user requirement. We basically start out with the tests from Test-ConditionalAccessBaseline.Tests.ps1 and put our own test inline. For that I created a File ContosoEntraTests.ps1 in the Maester custom tests folder and added the test there:

BeforeDiscovery {
    $EntraIDPlan = Get-MtLicenseInformation -Product "EntraID"
}

Describe "Conditional Access Baseline Policies Contoso" -Tag "CA", "Security", "All" -Skip:( $EntraIDPlan -eq "Free" ) {
    It "MT.1012.Contoso: At least one Conditional Access policy is configured to require MFA for risky sign-ins. See https://maester.dev/docs/tests/MT.1012" -Skip:( $EntraIDPlan -eq "P1" ) {
        $policies = Get-MtConditionalAccessPolicy | Where-Object { $_.state -eq "enabled" }
        # Remove policies that require password change, as they are related to user risk and not MFA on signin
        $policies = $policies | Where-Object { $_.grantcontrols.builtincontrols -notcontains 'passwordChange' }
        $policiesResult = New-Object System.Collections.ArrayList

        $result = $false
        # Policy Check angepasst und "-and $policy.conditions.users.includeUsers -eq "All" `" entfernt, da nicht alle bei Contoso eine Entra ID P2 Lizenz besitzen
            foreach ($policy in $policies) {
                if ( ( $policy.grantcontrols.builtincontrols -contains 'mfa' `
                        -or $policy.grantcontrols.authenticationStrength.requirementsSatisfied -contains 'mfa' ) `
                        -and $policy.conditions.applications.includeApplications -eq "All" `
                        -and "high" -in $policy.conditions.signInRiskLevels `
                        -and "medium" -in $policy.conditions.signInRiskLevels ) {
            $result = $true
            $currentresult = $true
            $policiesResult.Add($policy) | Out-Null
        } else {
            $currentresult = $false
        }
        Write-Verbose "$($policy.displayName) - $currentresult"
    }

    if ( $result ) {
        $testResult = "The following conditional access policies require multi-factor authentication for risky sign-ins`n`n%TestResult%"
    } else {
        $testResult = "No conditional access policy requires multi-factor authentication for risky sign-ins."
    }
    Add-MtTestResultDetail -Result $testResult -GraphObjects $policiesResult -GraphObjectType ConditionalAccess
    $result | Should -Be $true -Because "there is no policy that requires MFA for risky sign-ins"
    }

    #changed policy from require password change to block access and removed criterion that policy needs to include all users (not all users have entra id plan 2 license)
    It "MT.1013.Contoso: At least one Conditional Access policy is configured to block access when user risk is high. See https://maester.dev/docs/tests/MT.1013" -Skip:( $EntraIDPlan -eq "P1" ){
        $policies = Get-MtConditionalAccessPolicy | Where-Object { $_.state -eq "enabled" }
        # Only check policies that have password change as a grant control
        $policies = $policies | Where-Object { $_.grantcontrols.builtincontrols -contains 'passwordChange' }
        $policiesResult = New-Object System.Collections.ArrayList

        $result = $false
        # Policy Check angepasst und "-and $policy.conditions.users.includeUsers -eq "All" `" entfernt, da nicht alle bei Contoso eine Entra ID P2 Lizenz besitzen
        foreach ($policy in $policies) {
            if ( $policy.grantcontrols.builtincontrols -contains 'passwordChange' `
                    -and $policy.conditions.applications.includeApplications -eq "All" `
                    -and "high" -in $policy.conditions.userRiskLevels `
            ) {
                $result = $true
                $currentresult = $true
                $policiesResult.Add($policy) | Out-Null
            } else {
                $currentresult = $false
            }
            Write-Verbose "$($policy.displayName) - $currentresult"
        }

        if ( $result ) {
            $testResult = "The following conditional access policies require password change for risky users`n`n%TestResult%"
        } else {
            $testResult = "No conditional access policy requires a password change for risky users."
        }
        Add-MtTestResultDetail -Result $testResult -GraphObjects $policiesResult -GraphObjectType ConditionalAccess

        $result | Should -Be $true -Because "there is no policy that requires new password when user risk is high"
    }
}

As you can see, both tests are more or less the same as the tests from the Maester module. But they do not check if the policy is in place for all users.

So now when you run a complete test, you will see both tests from the Maester module still fail, but they are followed by the custom tests, as the results are alphabetically sorted.

I did not modify the original MT.1012 and MT.1013 tests, as I want to maintain test compatibility (and easily upgrade my Maester tests to the newest version).

PS Big thanks to Fabian Bader, who pointed me into the right direction!