Feature/azdo

[SebastianClaesson]

Description

Fixes #1368

Adding 31 Azure DevOps security tests.
Each of the tests is added to the new folder azdo in maester\public\maester

All of the tests have a markdown file, with :

• Rationale
• Remediation action
• Related links (Official links / blog posts describing the security issue)

Contribution Checklist

Before submitting this PR, please confirm you have completed the following:

• 📖 Read the guidelines for contributing to this repository.
• 🧪 Ensure the build and unit tests pass by running /powershell/tests/pester.ps1 on your local system.

[SamErde]

Shouldn't the detailed 401 go to users in the organization and the 404 go to users not in the organization?

[SebastianClaesson]

I do agree!
Seems the experience has changed since;

MicrosoftDocs/azure-devops-docs@91c4410#diff-a28e8dd823f1493651d0c6322e6b2dd976bccab79e279de5aea876ce20272729R44

I'll update with the new information the article.

[Copilot]

Pull request overview

This pull request adds 31 Azure DevOps security tests to the Maester project, providing comprehensive security assessments for Azure DevOps organizations. Each test includes both PowerShell implementation and markdown documentation describing the security rationale and remediation steps.

Changes:

• Added 31 new Azure DevOps security test functions covering authentication, access control, pipeline security, and resource management
• Created comprehensive markdown documentation for each test with remediation guidance
• Updated the module manifest to export all new test functions
• Added a test runner file that orchestrates all Azure DevOps security tests

Reviewed changes

Copilot reviewed 65 out of 65 changed files in this pull request and generated 28 comments.

Show a summary per file

File	Description
tests/Maester/Azdo/Test-Azdo.Tests.ps1	Test runner that executes all 31 Azure DevOps security tests
tests/Maester/Azdo/README.md	Overview documentation for Azure DevOps tests
powershell/public/maester/azdo/*.ps1	31 PowerShell functions implementing security checks
powershell/public/maester/azdo/*.md	31 markdown documentation files with rationale and remediation steps
powershell/Maester.psd1	Module manifest updated to export new functions

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 65 out of 65 changed files in this pull request and generated 11 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Return value logic conflict. When disableStageChooser is $true (secure state), the function returns $true, but the test expects $false (line 100 in Test-Azdo.Tests.ps1: "Should -Be $false"). Either the test expectation is wrong, or the function should return the inverse. Based on the test comment "Users should not be able to select stages," it seems the test expects $false to mean "stage chooser is disabled" (secure state), but the function returns $true when disabled. Consider inverting the return value: return -not $result to align with test expectations.

[Copilot]

The same return value logic issue exists here as in Test-AzdoOrganizationCreationClassicReleasePipeline. When disableClassicBuildPipelineCreation is $true (secure state), the function returns $false, which conflicts with the test expectation that secure state should return $false to indicate "creation should not be allowed". Review the intended semantics and align the return value with the test expectations.

Suggested change

	$result = $false
	}
	else {
	$resultMarkdown = "Classic build pipelines can be created / imported."
	$result = $true
	$result = $true
	}
	else {
	$resultMarkdown = "Classic build pipelines can be created / imported."
	$result = $false

[Copilot]

The "Because" message describes the behavior for "non-release pipelines" but the test is for "classic release pipelines". The message should state: "With this option enabled, you can reduce the scope of access for all classic release pipelines to the current project."

[Copilot]

Inconsistent connection checking across Azure DevOps tests. Some tests check for Azure DevOps connection using Get-ADOPSConnection (like Test-AzdoExternalGuestAccess, Test-AzdoEnforceAADConditionalAccess, etc.) while most tests only output a verbose message without actually checking the connection. This test should either check the connection status and skip with a proper message, or the Write-Verbose statement on line 27 is misleading. Consider implementing consistent connection checking across all AzDO tests, following the pattern in Test-AzdoExternalGuestAccess.ps1.

[Copilot]

The return value logic appears inverted. When disableClassicReleasePipelineCreation is $true (meaning creation is disabled, which is the desired secure state), the function returns $false, but the test expects $false when creation should NOT be allowed. This creates a logic conflict. If the setting name means "disable creation" and it's true, the function should return $true (secure state achieved), not $false. The same issue exists in Test-AzdoOrganizationCreationClassicBuildPipeline.ps1.

Suggested change

	$result = $false
	}
	else {
	$resultMarkdown = "Classic release pipelines, task groups, and deployment groups can be created / imported."
	$result = $true
	$result = $true
	}
	else {
	$resultMarkdown = "Classic release pipelines, task groups, and deployment groups can be created / imported."
	$result = $false

[Copilot]

Grammar error appears consistently across multiple markdown files: "to stops these" should be "to stop these". This phrase doesn't make sense in most contexts anyway - the remediation actions are about enabling/disabling policies, not stopping requests and notifications. Consider revising to match the actual action being taken or removing this phrase entirely. Found in: Test-AzdoThirdPartyAccessViaOauth.md, Test-AzdoSSHAuthentication.md, Test-AzdoPublicProject.md, Test-AzdoOrganizationTaskRestrictionsShellTaskArgumentValidation.md, Test-AzdoOrganizationTaskRestrictionsDisableNode6Task.md, Test-AzdoOrganizationTaskRestrictionsDisableMarketplaceTask.md, Test-AzdoOrganizationStageChooser.md, Test-AzdoOrganizationRepositorySettingsGravatarImage.md, Test-AzdoOrganizationBadgesArePrivate.md, Test-AzdoLogAuditEvent.md.

[SamErde]

Missing the actual check for an ADO connection here. 🙂

[SamErde]

Missing the actual check for an ADO connection here. 🙂

[SamErde]

Missing the actual check for an ADO connection here. 🙂

[SamErde]

Missing the actual check for an ADO connection here. 🙂

[SamErde]

Missing the actual check for an ADO connection here. 🙂

[SamErde]

Missing the actual check for an ADO connection here. 🙂

[SamErde]

Missing the actual check for an ADO connection here. 🙂

[SamErde]

Missing the actual check for an ADO connection here. 🙂

[SamErde]

Missing the actual check for an ADO connection here. 🙂

[SamErde]

Missing the actual check for an ADO connection here. 🙂

[SamErde]

Sorry for all of the comments! This is awesome work!

[SebastianClaesson]

@SebastianClaesson this is top notch work! Love it.

I would suggest two additions.

1. Add doc - Update the installation doc to include the optinal module to install and the command for connecting website/docs/installation.md . See https://maester.dev/docs/installation#optional-modules-and-permissions
2. Add a blog post that outlines that calls out the new tests to website/blog See https://maester.dev/blog

Thank you @merill , I'll add the docs and a blog post to showcase / outline the new tests!

[Copilot]

Pull request overview

Copilot reviewed 68 out of 68 changed files in this pull request and generated 12 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Inside Get-NestedAdoMembership, the first Write-Verbose message references $($_.Descriptor) even though $_ isn’t set in that scope; this will log an empty/incorrect descriptor. Use $Member.Descriptor there to match the rest of the function.

[Copilot]

Typo in result text: "Pipeliens" should be "Pipelines".

[Copilot]

Typo in title: "whne" should be "when".

[Copilot]

The comment header link in the description points to audit streaming (.../audit/auditing-streaming...) which doesn’t match this test’s purpose (Conditional Access enforcement). Updating it to the Conditional Access policy documentation would avoid confusing users reading the help.

Suggested change

	https://learn.microsoft.com/en-us/azure/devops/organizations/audit/auditing-streaming?view=azure-devops
	https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/manage-conditional-access?view=azure-devops&tabs=preview-page

[Copilot]

These Pester tests call the Test-Azdo* functions directly in the assertion. Many of those functions return $null when not connected to Azure DevOps, which will cause these assertions to fail (null is neither $true nor $false). Capture the function output into a variable and only assert when it’s non-null (same pattern used in other Maester test files).

[Copilot]

The here-string includes trailing backslashes at the end of lines, which will be emitted literally in the result output (they don’t act as line continuations inside a here-string). If the intent is multiline formatting, remove the trailing \ characters and rely on the here-string newlines (or build the string with explicit newlines).

[Copilot]

break after return $null is unreachable and can be removed. This pattern appears across the Azure DevOps test functions, so it would be good to clean it up throughout to avoid dead code.

Suggested change

break

[Copilot]

Minor grammatical issue in the result text: "Your tenant have not restricted ..." should be "Your tenant has not restricted ...".

[Copilot]

The policy queried is 'Policy.DisallowSecureShell', but the current logic treats $Policy.effectiveValue -eq $true as “SSH allowed” and returns $true. This appears inverted relative to the policy name and will likely report the opposite of the intended security posture. Consider inverting the boolean / message so that enabling the disallow policy yields the secure outcome.

[Copilot]

The policy queried is 'Policy.DisallowAadGuestUserAccess', but the current logic treats $Policy.effectiveValue -eq $true as “external guests have access” and returns $true. This looks inverted relative to the policy name and will likely mark a secure configuration as failing. Consider inverting the returned boolean and aligning the result text so that disallowing guest access corresponds to the secure outcome.

Suggested change

	$result = $Policy.effectiveValue
	if ($result) {
	$resultMarkdown = "External user(s) can be added to the organization to which they were invited and has immediate access. A guest user can add other guest users to the organization after being granted the Guest Inviter role in Microsoft Entra ID."
	} else {
	$resultMarkdown = "Well done. External users should not be allowed access to your Azure DevOps organization"
	# $Policy.effectiveValue is $true when guest access is disallowed.
	# $result represents the secure configuration: $true when external guest access is not allowed.
	$result = -not $Policy.effectiveValue
	if ($result) {
	$resultMarkdown = "Well done. External users should not be allowed access to your Azure DevOps organization"
	} else {
	$resultMarkdown = "External user(s) can be added to the organization to which they were invited and has immediate access. A guest user can add other guest users to the organization after being granted the Guest Inviter role in Microsoft Entra ID."