Add Copilot Studio support to Maester

[lnfernux]

Description

Adds 10 new Copilot Studio agent security tests (MT.1113-MT.1122) based on the Copilot Studio Agent Security Top 10 Risks. These tests query agent configuration from the Dataverse OData API and check for common security misconfigurations in Copilot Studio agents.

Also adds Dataverse as a new service option in Connect-Maester to support authentication to the Dataverse API, and preserves existing Az contexts (federated credentials, managed identity) so the interactive login prompt is skipped when a session already exists.

What's been tested

I've tested this locally using both interactive and SPN (clientId + secret). Interactive with Connect-Maester -Graph,Dataverse, SPN requires you to sign in with Connect-AzAccount and Connect-MgGraph before running Connect-Maester -Dataverse since Graph will be implicit (explicit??). Didn't test in CI/CD.

I set up 11 different agents in Copilot Studio plus the default one (this one is excluded from tests):

---

If you want to test it, here's the guidance on setting up the different agents:

Test	Agent Name Suggestion	Key Configuration
MT.1113	Broad Sharing	Share → "My organization" → set to "Viewer" permissions. Optionally enable Multi-tenant support in Settings → Authentication
MT.1114	Weak Auth	Set authentication to Authenticate manually, toggle Require users to sign in to off
MT.1114	No Auth	Set authentication to No authentication
MT.1115	Risky HTTP	Add an HTTP request node in a topic pointing to a non-443 port or plain http:// URL
MT.1116	Email Exfil	Add an Office 365 Outlook connector action (Send an email) in a topic
MT.1117	Dormant	Any published agent left unmodified for 180+ days (hard to test quickly — relies on LastModifiedTime)
MT.1118	Author Auth	Add a connector tool (e.g. SharePoint) and set its authentication to Agent author authentication under "Additional details" and setting "Credentials to use" to "Maker provided credentials"
MT.1119	Hardcoded Creds	Put a string like Bearer sk-abc123 or password=secret in a topic message/variable
MT.1120	MCP Tools	Add an MCP server tool integration to the agent
MT.1121	Missing Instructions	Enable Generative orchestration but leave the Instructions field empty
MT.1122	Orphaned	Create an agent with a co-owner, then disable/delete that owner's Entra account

I also had Claude whip up a test plan TEST-VALIDATION-PLAN.md that I went through locally.

Disclaimers

I'm not a good writer, so Claude helped build the skeletons for the .md files, the comments for the scripts and the tests. I've gone over the markdown files to make sure all the links works and has relevant/correct information. The regex for test MT.1119 detects everything I've tested, but man am I bad at regex.

Some things are open for debate and can freely be changed:

1. The naming scheme of Test-MtAIAgent* - can also be Test-MtCopilotStudio or something.
2. Test numbers of course.

Why no Defender XDR table?

This requires Copilot Studio licenses to publish the agents, so I didn't have the option to do this. Might be viable, would love to see someone with access to that test this.

Why no Graph?

I tried using these two endpoints:

1. GET graph.microsoft.com/copilot/admin/catalog/packages – Retrieves a complete inventory of all agents and apps (Microsoft, External, Shared, and Custom).
2. GET graph.microsoft.com/copilot/admin/catalog/packages/{id} – Fetches detailed metadata for a specific agent or app, including properties and manifest details.

Endpoints didn't exist (tried BETA also) and according to https://mc.merill.net/message/MC1173195 it's being added later, so either I'm dumb (I am, but still) or we can try this later.

What's included

New tests (10)

Test ID	Name	Risk
MT.1113	Test-MtAIAgentBroadSharing	Agent shared with "My organization" or with setting for "Multitenant support" enabled
MT.1114	Test-MtAIAgentNoAuthentication	No user authentication configured
MT.1115	Test-MtAIAgentRiskyHttpConfig	HTTP actions using plain HTTP or non-standard ports
MT.1116	Test-MtAIAgentEmailExfiltration	Email-sending tools with AI-controlled inputs
MT.1117	Test-MtAIAgentDormant	Published agent not modified in 180+ days
MT.1118	Test-MtAIAgentAuthorAuthentication	Connector tools using maker (author) credentials
MT.1119	Test-MtAIAgentHardCodedCredentials	Secrets embedded in topic definitions
MT.1120	Test-MtAIAgentMcpTools	Unreviewed MCP server integrations
MT.1121	Test-MtAIAgentMissingInstructions	Generative orchestration without custom instructions
MT.1122	Test-MtAIAgentOrphaned	Agent owner disabled or deleted in Entra ID

New internal function

• Get-MtAIAgentInfo - Queries the Dataverse OData API for Copilot Studio agent data, maps option set values, resolves owner UPNs, classifies topics vs tools, and caches results for reuse across tests.

Connect-Maester changes

• Added Dataverse to the -Service parameter ValidateSet
• Added Dataverse connectivity validation (checks config URL, tests token acquisition)
• Added Get-AzContext check to reuse existing Az sessions from federated credentials, managed identity, or prior Connect-AzAccount calls
• Updated Disconnect-Maester to handle Dataverse service

Documentation

• 10 test doc pages (website/docs/tests/maester/MT.1113.md - MT.1122.md)
• Updated website/docs/connect-maester/readme.md and the advanced connection docs with Dataverse section
• 10 result detail markdown files (powershell/public/maester/aiagent/Test-MtAIAgent*.md)

Configuration

• Added DataverseEnvironmentUrl to tests/maester-config.json GlobalSettings (empty default)
• Added DormantAgentDays config option (default: 180) for MT.1117

New files

powershell/internal/Get-MtAIAgentInfo.ps1           # Data retrieval from Dataverse
powershell/public/maester/aiagent/                   # 10 test .ps1 + 10 .md files
tests/Maester/AIAgent/Test-AIAgentSecurity.Tests.ps1 # Pester test wrapper
website/docs/tests/maester/MT.1113.md - MT.1122.md   # Website test pages

Modified files

powershell/Maester.psd1                       # Added 10 functions to FunctionsToExport
powershell/Maester.psm1                       # Added AIAgentInfo to session variable
powershell/internal/Clear-ModuleVariable.ps1  # Clear AIAgentInfo on reset
powershell/public/Connect-Maester.ps1         # Dataverse service + Az context reuse
powershell/public/Disconnect-Maester.ps1      # Dataverse disconnect
tests/maester-config.json                     # DataverseEnvironmentUrl setting
website/docs/connect-maester/readme.md        # Dataverse docs
website/docs/connect-maester/connect-maester-advanced.md        # Dataverse docs

How to test

Prerequisites

1. A Power Platform environment with Copilot Studio agents (at least one custom/unmanaged agent)
2. Az.Accounts module installed
3. The DataverseEnvironmentUrl set in tests/Custom/maester-config.json:

   {
     "GlobalSettings": {
       "DataverseEnvironmentUrl": "https://orgXXXXX.crmNN.dynamics.com"
     }
   }

4. The service principal must be registered as an application user in power platform with a security role that grants read access to the bot, botcomponent, systemuser, and connectionreference tables.

Create an Application User in Power Platform

1. Go to the Power Platform Admin Center → Manage → select your environment → Settings → Users + permissions → Application users
2. Click New app user → Add an app → select the app registration created above
3. Select the correct Business unit
4. Assign a security role with read access:
   • Basic User for simplicity, or
   • A custom role* (e.g. Maester Security Reader) with Organization-level Read on: Agent (bot), Agent component (botcomponent), User (systemuser), and Connection Reference (connectionreference)
5. Click Create

*Creating a custom role requires:

1. Going to the Power Platform Admin Center → Manage → select your environment → Settings → Users + permissions → Security roles
2. Click New role up top
3. Fill out the required information:
   • Role name: "Maester Security Reader"
   • Business unit: Default or select correct BU
   • Description: "Read-only access to bot, botcomponent, systemuser, and connectionreference tables for Maester AI agent security tests"
   • Member's privilege inheritance: Default

4. Save, then make sure the following access is given on the organization level:

Dataverse Table	UI Display Name	Privilege Needed
bot	Agent	Read (Organization)
botcomponent	Agent component	Read (Organization)
systemuser	User	Read (Organization)
connectionreference	Connection Reference	Read (Organization)

Configure Maester

Add the environment URL to maester-config.json:

{
  "GlobalSettings": {
    "DataverseEnvironmentUrl": "https://org12345.crm.dynamics.com"
  }
}

Interactive testing

Import-Module ./powershell/Maester.psd1 -Force
Connect-Maester -Service Graph,Dataverse
Invoke-Maester -Tag AIAgent

What to verify

• Module imports without errors
• Connect-Maester -Service Dataverse connects and validates token
• With existing Az context, no interactive login prompt appears
• Tests run and return expected results (See TEST-VALIDATION-PLAN.md)
• Tests skip cleanly when DataverseEnvironmentUrl is not configured
• HTML report renders AI Agent test results correctly

References

• Copilot Studio Agent Security Top 10 Risks
• Dataverse OData API
• Copilot Studio security and governance

Contribution Checklist

Before submitting this PR, please confirm you have completed the following:

• Read the guidelines for contributing to this repository.
• Ensure the build and unit tests pass by running /powershell/tests/pester.ps1 on your local system.

[Copilot]

Pull request overview

Adds Copilot Studio (Dataverse-backed) “AI Agent Security” coverage to Maester by introducing new MT.1113–MT.1122 tests, plus connection/docs updates to support authenticating and querying agent configuration from Dataverse.

Changes:

• Introduces Get-MtAIAgentInfo (Dataverse OData retrieval + caching) and 10 new Test-MtAIAgent* security tests (MT.1113–MT.1122).
• Extends Connect-Maester/Disconnect-Maester with a new Dataverse service option and Az-context reuse behavior.
• Adds Maester config + documentation updates for Dataverse setup and the new test pages/result details.

Reviewed changes

Copilot reviewed 42 out of 42 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
website/docs/tests/maester/MT.1113.md	New website test page for MT.1113 (broad sharing).
website/docs/tests/maester/MT.1114.md	New website test page for MT.1114 (missing/weak auth).
website/docs/tests/maester/MT.1115.md	New website test page for MT.1115 (risky HTTP config).
website/docs/tests/maester/MT.1116.md	New website test page for MT.1116 (email exfiltration).
website/docs/tests/maester/MT.1117.md	New website test page for MT.1117 (dormant agents).
website/docs/tests/maester/MT.1118.md	New website test page for MT.1118 (maker auth).
website/docs/tests/maester/MT.1119.md	New website test page for MT.1119 (hard-coded creds).
website/docs/tests/maester/MT.1120.md	New website test page for MT.1120 (MCP tools).
website/docs/tests/maester/MT.1121.md	New website test page for MT.1121 (missing instructions).
website/docs/tests/maester/MT.1122.md	New website test page for MT.1122 (orphaned ownership).
website/docs/sections/create-entra-app.md	Adds optional Dataverse permission/setup section for app registration.
website/docs/connect-maester/readme.md	Adds Dataverse connection guidance to Connect-Maester docs.
website/docs/connect-maester/connect-maester-advanced.md	Adds advanced Dataverse (SPN-style) guidance.
tests/maester-config.json	Adds DataverseEnvironmentUrl global setting + registers MT.1113–MT.1122 metadata.
tests/Maester/AIAgent/Test-AIAgentSecurity.Tests.ps1	New Pester wrapper to run MT.1113–MT.1122.
powershell/public/maester/aiagent/Test-MtAIAgentBroadSharing.ps1	New MT.1113 implementation.
powershell/public/maester/aiagent/Test-MtAIAgentBroadSharing.md	Result detail markdown for MT.1113.
powershell/public/maester/aiagent/Test-MtAIAgentNoAuthentication.ps1	New MT.1114 implementation.
powershell/public/maester/aiagent/Test-MtAIAgentNoAuthentication.md	Result detail markdown for MT.1114.
powershell/public/maester/aiagent/Test-MtAIAgentRiskyHttpConfig.ps1	New MT.1115 implementation.
powershell/public/maester/aiagent/Test-MtAIAgentRiskyHttpConfig.md	Result detail markdown for MT.1115.
powershell/public/maester/aiagent/Test-MtAIAgentEmailExfiltration.ps1	New MT.1116 implementation.
powershell/public/maester/aiagent/Test-MtAIAgentEmailExfiltration.md	Result detail markdown for MT.1116.
powershell/public/maester/aiagent/Test-MtAIAgentDormant.ps1	New MT.1117 implementation.
powershell/public/maester/aiagent/Test-MtAIAgentDormant.md	Result detail markdown for MT.1117.
powershell/public/maester/aiagent/Test-MtAIAgentAuthorAuthentication.ps1	New MT.1118 implementation.
powershell/public/maester/aiagent/Test-MtAIAgentAuthorAuthentication.md	Result detail markdown for MT.1118.
powershell/public/maester/aiagent/Test-MtAIAgentHardCodedCredentials.ps1	New MT.1119 implementation.
powershell/public/maester/aiagent/Test-MtAIAgentHardCodedCredentials.md	Result detail markdown for MT.1119.
powershell/public/maester/aiagent/Test-MtAIAgentMcpTools.ps1	New MT.1120 implementation.
powershell/public/maester/aiagent/Test-MtAIAgentMcpTools.md	Result detail markdown for MT.1120.
powershell/public/maester/aiagent/Test-MtAIAgentMissingInstructions.ps1	New MT.1121 implementation.
powershell/public/maester/aiagent/Test-MtAIAgentMissingInstructions.md	Result detail markdown for MT.1121.
powershell/public/maester/aiagent/Test-MtAIAgentOrphaned.ps1	New MT.1122 implementation.
powershell/public/maester/aiagent/Test-MtAIAgentOrphaned.md	Result detail markdown for MT.1122.
powershell/public/Connect-Maester.ps1	Adds Dataverse service option + Az-context reuse + token preflight attempt.
powershell/public/Disconnect-Maester.ps1	Disconnects Az when Dataverse was used; adds error handling.
powershell/internal/Get-MtAIAgentInfo.ps1	New Dataverse OData retrieval + mapping + caching for agent data.
powershell/internal/Clear-ModuleVariable.ps1	Clears cached AIAgentInfo on reset.
powershell/Maester.psm1	Adds AIAgentInfo to the session object.
powershell/Maester.psd1	Exports the 10 new Test-MtAIAgent* functions.
build/aitools/test-metadata/maester-config.json	Adds Dataverse environment URL to test metadata sample config.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[SamErde]

Thanks for the contribution, @lnfernux! I'm not able to test at this point but did do a quick review of the overall structure and it looks like you covered a lot! Something to consider with regards to the verbiage and labeling of this set of tests: The "Dataverse" refers to the enterprise data platform for Microsoft 365 as a whole. The description of/in these tests seems to equate "dataverse" with "Copilot Studio" and related components. What do you think about creating a little more clarity between the general dataverse tests and those that specifically pertain to Copilot Studio or AI-related features?

[lnfernux]

Thanks for the contribution, @lnfernux! I'm not able to test at this point but did do a quick review of the overall structure and it looks like you covered a lot! Something to consider with regards to the verbiage and labeling of this set of tests: The "Dataverse" refers to the enterprise data platform for Microsoft 365 as a whole. The description of/in these tests seems to equate "dataverse" with "Copilot Studio" and related components. What do you think about creating a little more clarity between the general dataverse tests and those that specifically pertain to Copilot Studio or AI-related features?

Yeah, I agree to a certain degree. The tests are only for Copilot Studio as of now, but this PR also uses the dataverse API (https://learn.microsoft.com/en-us/power-apps/developer/data-platform/webapi/overview) so that's why it refers to that.

I can make some adjustments to the test naming and labeling to reflect that the tests are for Copilot Studio, and the connection is via dataverse api.

[lnfernux]

@SamErde I've done a basic rework, where I've moved all Dataverse references into the latter part.

Basically now everything will be referred to as "Copilot Studio" tests, and refer to the dataverse api as the connection method, like "Copilot Studio (via dataverse API)". If you are not connected, error messages should still mention that the dataverse api url isn't configured (because that's still the API we are using).

Hopefully that looks and reads a bit better.

Quick edit:

If you need an SPN with read access to my test environment and I can fix that for you if you want to test.

[SamErde]

@SamErde I've done a basic rework, where I've moved all Dataverse references into the latter part.

Basically now everything will be referred to as "Copilot Studio" tests, and refer to the dataverse api as the connection method, like "Copilot Studio (via dataverse API)". If you are not connected, error messages should still mention that the dataverse api url isn't configured (because that's still the API we are using).

Hopefully that looks and reads a bit better.

Quick edit:

If you need an SPN with read access to my test environment and I can fix that for you if you want to test.

Nice, those updates look good.