docs: add policy-as-code tutorial chapter 4 (conditional policies)

[harinarayansrivatsan]

Summary

• Chapter 4 tutorial teaching conditional policies and conflict resolution
• Environment-aware rules: same agent behaves differently in dev vs prod
• Two-policy conflict scenario: security team blocks send_email, support team allows it
• Four conflict resolution strategies: deny_overrides, allow_overrides, priority_first_match, most_specific_wins
• Scope hierarchy (GLOBAL → TENANT → ORGANIZATION → AGENT) with specificity-based resolution
• Includes runnable Python example and three YAML policy files
• Updated Chapter 3 "What's missing?" section to link forward to Chapter 4
• Updated README table to link Chapter 4 and fix "coming soon" range to 05–07
• Normalized README.md line endings from CRLF to LF

Test plan

• Run python docs/tutorials/policy-as-code/examples/04_conditional_policies.py
• Verify all links between chapters work (Ch3 → Ch4, README → Ch4)
• Verify YAML snippets in markdown match the example files
• Verify example output in markdown matches actual script output

Ref #706

[github-actions]

🤖 AI Agent: security-scanner — Security Analysis of the Pull Request

Security Analysis of the Pull Request

This pull request primarily introduces a new tutorial chapter on conditional policies, conflict resolution strategies, and scope hierarchy for policy governance. While it is documentation-heavy, it includes runnable Python examples and YAML policy files, which must be reviewed for potential security vulnerabilities.

---

Findings

1. Prompt Injection Defense Bypass

Rating: 🔵 LOW
Analysis: The tutorial examples show how policies are evaluated based on input fields such as tool_name and environment. While the examples do not directly involve user input, in real-world scenarios, these fields could be populated by user-controlled data. If the system does not sanitize or validate these inputs, it could lead to prompt injection attacks, especially if the policies are used to govern AI agents that process natural language.
Recommendation: Ensure that all input fields (e.g., tool_name, environment) are sanitized and validated before policy evaluation. Add a note in the tutorial about the importance of input validation when implementing these policies in production.

---

2. Policy Engine Circumvention

Rating: 🟠 HIGH
Analysis: The PRIORITY_FIRST_MATCH conflict resolution strategy is inherently fragile because it resolves conflicts based on the order in which policies are loaded. This could allow an attacker to manipulate the order of policies (e.g., by injecting a higher-priority policy earlier in the list) to bypass security controls.
Recommendation: Strongly discourage the use of PRIORITY_FIRST_MATCH in production environments. Update the documentation to include a warning about the risks of this strategy and recommend using safer strategies like DENY_OVERRIDES or MOST_SPECIFIC_WINS.

---

3. Trust Chain Weaknesses

Rating: 🔵 LOW
Analysis: The tutorial does not mention any mechanisms for verifying the authenticity or integrity of the YAML policy files. If these files are tampered with, it could lead to unauthorized policy changes.
Recommendation: Add a section in the tutorial to emphasize the importance of securing policy files, such as using digital signatures, checksums, or storing them in a secure, access-controlled environment.

---

4. Credential Exposure

Rating: 🔵 LOW
Analysis: There is no evidence of sensitive information being exposed in the provided code or documentation. However, the tutorial does not explicitly warn against including sensitive information (e.g., API keys, secrets) in policy files or logs.
Recommendation: Add a note in the tutorial to caution users against including sensitive information in policy files or debug output.

---

5. Sandbox Escape

Rating: 🔵 LOW
Analysis: The tutorial does not involve code execution that could lead to a sandbox escape. However, the policies govern AI agents, which might execute commands or scripts. If these agents are not properly sandboxed, they could potentially execute malicious commands.
Recommendation: Add a note in the tutorial emphasizing the importance of running AI agents in a secure, isolated environment to prevent potential sandbox escapes.

---

6. Deserialization Attacks

Rating: 🟡 MEDIUM
Analysis: The PolicyDocument.from_yaml() method is used to load YAML policy files. If these files are not properly sanitized or validated, they could be exploited for deserialization attacks, especially if the YAML parser supports arbitrary object deserialization.
Recommendation: Ensure that the YAML parser used in PolicyDocument.from_yaml() does not allow the execution of arbitrary code during deserialization. If using a library like PyYAML, ensure it is configured to use safe_load instead of load. Add a note in the tutorial about the importance of using safe YAML parsing methods.

---

7. Race Conditions

Rating: 🔵 LOW
Analysis: The tutorial does not introduce any multi-threaded or concurrent operations that could lead to race conditions. However, in real-world implementations, concurrent policy evaluations could lead to time-of-check-to-time-of-use (TOCTOU) vulnerabilities if the policy state changes during evaluation.
Recommendation: Add a note in the tutorial to highlight the importance of ensuring atomicity in policy evaluations, especially in concurrent environments.

---

8. Supply Chain Risks

Rating: 🟡 MEDIUM
Analysis: The tutorial depends on the agent-os-kernel package, which is not reviewed in this pull request. If this package or its dependencies are compromised (e.g., through dependency confusion or typosquatting), it could introduce vulnerabilities.
Recommendation: Ensure that agent-os-kernel and its dependencies are audited for security vulnerabilities. Use dependency management tools like pip-audit or safety to regularly scan for known vulnerabilities.

---

Summary of Findings

Finding	Rating	Attack Vector	Recommendation
Prompt Injection Defense Bypass	🔵 LOW	Unsanitized input fields like tool_name or environment could lead to prompt injection attacks.	Validate and sanitize all input fields before policy evaluation.
Policy Engine Circumvention	🟠 HIGH	PRIORITY_FIRST_MATCH strategy is fragile and could be manipulated to bypass security controls.	Discourage its use in production; recommend safer strategies like DENY_OVERRIDES.
Trust Chain Weaknesses	🔵 LOW	YAML policy files could be tampered with if not secured.	Use digital signatures or checksums to verify policy file integrity.
Credential Exposure	🔵 LOW	No evidence of exposed credentials, but users might include sensitive data in policy files or logs.	Warn users against including sensitive information in policy files or debug output.
Sandbox Escape	🔵 LOW	Agents governed by these policies might execute commands unsafely if not sandboxed.	Emphasize the importance of running agents in secure, isolated environments.
Deserialization Attacks	🟡 MEDIUM	Unsafe YAML deserialization could allow arbitrary code execution.	Use safe YAML parsing methods like safe_load.
Race Conditions	🔵 LOW	Concurrent policy evaluations could lead to TOCTOU vulnerabilities.	Ensure atomicity in policy evaluations in concurrent environments.
Supply Chain Risks	🟡 MEDIUM	Dependency on agent-os-kernel could introduce vulnerabilities if it or its dependencies are unsafe.	Audit agent-os-kernel and its dependencies for known vulnerabilities.

---

Recommendations for the Pull Request

1. Documentation Updates:

   • Add warnings about the risks of using PRIORITY_FIRST_MATCH in production.
   • Include notes on input validation, safe YAML parsing, and securing policy files.
   • Highlight the importance of sandboxing agents and ensuring atomicity in policy evaluations.

2. Code Updates:

   • Verify that PolicyDocument.from_yaml() uses safe YAML parsing methods (e.g., safe_load).
   • Consider implementing safeguards against policy order manipulation when using PRIORITY_FIRST_MATCH.

3. Dependency Audit:

   • Perform a security audit of the agent-os-kernel package and its dependencies to mitigate supply chain risks.

By addressing these issues, the security of the toolkit and its downstream users can be significantly improved.

[github-actions]

Welcome to the Agent Governance Toolkit! Thanks for your first pull request.
Please ensure tests pass, code follows style (ruff check), and you have signed the CLA.
See our Contributing Guide.

[github-actions]

🤖 AI Agent: code-reviewer

Review Summary

The pull request introduces Chapter 4 of the "Policy-as-Code" tutorial, focusing on conditional policies and conflict resolution strategies. The tutorial is well-structured, provides clear examples, and includes runnable Python scripts and YAML policy files. The addition of conflict resolution strategies and scope hierarchy is a significant enhancement to the library's documentation. However, there are some areas that require attention to ensure correctness, security, and maintainability.

---

🔴 CRITICAL: Security Concerns

1. Conflict Resolution Strategy Default Behavior:

   • The PolicyConflictResolver class allows multiple strategies for resolving conflicts, including ALLOW_OVERRIDES. While this is documented as a business decision, it is inherently risky in security-sensitive contexts. Allowing any policy to override a deny decision could lead to security bypasses, especially if a malicious actor gains control over a lower-scope policy.
   • Action: Add explicit warnings in the documentation and code comments about the risks of using ALLOW_OVERRIDES. Consider implementing runtime checks or safeguards to ensure this strategy is not used in sensitive environments.

2. Default Actions in Policies:

   • The default action for the 04_env_policy.yaml is deny, which is safe. However, the default action for the 04_global_policy.yaml is allow. This could lead to unintended security bypasses if no rules match.
   • Action: Recommend changing the default action in 04_global_policy.yaml to deny for safer defaults.

3. Environment Field Validation:

   • The environment field in the example policies is used for conditional checks but lacks validation. If an agent sends an invalid or unexpected environment value, it could bypass rules.
   • Action: Ensure that the environment field is validated against a predefined list of acceptable values (e.g., ["development", "staging", "production"]) in the PolicyEvaluator.

---

🟡 WARNING: Potential Breaking Changes

1. Scope Hierarchy Behavior:

   • The introduction of MOST_SPECIFIC_WINS strategy and scope hierarchy (GLOBAL → TENANT → ORGANIZATION → AGENT) changes how conflicts are resolved. This could lead to different outcomes for existing policies if users switch to this strategy.
   • Action: Clearly document this behavior and provide migration guidance for users transitioning from other strategies.

2. Priority Tie Resolution:

   • The PRIORITY_FIRST_MATCH strategy resolves ties based on input order. This behavior is fragile and could lead to unexpected results if policies are loaded in a different order.
   • Action: Consider adding a warning in the documentation about the risks of relying on input order for tie resolution.

---

💡 Suggestions for Improvement

1. Thread Safety:

   • The PolicyEvaluator and PolicyConflictResolver classes should be reviewed for thread safety. If these classes are used in concurrent agent execution scenarios, ensure that they are stateless or implement locking mechanisms.
   • Action: Add unit tests to verify thread safety in concurrent execution scenarios.

2. OWASP Agentic Top 10 Compliance:

   • The tutorial does not explicitly address OWASP Agentic Top 10 risks, such as "Policy Injection" or "Improper Policy Enforcement."
   • Action: Add a section to the tutorial that discusses how the library mitigates these risks.

3. Type Safety and Validation:

   • The CandidateDecision and PolicyScope classes should be reviewed for type safety. Ensure that all fields are validated using Pydantic models or equivalent.
   • Action: Add Pydantic-based validation for these classes and include type hints in the documentation.

4. Backward Compatibility:

   • The tutorial introduces new concepts like PolicyConflictResolver and ConflictResolutionStrategy. Ensure that these additions do not break existing functionality.
   • Action: Add tests to verify backward compatibility with existing policies.

5. Policy Testing Framework:

   • The tutorial mentions policy testing in Chapter 6 but does not provide details. Consider adding a basic testing framework for users to validate their policies.
   • Action: Provide a simple pytest-based example for testing policies.

6. Documentation Enhancements:

   • The tutorial is comprehensive but could benefit from visual aids (e.g., diagrams for scope hierarchy and conflict resolution strategies).
   • Action: Add diagrams to illustrate the scope hierarchy and conflict resolution process.

---

Conclusion

The pull request is a valuable addition to the repository, enhancing the tutorial's depth and usability. However, addressing the critical security concerns and potential breaking changes is essential before merging. Implementing the suggested improvements will further strengthen the library's reliability and security posture.

[github-actions]

🤖 AI Agent: code-reviewer

Review Summary

This pull request introduces Chapter 4 of the "Policy-as-Code" tutorial, focusing on conditional policies and conflict resolution. The tutorial is well-structured, with clear examples, YAML configurations, and Python code snippets. The addition of conflict resolution strategies and scope hierarchy is a significant enhancement to the policy engine's capabilities. However, there are a few areas where improvements and clarifications are necessary.

---

🔴 CRITICAL

1. Policy Evaluation and Default Deny Behavior

   • The 04_env_policy.yaml file has a defaults section with action: deny. While this is a good security practice, the tutorial does not emphasize the importance of this default deny behavior. Failing to highlight this could lead to users creating policies with unintended permissive defaults, which is a security risk.
   • Action: Explicitly emphasize the importance of setting a default deny action in the tutorial text. Consider adding a warning about the risks of not doing so.

2. Priority Tie-Breaking

   • The PRIORITY_FIRST_MATCH strategy relies on input order to resolve ties. This behavior is fragile and could lead to unexpected results if policies are dynamically loaded or reordered. This could result in a security bypass if a lower-priority policy is inadvertently evaluated first.
   • Action: Add a warning in the tutorial about the risks of relying on input order for tie-breaking. Consider recommending that users avoid assigning the same priority to conflicting rules.

3. Environment Field Validation

   • The environment field in the YAML policy is critical for conditional policies. However, there is no mention of validation for this field in the tutorial or the provided code snippets. If the environment field is misspelled or contains unexpected values, the policy may not work as intended, leading to potential security gaps.
   • Action: Ensure that the PolicyEvaluator or PolicyDocument class validates the environment field against a predefined set of acceptable values (e.g., development, staging, production). Update the tutorial to reflect this validation.

---

🟡 WARNING

1. Backward Compatibility
   • The introduction of PolicyConflictResolver and ConflictResolutionStrategy is a significant addition to the library. If existing users have implemented custom conflict resolution logic, this change might lead to compatibility issues.
   • Action: Clearly document any changes to the public API in the release notes. Provide migration guidance for users who may have implemented custom conflict resolution logic.

---

💡 SUGGESTIONS

1. Thread Safety

   • The tutorial does not address whether the PolicyEvaluator and PolicyConflictResolver are thread-safe. This is critical for concurrent agent execution.
   • Action: Add a note in the tutorial and documentation about the thread safety of these components. If they are not thread-safe, provide guidance on how to use them safely in multi-threaded environments.

2. Error Handling in Examples

   • The Python examples in the tutorial do not include error handling for cases like missing or malformed YAML files.
   • Action: Add error handling to the examples to catch and report issues like file not found, invalid YAML syntax, or missing required fields.

3. Conflict Resolution Strategy Defaults

   • The tutorial does not specify a default conflict resolution strategy for the PolicyConflictResolver. This could lead to confusion for users who do not explicitly set a strategy.
   • Action: Clarify the default conflict resolution strategy in the tutorial and documentation. If no default exists, consider setting one (e.g., DENY_OVERRIDES for safety).

4. YAML Schema Validation

   • The tutorial does not mention whether the YAML policy files are validated against a schema before being loaded. This could lead to runtime errors or unexpected behavior if the YAML files are malformed.
   • Action: Ensure that the PolicyDocument.from_yaml method includes schema validation. Update the tutorial to explain this validation process.

5. Conflict Resolution Logging

   • The tutorial does not mention whether the PolicyConflictResolver logs its decision-making process. Logging this information would be valuable for debugging and auditing.
   • Action: Consider adding logging to the PolicyConflictResolver to record how conflicts are resolved and which strategy was used.

6. Example Output Validation

   • The tutorial includes example outputs for the Python scripts, but there is no automated test to verify that the actual output matches the documented output.
   • Action: Add automated tests to validate that the example scripts produce the expected output. This will ensure the tutorial remains accurate as the library evolves.

7. Clarify Scope Hierarchy

   • The tutorial introduces the scope hierarchy (GLOBAL → TENANT → ORGANIZATION → AGENT) but does not explain how these scopes are assigned in practice.
   • Action: Add a section explaining how users can assign scopes to policies and what the default scope is if none is specified.

---

Final Notes

This is a well-written and valuable addition to the tutorial series. Addressing the critical and suggested issues will not only improve the tutorial's clarity but also enhance the security and usability of the library.

[Copilot]

Pull request overview

Adds Chapter 4 to the Policy-as-Code tutorial, introducing environment-aware conditional rules and explicit conflict resolution between policies authored by different teams.

Changes:

• Added Chapter 4 markdown tutorial covering conditional policies, conflict scenarios, four resolution strategies, and scope specificity.
• Added runnable Chapter 4 Python example plus three YAML policy files (environment, global security, and support team).
• Updated tutorial navigation (README chapter list and Chapter 3 “What’s missing?” forward link to Chapter 4).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
docs/tutorials/policy-as-code/README.md	Adds Chapter 4 link and updates “coming soon” range.
docs/tutorials/policy-as-code/03-rate-limiting.md	Updates “What’s missing?” to point to Chapter 4 with revised framing.
docs/tutorials/policy-as-code/04-conditional-policies.md	New Chapter 4 tutorial content: conditional policies + conflict resolution + scopes.
docs/tutorials/policy-as-code/examples/04_conditional_policies.py	New runnable example demonstrating environment conditions and conflict resolution strategies.
docs/tutorials/policy-as-code/examples/04_env_policy.yaml	New environment-aware policy used by Chapter 4.
docs/tutorials/policy-as-code/examples/04_global_policy.yaml	New global security policy for conflict examples.
docs/tutorials/policy-as-code/examples/04_support_team_policy.yaml	New team policy that conflicts with global policy for demonstration.

[github-actions]

🤖 AI Agent: code-reviewer

Pull Request Review: docs: add policy-as-code tutorial chapter 4 (conditional policies)

---

🔴 CRITICAL: Security Concerns

1. Conflict Resolution Strategies:

   • The ALLOW_OVERRIDES strategy could lead to security bypasses if improperly configured. While this is a business decision, it should be accompanied by clear warnings in the documentation about its risks, especially in environments where security policies are critical (e.g., production). Consider adding a disclaimer in the tutorial emphasizing the potential for abuse or unintended consequences.

2. Default Actions:

   • The environment-aware policy (04_env_policy.yaml) uses a default action of deny. While this is generally safe, the tutorial mentions that staging falls through to the default. This could lead to confusion or misconfiguration in real-world scenarios. Ensure that users are explicitly warned about the importance of defining rules for all expected environments.

3. Scope Hierarchy:

   • The MOST_SPECIFIC_WINS strategy relies heavily on scope labels (GLOBAL, TENANT, ORGANIZATION, AGENT). If these labels are misused or misconfigured, it could lead to sandbox escape vectors where less secure policies override broader ones. Consider adding validation checks or warnings in the PolicyConflictResolver to ensure scopes are correctly applied.

---

🟡 WARNING: Backward Compatibility

1. Conflict Resolution API:

   • The introduction of PolicyConflictResolver and its strategies (DENY_OVERRIDES, ALLOW_OVERRIDES, etc.) may change how existing users interact with the policy engine. If users were previously relying on implicit priority-based resolution, this could lead to unexpected behavior. Ensure backward compatibility by documenting how to migrate existing policies to use the new conflict resolution strategies.

2. Normalized Line Endings:

   • The change from CRLF to LF in README.md could affect users on Windows systems who rely on specific line-ending formats. While this is minor, it may cause issues in environments where line endings are strictly enforced.

---

💡 Suggestions for Improvement

1. Policy Testing:

   • Add a section in the tutorial that demonstrates how to systematically test policies using pytest. For example, users could write tests to ensure that DENY_OVERRIDES always denies conflicting actions or that MOST_SPECIFIC_WINS resolves conflicts correctly based on scope.

2. Environment Validation:

   • Enhance the environment-aware policy example (04_env_policy.yaml) by including a validation mechanism for the environment field. For instance, restrict valid environments to a predefined list (development, staging, production) using Pydantic models.

3. Conflict Resolution Examples:

   • Expand the tutorial to include more complex conflict scenarios, such as three or more policies with overlapping rules. This would help users understand how the resolution strategies scale with complexity.

4. OWASP Agentic Top 10 Compliance:

   • Include a section in the tutorial that maps the conflict resolution strategies to OWASP Agentic Top 10 principles. For example, highlight how DENY_OVERRIDES aligns with "Fail Securely" and how MOST_SPECIFIC_WINS could be misused to bypass security controls.

5. Interactive Examples:

   • Provide an interactive Jupyter Notebook version of the tutorial. This would allow users to experiment with different policies and resolution strategies in real-time.

---

✅ Positive Observations

1. Documentation Quality:

   • The tutorial is well-written and provides clear, step-by-step instructions. The inclusion of YAML examples and Python code makes it easy for users to follow along.

2. Conflict Resolution Strategies:

   • The explanation of the four strategies (DENY_OVERRIDES, ALLOW_OVERRIDES, PRIORITY_FIRST_MATCH, MOST_SPECIFIC_WINS) is thorough and provides valuable insights into their trade-offs.

3. Runnable Examples:

   • The inclusion of a runnable Python script (04_conditional_policies.py) is excellent for hands-on learning.

4. Scope Hierarchy:

   • The tutorial does a good job explaining the scope hierarchy and its impact on conflict resolution. This is crucial for understanding how policies interact in complex organizations.

---

Action Items

1. Add security disclaimers for ALLOW_OVERRIDES and default actions in environment-aware policies.
2. Document migration steps for users transitioning to the new conflict resolution API.
3. Enhance the tutorial with testing examples and validation mechanisms.
4. Map conflict resolution strategies to OWASP Agentic Top 10 principles.
5. Consider providing an interactive Jupyter Notebook version of the tutorial.

---

Final Recommendation

Merge with the above changes to address critical security concerns and ensure backward compatibility.

[harinarayansrivatsan]

Quick update on the decisions I made for Chapter 4:

I expanded the scope beyond just environment-aware rules. I added conflict resolution in the same chapter since it naturally follows. I used the existing PolicyConflictResolver and PolicyScope hierarchy. Will gradually increase complexity to more production level policy enforcements.

[imran-siddique]

Great continuation of the tutorial series @harinarayansrivatsan! Chapter 4 looks well-structured with environment-based conditional policies.