BacklogGPT - Automate Backlog Creation in Azure DevOps with a Multi-Agent GenAI solution based on Logic Apps

Overview

This repository contains two GenAI agents that work together to automate the process of creating Features and User Stories for an Epic in Azure DevOps. This can be used to accelerate the backlog creation process. It won't replace the need for human input and review, but it can help generate a starting point for the backlog. So don't stop using your own brain and experience when creating backlogs !!! :-)

The first agent can generate a backlog with relevant features and user stories based on the Epic details:

The second agent refines the user stories by adding an extensive description including:

• Goals
• T-shirt sizing
• Business value
• User Experience
• Acceptance criteria
• Technical options to consider

Important! 1 This list of things you want in the description is not exhaustive and can be customized/expanded based on the requirements of the project. Important! 2 Make sure to ALWAYS review the generated backlog items. AI might be hallucinating and generate irrelevant or incorrect items. So always review the generated items before adding them to the backlog. Improtant! 3 In this repo I'm using Logic App consumption plan. If you will use this in production, you might want to consider using a different plan based on your requirements such as network security, performance, etc.

Benefits

• Automation: Streamlines the process of creating Features and User Stories.
• Consistency: Ensures that all backlog items are relevant and aligned with the Epic.
• Efficiency: Saves time and effort in backlog creation.
• Creativity: Generates new ideas and perspectives for backlog items where you might not have thought of. (Personally, I find this the most interesting benefit.)
• Costs: In this repo I'm using Logic App consumption plan for both agents. Using consumption plan one agent will costs you around 20 euros per month for the hosting. Setting up an agent could be done within 1 hour if all goes well. So, it's a cost-effective solution.

Requirements

• Azure Subscription
• Azure DevOps Project
• Access to a GPT-4 (LLM) endpoint

Note: Other models then GPT-4 might work as well, however the JSON template containing the system prompts are based on the GPT-4 specs. If you change the model, you might have to adjust the JSON templates as well. And, of course, you will have to test everyting thoroughly.

You can choose to import the Logic App using the Json templates from this repository or you can create the Logic App in the designer via the Azure Portal using the step by step guide below.

Getting Started - Using the JSON Import option

Setup the Backlog Generator Agent using Json file

1. Clone this repository.
2. Make sure you have an Azure Subscription and an Azure DevOps project.
3. In Azure Portal create an Azure OpenAI, deploy a GPT-4 model resource and copy the API key and endpoint URL for chat completions. (i.e. https://<YOUR ENDPOINT>.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview)
4. You will have to change all the parameters in the 1-BacklogGeneratorAgent.json file to set up the Logic App for your environment. The parameters include:
   • API KEY
   • YOUR ENDPOINT
   • YOUR DEVOPS ORGANIZATION
   • YOUR DEVOPS PROJECT
   • etc Make sure to review the complete file !
5. In Azure Portal, create a new Logic App and import the 1-BacklogGeneratorAgent.json file in the Development Tools section > Logic App code view.
6. Connect the Logic App to your Azure DevOps project.
7. Start automating your backlog creation process!

Setup the User Story Refinement Agent using Json file

1. Change the parameters (similar to the other Logic App) in the 2-UserStoryRefinementAgent.json file to set up the Logic App for your environment. The parameters are recognizable in this way <PARAMETER NAME>. You will have to replace them with your own values.
2. Connect the Logic App to your Azure DevOps project.
3. Start automating your user story refinement process!

Getting Started - Using the Logic App designer

Setup the Backlog Generator Agent using Logic App designer

1. In Azure Portal, create a new Logic App. Go to the Logic App Designer.
2. Add a trigger to start the Logic App. Search for "When a work item is created" and select the Azure DevOps connector.
3. Sign in to your Azure DevOps account and select the organization and project. Set the Work Item Type to Epic. Set the Priority to 1.
4. Add a new step to the Logic App and search for "Initialize variable". Set the name of the variable to WorkItemDescription and the type to String. Set the value to the Description of the work item from the previous step.
5. Add a new step to the Logic App and search for "HTTP". Set the method to POST and the URI to the OpenAI endpoint URL for chat completions. Set the headers to Content-Type and api-key with the value of <YOUR API KEY>. Set the body to the following JSON:

       {
       "frequency_penalty": 0,
       "max_tokens": 800,
       "messages": [
           {
           "content": "You are an AI assistant that helps people define one or more features and user stories derived from a use case description as part of an epic.",
           "role": "system"
           },
           {
           "content": "Generate a list of features and user stories for the below description in json format. A feature will have one or more user stories as child work items. Only output the json content. Use the following structure for the json file: Epic, Features, Feature, UserStories, Story. # Start Use case description:<br /> @{variables('WorkItemDescription')}<br /># End Use case description.",
           "role": "user"
           }
       ],
       "presence_penalty": 0,
       "stop": null,
       "temperature": 0.7,
       "top_p": 0.95
       }

6. Add a new step to the Logic App and search for "Parse JSON". Set the content to the body of the HTTP response from the previous step. Set the schema to the following JSON:

       {
           "properties": {
               "Epic": {
                   "type": "string"
               },
               "Features": {
                   "items": {
                       "properties": {
                           "Feature": {
                               "type": "string"
                           },
                           "UserStories": {
                               "items": {
                                   "properties": {
                                       "Story": {
                                           "type": "string"
                                       }
                                   },
                                   "required": [
                                       "Story"
                                   ],
                                   "type": "object"
                               },
                               "type": "array"
                           }
                       },
                       "required": [
                           "Feature",
                           "UserStories"
                       ],
                       "type": "object"
                   },
                   "type": "array"
               }
           },
           "type": "object"
       } 

7. Create a For Each loop and set the output Body Features of the Parse JSON step as the input.
8. Add a new step to the For Each loop and search for "Create a work item". Set the organization and project to your Azure DevOps project. Set the title to the Body Feature from the Parse JSON step. Set the work item type to Feature. Set the Link URL to URL from the trigger step. Set the Link Type to Hierarchy-reverse.
9. Within the For Each loop, add another For Each loop and set the output Body UserStories of the Parse JSON step as the input.
10. Add a new step to the inner For Each loop and search for "Create a work item". Set the organization and project to your Azure DevOps project. Set the title to the Story from the Parse JSON step. Set the work item type to User Story. Set the Link URL to URL from the trigger step. Set the Link Type to Hierarchy-reverse. Set the Priority to 4. The priority will be used as a switch for the triggers.
11. Outside all loops, add a new step to the Logic App and search for "Update a work item". Set the organization and project to your Azure DevOps project. Set the work item ID to the Id from the trigger step. Set the Priority to 2. Set the Title to the Title from the trigger step. Set the Description to the Description from the trigger step.
12. Save the Logic App.
13. Go to Azure DevOps and create an Epic with a description. If all goes well the Logic App will be triggered and a backlog containing Features and User Stories will be created.

Setup the User Story Refinement Agent using Logic App designer

1. In Azure Portal, create another Logic App. Go to the Logic App Designer.
2. Add a trigger to start the Logic App. Search for "When a work item is created" and select the Azure DevOps connector.
3. Sign in to your Azure DevOps account and select the organization and project. Select the work item type to User Story. Select the priority to 4.
4. Add a new step to the Logic App and search for "Initialize variable". Set the name of the variable to WorkItemDescription and the type to String. Set the value to the Description of the work item from the previous step.
5. Add another "Initialize variable" step and set the name of the variable to RefinedWorkItem and the type to String. Leave the value empty.
6. Add a new step to the Logic App and search for "HTTP". Set the method to POST and the URI to the OpenAI endpoint URL for chat completions. Set the headers to Content-Type and api-key with the value of <YOUR API KEY>. Set the body to the following JSON:

       {
       "frequency_penalty": 0,
       "max_tokens": 800,
       "messages": [
           {
           "content": "You are an AI assistant that helps people refine user stories.",
           "role": "system"
           },
           {
           "content": "Generate a description for the following user story: @{triggerBody()?['fields']?['System_Title']}. Use the following sections: Goals, T-shirt sizing, Business value, User experience, Acceptance criteria, Technical options to consider. Use HTML formatting.",
           "role": "user"
           }
       ],
       "presence_penalty": 0,
       "stop": null,
       "temperature": 0.7,
       "top_p": 0.95
       }

7. Add a new step of type "Set variable" and set the value of the variable RefinedWorkItem to the Body of the HTTP response from the previous step by using the expression body('HTTP')['choices'][0]['message']['content'].
8. Add a new step to the Logic App and search for "Update a work item". Set the organization and project to your Azure DevOps project. Set the work item ID to the Id from the trigger step. Set the Description to the RefinedWorkItem variable. Set the Priority to 2.

Contributing

Feel free to submit issues and pull requests to improve this project.

License

This project is licensed under the MIT License.