Agent knowledge base

[wisedev-pstach]

Description:
Currently, an agent can only have one data source at a time, as defined in the WithSource method:

public AgentContext WithSource(IAgentSource source, AgentSourceType type)
{
    _agent.Context.Source = new AgentSource()
    {
        Details = source,
        Type = type
    };
    return this;
}

The goal of this feature is to allow agents to work with multiple sources instead of being restricted to a single one. The agent should be able to reference various data sources and intelligently determine which ones to query based on the context of a prompt.

Implementation Plan:

• Modify the agent’s data model to support multiple sources instead of overwriting the previous source when calling WithSource.
• Introduce an index.json file that maintains a structured list of sources, helping the agent determine which ones to consult.
• Implement logic that allows the agent to dynamically choose relevant sources instead of searching everything blindly.

Example Use Case:

Prompt:

"Which employee bought 500kg of sausages?!"

Knowledge Base:

Contains the last 50 invoices, sorted by employee name.

Example index.json file: (only example, i think it should normally accept AgentSource objects)

{
  "files": [
    "mark_tx/32/2024",
    "mark_tx/42/2024",
    "edgar_tx/41/2024",
    "..."  // More files 
  ],
    "...",  // Placeholder for additional data sources, like databases or APIs/ webpages etc
  ]
}

Expected Behavior:

• The agent scans index.json and identifies relevant sources based on the question.
• It retrieves and processes data from structured files as needed.
• It determines that Edgar made the purchase and provides a response:

Answer: "It was Edgar!"

The index.json structure allows for flexible expansion, making it easy to add more files and other types of data sources as needed.