diff --git a/application/single_app/config.py b/application/single_app/config.py index caf09fc8..12906ce8 100644 --- a/application/single_app/config.py +++ b/application/single_app/config.py @@ -88,7 +88,7 @@ EXECUTOR_TYPE = 'thread' EXECUTOR_MAX_WORKERS = 30 SESSION_TYPE = 'filesystem' -VERSION = "0.236.012" +VERSION = "0.237.001" SECRET_KEY = os.getenv('SECRET_KEY', 'dev-secret-key-change-in-production') diff --git a/docs/explanation/features/v0.237.001/CONTROL_CENTER_APPLICATION_ROLES.md b/docs/explanation/features/v0.237.001/CONTROL_CENTER_APPLICATION_ROLES.md new file mode 100644 index 00000000..3d61f752 --- /dev/null +++ b/docs/explanation/features/v0.237.001/CONTROL_CENTER_APPLICATION_ROLES.md @@ -0,0 +1,154 @@ +# Control Center Application Roles + +## Overview + +Added two new application roles for finer-grained access control to the Control Center, enabling organizations to delegate administrative functions while maintaining security boundaries. + +**Version Implemented:** v0.237.001 + +## New Roles + +### Control Center Admin + +| Property | Value | +|----------|-------| +| **Role Name** | Control Center Admin | +| **Description** | Full administrative access to Control Center functionality | +| **Access Level** | Full read/write access to all Control Center features | + +**Permissions:** +- View all Control Center dashboards and metrics +- Manage user access and permissions +- Execute administrative operations (take ownership, transfer, delete) +- Approve/reject workflow requests +- Configure Control Center settings + +### Control Center Dashboard Reader + +| Property | Value | +|----------|-------| +| **Role Name** | Control Center Dashboard Reader | +| **Description** | Read-only access to Control Center dashboards | +| **Access Level** | View-only access to dashboards and metrics | + +**Permissions:** +- View Control Center dashboard +- View activity trends and metrics +- View user statistics +- View group and workspace information +- **Cannot** perform administrative actions +- **Cannot** modify settings or configurations + +## Use Cases + +### Scenario 1: IT Operations Team +- **Need**: Monitor system health and usage without admin capabilities +- **Solution**: Assign "Control Center Dashboard Reader" role +- **Benefit**: Visibility into metrics without risk of accidental changes + +### Scenario 2: Delegated Administration +- **Need**: Department leads manage their users' access +- **Solution**: Assign "Control Center Admin" role to specific individuals +- **Benefit**: Distributed administration without full application admin access + +### Scenario 3: Compliance Auditors +- **Need**: Review activity logs and usage patterns +- **Solution**: Assign "Control Center Dashboard Reader" role +- **Benefit**: Audit capability without modification access + +## Configuration + +### Adding Roles to Entra ID Enterprise Application + +1. Navigate to Azure Portal → Entra ID → Enterprise Applications +2. Find your SimpleChat application registration +3. Go to **App roles** +4. Add the new roles from `appRegistrationRoles.json` + +### Role Assignment + +```json +{ + "roles": [ + { + "allowedMemberTypes": ["User"], + "description": "Full administrative access to Control Center", + "displayName": "Control Center Admin", + "isEnabled": true, + "value": "ControlCenterAdmin" + }, + { + "allowedMemberTypes": ["User"], + "description": "Read-only access to Control Center dashboards", + "displayName": "Control Center Dashboard Reader", + "isEnabled": true, + "value": "ControlCenterDashboardReader" + } + ] +} +``` + +### Assigning Roles to Users + +1. Navigate to Enterprise Application → Users and groups +2. Click **Add user/group** +3. Select user(s) to assign +4. Select the appropriate role +5. Click **Assign** + +## Role Hierarchy + +``` +┌─────────────────────────────────────┐ +│ Admin │ ← Full application admin +│ (All permissions) │ +└─────────────────┬───────────────────┘ + │ +┌─────────────────▼───────────────────┐ +│ Control Center Admin │ ← Control Center admin only +│ (Full CC access, no app settings) │ +└─────────────────┬───────────────────┘ + │ +┌─────────────────▼───────────────────┐ +│ Control Center Dashboard Reader │ ← View-only access +│ (Read-only dashboard access) │ +└─────────────────────────────────────┘ +``` + +## Integration with Existing Roles + +| Existing Role | Control Center Access | +|---------------|----------------------| +| Admin | Full access (includes all CC permissions) | +| User | No Control Center access by default | +| Owner | Group-level access only | +| DocumentManager | No Control Center access | + +| New Role | Control Center Access | +|----------|----------------------| +| ControlCenterAdmin | Full CC admin access | +| ControlCenterDashboardReader | Read-only CC dashboard access | + +## Security Considerations + +1. **Principle of Least Privilege**: Assign Dashboard Reader by default, escalate to Admin only when needed +2. **Audit Trail**: All Control Center actions are logged regardless of role +3. **Role Separation**: Dashboard Reader cannot perform any destructive operations +4. **Admin Oversight**: Full Admin role retains visibility into all role assignments + +## Files Modified + +- `appRegistrationRoles.json` - Added new role definitions + +## Related Features + +- [Control Center](../v0.235.001/control_center.md) - Main Control Center functionality +- [Approval Workflow System](../v0.235.001/APPROVAL_WORKFLOW_SYSTEM.md) - Protected operations requiring approval +- [Enhanced User Management](../v0.235.001/ENHANCED_USER_MANAGEMENT.md) - User metrics and management + +## Migration Notes + +Existing deployments should: +1. Update the Entra ID app registration with new roles +2. Assign appropriate roles to users who need Control Center access +3. Review existing Admin role assignments for potential role refinement diff --git a/docs/explanation/features/v0.237.001/CONVERSATION_DEEP_LINKING.md b/docs/explanation/features/v0.237.001/CONVERSATION_DEEP_LINKING.md new file mode 100644 index 00000000..cebf392b --- /dev/null +++ b/docs/explanation/features/v0.237.001/CONVERSATION_DEEP_LINKING.md @@ -0,0 +1,141 @@ +# Conversation Deep Linking + +## Overview + +SimpleChat now supports conversation deep linking through URL query parameters. Users can share direct links to specific conversations, and the application will automatically navigate to and load the referenced conversation when the link is accessed. + +**Version Implemented:** v0.237.001 + +## Key Features + +- **Direct Conversation Links**: Share URLs that open a specific conversation +- **URL Parameter Support**: Supports both `conversationId` and `conversation_id` parameters +- **Automatic URL Updates**: Current conversation ID is automatically added to the URL +- **Browser History Integration**: Uses `replaceState` to update URLs without creating new history entries +- **Error Handling**: Graceful handling of invalid or inaccessible conversation IDs + +## How It Works + +### URL Format + +Conversations can be linked using either parameter format: + +``` +https://your-simplechat.com/?conversationId= +https://your-simplechat.com/?conversation_id= +``` + +### Automatic URL Updates + +When users select a conversation in the sidebar, the URL is automatically updated to include the conversation ID: + +```javascript +function updateConversationUrl(conversationId) { + if (!conversationId) return; + + try { + const url = new URL(window.location.href); + url.searchParams.set('conversationId', conversationId); + window.history.replaceState({}, '', url.toString()); + } catch (error) { + console.warn('Failed to update conversation URL:', error); + } +} +``` + +### Deep Link Loading + +On page load, the application checks for a `conversationId` parameter and loads that conversation: + +```javascript +// Deep-link: conversationId query param +const conversationId = getUrlParameter("conversationId") || getUrlParameter("conversation_id"); +if (conversationId) { + try { + await ensureConversationPresent(conversationId); + await selectConversation(conversationId); + } catch (err) { + console.error('Failed to load conversation from URL param:', err); + showToast('Could not open that conversation.', 'danger'); + } +} +``` + +## User Experience + +### Sharing Conversations + +1. Navigate to any conversation +2. Copy the URL from the browser address bar +3. Share the URL with colleagues +4. Recipients with access can open the link to view the conversation + +### Receiving Shared Links + +1. Click or paste a shared conversation link +2. The application loads and displays the referenced conversation +3. If the conversation doesn't exist or isn't accessible, an error toast is shown + +### Error Handling + +When a deep link fails to load: +- A toast notification appears: "Could not open that conversation." +- The user remains on the default view +- Console logging captures the error details for debugging + +## Technical Architecture + +### Frontend Components + +| File | Purpose | +|------|---------| +| [chat-onload.js](../../../../application/single_app/static/js/chat/chat-onload.js) | Handles deep link loading on page initialization | +| [chat-conversations.js](../../../../application/single_app/static/js/chat/chat-conversations.js) | `updateConversationUrl()` function for URL management | + +### Functions Involved + +| Function | Purpose | +|----------|---------| +| `getUrlParameter(name)` | Retrieves query parameter value from current URL | +| `ensureConversationPresent(id)` | Ensures conversation exists in the local list | +| `selectConversation(id)` | Loads and displays the specified conversation | +| `updateConversationUrl(id)` | Updates URL with current conversation ID | + +## Use Cases + +### Team Collaboration +- Share conversation links in chat or email for review +- Direct colleagues to specific AI interactions for discussion + +### Support and Troubleshooting +- Users can share conversation links with support staff +- Administrators can reference specific conversations in reports + +### Documentation +- Bookmark important conversations for future reference +- Create documentation links to example interactions + +## Security Considerations + +1. **Access Control**: Deep links respect existing conversation access permissions +2. **User Ownership**: Only accessible if the user has rights to the conversation +3. **No Authentication Bypass**: Users must still be logged in to access conversations +4. **Workspace Boundaries**: Workspace permissions still apply + +## Browser Compatibility + +- Uses standard `URL` and `URLSearchParams` APIs +- `history.replaceState()` for seamless URL updates +- Compatible with all modern browsers + +## Known Limitations + +- Deep links only work for conversations the current user has access to +- Links to deleted conversations will show an error +- Group/public workspace conversations require appropriate membership + +## Related Features + +- Conversation management and history +- Sidebar conversation navigation +- Chat workspace functionality diff --git a/docs/explanation/features/v0.237.001/PLUGIN_AUTH_TYPE_CONSTRAINTS.md b/docs/explanation/features/v0.237.001/PLUGIN_AUTH_TYPE_CONSTRAINTS.md new file mode 100644 index 00000000..9d2ea6e6 --- /dev/null +++ b/docs/explanation/features/v0.237.001/PLUGIN_AUTH_TYPE_CONSTRAINTS.md @@ -0,0 +1,202 @@ +# Plugin Authentication Type Constraints + +## Overview + +SimpleChat now enforces authentication type constraints per plugin type. Different plugin types may support different authentication methods based on their requirements and the APIs they integrate with. This feature provides a structured way to define and retrieve allowed authentication types for each plugin type. + +**Version Implemented:** v0.237.001 + +## Key Features + +- **Per-Plugin Auth Types**: Each plugin type can define its own allowed authentication types +- **Schema-Based Defaults**: Falls back to global AuthType enum from plugin.schema.json +- **Definition File Overrides**: Plugin-specific definition files can restrict available auth types +- **API Endpoint**: RESTful endpoint to query allowed auth types for any plugin type + +## How It Works + +### Authentication Type Resolution + +The system resolves allowed authentication types in this order: + +1. **Check Plugin Definition File**: `{plugin_type}.definition.json` + - If `allowedAuthTypes` array exists and is non-empty, use it +2. **Fallback to Global Schema**: `plugin.schema.json` + - Use the `AuthType` enum from definitions + +### API Endpoint + +``` +GET /api/plugins/{plugin_type}/auth-types +``` + +**Response:** +```json +{ + "allowedAuthTypes": ["none", "api_key", "oauth2", "basic"], + "source": "definition" +} +``` + +**Response Fields:** +| Field | Description | +|-------|-------------| +| `allowedAuthTypes` | Array of allowed authentication type strings | +| `source` | Where the types came from: "definition" or "schema" | + +## Configuration Files + +### Plugin Schema (Global Defaults) + +Location: `static/json/schemas/plugin.schema.json` + +```json +{ + "definitions": { + "AuthType": { + "enum": ["none", "api_key", "oauth2", "basic", "bearer", "custom"] + } + } +} +``` + +### Plugin Definition Files (Per-Plugin Overrides) + +Location: `static/json/schemas/{plugin_type}.definition.json` + +Example for a plugin that only supports API key authentication: + +```json +{ + "name": "weather_plugin", + "displayName": "Weather API", + "description": "Get weather information", + "allowedAuthTypes": ["none", "api_key"] +} +``` + +## Technical Architecture + +### Backend Implementation + +Location: [route_backend_plugins.py](../../../../application/single_app/route_backend_plugins.py) + +```python +@bpap.route('/api/plugins//auth-types', methods=['GET']) +@login_required +@user_required +def get_plugin_auth_types(plugin_type): + """ + Returns allowed auth types for a plugin type. Uses definition file if present, + otherwise falls back to AuthType enum in plugin.schema.json. + """ + schema_dir = os.path.join(current_app.root_path, 'static', 'json', 'schemas') + safe_type = re.sub(r'[^a-zA-Z0-9_]', '_', plugin_type).lower() + + # Try to load from plugin definition file + definition_path = os.path.join(schema_dir, f'{safe_type}.definition.json') + schema_path = os.path.join(schema_dir, 'plugin.schema.json') + + allowed_auth_types = [] + source = "schema" + + # Load defaults from schema + try: + with open(schema_path, 'r', encoding='utf-8') as schema_file: + schema = json.load(schema_file) + allowed_auth_types = ( + schema + .get('definitions', {}) + .get('AuthType', {}) + .get('enum', []) + ) + except Exception: + allowed_auth_types = [] + + # Override with definition file if present + if os.path.exists(definition_path): + try: + with open(definition_path, 'r', encoding='utf-8') as definition_file: + definition = json.load(definition_file) + allowed_from_definition = definition.get('allowedAuthTypes') + if isinstance(allowed_from_definition, list) and allowed_from_definition: + allowed_auth_types = allowed_from_definition + source = "definition" + except Exception: + pass + + return jsonify({ + "allowedAuthTypes": allowed_auth_types, + "source": source + }) +``` + +### Security + +- Plugin type is sanitized to prevent path traversal +- Only alphanumeric characters and underscores are allowed in plugin type names +- Endpoint requires user authentication + +## Common Authentication Types + +| Type | Description | Use Case | +|------|-------------|----------| +| `none` | No authentication required | Public APIs | +| `api_key` | API key in header or query | Most REST APIs | +| `oauth2` | OAuth 2.0 flow | Microsoft Graph, Google APIs | +| `basic` | Basic HTTP authentication | Legacy systems | +| `bearer` | Bearer token authentication | JWT-based APIs | +| `custom` | Custom authentication handler | Special requirements | + +## Use Cases + +### Restricting Auth for Internal Plugins + +An internal plugin might only support specific authentication: + +```json +{ + "name": "internal_hr_system", + "allowedAuthTypes": ["oauth2"] +} +``` + +### Simple Public API Plugin + +A public weather API might need no authentication: + +```json +{ + "name": "public_weather", + "allowedAuthTypes": ["none", "api_key"] +} +``` + +## Frontend Integration + +The frontend can query auth types to: +1. Display only valid authentication options in plugin configuration UI +2. Validate user selections before saving +3. Show appropriate configuration fields based on auth type + +Example usage: + +```javascript +async function loadAuthTypes(pluginType) { + const response = await fetch(`/api/plugins/${pluginType}/auth-types`); + const data = await response.json(); + return data.allowedAuthTypes; +} +``` + +## Known Limitations + +- Auth types must be predefined in the schema +- Custom auth implementations require additional plugin code +- Definition files must be manually created for each plugin type + +## Related Features + +- Plugin Management +- Action/Plugin Registration +- OpenAPI Plugin Integration diff --git a/docs/explanation/features/v0.237.001/PRIVATE_NETWORKING_SUPPORT.md b/docs/explanation/features/v0.237.001/PRIVATE_NETWORKING_SUPPORT.md new file mode 100644 index 00000000..5379b73d --- /dev/null +++ b/docs/explanation/features/v0.237.001/PRIVATE_NETWORKING_SUPPORT.md @@ -0,0 +1,179 @@ +# Private Networking Support + +## Overview + +Comprehensive private networking support for SimpleChat deployments via Azure Developer CLI (AZD) and Bicep infrastructure-as-code. This feature enables secure, isolated deployments with private endpoints, virtual networks, and private DNS zones. + +**Version Implemented:** v0.237.001 + +## Key Features + +- **Private Endpoint Support**: All Azure PaaS services can be configured with private endpoints +- **Virtual Network Integration**: Full VNet integration for App Service and dependent resources +- **Private DNS Zones**: Automated DNS zone configuration for private endpoint resolution +- **AZD Integration**: Seamless deployment via `azd up` with private networking enabled +- **Bicep Automation**: Infrastructure-as-code templates for reproducible deployments +- **Post-Deployment Security**: Automatic disabling of public network access when private networking is enabled + +## Architecture + +### Network Topology + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Virtual Network │ +│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ +│ │ App Service │ │ Private DNS │ │ Private │ │ +│ │ Subnet │ │ Zones │ │ Endpoints │ │ +│ │ │ │ │ │ Subnet │ │ +│ │ ┌───────────┐ │ │ - Cosmos DB │ │ │ │ +│ │ │SimpleChat │ │ │ - OpenAI │ │ ┌───────────┐ │ │ +│ │ │ App │──────│ - AI Search │──────│ Cosmos DB │ │ │ +│ │ └───────────┘ │ │ - Storage │ │ └───────────┘ │ │ +│ │ │ │ - Key Vault │ │ │ │ +│ └─────────────────┘ └─────────────────┘ │ ┌───────────┐ │ │ +│ │ │ Azure │ │ │ +│ │ │ OpenAI │ │ │ +│ │ └───────────┘ │ │ +│ │ │ │ +│ │ ┌───────────┐ │ │ +│ │ │ AI Search │ │ │ +│ │ └───────────┘ │ │ +│ └─────────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Supported Private Endpoints + +| Service | Private DNS Zone | +|---------|-----------------| +| Azure Cosmos DB | `privatelink.documents.azure.com` | +| Azure OpenAI | `privatelink.openai.azure.com` | +| Azure AI Search | `privatelink.search.windows.net` | +| Azure Blob Storage | `privatelink.blob.core.windows.net` | +| Azure Key Vault | `privatelink.vaultcore.azure.net` | +| Azure Document Intelligence | `privatelink.cognitiveservices.azure.com` | + +## Deployment + +### Prerequisites + +1. **Azure Subscription** with appropriate permissions +2. **Azure Developer CLI (AZD)** installed +3. **Azure CLI** installed and authenticated +4. **Permissions**: Contributor or higher on the subscription/resource group + +### AZD Deployment + +```bash +# Clone the repository +git clone https://github.com/microsoft/simplechat.git +cd simplechat/deployers + +# Initialize AZD (first time) +azd init + +# Enable private networking +azd env set ENABLE_PRIVATE_NETWORKING true + +# Deploy with private networking +azd up +``` + +### Bicep Deployment + +```bash +# Deploy with private networking parameter +az deployment sub create \ + --location eastus \ + --template-file main.bicep \ + --parameters enablePrivateNetworking=true +``` + +## Configuration Options + +### Environment Variables + +| Variable | Description | Default | +|----------|-------------|---------| +| `ENABLE_PRIVATE_NETWORKING` | Enable private endpoints for all services | `false` | +| `VNET_ADDRESS_SPACE` | Virtual network address space | `10.0.0.0/16` | +| `APP_SUBNET_PREFIX` | App Service subnet prefix | `10.0.1.0/24` | +| `PRIVATE_ENDPOINT_SUBNET_PREFIX` | Private endpoints subnet prefix | `10.0.2.0/24` | + +## Deployment Hooks + +### Post-Provision Hook +- Creates private DNS zones +- Configures private endpoints +- Sets up VNet integration + +### Pre-Deploy Hook +- Validates network configuration +- Ensures DNS resolution is working + +### Post-Up Hook +- **NEW**: Automatically disables public network access for resources when private networking is enabled +- Validates connectivity through private endpoints +- Outputs connection validation results + +## Azure Government Considerations + +### Regional Availability +- Private endpoints available in all USGov regions +- Some services may have regional restrictions + +### Model Configuration +- Azure OpenAI models may differ in government regions +- Configure model overrides as needed + +### Service Limitations +- Some preview features may not be available +- Check Azure Government documentation for current status + +## Error Handling + +The deployment scripts include: +- **Stepwise logging**: Detailed output for each deployment phase +- **Explicit error handling**: Failures caught early with clear messages +- **Troubleshooting guidance**: Helpful error messages for common issues + +## Post-Deployment Validation + +After deployment, validate: + +1. **DNS Resolution**: Private DNS zones resolve correctly +2. **Network Connectivity**: App Service can reach all services via private endpoints +3. **AI Model Connections**: Test chat functionality +4. **Search Integration**: Verify AI Search connectivity +5. **Document Processing**: Test Document Intelligence + +## Security Benefits + +1. **No Public Exposure**: Services not accessible from public internet +2. **Network Isolation**: All traffic stays within Azure backbone +3. **Reduced Attack Surface**: Minimized exposure to external threats +4. **Compliance**: Meets enterprise security requirements +5. **Data Protection**: Data never traverses public networks + +## Known Issues and Workarounds + +1. **DNS Propagation Delay**: Allow 5-10 minutes for DNS changes to propagate +2. **VNet Peering**: Additional configuration needed if peering with existing VNets +3. **On-Premises Connectivity**: Requires ExpressRoute or VPN Gateway for hybrid scenarios + +## Files Modified + +### Deployment Files +- `deployers/azure.yaml` - Enhanced hooks with logging and error handling +- `deployers/bicep/*.bicep` - Private networking Bicep templates + +### Documentation +- `deployers/bicep/README.md` - Enhanced prerequisites and USGov guidance +- `OneClickDeploy.md` - Corrected deployment button links + +## Related Documentation + +- [Azure Private Endpoints Documentation](https://docs.microsoft.com/azure/private-link/private-endpoint-overview) +- [App Service VNet Integration](https://docs.microsoft.com/azure/app-service/web-sites-integrate-with-vnet) +- [Private DNS Zones](https://docs.microsoft.com/azure/dns/private-dns-overview) diff --git a/docs/explanation/features/v0.237.001/RETENTION_POLICY_DEFAULTS.md b/docs/explanation/features/v0.237.001/RETENTION_POLICY_DEFAULTS.md new file mode 100644 index 00000000..cdca4ce5 --- /dev/null +++ b/docs/explanation/features/v0.237.001/RETENTION_POLICY_DEFAULTS.md @@ -0,0 +1,207 @@ +# RETENTION_POLICY_DEFAULTS.md + +**Feature**: Admin-Configurable Default Retention Policies +**Version**: v0.237.001 + +## Overview and Purpose + +The Retention Policy Defaults feature allows administrators to configure organization-wide default retention periods for conversations and documents across all workspace types (personal, group, and public). Users can choose to use the organization default or set their own custom retention period. Administrators also have the ability to force push defaults to override all custom policies. + +## Key Features + +- **Organization Defaults**: Set default retention periods for conversations and documents per workspace type +- **User Choice**: Users see "Using organization default (X days)" option and can override with custom settings +- **Conditional Display**: Default settings only appear for enabled workspace types +- **Force Push**: Administrators can push organization defaults to all workspaces, overriding custom settings +- **Activity Logging**: Force push actions are logged for audit purposes +- **Settings Auto-Save**: Force push automatically saves pending settings changes before executing + +## Technical Specifications + +### Architecture Overview + +The feature integrates with the existing retention policy system and adds: + +1. **Backend Settings** - 6 new settings fields for default retention values +2. **Admin UI** - Dropdown selectors in Admin Settings for each workspace type +3. **API Endpoints** - New endpoints for fetching defaults and force pushing +4. **User UI Integration** - Updated profile, control center, and workspace manager +5. **Execution Logic** - Resolution of 'default' values at policy execution time + +### New Settings Fields + +Added to `functions_settings.py`: + +| Setting | Default Value | Description | +|---------|---------------|-------------| +| `default_retention_conversation_personal` | `'none'` | Default conversation retention for personal workspaces | +| `default_retention_document_personal` | `'none'` | Default document retention for personal workspaces | +| `default_retention_conversation_group` | `'none'` | Default conversation retention for group workspaces | +| `default_retention_document_group` | `'none'` | Default document retention for group workspaces | +| `default_retention_conversation_public` | `'none'` | Default conversation retention for public workspaces | +| `default_retention_document_public` | `'none'` | Default document retention for public workspaces | + +**Note**: Value `'none'` means no automatic deletion. Numeric values represent days. + +### API Endpoints + +#### Get Retention Defaults + +**Endpoint**: `GET /api/retention-policy/defaults/` + +**Parameters**: +- `workspace_type`: One of `personal`, `group`, or `public` + +**Response**: +```json +{ + "success": true, + "workspace_type": "personal", + "defaults": { + "conversation_retention_days": "none", + "document_retention_days": "30" + } +} +``` + +**Authentication**: Requires user authentication (`@login_required`) + +#### Force Push Retention Defaults + +**Endpoint**: `POST /api/admin/retention-policy/force-push` + +**Request Body**: +```json +{ + "scopes": ["personal", "group", "public"] +} +``` + +**Response**: +```json +{ + "success": true, + "message": "Defaults pushed to 150 items", + "updated_count": 150, + "scopes": ["personal", "group"], + "details": { + "personal": 100, + "group": 50 + } +} +``` + +**Authentication**: Requires admin authentication (`@admin_required`) + +### File Structure + +**Backend Files**: +- `functions_settings.py` - New default retention settings fields +- `route_frontend_admin_settings.py` - Handling for saving new settings +- `route_backend_retention_policy.py` - New API endpoints +- `functions_retention_policy.py` - `resolve_retention_value()` helper function +- `functions_activity_logging.py` - `log_retention_policy_force_push()` function + +**Frontend Files**: +- `templates/admin_settings.html` - Default Retention Policies section, Force Push modal +- `templates/profile.html` - Updated retention dropdowns with org default option +- `templates/control_center.html` - Updated group and public workspace retention UI +- `static/js/workspace-manager.js` - Public workspace retention settings + +## Usage Instructions + +### Configuring Organization Defaults (Admin) + +1. Navigate to **Admin Settings** > **Content** tab +2. Scroll to the **Retention Policy** section +3. For each enabled workspace type, you'll see: + - **Default Conversation Retention Days**: How long conversations are kept + - **Default Document Retention Days**: How long documents are kept +4. Select the desired defaults from the dropdown (1 day to 10 years, or "Don't delete") +5. Click **Save All Settings** + +### Force Pushing Defaults (Admin) + +1. In the **Default Retention Policies** section, click **Force Push Defaults to All** +2. In the modal, select which workspace types to update +3. Review the warning about overriding custom policies +4. Click **Force Push** to confirm +5. The system will: + - First save any pending settings changes + - Then push defaults to all selected workspaces + - Display a summary of updated items +6. Click **Close** when complete + +### User Experience + +Users see updated retention options in their workspace settings: + +- **Using organization default (X days)** - Uses the admin-configured default +- **Don't delete** - Keep items indefinitely +- **Custom values** - 1 day to 10 years + +When "Using organization default" is selected: +- The actual default value is shown (e.g., "30 days") +- If the admin changes the default, the user's policy automatically follows +- Users can override by selecting a specific value + +## Activity Logging + +Force push actions are logged to the `activity_logs` container with: + +- **Activity Type**: `retention_policy_force_push` +- **Admin Info**: User ID and email of admin who executed +- **Scopes**: Which workspace types were affected +- **Results**: Breakdown of updates per workspace type +- **Total Updated**: Number of workspaces/users updated +- **Timestamp**: When the action occurred + +## UI/UX Details + +### Admin Settings Modal Flow + +1. **Initial State**: Shows Cancel and Force Push buttons +2. **Processing**: + - Status shows "Saving settings first..." + - Then "Pushing defaults to workspaces..." +3. **Completed**: + - Cancel and Force Push buttons hide + - Only Close button visible + - Results summary displayed + +### Conditional Visibility + +Default retention dropdowns only appear when the corresponding workspace type is enabled: +- Personal workspace defaults shown only when `enable_retention_policy_personal` is checked +- Group workspace defaults shown only when `enable_retention_policy_group` is checked +- Public workspace defaults shown only when `enable_retention_policy_public` is checked + +## Testing and Validation + +### Test Coverage + +The feature can be validated by: +1. Setting organization defaults in Admin Settings +2. Verifying the defaults appear in user-facing dropdowns +3. Testing the Force Push functionality +4. Checking activity logs for audit records +5. Verifying retention policy execution respects 'default' values + +### Performance Considerations + +- Force push iterates through all users/groups/workspaces +- Large deployments may take several seconds to complete +- Progress indicator shown during execution +- Non-blocking - admin can continue using the application + +## Known Limitations + +- Force push is an all-or-nothing operation per workspace type +- No option to selectively target specific users/groups +- Requires page refresh to see updated defaults in user UI after admin changes + +## Related Documentation + +- Retention Policy system documentation +- Admin Settings configuration guide +- Activity Logging reference diff --git a/docs/explanation/features/v0.237.001/USER_AGREEMENT.md b/docs/explanation/features/v0.237.001/USER_AGREEMENT.md new file mode 100644 index 00000000..d72d6533 --- /dev/null +++ b/docs/explanation/features/v0.237.001/USER_AGREEMENT.md @@ -0,0 +1,223 @@ +# User Agreement Feature + +## Overview + +The User Agreement feature allows administrators to configure a global agreement that users must accept before uploading files to workspaces. This provides organizations with a mechanism to ensure users acknowledge terms, policies, or guidelines before contributing documents to the system. + +**Version Implemented:** v0.237.001 + +## Key Features + +- **Global Admin Configuration**: Single configuration point in Admin Settings → Workspaces tab +- **Workspace Type Selection**: Apply agreement to personal workspaces, group workspaces, public workspaces, and/or chat +- **Markdown Support**: Agreement text supports Markdown formatting for rich content +- **Word Limit**: 200-word limit with real-time word count display +- **Daily Acceptance Option**: Optional setting to only prompt users once per day +- **Activity Logging**: All acceptances are logged for compliance tracking + +## Configuration + +### Accessing User Agreement Settings + +1. Navigate to **Admin Settings** from the sidebar +2. Select the **Workspaces** tab +3. Scroll to the **User Agreement** section + +### Configuration Options + +| Setting | Description | +|---------|-------------| +| **Enable User Agreement** | Master toggle to enable/disable the feature | +| **Apply To** | Checkboxes to select which workspace types require agreement (Personal, Group, Public, Chat) | +| **Agreement Text** | Markdown-formatted text displayed to users (max 200 words) | +| **Enable Daily Acceptance** | When enabled, users only need to accept once per day instead of every upload | + +### Example Agreement Text + +```markdown +## File Upload Agreement + +By uploading files to this workspace, you agree to the following: + +1. **Ownership**: You have the right to share this content +2. **Confidentiality**: You will not upload confidential information without authorization +3. **Compliance**: All uploads comply with organizational policies + +For questions, contact your administrator. +``` + +## User Experience + +### File Upload Flow + +1. User initiates a file upload (drag-and-drop or file picker) +2. System checks if User Agreement is enabled for that workspace type +3. If enabled and user hasn't accepted today (when daily acceptance is on): + - Modal appears with agreement text + - User can **Accept & Upload** or **Cancel** +4. Upon acceptance: + - Acceptance is logged to activity logs + - File upload proceeds normally +5. If cancelled: + - Upload is aborted + - No files are uploaded + +### Modal Interface + +The User Agreement modal displays: +- Agreement title +- Rendered Markdown content (sanitized via DOMPurify) +- Daily acceptance info (when enabled): "You only need to accept once per day" +- **Cancel** button - Dismisses modal, cancels upload +- **Accept & Upload** button - Records acceptance, proceeds with upload + +## Technical Architecture + +### Backend Components + +| File | Purpose | +|------|---------| +| `route_frontend_admin_settings.py` | Handles form submission for User Agreement settings | +| `route_backend_user_agreement.py` | API endpoints for checking/accepting agreements | +| `functions_activity_logging.py` | `log_user_agreement_accepted()` and `has_user_accepted_agreement_today()` | + +### Frontend Components + +| File | Purpose | +|------|---------| +| `admin_settings.html` | Configuration UI in Workspaces tab | +| `base.html` | User Agreement upload modal | +| `user-agreement.js` | `UserAgreementManager` module for handling upload checks | + +### API Endpoints + +#### Check Agreement Status +``` +GET /api/user_agreement/check?workspace_type={type}&workspace_id={id}&action_context=file_upload +``` + +**Response:** +```json +{ + "needsAgreement": true, + "agreementText": "## Agreement\n\nYour agreement text...", + "enableDailyAcceptance": true +} +``` + +#### Record Acceptance +``` +POST /api/user_agreement/accept +Content-Type: application/json + +{ + "workspace_type": "personal", + "workspace_id": "default", + "action_context": "file_upload" +} +``` + +**Response:** +```json +{ + "success": true, + "message": "User agreement accepted" +} +``` + +### Settings Data Model + +Settings are stored in `app_settings` with the following keys: + +```python +{ + "enable_user_agreement": False, # Master toggle + "user_agreement_text": "", # Markdown content + "user_agreement_apply_to": [], # List: ["personal", "group", "public", "chat"] + "enable_user_agreement_daily": False # Daily acceptance toggle +} +``` + +### Activity Log Entry + +When a user accepts the agreement, an activity log entry is created: + +```python +{ + "activity_type": "user_agreement_accepted", + "user_id": "user@example.com", + "workspace_type": "personal", + "workspace_id": "default", + "action_context": "file_upload", + "timestamp": "2026-01-21T10:30:00Z" +} +``` + +## Integration Points + +### Workspace Upload Handlers + +The following files integrate with `UserAgreementManager`: + +| File | Workspace Type | Integration Point | +|------|----------------|-------------------| +| `workspace-documents.js` | Personal | `handleFileUpload()` | +| `group_workspaces.html` | Group | `uploadFiles()` | +| `public_workspace.js` | Public | `handleFileUpload()` | +| `chat-input-actions.js` | Chat | `handleFileSelect()` | + +### Usage Pattern + +```javascript +// Example integration in upload handler +async function handleFileUpload(files) { + // Check user agreement before upload + UserAgreementManager.checkBeforeUpload( + 'personal', // workspace type + 'default', // workspace id + files, // files to upload + function(approvedFiles) { + // This callback runs after user accepts + proceedWithUpload(approvedFiles); + } + ); +} +``` + +## Security Considerations + +- Agreement text is sanitized using DOMPurify before rendering +- All API endpoints require user authentication +- Admin settings are protected by `@admin_required` decorator +- Activity logs provide audit trail for compliance + +## Dependencies + +- **marked.js**: Markdown parsing +- **DOMPurify**: HTML sanitization +- **Bootstrap 5**: Modal component + +## Sidebar Navigation + +The User Agreement settings are accessible via: +- **Admin Settings** → **Workspaces** submenu → **User Agreement** + +This follows the same pattern as other workspace-related admin settings like Retention Policy. + +## Testing + +To test the feature: + +1. Enable User Agreement in Admin Settings → Workspaces +2. Select at least one workspace type (e.g., Personal Workspaces) +3. Enter agreement text +4. Navigate to a personal workspace +5. Attempt to upload a file +6. Verify the agreement modal appears +7. Accept and verify the upload proceeds +8. Check Activity Logs for acceptance entry + +## Related Features + +- [Activity Logging](../v0.229.001/ACTION_LOGGING_AND_CITATION.md) - Acceptance tracking +- [Public Workspaces](../v0.229.001/PUBLIC_WORKSPACES.md) - Workspace types diff --git a/docs/explanation/features/v0.237.001/WEB_SEARCH_AZURE_AI_FOUNDRY.md b/docs/explanation/features/v0.237.001/WEB_SEARCH_AZURE_AI_FOUNDRY.md new file mode 100644 index 00000000..2e8e3b7c --- /dev/null +++ b/docs/explanation/features/v0.237.001/WEB_SEARCH_AZURE_AI_FOUNDRY.md @@ -0,0 +1,187 @@ +# Web Search via Azure AI Foundry Agents + +## Overview + +SimpleChat now supports web search capability through Azure AI Foundry agents using the Grounding with Bing Search service. This feature enables AI responses to be augmented with real-time web search results, providing users with up-to-date information beyond the model's training data. + +**Version Implemented:** v0.237.001 + +## Key Features + +- **Azure AI Foundry Integration**: Leverages Azure AI Foundry's Grounding with Bing Search capability +- **Admin Consent Flow**: Requires explicit administrator consent before enabling due to data processing considerations +- **Activity Logging**: All consent acceptances are logged for compliance and audit purposes +- **Setup Guide Modal**: Comprehensive in-app configuration guide with step-by-step instructions +- **User Data Notice**: Admin-configurable notification banner informing users when their message will be sent to Bing +- **Graceful Error Handling**: Informs users when web search fails rather than answering from outdated training data +- **Seamless Experience**: Web search results are automatically integrated into AI responses + +## Admin Consent Requirement + +Before web search can be enabled, administrators must acknowledge important data handling considerations: + +### Consent Message + +> When you use Grounding with Bing Search, your customer data is transferred outside of the Azure compliance boundary to the Grounding with Bing Search service. Grounding with Bing Search is not subject to the same data processing terms (including location of processing) and does not have the same compliance standards and certifications as the Azure AI Agent Service, as described in the Grounding with Bing Search TOU. + +### Why Consent is Required + +1. **Data Transfer**: Customer data is transferred outside the Azure compliance boundary +2. **Different Terms**: Grounding with Bing Search has different data processing terms +3. **Compliance Considerations**: Different compliance standards and certifications apply +4. **Organizational Responsibility**: Organizations must assess whether this meets their requirements + +## Configuration + +### Enabling Web Search + +1. Navigate to **Admin Settings** from the sidebar +2. Go to the **Search** or **Agents** section +3. Locate the **Web Search** toggle +4. Read and accept the consent message +5. Enable web search + +### Settings Stored + +| Setting | Description | +|---------|-------------| +| `enable_web_search` | Master toggle for web search capability | +| `web_search_consent_accepted` | Tracks whether consent has been accepted | +| `enable_web_search_user_notice` | Toggle for showing user notification when web search is activated | +| `web_search_user_notice_text` | Customizable notification message shown to users | + +## User Data Notice + +Administrators can enable a notification banner that appears when users activate web search, informing them about data being sent to Bing. + +### Configuration + +1. Navigate to **Admin Settings** > **Search and Extract** tab +2. Locate the **User Data Notice** card in the Web Search section +3. Enable the **Show User Notice** toggle +4. Customize the notification text (optional) + +### Default Notice Text + +> Your message will be sent to Microsoft Bing for web search. Only your current message is sent, not your conversation history. + +### Behavior + +- **Appears**: When user clicks the "Web" button to activate web search +- **Dismissible**: Users can dismiss the notice via the X button +- **Session-based**: Dismissal persists for the browser session only +- **Hides automatically**: When web search is deactivated + +## Setup Guide Modal + +The admin settings include a comprehensive setup guide modal with: + +### Pricing Information + +| Metric | Value | +|--------|-------| +| **Cost** | $14 per 1,000 transactions | +| **Rate Limit** | 150 transactions/second | +| **Daily Limit** | 1,000,000 transactions/day | + +### Step-by-Step Instructions + +1. Create an Azure AI Foundry project +2. Navigate to Agents section +3. Create a new agent with Bing grounding tool +4. Configure result count to 10 +5. Add recommended agent instructions +6. Copy the agent ID to SimpleChat admin settings +7. Configure Azure AI Foundry connection settings + +### Access + +Click the **Setup Guide** button in the Web Search admin settings section to open the modal. + +## Technical Architecture + +### Backend Components + +| File | Purpose | +|------|---------| +| [route_frontend_admin_settings.py](../../../../application/single_app/route_frontend_admin_settings.py) | Handles consent flow and settings persistence | +| [route_backend_chats.py](../../../../application/single_app/route_backend_chats.py) | `perform_web_search()` with graceful error handling | +| [functions_activity_logging.py](../../../../application/single_app/functions_activity_logging.py) | `log_web_search_consent_acceptance()` for audit logging | +| [functions_settings.py](../../../../application/single_app/functions_settings.py) | Default settings including user notice configuration | + +### Frontend Components + +| File | Purpose | +|------|---------| +| [admin_settings.html](../../../../application/single_app/templates/admin_settings.html) | Admin UI for web search configuration | +| [_web_search_foundry_info.html](../../../../application/single_app/templates/_web_search_foundry_info.html) | Setup guide modal with pricing and instructions | +| [chats.html](../../../../application/single_app/templates/chats.html) | User notice container in chat interface | +| [chat-input-actions.js](../../../../application/single_app/static/js/chat-input-actions.js) | Notice show/hide logic with session dismissal | + +### Consent Flow Logic + +```python +# Simplified flow +web_search_consent_accepted = form_data.get('web_search_consent_accepted') == 'true' +requested_enable_web_search = form_data.get('enable_web_search') == 'on' +enable_web_search = requested_enable_web_search and web_search_consent_accepted + +# Log consent if newly accepted +if enable_web_search and web_search_consent_accepted and not settings.get('web_search_consent_accepted'): + log_web_search_consent_acceptance( + user_id=user_id, + admin_email=admin_email, + consent_text=web_search_consent_message, + source='admin_settings' + ) +``` + +### Activity Log Entry + +When consent is accepted, the following information is logged: +- Admin user ID +- Admin email address +- Full consent text +- Source of consent (admin_settings) +- Timestamp + +## User Experience + +### For End Users + +- Web search is transparent when enabled +- AI responses automatically incorporate relevant web search results +- Citations from web sources are displayed alongside responses +- Optional notification banner when activating web search (if enabled by admin) +- Graceful error messages when web search fails + +### For Administrators + +- Clear consent flow before enabling +- One-time consent acceptance (persisted in settings) +- Audit trail of consent acceptance +- Comprehensive setup guide with pricing information +- Configurable user notification for transparency + +## Security Considerations + +1. **Consent Tracking**: All consent acceptances are logged for compliance +2. **Admin-Only Configuration**: Only administrators can enable web search +3. **Data Awareness**: Clear communication about data handling implications +4. **Revocability**: Web search can be disabled at any time + +## Related Features + +- [Azure AI Foundry Agent Support](AZURE_AI_FOUNDRY_AGENT_SUPPORT.md) +- Agent-based chat with real-time information + +## Dependencies + +- Azure AI Foundry account with Grounding with Bing Search enabled +- Proper Azure AI Foundry configuration in SimpleChat + +## Known Limitations + +- Web search results depend on Bing Search availability +- Results may vary based on Bing's index freshness +- Subject to Bing Search Terms of Use diff --git a/docs/explanation/fixes/v0.237.001/AGENT_PAYLOAD_FIELD_LENGTHS_FIX.md b/docs/explanation/fixes/v0.237.001/AGENT_PAYLOAD_FIELD_LENGTHS_FIX.md new file mode 100644 index 00000000..e3119d5c --- /dev/null +++ b/docs/explanation/fixes/v0.237.001/AGENT_PAYLOAD_FIELD_LENGTHS_FIX.md @@ -0,0 +1,27 @@ +# Agent Payload Field Lengths Fix (Version v0.237.001) + +## Header Information +- **Fix Title:** Agent payload field length validation +- **Issue Description:** Agent payload validation did not enforce length limits, allowing oversized values into storage. +- **Root Cause Analysis:** Length checks existed but were never invoked in `sanitize_agent_payload`, and no limits covered Azure-specific fields. +- **Fixed/Implemented in version:** **0.237.009** +- **Config Version Updated:** `config.py` VERSION set to **0.237.009** + +## Technical Details +- **Files Modified:** + - application/single_app/functions_agent_payload.py + - application/single_app/config.py +- **Code Changes Summary:** + - Added max length recommendations for Azure OpenAI and APIM fields. + - Validated field lengths in `sanitize_agent_payload` and for Foundry settings. + - Bumped application version in config.py. +- **Testing Approach:** + - Added a functional test to confirm validation wiring and limits are present. + +## Validation +- **Test Results:** functional_tests/test_agent_payload_field_lengths.py +- **Before/After Comparison:** + - Before: Oversized agent fields could pass validation. + - After: Oversized fields raise `AgentPayloadError` with a clear message. +- **User Experience Improvements:** + - Prevents invalid payloads and provides consistent validation feedback. diff --git a/docs/explanation/fixes/v0.237.001/AGENT_TEMPLATE_MAX_LENGTHS_FIX.md b/docs/explanation/fixes/v0.237.001/AGENT_TEMPLATE_MAX_LENGTHS_FIX.md new file mode 100644 index 00000000..7748a093 --- /dev/null +++ b/docs/explanation/fixes/v0.237.001/AGENT_TEMPLATE_MAX_LENGTHS_FIX.md @@ -0,0 +1,27 @@ +# Agent Template Max Lengths Fix (v0.237.001) + +## Header Information +- **Fix Title:** Agent template max length validation +- **Issue Description:** Agent template updates did not enforce length limits, allowing oversized fields into storage. +- **Root Cause Analysis:** Length checks were missing from the update path in `update_agent_template`. +- **Fixed/Implemented in version:** **0.237.010** +- **Config Version Updated:** `config.py` VERSION set to **0.237.010** + +## Technical Details +- **Files Modified:** + - application/single_app/functions_agent_templates.py + - application/single_app/config.py +- **Code Changes Summary:** + - Added max length constants for template fields and list items. + - Validated lengths during template updates. + - Bumped application version in config.py. +- **Testing Approach:** + - Added a functional test to validate length validation wiring. + +## Validation +- **Test Results:** functional_tests/test_agent_template_length_validation.py +- **Before/After Comparison:** + - Before: Oversized template fields could be saved. + - After: Oversized fields raise a validation error before persistence. +- **User Experience Improvements:** + - Consistent template validation and clearer error feedback. diff --git a/docs/explanation/fixes/v0.237.001/CONTROL_CENTER_DATE_LABELS_FIX.md b/docs/explanation/fixes/v0.237.001/CONTROL_CENTER_DATE_LABELS_FIX.md new file mode 100644 index 00000000..1add8e46 --- /dev/null +++ b/docs/explanation/fixes/v0.237.001/CONTROL_CENTER_DATE_LABELS_FIX.md @@ -0,0 +1,27 @@ +# Control Center Date Labels Fix (v0.237.001) + +## Header Information +- **Fix Title:** Control Center Date Labels Fix +- **Issue Description:** Control Center charts displayed dates one day behind due to UTC parsing of date keys. +- **Root Cause Analysis:** The frontend parsed YYYY-MM-DD strings with `new Date(...)`, which treats the value as UTC and shifts the day in local timezones. +- **Fixed/Implemented in version:** **0.235.074** +- **Config Version Updated:** `config.py` VERSION set to **0.235.074** + +## Technical Details +- **Files Modified:** + - application/single_app/static/js/control-center.js + - application/single_app/config.py +- **Code Changes Summary:** + - Added a local date parsing helper for YYYY-MM-DD keys. + - Updated chart label and tooltip rendering to use local date parsing. + - Bumped application version in config.py. +- **Testing Approach:** + - Added a functional test to validate the date parsing helper is present in the chart logic. + +## Validation +- **Test Results:** functional_tests/test_control_center_date_labels_fix.py +- **Before/After Comparison:** + - Before: Date labels in charts appeared one day behind in local timezones. + - After: Date labels match the correct local date (e.g., Jan 21 for today). +- **User Experience Improvements:** + - Accurate daily labels across all activity charts. diff --git a/docs/explanation/fixes/v0.237.001/RETENTION_POLICY_NOTFOUND_FIX.md b/docs/explanation/fixes/v0.237.001/RETENTION_POLICY_NOTFOUND_FIX.md new file mode 100644 index 00000000..e264920f --- /dev/null +++ b/docs/explanation/fixes/v0.237.001/RETENTION_POLICY_NOTFOUND_FIX.md @@ -0,0 +1,95 @@ +# Retention Policy NotFound Error Fix + +## Issue Description + +The retention policy deletion process was logging errors when attempting to delete conversations or documents that had already been deleted (e.g., by another process or user action between the query and delete operations). + +### Error Observed +``` +DEBUG: [Log] delete_aged_conversations_deletion_error -- {'error': '(NotFound) Entity with the specified id does not exist in the system. +``` + +### Root Cause + +This is a **race condition** scenario where: +1. The retention policy queries for aged conversations/documents +2. Between the query and the delete operation, the item is deleted by another process (user action, concurrent retention execution, etc.) +3. The delete operation fails with `CosmosResourceNotFoundError` (404 NotFound) + +## Fix Applied + +**Version:v0.237.001** + +The fix adds specific handling for `CosmosResourceNotFoundError` in both conversation and document deletion loops: + +### Conversations +- When reading a conversation before archiving: If not found, log debug message and count as already deleted +- When deleting messages: Catch NotFound and continue (message already gone) +- When deleting conversation: Catch NotFound and continue (conversation already gone) + +### Documents +- When deleting document chunks: Catch NotFound and continue +- When deleting document: Catch NotFound and continue +- Outer try/catch also handles NotFound to count as successful deletion + +## Files Modified + +- [functions_retention_policy.py](../../../application/single_app/functions_retention_policy.py) + - `delete_aged_conversations()` - Added CosmosResourceNotFoundError handling + - `delete_aged_documents()` - Added CosmosResourceNotFoundError handling + +## Technical Details + +### Before Fix +```python +# Read would throw exception if item was deleted between query and read +conversation_item = container.read_item( + item=conversation_id, + partition_key=conversation_id +) +# Delete would throw exception if item was deleted +container.delete_item( + item=conversation_id, + partition_key=conversation_id +) +``` + +### After Fix +```python +try: + conversation_item = container.read_item( + item=conversation_id, + partition_key=conversation_id + ) +except CosmosResourceNotFoundError: + # Already deleted - this is fine, count as success + debug_print(f"Conversation {conversation_id} already deleted (not found during read), skipping") + deleted_details.append({...}) + continue + +# ... archiving and message deletion ... + +try: + container.delete_item( + item=conversation_id, + partition_key=conversation_id + ) +except CosmosResourceNotFoundError: + # Already deleted between read and delete - this is fine + debug_print(f"Conversation {conversation_id} already deleted (not found during delete)") +``` + +## Benefits + +1. **No false error logs**: Items that are already deleted no longer generate error entries +2. **Accurate counts**: Already-deleted items are properly counted as successful deletions +3. **Graceful handling**: Race conditions are handled without disrupting the overall retention process +4. **Better debugging**: Debug messages clearly indicate when items were already deleted + +## Testing + +Test by: +1. Enabling retention policy with a short retention period +2. Running the retention policy execution +3. Verify no NotFound errors are logged +4. Verify deletion counts accurately reflect processed items diff --git a/docs/explanation/fixes/v0.237.001/SOVEREIGN_CLOUD_COGNITIVE_SERVICES_SCOPE_FIX.md b/docs/explanation/fixes/v0.237.001/SOVEREIGN_CLOUD_COGNITIVE_SERVICES_SCOPE_FIX.md new file mode 100644 index 00000000..4316dfcb --- /dev/null +++ b/docs/explanation/fixes/v0.237.001/SOVEREIGN_CLOUD_COGNITIVE_SERVICES_SCOPE_FIX.md @@ -0,0 +1,94 @@ +# Sovereign Cloud Cognitive Services Scope Fix + +## Overview + +Fixed hardcoded commercial Azure cognitive services scope references in chat streaming and Smart HTTP Plugin that prevented proper authentication in Azure Government (MAG) and custom cloud environments. + +**Version Implemented:** v0.237.001 + +**Related Issue:** [#616](https://github.com/microsoft/simplechat/issues/616#issue-3835164022) + +## Problem + +The `chat_stream_api` and `smart_http_plugin` contained hardcoded references to commercial Azure cognitive services scope URLs. This caused authentication failures when running SimpleChat in: +- Azure Government (MAG) environments +- Custom/sovereign cloud deployments + +### Error Symptoms + +Users in MAG environments encountered authentication errors when: +- Using chat with streaming enabled +- Making Smart HTTP Plugin calls + +The error occurred because the code attempted to authenticate against commercial Azure endpoints instead of the appropriate government or custom cloud endpoints. + +## Root Cause + +The authentication scope was hardcoded as the commercial cognitive services URL rather than using the configurable value from `config.py`. This meant: +- Commercial: `https://cognitiveservices.azure.com/.default` +- Government: Should be `https://cognitiveservices.azure.us/.default` +- Custom: Should use environment-specific scope + +## Solution + +Replaced all hardcoded cognitive services scope references with the configurable variable from `config.py`: +- `AZURE_OPENAI_TOKEN_SCOPE` environment variable +- Dynamically resolved based on cloud environment + +### Files Modified + +1. **chat_stream_api** (streaming chat implementation) + - Replaced hardcoded scope with `config.AZURE_OPENAI_TOKEN_SCOPE` + +2. **smart_http_plugin** (Smart HTTP Plugin) + - Replaced hardcoded scope with configurable variable + +## Cloud Environment Support + +| Cloud Environment | Cognitive Services Scope | +|-------------------|-------------------------| +| Commercial | `https://cognitiveservices.azure.com/.default` | +| Government (MAG) | `https://cognitiveservices.azure.us/.default` | +| China | `https://cognitiveservices.azure.cn/.default` | +| Custom | Configurable via environment variable | + +## Testing + +### Azure Government Validation + +1. Deploy SimpleChat to Azure Government environment +2. Configure appropriate Azure OpenAI resources +3. Enable streaming in chat settings +4. Send a chat message with streaming enabled +5. Verify response streams correctly without authentication errors + +### Commercial Cloud Validation + +1. Verify existing commercial deployments continue to function +2. Test streaming chat functionality +3. Test Smart HTTP Plugin calls + +## Impact + +- **Azure Government**: Full streaming and plugin functionality now works correctly +- **Custom Clouds**: Deployments can configure appropriate scope for their environment +- **Commercial**: No change to existing behavior + +## Configuration + +The cognitive services scope is configured via: + +```python +# config.py +AZURE_OPENAI_TOKEN_SCOPE = os.getenv('AZURE_OPENAI_TOKEN_SCOPE', 'https://cognitiveservices.azure.com/.default') +``` + +For Azure Government, set: +``` +AZURE_OPENAI_TOKEN_SCOPE=https://cognitiveservices.azure.us/.default +``` + +## Related + +- Sovereign Cloud Managed Identity Authentication Fix (v0.229.001) +- Azure Government Support documentation diff --git a/docs/explanation/fixes/v0.237.001/USER_SEARCH_TOAST_INLINE_MESSAGES_FIX.md b/docs/explanation/fixes/v0.237.001/USER_SEARCH_TOAST_INLINE_MESSAGES_FIX.md new file mode 100644 index 00000000..7ba7ed08 --- /dev/null +++ b/docs/explanation/fixes/v0.237.001/USER_SEARCH_TOAST_INLINE_MESSAGES_FIX.md @@ -0,0 +1,68 @@ +# User Search Toast and Inline Messages Fix + +## Overview + +Updated the `searchUsers()` function to use inline and toast messages instead of browser alert pop-ups, improving user experience and aligning with modern UI patterns. + +**Version Implemented:** v0.237.001 + +**Related PR:** [#608](https://github.com/microsoft/simplechat/pull/608#discussion_r2701900020) + +## Problem + +The user search functionality in group management used browser `alert()` pop-ups for all feedback messages (empty search, no users found, errors). This created a disruptive user experience and was inconsistent with the toast notification patterns used elsewhere in the application. + +## Solution + +Refactored the `searchUsers()` function to display feedback using: +- **Inline messages**: Primary feedback shown directly in the search results area +- **Toast notifications**: Used only for errors, in addition to inline messaging + +## User Experience + +### Empty Search Query +When users click search without entering a query: +- Inline message displayed in the search results area +- No disruptive alert pop-up + +### No Users Found +When the search returns no results: +- Informative inline message in the results area +- Clear indication that no matching users exist + +### Users Found +When one or more users are found: +- Results displayed in the search results area +- Success feedback integrated naturally into the flow + +### Error Handling +When an error occurs: +- Inline error message displayed +- Toast notification also shown for visibility +- Consistent with application error handling patterns + +## Technical Details + +### Files Modified +- Group management JavaScript (search user functionality) + +### Changes +- Replaced `alert()` calls with inline message rendering +- Added toast notification for error cases only +- Maintained consistent styling with existing UI patterns + +## Benefits + +1. **Non-disruptive UX**: Users can continue working without dismissing pop-ups +2. **Contextual feedback**: Messages appear where users are looking (in the search area) +3. **Consistency**: Aligns with toast notification patterns used elsewhere +4. **Accessibility**: Better screen reader support with inline messages +5. **Modern UI**: Follows contemporary web application design patterns + +## Testing + +1. Open group management → Add Members +2. Click Search without entering a query → Verify inline "empty search" message +3. Search for a non-existent user → Verify inline "no users found" message +4. Search for an existing user → Verify results display correctly +5. Simulate network error → Verify both inline message and toast appear diff --git a/docs/explanation/fixes/v0.237.001/WEB_SEARCH_FAILURE_GRACEFUL_HANDLING_FIX.md b/docs/explanation/fixes/v0.237.001/WEB_SEARCH_FAILURE_GRACEFUL_HANDLING_FIX.md new file mode 100644 index 00000000..ad18d356 --- /dev/null +++ b/docs/explanation/fixes/v0.237.001/WEB_SEARCH_FAILURE_GRACEFUL_HANDLING_FIX.md @@ -0,0 +1,184 @@ +# Web Search Failure Graceful Handling Fix + +## Overview + +Fixed an issue where Azure AI Foundry web search agent failures would cause the AI model to answer questions using outdated training data instead of informing the user that the web search failed. + +**Version Implemented:** v0.237.001 + +## Problem + +When using the Azure AI Foundry web search agent (Bing grounding), if the web search operation failed for any reason (network issues, configuration errors, API failures), the conversation would continue without web search results. The AI model would then answer the user's question based on its training data, which could be outdated or incorrect. + +### Example Scenario + +**User asks:** "Who is the current President of the United States?" + +**Before fix (incorrect behavior):** +- Web search fails silently due to agent configuration issue +- Model answers from training data: "Joe Biden is the current President" +- User receives confident but potentially outdated/incorrect information +- No indication that web search failed + +**After fix (correct behavior):** +- Web search fails +- System message injected instructing model to inform user of failure +- Model responds: "I'm sorry, but the web search encountered an error and I couldn't retrieve current information. Please try again later." +- User is aware the information may be unavailable + +### Error Symptoms + +Users would receive: +- Answers based on outdated training data cutoff dates +- Incorrect information for time-sensitive queries +- No indication that web search was attempted but failed +- Confidently stated incorrect facts + +## Root Cause + +The `perform_web_search` function in `route_backend_chats.py` did not communicate failure status back to the calling code. When exceptions occurred during web search: +1. Errors were logged but not acted upon +2. The function returned `None` in all cases (success and failure) +3. No mechanism existed to inform the model about search failures +4. The conversation proceeded as if web search was not configured + +## Solution + +Implemented a comprehensive failure handling mechanism: + +### 1. Return Value Indication + +Modified `perform_web_search` to return a boolean status: +- `True` - Web search succeeded or was intentionally skipped (disabled, empty query) +- `False` - Web search failed due to an error + +### 2. System Message Injection on Failure + +When web search fails, a system message is added to the conversation context instructing the model to: +- Acknowledge the search failure to the user +- Not attempt to answer using training data +- Suggest the user try again later + +### 3. Error-Specific Messages + +Different failure scenarios receive appropriate messages: + +| Failure Type | System Message | +|--------------|----------------| +| Agent ID Not Configured | "Web search agent is not configured. Please inform the user that web search is currently unavailable." | +| Foundry Invocation Error | "Web search failed: [error details]. Please inform the user that the web search encountered an error and you cannot provide real-time information for this query." | +| Unexpected Exception | "Web search failed with an unexpected error: [error]. Please inform the user that the web search encountered an error and suggest they try again later." | + +### Files Modified + +**route_backend_chats.py** +- Modified `perform_web_search` function to return boolean status +- Added system message injection on all failure paths +- Updated exception handlers to set appropriate failure messages + +## Code Changes + +### Return Value Pattern + +```python +def perform_web_search(conversation_id, source, query, web_search_results_container): + """ + Now returns: + - True: Web search succeeded or was intentionally skipped + - False: Web search failed due to an error + """ + + # Success path + return True + + # Failure path - inject system message and return False + web_search_results_container.append({ + 'role': 'system', + 'content': 'Web search failed: [error]. Please inform the user...' + }) + return False +``` + +### System Message Structure + +When failure occurs, a message is appended to the conversation: +```python +{ + 'role': 'system', + 'content': 'Web search failed with an unexpected error: [error details]. ' + 'Please inform the user that the web search encountered an error ' + 'and you cannot provide real-time information for this query. ' + 'Suggest they try again later.' +} +``` + +## Testing + +### Failure Scenario Validation + +1. **Missing Agent Configuration** + - Remove web search agent ID from settings + - Send a query to web search-enabled agent + - Verify user receives message about unavailable web search + +2. **Network/API Failure** + - Simulate network connectivity issue + - Send a query to web search agent + - Verify user receives error message instead of outdated answer + +3. **Success Scenario (Regression)** + - Configure valid web search agent + - Send a query requesting current information + - Verify web search results are returned with citations + +### Test Commands + +```python +# Test query for web search +"Who is the current President of the United States?" +"What is the current weather in Seattle?" +"What are today's top news headlines?" +``` + +## Impact + +- **User Experience**: Users are now informed when web search fails instead of receiving potentially incorrect information +- **Transparency**: Clear indication when real-time information cannot be retrieved +- **Trust**: Users can make informed decisions about the reliability of responses +- **Error Visibility**: Administrators can identify web search configuration issues through user reports + +## Configuration + +Web search requires proper Azure AI Foundry configuration: + +```python +# Required settings +FOUNDRY_WEB_SEARCH_AGENT_ID = "asst_xxxxxxxxxxxxx" # Foundry agent with Bing grounding +AZURE_AI_PROJECT_CONNECTION_STRING = "..." # Project connection string +``` + +## Debug Logging + +Enhanced debug logging was also added to `perform_web_search` to aid troubleshooting: + +```python +debug_print(f"🌐 Starting web search for conversation: {conversation_id}") +debug_print(f"📊 Web search query: '{query}'") +debug_print(f"✅ Web search completed successfully with {len(citations)} citations") +debug_print(f"❌ Web search failed: {error_details}") +``` + +Enable debug logging by setting: +```python +DEBUG_LOG_ENABLED = True +``` + +## Related + +- [Azure AI Foundry Agent Support](../features/v0.236.011/AZURE_AI_FOUNDRY_AGENT_SUPPORT.md) +- Bing Grounding Tool Configuration +- Error Handling Best Practices + +## Migration Notes + +This is a behavioral change that improves user experience. No configuration changes are required. Existing web search functionality will continue to work, with improved failure handling when errors occur. diff --git a/docs/explanation/release_notes.md b/docs/explanation/release_notes.md index 2d1e0e94..df88ebcd 100644 --- a/docs/explanation/release_notes.md +++ b/docs/explanation/release_notes.md @@ -1,7 +1,7 @@ # Feature Release -### **(v0.236.011)** +### **(v0.237.001)** #### New Features