Skip to content

ayman-m/xlog

BranchesTags

Repository files navigation

made-with made-with-Python made-with-FastAPI made-with-GraphQL Docker Pulls scanned-with snyk codeql

XLog

XLog is a security testing and simulation platform that combines synthetic log generation, scenario-based attack telemetry, and AI-orchestrated workflows through MCP.

XLog Agent UI

Capabilities

  • Synthetic log generation in SYSLOG, CEF, LEEF, WINEVENT, JSON, Incident, XSIAM Parsed, and XSIAM CEF formats.
  • Scenario-based telemetry with multi-step MITRE ATT&CK tactics from JSON scenarios or on-the-fly scenario input.
  • Streaming workers that continuously send logs to UDP, TCP, HTTP(S), XSIAM PAPI, or XSIAM webhook collectors.
  • Field catalog and observables tooling to discover supported fields and generate realistic observables and technology stacks.
  • Simulation skills library with CRUD tooling for skill files (foundation, scenarios, validation, workflows).
  • MCP integrations for CALDERA (abilities, adversaries, operations, agents, payloads) and XSIAM (XQL, datasets, lookups, assets, cases, issues).
  • Web agent for chat-based orchestration and skills management built in Next.js with Gemini/Vertex support.

Project Structure

xlog/
├── app/                    # GraphQL API for log generation and workers
├── scenarios/              # Scenario definitions (ready and drafts)
├── mcp/
│   ├── server/             # MCP server exposing XLog, CALDERA, XSIAM tools
│   └── agent/              # Next.js MCP chat and skills UI
├── scripts/                # Utility scripts
├── examples/               # GraphQL request examples
└── img/                    # Documentation images

Quick Start

  1. Create your environment file: 
    cp .env.example .env
  2. Edit .env with your values.
    • Choose a strong shared token and set MCP_TOKEN (the MCP server and agent use it for auth).
  3. If using Vertex AI, create a GCP service account and grant Vertex AI access (for example Vertex AI User), then put its JSON credentials in .env under GOOGLE_APPLICATION_CREDENTIALS.
  4. Start the full stack with Docker Compose: 
    docker compose up -d
  5. Verify containers are running (and healthy when healthchecks are defined): 
    docker compose ps
docker inspect --format '{{.Name}} -> {{if .State.Health}}{{.State.Health.Status}}{{else}}no-healthcheck{{end}}' xlog xlog_mcp xlog_agent caldera
  6. Verify services:
    • XLog GraphQL API: http://localhost:8999/
    • MCP server: http://localhost:8080/
    • Agent UI: http://localhost:3000/
    • Caldera: http://localhost:8888/
  7. Stop the stack when needed: 
    docker compose down

.env Notes

  • Base template: .env.example
  • Place .env in the same directory as docker-compose.yml (project root). If needed, use docker compose --env-file <path> up -d.
  • Set MCP_TOKEN to a strong random value (long, hard to guess).
  • For Vertex AI auth, use a GCP service account JSON credential in GOOGLE_APPLICATION_CREDENTIALS (service account must have Vertex AI permissions).
  • Keep MCP_URL=http://xlog-mcp:8080/api/v1/stream/mcp for container-to-container communication.
  • For external clients, set MCP_URL to the host IP running the containers (do not use localhost or 127.0.0.1), for example: http://10.10.0.6:8080/api/v1/stream/mcp.
  • Set credentials and tokens before startup (for example: MCP_TOKEN, UI_USER, UI_PASSWORD, and your model/API credentials).

Caldera Connectivity

  • If you use the Caldera container included in this compose stack, set CALDERA_URL=http://caldera:8888 in .env (container-to-container URL).
  • To access Caldera UI from another machine, use the IP of the host running Docker, for example: http://10.10.0.6:8888 (not localhost from a remote machine).
  • If you do not use the bundled Caldera image and instead use a remote Caldera server, set CALDERA_URL to that remote host URL, for example: http://<remote-caldera-ip>:8888.

Technology Stack Configuration (TECHNOLOGY_STACK)

TECHNOLOGY_STACK in .env is a JSON object used by the agent to understand your environment and choose defaults.

Most important requirement:

  • Update log_destination to your actual syslog receiver.
  • The agent uses log_destination.full_address as the default destination when sending simulated logs if the user does not explicitly provide a destination.

Example:

{
  "stack_name": "Enterprise Security Stack",
  "log_destination": {
    "type": "syslog",
    "protocol": "udp",
    "host": "10.10.0.8",
    "port": 514,
    "full_address": "udp:10.10.0.8:514"
  },
  "vendors": [
    {
      "vendor": "Fortinet",
      "product": "FortiGate",
      "category": "Firewall",
      "formats": ["CEF", "SYSLOG", "JSON"]
    }
  ]
}

If this value is not updated, simulated logs may be sent to the wrong default syslog destination.

Generate SSL_CERT_PEM and SSL_KEY_PEM

Use one-line PEM values in .env (newlines escaped as \\n).

Windows PowerShell:

& {
    # 1. Create the Certificate
    $cert = New-SelfSignedCertificate -DnsName "localhost" `
        -CertStoreLocation "cert:\CurrentUser\My" `
        -NotAfter (Get-Date).AddYears(1) `
        -KeyExportPolicy Exportable

    # 2. Export Certificate to PEM
    $certBytes = $cert.Export([System.Security.Cryptography.X509Certificates.X509ContentType]::Cert)
    $certB64 = [Convert]::ToBase64String($certBytes, [System.Base64FormattingOptions]::InsertLineBreaks)
    $certPem = "-----BEGIN CERTIFICATE-----`n$certB64`n-----END CERTIFICATE-----"
    $certPem | Set-Content -Path "localhost.crt"

    # 3. Export Private Key to PKCS#8 PEM
    $rsa = [System.Security.Cryptography.X509Certificates.RSACertificateExtensions]::GetRSAPrivateKey($cert)
    $keyBytes = $null

    if ($rsa.PSObject.Methods.Name -contains "ExportPkcs8PrivateKey") {
        $keyBytes = $rsa.ExportPkcs8PrivateKey()
    }
    elseif ($rsa -is [System.Security.Cryptography.RSACng]) {
        $keyBytes = $rsa.Key.Export([System.Security.Cryptography.CngKeyBlobFormat]::Pkcs8PrivateBlob)
    }

    if ($null -ne $keyBytes) {
        $keyB64 = [Convert]::ToBase64String($keyBytes, [System.Base64FormattingOptions]::InsertLineBreaks)
        $keyPem = "-----BEGIN PRIVATE KEY-----`n$keyB64`n-----END PRIVATE KEY-----"
        $keyPem | Set-Content -Path "localhost.key"

        # 4. Generate .env safe strings
        $certEnv = $certPem -replace "`r?`n", "\n"
        $keyEnv  = $keyPem -replace "`r?`n", "\n"

        Write-Host "`n--- COPY THESE INTO YOUR .ENV FILE ---`n" -ForegroundColor Green
        Write-Output "SSL_CERT_PEM=""$certEnv"""
        Write-Output "SSL_KEY_PEM=""$keyEnv"""
    } else {
        Write-Error "Failed to extract private key bytes."
    }
}

Linux:

# Generate self-signed cert/key for localhost
openssl req -x509 -newkey rsa:2048 -sha256 -days 365 -nodes \
  -keyout localhost.key -out localhost.crt -subj "/CN=localhost"

# Convert files to .env-safe one-line values
CERT_ENV=$(awk 'NF {sub(/\r/, ""); printf "%s\\\\n",$0;}' localhost.crt)
KEY_ENV=$(awk 'NF {sub(/\r/, ""); printf "%s\\\\n",$0;}' localhost.key)

echo "SSL_CERT_PEM=${CERT_ENV}"
echo "SSL_KEY_PEM=${KEY_ENV}"

Copy the printed SSL_CERT_PEM=... and SSL_KEY_PEM=... lines into your .env.

Cortex XSIAM Integration

To enable XSIAM operations from the MCP server and agent:

  1. In Cortex XSIAM, create a new Standard API Key.
  2. Add the API key values to .env:
    • CORTEX_MCP_PAPI_URL
    • CORTEX_MCP_PAPI_AUTH_HEADER
    • CORTEX_MCP_PAPI_AUTH_ID
  3. In Cortex XSIAM, create a new issue that will be used as the remote execution context (issue war room).
  4. Put that issue ID in .env as:
    • PLAYGROUND_ID

PLAYGROUND_ID is required for agent-triggered remote XSIAM command execution in the issue war room context.

Webhook Collector Integration (WEBHOOK_ENDPOINT)

Use webhook delivery when you want logs sent over HTTP as an additional mechanism alongside SYSLOG/TCP/UDP destinations.

  1. Create a Cortex XSIAM HTTP Collector (or any compatible webhook collector).
  2. For Cortex XSIAM HTTP Collector, configure:
    • Compression: disabled (uncompressed)
    • Log format: json
    • Vendor/Product: any values that match your use case
  3. Copy the collector endpoint URL and authentication key.
  4. Set in .env:
    • WEBHOOK_ENDPOINT=<collector_endpoint_url>
    • WEBHOOK_KEY=<collector_authentication_key>

Authentication header behavior in XLog:

  • XLog sends the webhook key in header: Authorization
  • Source: app/schema.py (_get_webhook_headers)

If you use a non-XSIAM webhook collector, it must accept the same Authorization header for auth, or you need to adapt the receiver/proxy to map this header accordingly.

GraphQL API

The GraphQL endpoint is served at http://localhost:8999/ when running via Docker Compose.

Queries and Mutations

  • getSupportedFields -> returns the supported fields list used for observables and required fields.
  • generateFakeData(requestInput: DataFakerInput) -> returns a batch of synthetic logs.
  • generateScenarioFakeData(requestInput: DetailedScenarioInput) -> returns multi-step scenario logs without starting workers.
  • createDataWorker(requestInput: DataWorkerCreateInput) -> starts a streaming worker.
  • createScenarioWorker(requestInput: ScenarioWorkerCreateInput) -> starts workers from a scenarios/ready/*.json file.
  • createScenarioWorkerFromQuery(requestInput: ScenarioQueryWorkerCreateInput) -> starts workers from inline scenario steps.
  • listWorkers -> lists active workers.
  • actionWorker(requestInput: DataWorkerActionInput) -> stops a worker or checks status.
  • generateObservables(requestInput: GenerateObservablesInput) -> generates observables from threat intel feeds.

Input Types

DataFakerInput

  • type (required): SYSLOG, CEF, LEEF, WINEVENT, JSON, Incident, XSIAM_Parsed, XSIAM_CEF
  • count (default 1)
  • vendor, product, version
  • datetimeIso (YYYY-MM-DD HH:MM:SS)
  • fields (comma-separated field list)
  • observablesDict (object of supported fields)
  • requiredFields (list of supported field enums)

DetailedScenarioInput

  • name (required)
  • tags (optional list)
  • steps (list of DetailedScenarioStep)

DetailedScenarioStep:

  • tactic, tacticId, technique, techniqueId, procedure, type
  • logs (list of DataFakerInput)

DataWorkerCreateInput

  • type (required): SYSLOG, CEF, LEEF, WINEVENT, JSON, Incident, XSIAM_Parsed, XSIAM_CEF
  • destination (required): udp:host:port, tcp:host:port, https://..., XSIAM, or XSIAM_WEBHOOK
  • count (default 1), interval (default 2)
  • vendor, product, version
  • fields, observablesDict, requiredFields, datetimeIso
  • verifySsl (default false)

XSIAM Custom HTTP Collector Requirements

When sending logs to a XSIAM custom HTTP collector (for example, alongside a SYSLOG destination), configure the collector with:

  • Compression: uncompressed
  • Log format: json

ScenarioWorkerCreateInput

  • scenario (required): filename without .json in scenarios/ready/
  • destination (required)
  • count, interval, vendor, datetimeIso, verifySsl

ScenarioQueryWorkerCreateInput

  • name (required)
  • destination (required)
  • tags (optional list)
  • steps (list of DetailedQueryScenarioStep)

DetailedQueryScenarioStep:

  • tactic, tacticId, technique, techniqueId, procedure, type
  • logs (list of WorkerFakerInput)

WorkerFakerInput:

  • type (required), count, interval
  • vendor, product, version
  • datetimeIso, fields, observablesDict, requiredFields, verifySsl

DataWorkerActionInput

  • worker (required)
  • action (required): STOP or STATUS

GenerateObservablesInput

  • count (required)
  • observableType (required): IP, URL, SHA256, CVE, TERMS
  • known (default BAD): BAD or GOOD

Example Queries

Generate fake data (SYSLOG):

query Example($input: DataFakerInput!) {
  generateFakeData(requestInput: $input) {
    count
    type
    data
  }
}

Generate fake data with observables (JSON):

query ExampleJson($input: DataFakerInput!) {
  generateFakeData(requestInput: $input) {
    count
    type
    data
  }
}

Example variables:

{
  "input": {
    "type": "JSON",
    "count": 2,
    "datetimeIso": "2025-01-05 18:29:25",
    "observablesDict": {
      "remoteIp": "203.0.113.10",
      "localIp": "10.0.0.5",
      "user": "svc-backup",
      "url": "https://example.com/login"
    }
  }
}

Create a worker:

query CreateWorker($input: DataWorkerCreateInput!) {
  createDataWorker(requestInput: $input) {
    worker
    status
    type
    destination
  }
}

List workers:

query ListWorkers {
  listWorkers {
    worker
    status
    type
    interval
    destination
  }
}

Stop a worker:

query StopWorker($input: DataWorkerActionInput!) {
  actionWorker(requestInput: $input) {
    worker
    status
  }
}

Generate scenario fake data:

query Scenario($input: DetailedScenarioInput!) {
  generateScenarioFakeData(requestInput: $input) {
    name
    steps
  }
}

Documentation

  • scenarios/README.md - Scenario format and examples
  • mcp/server/README.md - MCP server capabilities and setup
  • mcp/agent/README.md - Next.js agent setup and usage
  • mcp/server/skills/README.md - Skill library structure

About

A log generator service with an API interface , MCP and an Agent, to generate logs and simulate attacks using caldera.

Resources

Readme

License

MIT license
Activity

Stars

2 stars

Watchers

2 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages