Iowarp dev

docs/api/python.md

@@ -1,42 +1,16 @@
-# Context Exploration Engine - Python API Documentation
-
-## Overview
-
-The Context Exploration Engine (CEE) provides a high-level Python API for managing and exploring data contexts in IOWarp. The API is accessible through the `wrp_cee` Python module and offers a simple interface for data assimilation, querying, retrieval, and cleanup operations.
-
-**Key Feature:** The CEE API automatically initializes the IOWarp runtime when you create a `ContextInterface` instance. You don't need to manually initialize Chimaera, CTE, or CAE - the `ContextInterface` constructor handles all of this internally.
-
-## Installation
-
-### From pip (Recommended)
-
-```bash
-pip install iowarp-core
-```
-
-This installs the `iowarp_core` package (runtime utilities, CLI) and the `wrp_cee` Python extension (context exploration API). All native dependencies are bundled — no system libraries or build tools required.
-
-### From Source
-
-Build IOWarp with Python bindings enabled:
+---
+sidebar_position: 1
+title: Context Exploration
+description: API reference for data assimilation, querying, retrieval, and cleanup in IOWarp.
+---
 
-```bash
-cmake --preset=debug -DWRP_CORE_ENABLE_PYTHON=ON
-cmake --build build -j$(nproc)
-sudo cmake --install build
-```
+# Context Exploration
 
-The `wrp_cee` module will be installed to your Python site-packages directory.
+## Overview
 
-### Verification
+The `wrp_cee` Python module provides a high-level API for managing and exploring data contexts in IOWarp. It offers a simple interface for data assimilation, querying, retrieval, and cleanup operations.
 
-```python
-import iowarp_core
-print("IOWarp version:", iowarp_core.get_version())
-
-import wrp_cee
-print("CEE API loaded successfully!")
-```
+**Key Feature:** The API automatically initializes the IOWarp runtime when you create a `ContextInterface` instance. You don't need to manually initialize Chimaera, CTE, or CAE — the `ContextInterface` constructor handles all of this internally.
 
 ## Module: `wrp_cee`
 
@@ -178,10 +152,8 @@ ctx_interface = wrp_cee.ContextInterface()
 **Parameters:** None
 
 **Notes:**
-- Automatically initializes CAE client (which in turn initializes CTE and Chimaera)
-- Verifies Chimaera IPC is available
-- Sets `is_initialized_` flag on success
-- Assumes runtime configuration is already set via environment variables (e.g., `CHI_SERVER_CONF`)
+- Automatically initializes the full IOWarp runtime stack
+- Requires runtime configuration via environment variables (e.g., `CHI_SERVER_CONF`)
 
 **Typical Environment Setup:**
 
@@ -221,7 +193,7 @@ result = ctx_interface.context_bundle(bundle)
 
 **Description:**
 
-Assimilates one or more data objects into IOWarp. Each `AssimilationCtx` in the bundle describes a source file/dataset to assimilate and where to store it. The method calls the CAE's `ParseOmni` function which schedules assimilation tasks for each context.
+Assimilates one or more data objects into IOWarp. Each `AssimilationCtx` in the bundle describes a source file/dataset to assimilate and where to store it.
 
 **Example:**
 
@@ -276,7 +248,7 @@ blob_names = ctx_interface.context_query(tag_re, blob_re, max_results=0)
 
 **Description:**
 
-Queries the CTE system for blobs matching the specified regex patterns. Uses `BlobQuery` with `Broadcast` pool query to search across all nodes. Returns only the blob names, not the data.
+Queries for blobs matching the specified regex patterns across all nodes. Returns only the blob names, not the data.
 
 **Example:**
 
@@ -320,7 +292,7 @@ packed_data = ctx_interface.context_retrieve(
   - Default: `1024`
 - **`max_context_size`** (int, optional): Maximum total context size in bytes
   - Default: `268435456` (256MB)
-- **`batch_size`** (int, optional): Number of concurrent `AsyncGetBlob` operations
+- **`batch_size`** (int, optional): Number of concurrent retrieval operations
   - Controls parallelism
   - Default: `32`
 
@@ -332,14 +304,12 @@ packed_data = ctx_interface.context_retrieve(
 **Description:**
 
 Retrieves blob data matching the specified patterns and packs it into a single binary buffer. The method:
-1. Uses `BlobQuery` to find matching blobs
+1. Finds matching blobs
 2. Allocates a buffer of size `max_context_size`
-3. Retrieves blobs in batches using `AsyncGetBlob`
+3. Retrieves blobs in batches for efficiency
 4. Packs data sequentially into the buffer
 5. Returns the packed data as a string
 
-Blobs are processed in batches for efficiency. The buffer is automatically allocated and freed.
-
 **Example:**
 
 ```python
@@ -388,7 +358,7 @@ result = ctx_interface.context_destroy(context_names)
 
 **Description:**
 
-Deletes the specified contexts from the CTE system. Each context name is treated as a tag name and deleted using CTE's `DelTag` API. This operation removes the tag and all associated blobs.
+Deletes the specified contexts. Each context name is treated as a tag name. This operation removes the tag and all associated blobs.
 
 **Example:**
 
@@ -415,7 +385,7 @@ else:
 
 ```python
 #!/usr/bin/env python3
-"""Complete CEE API example"""
+"""Complete Python API example"""
 
 import wrp_cee as cee
 import os
@@ -480,9 +450,9 @@ dst="iowarp://my_tag"              # Wrong! Don't use ://
 
 ## Runtime Assumptions
 
-The CEE Python API assumes:
+The Python API assumes:
 
-1. **Runtime is Started:** The IOWarp runtime (Chimaera server) should be running, or will be started by the `ContextInterface` constructor.
+1. **Runtime is Started:** The IOWarp runtime should be running, or will be started by the `ContextInterface` constructor.
 
 2. **Configuration Available:** Runtime configuration is available via environment variable:
    ```bash
@@ -491,12 +461,6 @@ The CEE Python API assumes:
 
 3. **Proper Permissions:** Your Python process has permission to access shared memory segments and connect to the runtime.
 
-4. **Dependencies Initialized:** When you create a `ContextInterface`, it will:
-   - Initialize CAE client
-   - Initialize CTE client (via CAE)
-   - Initialize Chimaera client (via CTE)
-   - Verify IPC manager is available
-
 ---
 
 ## Error Handling
@@ -529,7 +493,5 @@ if result != 0:
 
 ## See Also
 
-- **C++ API Documentation:** `context-exploration-engine/api/include/wrp_cee/api/context_interface.h`
-- **Unit Tests:** `context-exploration-engine/api/test/test_context_interface.py`
-- **Demo Script:** `context-exploration-engine/api/demo/simple_assimilation_demo.py`
-- **CTE Documentation:** `context-transfer-engine/docs/cte/cte.md`
+- [Quick Start Guide](../getting-started/quick-start) — End-to-end walkthrough
+- [Configuration Reference](../deployment/configuration) — Runtime and storage tier setup

docs/getting-started/installation.mdx

@@ -1,7 +1,7 @@
 ---
 sidebar_position: 1
 title: Installation
-description: Install IOWarp via pip, Docker, native build, or Spack package manager.
+description: Install IOWarp via Conda, Docker, Spack, or pip.
 ---
 
 # Installation
@@ -12,43 +12,57 @@ import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';
 
 <Tabs>
-<TabItem value="pip" label="🐍 Pip (Recommended)" default>
+<TabItem value="conda" label="Conda (Recommended)" default>
 
-Install the IOWarp Python package:
+Clone and build IOWarp Core from source using the automated installer.
+This uses conda internally to manage dependencies and produces a full-featured
+build with all optional components.
 
 ```bash
-pip install iowarp-core
+git clone --recurse-submodules https://github.com/iowarp/iowarp-core.git
+cd iowarp-core
+./install.sh
 ```
 
-Verify the installation:
+Activate the environment in every new terminal:
 
-```python
-import iowarp_core
-print(iowarp_core.get_version())
+```bash
+conda activate iowarp
 ```
 
-### Start the Runtime
+#### Build Variants
 
-IOWarp includes the `chimaera` CLI for managing the runtime:
+Pass a variant name to enable additional features:
+
+| Variant | Command | What it enables |
+|---------|---------|-----------------|
+| Release (default) | `./install.sh` | CPU-only, all engines |
+| Debug | `./install.sh debug` | Debug symbols, sanitizers |
+| CUDA | `./install.sh cuda` | NVIDIA GPU acceleration |
+| ROCm | `./install.sh rocm` | AMD GPU acceleration |
+| MPI | `./install.sh mpi` | Distributed multi-node |
+| Full | `./install.sh full` | CUDA + MPI + everything |
+
+### Verify the installation
 
 ```bash
-chimaera runtime start
+conda activate iowarp
+chimaera --help
 ```
 
 ### What's Included
 
-The pip package is self-contained with all dependencies statically linked. It includes:
+The Conda install provides the complete IOWarp stack:
 
-- **Python API** — `import iowarp_core` and `import wrp_cee` for context management
+- **All engines** — CTE, CAE, CEE with full feature support
+- **Python API** — `import wrp_cee` for context management
 - **CLI** — `chimaera` command for runtime management
-- **Shared libraries** — All IOWarp runtime libraries bundled in the package
-
-:::info
-No system dependencies are required beyond a standard C/C++ runtime (glibc). The package works on any Linux x86_64 or aarch64 system with Python 3.10+.
-:::
+- **GPU support** — CUDA and ROCm variants available
+- **MPI support** — Distributed multi-node deployments
+- **HDF5 support** — Scientific data format ingestion
 
 </TabItem>
-<TabItem value="docker" label="🐳 Docker">
+<TabItem value="docker" label="Docker">
 
 Pull and run the IOWarp Docker image:
 
@@ -92,23 +106,9 @@ IOWarp uses `memfd_create()` for shared memory, so no special `/dev/shm` configu
 :::
 
 </TabItem>
-<TabItem value="native" label="⚡ Native Install">
-
-Use the standalone installer script:
-
-```bash
-curl -fsSL https://raw.githubusercontent.com/iowarp/iowarp-install/main/install.sh | bash
-```
-
-This will:
-- Clone and build IOWarp core with all submodules
-- Install the IOWarp CLIO Kit
-- Set up the complete IOWarp environment
-
-</TabItem>
-<TabItem value="spack" label="📦 Spack">
+<TabItem value="spack" label="Spack">
 
-1. Install Spack (v0.23 recommended):
+1. Install Spack (v0.22.3+ recommended):
 
 ```bash
 git clone https://github.com/spack/spack.git -b v0.22.3
@@ -130,11 +130,53 @@ spack repo add iowarp-install/iowarp-spack
 spack install iowarp
 ```
 
+</TabItem>
+<TabItem value="pip" label="Pip (Experimental)">
+
+:::caution Experimental
+The pip package is experimental. It does not include GPU (CUDA/ROCm),
+MPI, or HDF5 support. For production use, prefer the
+[Conda install](#) method.
+:::
+
+Install the IOWarp Python package:
+
+```bash
+pip install iowarp-core
+```
+
+Verify the installation:
+
+```python
+import iowarp_core
+print(iowarp_core.get_version())
+```
+
+### Start the Runtime
+
+IOWarp includes the `chimaera` CLI for managing the runtime:
+
+```bash
+chimaera runtime start
+```
+
+### What's Included
+
+The pip package is self-contained with all dependencies statically linked. It includes:
+
+- **Python API** — `import iowarp_core` and `import wrp_cee` for context management
+- **CLI** — `chimaera` command for runtime management
+- **Shared libraries** — All IOWarp runtime libraries bundled in the package
+
+:::info
+No system dependencies are required beyond a standard C/C++ runtime (glibc). The package works on any Linux x86_64 or aarch64 system with Python 3.10+.
+:::
+
 </TabItem>
 </Tabs>
 
 ## Next Steps
 
-- [Quick Start Tutorial](./quick-start) — Run your first benchmark
+- [Quick Start Tutorial](./quick-start) — Start the runtime and run your first example
 - [Configuration Reference](../deployment/configuration) — Customize your deployment
 - [CLIO Kit](../clio-kit/mcp-servers) — Explore MCP servers for AI agents