From 3ede2fb5d73d8b314a74540839aff65a2ddc1d1a Mon Sep 17 00:00:00 2001 From: Alexander Dusenbery Date: Wed, 28 Jan 2026 11:20:36 -0500 Subject: [PATCH] feat: use a virtualenv around all python Make targets --- CLAUDE.md | 117 +++++++++++++++++++++++++++++++++++++ Makefile | 76 ++++++++++++++++++------ requirements/base.txt | 6 +- requirements/pip.txt | 6 +- requirements/pip_tools.txt | 8 ++- requirements/test.txt | 14 ++--- 6 files changed, 194 insertions(+), 33 deletions(-) create mode 100644 CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..a6ae6da --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,117 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +This repository contains specifications and deployment scripts for managing Open edX REST API routing through an API management layer. **This repository does not contain source code for running an API management service** - it contains Swagger/OpenAPI specifications and AWS API Gateway deployment scripts. + +The primary purpose is to: +- Define a unified API interface for Open edX REST endpoints +- Provide deployment automation for AWS API Gateway +- Enable routing to various Open edX services (edx-platform, IDAs) through a single access point + +## Development Commands + +### Setup + +This project requires Python 3.11. The Makefile will automatically: +1. Check if Python 3.11 is installed +2. On Ubuntu/Debian systems with apt-get: automatically install Python 3.11 via deadsnakes PPA (requires sudo) +3. On other systems: provide installation instructions + +```bash +make venv # Create Python 3.11 virtualenv (auto-created by other targets) +make requirements # Install Python dependencies for local development +``` + +**Note**: On Ubuntu, the first run will install Python 3.11, python3.11-venv, and python3.11-dev packages via apt-get. + +### Testing +```bash +make test # Run all tests (quality, Python tests, and Swagger tests) +make test_python # Run Python tests only +make test_swagger # Spin up stub server and run integration tests +``` + +### Quality Checks +```bash +make quality # Run PEP8 and Pylint on scripts/aws directory +``` + +### Building Swagger Documentation +```bash +make build # Flatten Swagger docs into build artifacts + # Requires Java 7+ installed + # Downloads swagger-codegen-cli.jar and generates flattened docs +``` + +### Dependency Management +```bash +make upgrade # Update all requirements/*.txt files with latest packages +``` + +### Cleanup +```bash +make clean # Remove Python bytecode and build artifacts +``` + +## Architecture + +### Swagger Specifications + +The API definitions use nested Swagger 2.0 specifications with remote references: + +- **`swagger/api.yaml`**: Main specification that defines the complete Open edX public API surface +- **`swagger/index.yaml`**: Index endpoint specification +- **`swagger/heartbeat.yaml`**: Health check endpoint +- **`swagger/oauth.yaml`**: OAuth2 token endpoint + +The main `api.yaml` uses `$ref` to pull in both local files and remote specifications from upstream services (e.g., course-discovery, edx-enterprise). This allows service teams to maintain their own API specs while the api-manager composes them into a unified interface. + +### AWS Deployment Scripts + +Located in `scripts/aws/`, these Python scripts manage AWS API Gateway deployments using a ring deployment strategy: + +1. **`bootstrap.py`**: Creates a new API Gateway RestApi and deploys a hello-world bootstrap stage +2. **`deploy.py`**: Uploads flattened Swagger to API Gateway in the next stage of the ring rotation +3. **`flip.py`**: Updates the custom domain to point to a specific stage (activation/rollback) +4. **`monitor.py`**: Monitoring utilities for API Gateway instances +5. **`common/deploy.py`**: Shared deployment logic (stage rotation, API updates, throttling configuration) + +#### Deployment Flow + +1. Bootstrap: Create initial API Gateway with custom domain +2. Build: Generate flattened Swagger JSON using swagger-codegen +3. Deploy: Upload to next stage in ring (e.g., red → black → red) +4. Flip: Point domain to new stage or rollback to previous + +#### Ring Deployment Strategy + +The deployment scripts use ordered stages (e.g., "red" and "black") to enable zero-downtime deployments. The live stage serves traffic while the next stage is updated, then traffic is flipped to the new stage. + +### Test Structure + +- **`scripts/aws/tests/`**: Unit tests for deployment scripts (bootstrap, deploy, flip) +- **`tests/`**: Integration tests that validate Swagger specifications against a stub server + +## AWS Configuration + +When working with AWS deployment scripts, ensure these environment variables are set: +- `AWS_REGION` +- `AWS_ACCESS_KEY_ID` +- `AWS_SECRET_ACCESS_KEY` + +## Key Files + +- **`Makefile`**: Build, test, and quality commands +- **`swagger/api.yaml`**: Main API specification with nested references +- **`scripts/aws/bootstrap.json`**: Minimal hello-world API for bootstrap stage +- **`.pep8`** and **`.pylintrc`**: Code quality configuration + +## Important Notes + +- Java 7+ is required for building Swagger documentation +- The repository uses pip-tools for dependency management (requirements/*.in → requirements/*.txt) +- AWS API Gateway is the reference implementation, but the specs are vendor-agnostic +- Stage variables in API Gateway must be configured for your specific Open edX installation diff --git a/Makefile b/Makefile index fda3095..7df36d4 100644 --- a/Makefile +++ b/Makefile @@ -1,12 +1,21 @@ SWAGGER_CODEGEN_JAR := https://repo1.maven.org/maven2/io/swagger/swagger-codegen-cli/2.4.13/swagger-codegen-cli-2.4.13.jar BUILD_OUTPUT_DIR := swagger-build-artifacts STUB_SERVER_DIR := edx-api-stub-server +VENV := venv +PYTHON_VERSION := 3.11 +PYTHON := $(VENV)/bin/python +PIP := $(VENV)/bin/pip + +# Detect available Python and OS +PYTHON311 := $(shell command -v python3.11 2> /dev/null) +HAS_APT := $(shell command -v apt-get 2> /dev/null) help: @echo ' ' @echo 'Makefile for api-manager ' @echo ' ' @echo 'Usage: ' + @echo ' make venv Create Python 3.11 virtualenv ' @echo ' make clean Delete generated python byte code and testing remnants ' @echo ' make requirements Install requirements for local development ' @echo ' make quality Run PEP8 and Pylint ' @@ -14,19 +23,50 @@ help: @echo ' make test Run all tests ' @echo ' ' +venv: + @echo "Checking for Python $(PYTHON_VERSION)..." +ifndef PYTHON311 + @echo "Python 3.11 not found, attempting to install..." +ifeq ($(HAS_APT),) + @echo "ERROR: Python $(PYTHON_VERSION) is not installed and apt-get is not available." + @echo "" + @echo "Please install Python $(PYTHON_VERSION) manually:" + @echo " macOS (using Homebrew): brew install python@$(PYTHON_VERSION)" + @echo "" + @exit 1 +endif + @echo "Installing Python $(PYTHON_VERSION) using apt-get..." + sudo apt-get update + sudo apt-get install -y software-properties-common + sudo add-apt-repository -y ppa:deadsnakes/ppa + sudo apt-get update + sudo apt-get install -y python$(PYTHON_VERSION) python$(PYTHON_VERSION)-venv python$(PYTHON_VERSION)-dev + @echo "Python $(PYTHON_VERSION) installed successfully" +else + @echo "Found python3.11 at $(PYTHON311)" +endif + @if [ ! -d "$(VENV)" ]; then \ + echo "Creating virtual environment in $(VENV)"; \ + python3.11 -m venv $(VENV); \ + $(PIP) install --upgrade pip; \ + else \ + echo "Virtual environment $(VENV) already exists, skipping creation"; \ + fi + clean: find . -name '*.pyc' -delete find . -name '__pycache__' -type d -delete rm -rf $(BUILD_OUTPUT_DIR) rm -rf $(STUB_SERVER_DIR) + rm -rf $(VENV) -requirements: - pip install -r requirements/pip.txt - pip install -qr requirements/test.txt --exists-action w +requirements: venv + $(PIP) install -r requirements/pip.txt + $(PIP) install -qr requirements/test.txt --exists-action w -quality: - pep8 --config=.pep8 scripts/aws - pylint --rcfile=.pylintrc scripts/aws +quality: venv requirements + $(VENV)/bin/pep8 --config=.pep8 scripts/aws + $(VENV)/bin/pylint --rcfile=.pylintrc scripts/aws # Download the swagger codegen jar # TODO: verify via checksum that the file is valid. @@ -43,15 +83,15 @@ build-gocd: codegen.download /gocd-jre/bin/java -jar swagger-codegen-cli.jar generate -l swagger -i swagger/api.yaml -o $(BUILD_OUTPUT_DIR) test_python: clean requirements - cd scripts/aws && python -m pytest + cd scripts/aws && ../../$(PYTHON) -m pytest # Spin up a stub server and hit it with tests -test_swagger: codegen.download +test_swagger: codegen.download venv requirements java -jar swagger-codegen-cli.jar generate -l nodejs-server -i swagger/api.yaml -o $(STUB_SERVER_DIR) cd $(STUB_SERVER_DIR) && npm install cd $(STUB_SERVER_DIR) && NODE_ENV=development node index.js 2> /dev/null & sleep 5 - pyresttest --url=http://localhost:8080 --test=tests/test_all.yaml + $(VENV)/bin/pyresttest --url=http://localhost:8080 --test=tests/test_all.yaml killall -9 node test: clean @@ -60,14 +100,14 @@ test: clean make test_swagger # Targets in a Makefile which do not produce an output file with the same name as the target name -.PHONY: help requirements clean quality test +.PHONY: help venv requirements clean quality test codegen.download build build-gocd test_python test_swagger upgrade upgrade: export CUSTOM_COMPILE_COMMAND=make upgrade -upgrade: ## update the requirements/*.txt files with the latest packages satisfying requirements/*.in - pip install -q -r requirements/pip_tools.txt - pip-compile --upgrade --allow-unsafe --rebuild -o requirements/pip.txt requirements/pip.in - pip-compile --upgrade -o requirements/pip_tools.txt requirements/pip_tools.in - pip install -qr requirements/pip.txt - pip install -qr requirements/pip_tools.txt - pip-compile --upgrade -o requirements/base.txt requirements/base.in - pip-compile --upgrade -o requirements/test.txt requirements/test.in +upgrade: venv ## update the requirements/*.txt files with the latest packages satisfying requirements/*.in + $(PIP) install -q -r requirements/pip_tools.txt + $(VENV)/bin/pip-compile --upgrade --allow-unsafe --rebuild -o requirements/pip.txt requirements/pip.in + $(VENV)/bin/pip-compile --upgrade -o requirements/pip_tools.txt requirements/pip_tools.in + $(PIP) install -qr requirements/pip.txt + $(PIP) install -qr requirements/pip_tools.txt + $(VENV)/bin/pip-compile --upgrade -o requirements/base.txt requirements/base.in + $(VENV)/bin/pip-compile --upgrade -o requirements/test.txt requirements/test.in diff --git a/requirements/base.txt b/requirements/base.txt index 0685389..34405b4 100644 --- a/requirements/base.txt +++ b/requirements/base.txt @@ -6,9 +6,9 @@ # boto==2.49.0 # via google-compute-engine -boto3==1.42.31 +boto3==1.42.36 # via -r requirements/base.in -botocore==1.42.31 +botocore==1.42.36 # via # -r requirements/base.in # boto3 @@ -19,7 +19,7 @@ google-compute-engine==2.8.13 # via -r requirements/base.in jinja2==3.1.6 # via -r requirements/base.in -jmespath==1.0.1 +jmespath==1.1.0 # via # -r requirements/base.in # boto3 diff --git a/requirements/pip.txt b/requirements/pip.txt index 929f3ee..9fbfa81 100644 --- a/requirements/pip.txt +++ b/requirements/pip.txt @@ -4,11 +4,13 @@ # # make upgrade # -wheel==0.45.1 +packaging==26.0 + # via wheel +wheel==0.46.3 # via -r requirements/pip.in # The following packages are considered to be unsafe in a requirements file: pip==25.3 # via -r requirements/pip.in -setuptools==80.9.0 +setuptools==80.10.2 # via -r requirements/pip.in diff --git a/requirements/pip_tools.txt b/requirements/pip_tools.txt index 572bb75..97cff3f 100644 --- a/requirements/pip_tools.txt +++ b/requirements/pip_tools.txt @@ -8,15 +8,17 @@ build==1.4.0 # via pip-tools click==8.3.1 # via pip-tools -packaging==25.0 - # via build +packaging==26.0 + # via + # build + # wheel pip-tools==7.5.2 # via -r requirements/pip_tools.in pyproject-hooks==1.2.0 # via # build # pip-tools -wheel==0.45.1 +wheel==0.46.3 # via pip-tools # The following packages are considered to be unsafe in a requirements file: diff --git a/requirements/test.txt b/requirements/test.txt index facd119..911d5a7 100644 --- a/requirements/test.txt +++ b/requirements/test.txt @@ -20,12 +20,12 @@ boto==2.49.0 # via # -r requirements/base.txt # google-compute-engine -boto3==1.42.31 +boto3==1.42.36 # via # -r requirements/base.txt # aws-sam-translator # moto -botocore==1.42.31 +botocore==1.42.36 # via # -r requirements/base.txt # aws-xray-sdk @@ -40,7 +40,7 @@ cfn-lint==1.41.0 # via moto charset-normalizer==3.4.4 # via requests -cryptography==46.0.3 +cryptography==46.0.4 # via # joserfc # moto @@ -68,7 +68,7 @@ jinja2==3.1.6 # via # -r requirements/base.txt # moto -jmespath==1.0.1 +jmespath==1.1.0 # via # -r requirements/base.txt # boto3 @@ -111,7 +111,7 @@ openapi-schema-validator==0.6.3 # via openapi-spec-validator openapi-spec-validator==0.7.2 # via moto -packaging==25.0 +packaging==26.0 # via pytest pathable==0.4.4 # via jsonschema-path @@ -123,7 +123,7 @@ pluggy==1.6.0 # via pytest py-partiql-parser==0.6.3 # via moto -pycparser==2.23 +pycparser==3.0 # via cffi pycurl==7.45.7 # via pyresttest @@ -135,7 +135,7 @@ pygments==2.19.2 # via pytest pylint==4.0.4 # via -r requirements/test.in -pyparsing==3.3.1 +pyparsing==3.3.2 # via moto pyresttest==1.7.1 # via -r requirements/test.in