Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions .github/workflows/publish-package.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Publish Python Package to GCP Artifact Registry

on:
push:
tags:
- 'v*' # Trigger on tags like v0.1.0, v1.2.3, etc.

jobs:
build-and-publish:
name: Build and publish Python package to GCP Artifact Registry
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'

- name: Install dependencies and build package
run: |
cd src/python
./build.sh

- id: 'auth'
uses: 'google-github-actions/auth@v2'
with:
credentials_json: '${{ secrets.GCP_SA_KEY }}'

- name: Publish to Artifact Registry
run: |
pip install twine
cd src/python/role_play
twine upload --repository-url https://${{ secrets.GCP_REGION }}-python.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/${{ secrets.GCP_ARTIFACT_REGISTRY_REPO }}/ dist/*
22 changes: 22 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,13 @@ cd src/ts/role_play/ui && npm install && npm run dev
make test-chat # Chat module tests only
make test-coverage-html # Generate HTML coverage report
make test-specific TEST_PATH="test/python/unit/chat/test_chat_logger.py"

# Package Testing
make test-package-all # Run all package tests (build, install, inspect)
make test-package-build # Test package build process
make test-package-install # Test package installation
make inspect-package # Inspect package contents
make test-gcp-upload # Test GCP upload (interactive)
```

**Environment**: `ENV=dev|beta|prod`, configs in `config/{env}.yaml`
Expand Down Expand Up @@ -275,6 +282,21 @@ make test-specific TEST_PATH="test/python/unit/chat/test_chat_logger.py"
- [x] **Debug Utility Tests**: Complete test suite for audio reassembly and WAV generation functions
- [x] **Documentation**: Complete README.md with usage examples and troubleshooting guides

### Python Package Publishing Infrastructure (Completed)
- [x] **Modern Package Configuration**: Created `pyproject.toml` with complete metadata and build settings
- [x] **Resource Inclusion**: Added `MANIFEST.in` for proper resource distribution in packages
- [x] **Dependency Management**: Separated requirements into core, dev, test, and all-inclusive files
- [x] **GitHub Actions Workflow**: Automated publishing to GCP Artifact Registry on version tags
- [x] **Comprehensive Testing Framework**: Created 4 test scripts for complete validation:
- `test/python/packaging/test-build.sh` - Package build process testing
- `test/python/packaging/test-install.sh` - Installation testing in clean environments
- `test/python/packaging/inspect-package.sh` - Package content inspection and validation
- `test/python/packaging/test-gcp-upload.sh` - GCP Artifact Registry upload testing
- [x] **Makefile Integration**: Added `make test-package-*` targets for easy package testing
- [x] **Import Structure Validation**: Confirmed relative imports work correctly in both development and packaged contexts
- [x] **Documentation**: Created `PACKAGE_TESTING.md` and updated README with packaging instructions
- [x] **Clean Project Structure**: Moved test scripts to proper `test/` directory following standard conventions
- [x] **Corporate Licensing**: Updated all LICENSE files to reflect CatTail Software copyright

## Implementation Phases
1. Core Infrastructure → 2. Authentication → 3. Handlers → 4. WebSocket/Audio → 5. Polish
Expand Down
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
MIT License

Copyright (c) 2025 Yenchi Lin
Copyright (c) 2025 CatTail Software

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
53 changes: 41 additions & 12 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -380,19 +380,48 @@ dev-setup: load-env-mk
@echo ""
@echo "Or use PyCharm to run src/python/run_server.py"

# --- Package Testing ---
.PHONY: test-package-build
test-package-build: ## Test local package build process
@echo "Running package build test..."
@./test/python/packaging/test-build.sh

.PHONY: test-package-install
test-package-install: ## Test package installation in clean environment
@echo "Running package installation test..."
@./test/python/packaging/test-install.sh

.PHONY: inspect-package
inspect-package: ## Inspect package contents and structure
@echo "Inspecting package contents..."
@./test/python/packaging/inspect-package.sh

.PHONY: test-gcp-upload
test-gcp-upload: ## Test GCP Artifact Registry upload (interactive)
@echo "Running GCP upload test..."
@./test/python/packaging/test-gcp-upload.sh

.PHONY: test-package-all
test-package-all: test-package-build inspect-package test-package-install ## Run all package tests (except GCP)
@echo "All package tests completed successfully!"

.PHONY: build-package
build-package: ## Build the Python package
@echo "Building Python package..."
@cd src/python && ./build.sh

.PHONY: test-package-venv
test-package-venv: ## Activate venv and run package tests (alternative command)
@echo "Activating virtual environment and running package tests..."
@source venv/bin/activate && $(MAKE) test-package-all

# --- Release Management ---
.PHONY: tag-git-release
tag-git-release: # Expects NEW_GIT_TAG to be set, e.g., make tag-git-release NEW_GIT_TAG=v1.0.0
ifndef NEW_GIT_TAG
$(error NEW_GIT_TAG is not set. Usage: make tag-git-release NEW_GIT_TAG=vX.Y.Z)
endif
@echo "Creating Git tag: $(NEW_GIT_TAG)"
@read -p "Enter commit message for tag $(NEW_GIT_TAG) (Press Enter for default: 'Release $(NEW_GIT_TAG)'): " msg; \
COMMIT_MSG=$${msg:-Release $(NEW_GIT_TAG)}; \
git tag -a "$(NEW_GIT_TAG)" -m "$$COMMIT_MSG"
@echo "Pushing Git tag $(NEW_GIT_TAG) to origin..."
@git push origin "$(NEW_GIT_TAG)"
@echo "Git tag $(NEW_GIT_TAG) created and pushed."
.PHONY: release
release:
@echo "To create a new release, create and push a new git tag."
@echo "Example: git tag v0.1.0 && git push origin v0.1.0"
@echo ""
@echo "Before releasing, run: make test-package-all"

# --- GCP Setup ---
.PHONY: setup-gcp-infra
Expand Down
256 changes: 256 additions & 0 deletions PACKAGE_TESTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,256 @@
# Package Testing Guide

This guide provides comprehensive testing instructions for the `role_play_system` Python package before publishing to GCP Artifact Registry.

## Quick Start

**Option 1: Scripts handle venv automatically**
```bash
# Test everything locally (scripts auto-activate venv)
make test-package-all

# Test GCP upload (interactive)
make test-gcp-upload
```

**Option 2: Manually activate venv first**
```bash
# Activate virtual environment first
source venv/bin/activate

# Then run tests
make test-package-all
```

**Option 3: Use the venv-aware target**
```bash
# Alternative command that ensures venv activation
make test-package-venv
```

## Testing Scripts

### 1. Build Testing (`test-build.sh`)

Tests the package build process and validates artifacts:

```bash
./test/python/packaging/test-build.sh
```

**What it tests:**
- Clean build process
- Artifact creation (.whl and .tar.gz)
- Package metadata validation with twine
- File contents and sizes
- Required files inclusion

### 2. Installation Testing (`test-install.sh`)

Tests package installation in a clean environment:

```bash
./test/python/packaging/test-install.sh
```

**What it tests:**
- Virtual environment creation
- Package installation from wheel
- Module imports
- Dependency installation
- Package metadata verification
- Clean uninstallation

### 3. Content Inspection (`inspect-package.sh`)

Detailed inspection of package contents:

```bash
./test/python/packaging/inspect-package.sh
```

**What it shows:**
- Detailed file listings
- Module structure
- Essential file presence
- Unwanted file detection
- Dependency information
- Comparison between wheel and tarball

### 4. GCP Upload Testing (`test-gcp-upload.sh`)

Interactive GCP Artifact Registry testing:

```bash
./test/python/packaging/test-gcp-upload.sh
```

**What it tests:**
- GCP CLI installation and auth
- Project access and permissions
- Artifact Registry API enablement
- Test repository creation
- Twine configuration
- Actual upload testing (optional)
- Installation from Artifact Registry

## Step-by-Step Testing Process

### Phase 1: Local Testing

1. **Build the package:**
```bash
make build-package
```

2. **Run all local tests:**
```bash
make test-package-all
```

3. **Inspect package contents:**
```bash
make inspect-package
```

### Phase 2: GCP Testing

1. **Set up GCP authentication:**
```bash
gcloud auth login
gcloud auth application-default login
```

2. **Run GCP tests:**
```bash
make test-gcp-upload
```

3. **Follow the interactive prompts to:**
- Configure project settings
- Create test repository
- Perform test upload
- Verify installation from registry

### Phase 3: GitHub Actions Testing

1. **Verify workflow syntax:**
```bash
# Install act for local testing (optional)
brew install act # or similar for your OS
act -n # dry run
```

2. **Check required secrets are set:**
- `GCP_PROJECT_ID`
- `GCP_SA_KEY`
- `GCP_REGION`
- `GCP_ARTIFACT_REGISTRY_REPO`

3. **Test with a pre-release tag:**
```bash
git tag v0.1.0-test
git push origin v0.1.0-test
```

## Common Issues and Solutions

### Build Issues

**Problem:** `python3 -m build` fails
- **Solution:** Install build tools: `pip install build setuptools wheel`

**Problem:** Missing dependencies in package
- **Solution:** Check `pyproject.toml` dependencies list matches `requirements.txt`

### Installation Issues

**Problem:** Import errors after installation
- **Solution:** Verify `__init__.py` files exist in all packages

**Problem:** Missing dependencies
- **Solution:** Check dependency specification in `pyproject.toml`

### GCP Issues

**Problem:** Authentication failures
- **Solution:** Run `gcloud auth login` and `gcloud auth application-default login`

**Problem:** Repository not found
- **Solution:** Create repository: `gcloud artifacts repositories create python-packages --repository-format=python --location=us-central1`

**Problem:** Permission denied
- **Solution:** Ensure service account has `artifactregistry.writer` role

## Pre-Release Checklist

Before creating a release tag:

- [ ] `make test-package-all` passes
- [ ] Package installs and imports correctly
- [ ] All required files are included in package
- [ ] No unwanted files (cache, git, etc.) in package
- [ ] GCP authentication and permissions configured
- [ ] Test repository upload successful
- [ ] GitHub secrets configured correctly
- [ ] Version number updated in `pyproject.toml`

## Release Process

When ready to release:

1. **Update version:**
```bash
# Edit src/python/role_play/pyproject.toml
version = "0.1.1" # or appropriate version
```

2. **Final testing:**
```bash
make test-package-all
```

3. **Create and push tag:**
```bash
git add .
git commit -m "chore: bump version to 0.1.1"
git tag v0.1.1
git push origin main
git push origin v0.1.1
```

4. **Monitor GitHub Actions:**
- Check workflow execution in GitHub
- Verify package appears in Artifact Registry
- Test installation from production registry

## Cleanup

After testing, clean up temporary files:

```bash
# Remove build artifacts
cd src/python/role_play
rm -rf dist/ build/ *.egg-info

# Remove test repositories (if created)
gcloud artifacts repositories delete python-test --location=us-central1

# Remove test tags
git tag -d v0.1.0-test
git push origin --delete v0.1.0-test
```

## Getting Help

If you encounter issues:

1. Check this guide for common solutions
2. Verify all prerequisites are installed
3. Check GCP console for detailed error messages
4. Review GitHub Actions logs for workflow issues

For additional help, refer to:
- [Python Packaging Guide](https://packaging.python.org/)
- [GCP Artifact Registry Documentation](https://cloud.google.com/artifact-registry/docs)
- [Twine Documentation](https://twine.readthedocs.io/)
Loading