IDTAP Python API Development Guide

Overview

The Python API (idtap) is a sophisticated client library for interacting with the IDTAP (Interactive Digital Transcription and Analysis Platform) server, specifically designed for transcribing, analyzing, and managing Hindustani music recordings.

Key Development Points

Dependencies Management

Keep Pipfile and pyproject.toml in sync - this is critical!
Add new packages: pipenv install package-name
Then manually add to pyproject.toml dependencies array
Core deps: requests, pyhumps, keyring, cryptography, PyJWT, pymongo, google-auth-oauthlib

Testing

Unit tests: pytest python/idtap/tests/ (uses responses for HTTP mocking)
Integration tests: python python/api_testing/api_test.py (requires live server auth)
Test structure: Complete coverage of data models, client functionality, and authentication

⚠️ IMPORTANT FOR CLAUDE: Before running the full test suite (pytest idtap/tests/), ALWAYS warn Jon first!

Some tests may require browser authorization for OAuth authentication
Running tests without warning can waste time waiting for authorization that Jon doesn't realize is needed
Best practice: Ask "Ready to run the full test suite? (May require browser authorization)" before executing

Build/Package/Publish - AUTOMATED via GitHub Actions

⚠️ IMPORTANT: Manual publishing is now automated. See "Automated Version Management" section below.

Manual publishing (for local testing only):

python -m build
python -m twine upload dist/*  # or --repository testpypi for testing

Architecture

Main Components

SwaraClient (client.py) - Main HTTP client with OAuth authentication
Data Models (/classes/) - Rich musical transcription classes (Piece, Phrase, Trajectory, etc.)
Authentication (auth.py + secure_storage.py) - OAuth flow with secure token storage
Utils (utils.py) - camelCase ↔ snake_case conversion

Key Classes

Piece: Central transcription container with multi-track support, sections, audio association
SwaraClient: API interface with methods for transcription CRUD, audio download, permissions
Musical Elements: Phrase, Trajectory, Pitch, Raga, Section, Meter, Articulation

Security/Authentication

OAuth Flow: Server-based OAuth (not direct Google) → local HTTP server → secure storage
Storage Layers: OS Keyring (primary) → AES-256 encrypted file (fallback) → plaintext (legacy)
CSRF Protection: State parameter validation
Permissions: User-based access control with public/private visibility

Development Patterns

Code Conventions

snake_case for Python code
camelCase ↔ snake_case conversion via pyhumps for API communication
Type hints throughout
Backwards compatibility maintained (especially token storage migration)

Serialization Pattern

class DataModel:
    def to_json(self) -> Dict[str, Any]:
        # Convert to dict with camelCase keys for API
        
    @staticmethod 
    def from_json(obj: Dict[str, Any]) -> 'DataModel':
        # Parse from API response with snake_case conversion

Package Structure

python/idtap/
├── __init__.py           # Public API exports
├── client.py             # HTTP client (SwaraClient)
├── auth.py               # OAuth authentication
├── secure_storage.py     # Token security layers
├── enums.py              # Instrument types, etc.
├── utils.py              # camelCase conversion utilities
├── classes/              # Musical data models
└── tests/                # Unit tests

API Endpoints (via SwaraClient)

Transcriptions: GET/POST /api/transcription/{id}, GET /api/transcriptions
Data Export: GET /api/transcription/{id}/json, /api/transcription/{id}/excel
Audio: GET /audio/{format}/{id}.{format}
Permissions: POST /api/visibility
OAuth: GET /oauth/authorize, POST /oauth/token

Musical Domain Knowledge

Hindustani Music Focus: Transcription system for Indian classical music
Complex Data Models: Supports microtonal pitches, ragas, articulations, meter cycles
Multi-instrument: Sitar, Vocal (Male/Female) with instrument-specific features
Analytical Tools: Trajectory categorization, phrase grouping, temporal analysis

Automated Version Management & Publishing

🤖 How It Works (python-semantic-release + GitHub Actions)

PATCH-ONLY MODE: All commits automatically increment patch version (0.1.14 → 0.1.15) regardless of commit message.

Automatic Workflow

Make changes → Commit with any message
Create PR → GitHub Actions test + upload to TestPyPI automatically
Merge to main → Version bumps + PyPI published + GitHub release created
Zero manual intervention required!

Version Locations (Auto-Updated)

idtap/__init__.py:3 - __version__ = "0.1.14"
pyproject.toml:7 - version = "0.1.14"
docs/conf.py:16-17 - release = '0.1.14' and version = '0.1.14'

🎛️ Manual Version Control Commands

For normal development (patch bumps):

# Any commit message works - always bumps patch
git commit -m "fix something"           # 0.1.14 → 0.1.15
git commit -m "add new feature"         # 0.1.14 → 0.1.15 
git commit -m "update docs"             # 0.1.14 → 0.1.15

When Jon wants to control version bumps manually:

# Force minor version bump (0.1.15 → 0.2.0)
semantic-release version --increment minor

# Force major version bump (0.2.0 → 1.0.0)  
semantic-release version --increment major

# Dry run to see what would happen
semantic-release version --print --no-commit --no-tag --no-push --no-vcs-release

🔧 Configuration Details

Location: pyproject.toml [tool.semantic_release] section

patch_tags: ALL commit types → patch version increment
minor_tags: [] (empty - no automatic minor bumps)
major_tags: [] (empty - no automatic major bumps)
Files updated: __init__.py, pyproject.toml, docs/conf.py

🚀 GitHub Actions Workflows

.github/workflows/test-pr.yml:

Runs on every PR
Tests + builds package + uploads to TestPyPI
Comments PR with TestPyPI install link

.github/workflows/release.yml:

Runs on merge to main
Tests → version bump → build → PyPI publish → GitHub release

📋 Required Setup (One-Time)

GitHub Secrets:

TESTPYPI_API_TOKEN - For PR testing uploads

PyPI Trusted Publisher (configured):

Owner: UCSC-IDTAP
Repository: Python-API
Workflow: release.yml
Uses OIDC - no API tokens needed for production PyPI

⚠️ CLAUDE: When Jon asks to update version in special way, use the manual commands above!

Development Workflow (Updated for Automation)

Data Model Development: Create/modify classes in /classes/ with proper serialization
Client Method Development: Add HTTP methods in client.py with authentication
Testing: Write unit tests (mocked) + integration tests (live API)
PR Creation: GitHub Actions automatically test + upload to TestPyPI
Merge to main: Automatic version bump + PyPI publish + GitHub release
No manual version management needed!

Installation Commands

# Development
pip install -e python/
pipenv install --dev

# Testing  
pytest python/idtap/tests/
python python/api_testing/api_test.py

# Package management
pipenv install package-name  # then manually add to pyproject.toml

This API provides a production-ready foundation for complex musical transcription analysis with modern security practices and comprehensive testing coverage.

PyPI Publishing Guide

Quick Reference Checklist

Before ANY publish, run these commands:

# 1. Fix critical bugs first (see CRITICAL FIXES section below)
# 2. Update versions to match in BOTH files
# 3. Run tests
pytest idtap/tests/
# 4. Clean build
rm -rf dist/ build/ *.egg-info/
# 5. Build
python -m build
# 6. Test upload (optional but recommended)
python -m twine upload --repository testpypi dist/*
# 7. Production upload
python -m twine upload dist/*
# 8. Tag and push
git tag vX.X.X && git push origin vX.X.X

Complete Steps to Publish New Version to PyPI

Prerequisites

PyPI account with API token configured
build and twine packages installed (pipenv install build twine)
All changes committed and pushed to GitHub

Step-by-Step Publishing Process

1. Update Version Numbers (CRITICAL - Must be done in both places)

A. Update idtap/__init__.py:

__version__ = "0.1.13"  # Increment from current "0.1.12"

B. Update pyproject.toml:

[project]
version = "0.1.13"  # Must match __init__.py exactly

Version Increment Rules:

Patch (0.1.7 → 0.1.8): Bug fixes, small improvements, query system enhancements
Minor (0.1.8 → 0.2.0): New features, API additions (backwards compatible)
Major (0.2.0 → 1.0.0): Breaking changes, API modifications

2. Update Dependencies (if needed)

A. Sync Pipfile and pyproject.toml:

# If adding new packages:
pipenv install new-package-name
# Then manually add to pyproject.toml dependencies array

B. Check dependency versions in pyproject.toml:

dependencies = [
    "requests>=2.31.0",
    "pyhumps>=3.8.0",
    # ... ensure all deps are current
]

3. Pre-Publishing Checks

A. Run Full Test Suite:

pytest idtap/tests/  # Must pass all tests

B. Test Package Installation Locally:

pip install -e .  # Test editable install
python -c "import idtap; print(idtap.__version__)"  # Verify version

C. Check Package Structure:

python -m build --sdist --wheel .  # Build without uploading
# Check dist/ directory for .whl and .tar.gz files

4. Build Package

A. Clean Previous Builds:

rm -rf dist/ build/ *.egg-info/

B. Build Distribution Files:

python -m build

Creates dist/idtap-X.X.X-py3-none-any.whl (wheel)
Creates dist/idtap-X.X.X.tar.gz (source distribution)

5. Test on TestPyPI (Recommended)

A. Authentication Setup: TestPyPI and production PyPI require separate API tokens. Configure in .envrc (gitignored):

# TestPyPI token (get from https://test.pypi.org/)
export TWINE_TESTPYPI_PASSWORD="pypi-[YOUR_TESTPYPI_TOKEN]"

# Production PyPI token (get from https://pypi.org/)
export TWINE_PASSWORD="pypi-[YOUR_PRODUCTION_TOKEN]"

# Twine username for both
export TWINE_USERNAME="__token__"

B. Upload to TestPyPI:

TWINE_PASSWORD="$TWINE_TESTPYPI_PASSWORD" python -m twine upload --repository testpypi dist/*

C. Test Installation from TestPyPI:

pip install --index-url https://test.pypi.org/simple/ idtap

D. Verify TestPyPI Installation:

python -c "import idtap; print(idtap.__version__)"

6. Publish to Production PyPI

A. Upload to PyPI:

python -m twine upload dist/*

B. Verify Upload:

Check https://pypi.org/project/idtap/
Verify new version is listed
Check that description/metadata displays correctly

7. Post-Publishing Verification

A. Test Production Installation:

pip install --upgrade idtap
python -c "import idtap; print(idtap.__version__)"

B. Test Key Functionality:

from idtap.classes.raga import Raga
from idtap import SwaraClient

# Test that new validation works
try:
    Raga({'rules': {}})  # Should raise helpful error
except ValueError as e:
    print(f"✅ Validation working: {e}")

8. Git Tagging and Cleanup

A. Create Git Tag:

git tag v0.1.8  # Match the version number
git push origin v0.1.8

B. Clean Build Artifacts:

rm -rf dist/ build/ *.egg-info/

Troubleshooting Common Issues

Version Conflicts

Error: "File already exists" on PyPI upload
Fix: Ensure version number is incremented in BOTH __init__.py AND pyproject.toml

Package Structure Issues

Error: Import errors after installation
Fix: Check pyproject.toml [tool.setuptools.packages.find] section
Historical Issue (FIXED): Config used to look for idtap* but package is idtap*
Current Status: ✅ Fixed - now correctly configured to include idtap*

Dependency Conflicts

Error: Dependencies not installing correctly
Fix: Ensure Pipfile and pyproject.toml dependencies are exactly synchronized

Authentication Issues

Error: 403 Forbidden on upload
Fix: Configure PyPI API token: python -m twine configure

PyPI Configuration Files

API Token Setup (one-time):

# Create ~/.pypirc with API token
[distutils]
index-servers = pypi testpypi

[pypi]
username = __token__
password = pypi-YOUR_API_TOKEN_HERE

[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__
password = pypi-YOUR_TEST_API_TOKEN_HERE

Current Package Status

Package Name: idtap (changed from idtap-api)
Current Version: 0.1.13 (synced in both pyproject.toml and init.py) ✅
Package Structure: Fixed - now correctly includes idtap* instead of idtap* ✅
Package Data: Fixed - now correctly references idtap directory ✅
Python Support: >= 3.10
Key Dependencies: requests, pyhumps, keyring, cryptography, PyJWT

✅ All critical packaging bugs have been resolved!

Post-Validation Release Notes Template

For the current comprehensive validation release:

## v0.1.8 - Query System Improvements

### 🚀 New Features
- Enhanced query system functionality
- Improved authentication token handling investigation
- Better development workflow for query branch integration

### 🐛 Bug Fixes  
- Resolved query system integration issues
- Improved error handling in authentication flow

### 🧪 Testing
- All existing tests continue to pass
- Query system thoroughly tested and validated

### ⚠️ Breaking Changes
None - fully backwards compatible with existing usage

## v0.1.6 - Enhanced Parameter Validation

### 🚀 New Features
- Comprehensive parameter validation for Raga constructor
- Helpful error messages for common parameter mistakes
- Type validation for all constructor parameters
- Structure validation for rule_set and tuning dictionaries

### 🐛 Bug Fixes  
- Prevents silent parameter failures that led to incorrect musical analysis
- Fixed unused variable warnings in existing code

### 🧪 Testing
- Added 29 comprehensive validation test cases
- All 254 tests in full suite passing
- Tests cover every possible invalid input scenario

### ⚠️ Breaking Changes
None - fully backwards compatible with existing valid usage

FilesExpand file tree

CLAUDE.md

Latest commit

History