test_math_pipeline_runs_e2e fails locally: DynamoDB endpoint + RepresentativeComments export

[jucor]

Summary

test_math_pipeline_runs_e2e fails locally when DynamoDB, MinIO, and Postgres are all running. Two issues discovered:

1. DYNAMODB_ENDPOINT not loaded from delphi/.env

Neither conftest.py nor run_math_pipeline.py calls load_dotenv(), so env vars from delphi/.env aren't picked up when running tests outside Docker. The test fixture (dynamodb_resource) hardcodes a fallback to http://localhost:8000, but the pipeline code (run_math_pipeline.py:389) uses os.environ.get('DYNAMODB_ENDPOINT') which returns None — causing boto3 to connect to real AWS instead of local DynamoDB.

In CI this works because docker-compose.test.yml injects DYNAMODB_ENDPOINT=http://dynamodb:8000 into the container env.

Fix options:

• Add load_dotenv(find_dotenv()) to conftest.py
• Or set DYNAMODB_ENDPOINT in pyproject.toml via [tool.pytest.ini_options] env (requires pytest-env)

2. ~/.aws/credentials overrides dummy defaults

Both the test and pipeline default AWS_ACCESS_KEY_ID to 'dummy' via os.environ.get(...). But when ~/.aws/credentials exists with real AWS creds, boto3's credential chain picks those up before the code's explicit aws_access_key_id='dummy' parameter — but only for clients that don't pass explicit creds (e.g., if DYNAMODB_ENDPOINT is None and boto3 tries real AWS).

3. RepresentativeComments not exported to DynamoDB

Even with correct env vars, the pipeline computes repness but the Delphi_RepresentativeComments table is empty after the e2e test run. PCA and KMeans results are written successfully. This suggests export_to_dynamodb() either skips or fails silently on representative comments.

Reproduction

cd delphi
DYNAMODB_ENDPOINT=http://localhost:8000 \
  AWS_ACCESS_KEY_ID=dummy \
  AWS_SECRET_ACCESS_KEY=dummy \
  uv run pytest tests/test_math_pipeline_runs_e2e.py -v --tb=long

Environment

• Local dev with Docker services running (DynamoDB, MinIO, Postgres)
• Tests run outside Docker (via uv run pytest)

[jucor]

Update: Root cause narrowed

Issue 1 (env vars) confirmed

DYNAMODB_ENDPOINT and AWS creds from delphi/.env aren't loaded because nothing calls load_dotenv(). CI works because docker-compose.test.yml injects DYNAMODB_ENDPOINT=http://dynamodb:8000 into the container env (line 239).

Issue 3 (RepresentativeComments empty) — deeper than expected

Even with all env vars set correctly (DYNAMODB_ENDPOINT=http://localhost:8000, AWS_ACCESS_KEY_ID=dummy), the pipeline logs "Step 5 completed: Written representativeness data for 123456789 in 2.68s" but the DynamoDB scan finds zero items for that zid.

• PCA, KMeans, CommentRouting, and Projections are all written correctly
• The batch_writer reports no errors
• The data simply isn't in the table after the test completes
• This passes in CI (Docker container environment)
• No type mismatch: both write and scan use str(zid)

Suspect: either batch_writer() silently drops items (schema validation?), or the DynamoDBClient's _ensure_tables_exist() interacts oddly with the existing local tables. Needs further investigation.