Serverless text search

Full-text search on AWS using only Lambda and S3 - no Elasticsearch, OpenSearch or Athena. A parent Lambda splits an S3 object into byte ranges and fans out child Lambdas to scan them in parallel, so it costs nothing at idle and scales per request.

Proof of concept accompanying a blog series on serverless search.

Formats

Records can be pipe (flattened SQL-style rows), jsonl (one JSON object per line), or prose (multi-line documents separated by a 0x1E record separator). Byte ranges are aligned to the record delimiter so nothing is split. An optional bloom-filter pre-pass (build_bloom.py) skips chunks that cannot contain a term - a big win for rare terms, nothing for common ones.

Layout

src/     parent_lambda.py, child_lambda.py, bloom.py
scripts/ deploy.sh, run_search.sh
tools/   generate_data.py, build_corpus.py, build_bloom.py, benchmark.py
tests/   test_parent_lambda.py, test_child_lambda.py

Usage

# generate test data (none of it is real)
python3 tools/generate_data.py --count 50000 --seed 7 --outfile test_data/images.txt

# deploy (proof of concept - a real deploy would use CDK/SAM/Terraform)
bash scripts/deploy.sh <aws-profile>

# search
aws lambda invoke --function-name parent-text-search-lambda \
  --cli-binary-format raw-in-base64-out \
  --payload '{"search_terms":["kyoto"],"account_id":"test123","format":"pipe","use_bloom":true}' \
  response.json && cat response.json

tools/benchmark.py sweeps fan-out width and bloom pruning; tests/ mock S3 and run locally.

Caveats

A proof of concept, not production. The parent aggregates all results itself (the bottleneck at scale), bloom pruning is sound for exact whole-word terms only, and the bloom index is a full rebuild.