Skip to content

feat(etl-api): Add publication endpoints#748

Merged
iambriccardo merged 8 commits into
supabase:mainfrom
ttatsato:add-publication-endpoints
May 22, 2026
Merged

feat(etl-api): Add publication endpoints#748
iambriccardo merged 8 commits into
supabase:mainfrom
ttatsato:add-publication-endpoints

Conversation

@ttatsato

@ttatsato ttatsato commented May 18, 2026

Copy link
Copy Markdown
Contributor

What kind of change does this PR introduce?

Feature (new endpoints)

What is the current behavior?

The only way to modify a publication's table list is via POST /v1/sources/{source_id}/publications/{publication_name}, which performs a full replacement (SET TABLE).

There is currently no way to:

  • Add a single table to a publication without resending the full list
  • Remove a single table from a publication without resending the full list
  • Perform an idempotent full replacement using a semantically correct HTTP method

Related issue: #459

What is the new behavior?

Adds three new endpoints under /v1/sources/{source_id}/publications/{publication_name}/tables:

Method SQL Semantics
POST .../tables ALTER PUBLICATION x ADD TABLE ONLY ... Incremental add
DELETE .../tables ALTER PUBLICATION x DROP TABLE ONLY ... Incremental remove
PUT .../tables ALTER PUBLICATION x SET TABLE ONLY ... Idempotent full replacement (successor to the deprecated POST)

All three endpoints accept the same request body:

{ "tables": [{ "schema": "public", "name": "table_a" }] }

The PUT endpoint internally reuses data::publications::update_publication, so its behavior is identical to the deprecated POST /publications/{name} endpoint but at a semantically correct URL.

Additional context

This implements Option 2 from #459 (granular endpoints).

Broader publication features discussed in the issue thread (operation filtering, FOR ALL TABLES IN SCHEMA, column/row filtering) are intentionally out of scope and tracked separately.

How to test

Note: The setup below assumes the created source gets id: 1. Adjust the source ID in subsequent requests if your environment differs.

Setup

# 1. Start Docker
./scripts/init.sh

# 1.5 Run API server
# other terminal
APP_ENVIRONMENT=dev cargo run

# 2. Create tables
psql postgres://postgres:postgres@localhost:5430/postgres -c "
  create table if not exists public.table_a(id int primary key, name text);
  create table if not exists public.table_b(id int primary key, name text);
  create table if not exists public.table_c(id int primary key, name text);"

# 3. Create tenant/source
KEY='XOUbHmWbt9h7nWl15wWwyWQnctmFGNjpawMc3lT5CFs='
curl -X POST "http://127.0.0.1:8080/v1/tenants-sources" \
  -H "Authorization: Bearer $KEY" \
  -H "tenant_id: my-tenant" \
  -H "Content-Type: application/json" \
  -d '{"tenant_id":"my-tenant","tenant_name":"My Tenant","source_name":"Local Postgres",
       "source_config":{"host":"localhost","port":5430,"name":"postgres","username":"postgres","password":"postgres"}}'

# 4. Create a publication starting with table_a
curl -X POST "http://127.0.0.1:8080/v1/sources/1/publications" \
  -H "Authorization: Bearer $KEY" -H "tenant_id: my-tenant" \
  -H "Content-Type: application/json" \
  -d '{"name":"test_pub","tables":[{"schema":"public","name":"table_a"}]}'

POST .../tables (incremental add)

curl -i -X POST "http://127.0.0.1:8080/v1/sources/1/publications/test_pub/tables" \
  -H "Authorization: Bearer $KEY" -H "tenant_id: my-tenant" \
  -H "Content-Type: application/json" \
  -d '{"tables":[{"schema":"public","name":"table_b"}]}'
# → 200 OK

curl -s "http://127.0.0.1:8080/v1/sources/1/publications/test_pub" \
  -H "Authorization: Bearer $KEY" -H "tenant_id: my-tenant" | jq .
# → tables: [table_a, table_b]  (table_a is preserved)

DELETE .../tables (incremental remove)

curl -i -X DELETE "http://127.0.0.1:8080/v1/sources/1/publications/test_pub/tables" \
  -H "Authorization: Bearer $KEY" -H "tenant_id: my-tenant" \
  -H "Content-Type: application/json" \
  -d '{"tables":[{"schema":"public","name":"table_a"}]}'
# → 200 OK

curl -s "http://127.0.0.1:8080/v1/sources/1/publications/test_pub" \
  -H "Authorization: Bearer $KEY" -H "tenant_id: my-tenant" | jq .
# → tables: [table_b]  (table_a removed, table_b preserved)

PUT .../tables (idempotent full replacement)

curl -i -X PUT "http://127.0.0.1:8080/v1/sources/1/publications/test_pub/tables" \
  -H "Authorization: Bearer $KEY" -H "tenant_id: my-tenant" \
  -H "Content-Type: application/json" \
  -d '{"tables":[{"schema":"public","name":"table_c"}]}'
# → 200 OK

curl -s "http://127.0.0.1:8080/v1/sources/1/publications/test_pub" \
  -H "Authorization: Bearer $KEY" -H "tenant_id: my-tenant" | jq .
# → tables: [table_c]  (full replacement)

# Idempotency check: re-run the same PUT
curl -i -X PUT "http://127.0.0.1:8080/v1/sources/1/publications/test_pub/tables" \
  -H "Authorization: Bearer $KEY" -H "tenant_id: my-tenant" \
  -H "Content-Type: application/json" \
  -d '{"tables":[{"schema":"public","name":"table_c"}]}'
# → 200 OK, state unchanged

When publication is not found (return 404)

curl -i -X POST "http://127.0.0.1:8080/v1/sources/1/pubications/test_pub_missing/tables" \  
-H "Authorization: Bearer $KEY" -H "tenant_id: my-tenant" \ 
-H "Content-Type: application/json" \ 
-d '{"tables":[{"schema":"public","name":"table_b"}]}'

HTTP/1.1 404 Not Found
content-length: 65
content-type: application/json
date: Tue, 19 May 2026 23:02:37 GMT

{"message":"The publication \"test_pub_missing\" was not found."}%                                              
curl -i -X PUT "http://127.0.0.1:8080/v1/sources/1/publcations/test_pub_missing/tables" \  -H "Authorization: Bearer $KEY" -H "tenant_id: my-tenant" \  -H "Content-Type: application/json" \  -d '{"tables":[{"schema":"public","name":"table_b"}]}'

HTTP/1.1 404 Not Found
content-length: 65
content-type: application/json
date: Tue, 19 May 2026 23:01:10 GMT

{"message":"The publication \"test_pub_missing\" was not found."}%                                              

curl -i -X DELETE "http://127.0.0.1:8080/v1/sources/1/pblications/test_pub_missing/tables" \  -H "Authorization: Bearer $KEY" -H "tenant_id: my-tenant" \  -H "Content-Type: application/json" \  -d '{"tables":[{"schema":"public","name":"table_b"}]}'

HTTP/1.1 404 Not Found
content-length: 65
content-type: application/json
date: Tue, 19 May 2026 23:05:37 GMT

{"message":"The publication \"test_pub_missing\" was not found."}%                                                        

curl -i -X POST "http://127.0.0.1:8080/v1/sources/1/pubications/test_pub_missing" \  -H "Authorization: Bearer $KEY" -H "tenant_id: my-tenant" \  -H "Content-Type: application/json" \  -d '{"tables":[{"schema":"public","name":"table_b"}]}'

HTTP/1.1 404 Not Found
content-length: 65
content-type: application/json
date: Tue, 19 May 2026 23:14:24 GMT

{"message":"The publication \"test_pub_missing\" was not found."}%                                                                          

Note

  • This endpoint was marked deprecated in ref(etl-api): deprecate POST update-publication endpoint #743 due to misleading semantics (POST implies create/append, not replace).
  • When the request body contains invalid tables (empty array or non-existent table names), these endpoints still return 500. I have not addressed that in this PR.

@ttatsato ttatsato marked this pull request as ready for review May 18, 2026 13:02
@ttatsato ttatsato requested a review from a team as a code owner May 18, 2026 13:02
@ttatsato ttatsato changed the title Add publication endpoints feat(etl-api): Add publication endpoints May 18, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 835ca5fe8f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/etl-api/src/routes/sources/publications.rs
Comment thread crates/etl-api/src/data/publications.rs Outdated
Comment thread crates/etl-api/src/data/publications.rs

@bnjjj bnjjj left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only looked at the code. LGTM but as @iambriccardo was involved in the original issue I would like him to review

@iambriccardo iambriccardo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one comment: Missing publications return 500 from the new /tables endpoints instead of the advertised 404.

But otherwise LGTM, thanks for this! We have tickets to make the frontend adopt this endpoint!

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 05991c61c9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/etl-api/src/data/publications.rs
@ttatsato ttatsato temporarily deployed to integration-tests-secrets May 20, 2026 06:19 — with GitHub Actions Inactive
@ttatsato ttatsato temporarily deployed to integration-tests-secrets May 20, 2026 06:19 — with GitHub Actions Inactive
@ttatsato ttatsato temporarily deployed to integration-tests-secrets May 20, 2026 06:19 — with GitHub Actions Inactive
@ttatsato ttatsato temporarily deployed to integration-tests-secrets May 20, 2026 06:19 — with GitHub Actions Inactive
@ttatsato ttatsato temporarily deployed to integration-tests-secrets May 20, 2026 06:19 — with GitHub Actions Inactive
@ttatsato ttatsato temporarily deployed to integration-tests-secrets May 20, 2026 06:19 — with GitHub Actions Inactive
@ttatsato ttatsato requested a review from iambriccardo May 21, 2026 00:03
@ttatsato

Copy link
Copy Markdown
Contributor Author

@iambriccardo

Thank you for reviewing this.
I've fixed the response status code when the publication is missing.
Could you review it again?
[cf911dc...05991c6]

Note:
The Codex's comment was not addressed in this PR; I've created a separate issue for it.

#748 (comment)

@coveralls

coveralls commented May 21, 2026

Copy link
Copy Markdown

Coverage Status

coverage: 70.402% (-3.1%) from 73.508% — ttatsato:add-publication-endpoints into supabase:main

@iambriccardo iambriccardo enabled auto-merge (squash) May 22, 2026 08:40
@iambriccardo iambriccardo merged commit 3f79489 into supabase:main May 22, 2026
13 checks passed
ttatsato added a commit to ttatsato/supabase-etl that referenced this pull request May 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants