feat: separate worker into ingestion and export instances(MAPCO-9571) by almog8k · Pull Request #95 · MapColonies/polygon-parts-worker

almog8k · 2026-06-22T08:30:03Z

Question	Answer
Tests added	✔
Chore	✔

Summary

The worker previously polled all four job types in a single deployment (3 ingestion jobs + Export). This change lets a worker process run as a single instanceType — ingestion or export — so the two domains can be deployed and scaled independently, each carrying only the resources its domain needs.

Changes

Domain selection — a new instanceType config value (ingestion | export) drives everything:
- JobProcessor polls only the job types of its domain.
- containerConfig validates instanceType at startup, registers only that domain's handler, and validates only the directories that domain uses.
- handlerFactory resolves a single INGESTION/EXPORT handler token (new/update/swap all map to the one ingestion handler).
Config — instanceType key + INSTANCE_TYPE env mapping (default ingestion).
Helm — renders only the relevant domain's env vars, volumes and S3 secret per instanceType, with a guard that fails the render fast on a missing/invalid value. Also restores the missing common.metrics.merged helper.
Deps — bumps @map-colonies/raster-shared to ^8.2.0 for the shared InstanceType const/type.
Tests — ingestion/export polling cases; fixtures/mocks updated for the single handler tokens.

Verification

npm test (tsc + 127 unit + 23 integration) passes; eslint clean.
helm template for each instanceType renders the correct domain-only env/volumes; the guard fails fast when instanceType is unset or invalid.

Provides the shared InstanceType const and type used to select the worker domain.

Each worker process now runs as a single instanceType (ingestion or export), read from config: - JobProcessor builds its poll set from the instance domain only. - containerConfig validates instanceType at startup, registers only that domain's handler, and validates only the directories that domain needs. - handlerFactory resolves a single INGESTION or EXPORT token instead of per-job-type tokens. - InstanceType const/type are now sourced from @map-colonies/raster-shared.

Adds the instanceType key (default ingestion) and its INSTANCE_TYPE env mapping.

configmap.yaml and deployment.yaml reference common.metrics.merged, which was missing and broke rendering.

Adds the instanceType value and renders only the relevant domain's env vars, volumes and S3 secret. A guard fails the render fast when instanceType is missing or invalid.

Adds ingestion/export polling cases and updates fixtures/mocks for the single INGESTION/EXPORT handler tokens.

CL-SHLOMIKONCHA · 2026-06-22T12:12:48Z

@@ -1,4 +1,5 @@
 {
+  "instanceType": "ingestion",


avoid set default value of worker by leave it empty spherically when you have this helper schema that validate for the supported values

Changed to empty string

CL-SHLOMIKONCHA · 2026-06-22T12:16:34Z

+{{- include "polygon-parts-worker.validateInstanceType" . -}}
 {{- $releaseName := .Release.Name -}}
 {{- $chartName := include "polygon-parts-worker.name" . -}}
 {{- $metrics := (include "common.metrics.merged" .) | fromYaml }}


out of context but please remove as nothing uses the $metrics in the file

Will handle in another pr

CL-SHLOMIKONCHA · 2026-06-25T07:48:37Z

 {{- include "common.tplvalues.merge" ( dict "values" ( list .Values.tracing .Values.global.tracing ) "context" . ) }}
 {{- end -}}

+{{- define "common.metrics.merged" -}}


although it out of context
do we need it for the metrics buckets? if so, can you add the "buckets" under the mclabel prometheus scope as we did with "overseer":
https://github.com/MapColonies/overseer/pull/101/changes#diff-81801117ec01136c8a49bfcd42af8e104f4efad1f2daccda9da9d960eaad5279

see values.yaml buckets list within mclabels scope

Will handle in another pr

CL-SHLOMIKONCHA · 2026-06-25T07:50:00Z

+  VALIDATION_REPORTS_PATH: {{ $validationReportsLocation | quote }}
  INGESTION_SOURCES_DIR_PATH: {{ printf "/%s" $fs.ingestionSourcePvc.mountPath | quote }}
+  REPORT_STORAGE_PROVIDER: {{ $storage.reportProvider }}
+  DOWNLOAD_SERVER_PUBLIC_DNS: {{ $serviceUrls.downloadServerPublicDNS | quote }}


im not sure, can you check if this env is used by the export job aswell?

DOWNLOAD_SERVER_PUBLIC_DNS is used only in ingestion scope

CL-SHLOMIKONCHA · 2026-06-28T06:34:42Z


  public static mockJobManagerSearchTasks(jobType: string, taskTypes: string[], task: ITaskResponse<unknown>): void {
-    for (const handler of Object.values(handlers)) {
+    for (const currentJobType of Object.values(jobTypes)) {


avoid "current" leave it as "jobType"

changed to "jobType"

CL-SHLOMIKONCHA · 2026-06-28T06:34:59Z

  });

  afterEach(() => {
    nock.cleanAll();


add the eslint rule to ignore nock imports

Added nock imports ignore

CL-SHLOMIKONCHA · 2026-06-28T06:37:13Z

lets talk about the metrics issue, we can separate it from this PR and handle it right after this one

almog8k · 2026-06-28T07:44:28Z

lets talk about the metrics issue, we can separate it from this PR and handle it right after this one

Ok, lets handle it in separate pr.

almog8k added 6 commits June 22, 2026 11:28

chore(deps): bump @map-colonies/raster-shared to ^8.2.0

07d507d

Provides the shared InstanceType const and type used to select the worker domain.

feat(config): add instanceType selector

5d14449

Adds the instanceType key (default ingestion) and its INSTANCE_TYPE env mapping.

fix(helm): restore common.metrics.merged template helper

a429352

configmap.yaml and deployment.yaml reference common.metrics.merged, which was missing and broke rendering.

feat(helm): render worker resources per instanceType

8820d22

Adds the instanceType value and renders only the relevant domain's env vars, volumes and S3 secret. A guard fails the render fast when instanceType is missing or invalid.

test: cover instanceType polling and handler token changes

0590d90

Adds ingestion/export polling cases and updates fixtures/mocks for the single INGESTION/EXPORT handler tokens.

mapcolonies-devops assigned almog8k Jun 22, 2026

almog8k changed the title ~~feat: separate worker into ingestion and export instances~~ feat: separate worker into ingestion and export instances(MAPCO-9571) Jun 22, 2026

CL-SHLOMIKONCHA requested changes Jun 28, 2026

View reviewed changes

test: reset default instanceType and fix mock variable shadowing

156193b

almog8k force-pushed the feat/seperate-deployments branch from ce55e99 to 156193b Compare June 28, 2026 07:52

Uh oh!

Conversation

almog8k commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Verification

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

almog8k Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CL-SHLOMIKONCHA commented Jun 28, 2026

Uh oh!

almog8k commented Jun 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

almog8k commented Jun 22, 2026 •

edited

Loading

almog8k Jun 28, 2026 •

edited

Loading