Implement Redis-based data ingestion service with event handlers and Synapse integration by Patrik-Corri · Pull Request #153 · Team-Deepiri/deepiri-platform

Patrik-Corri · 2026-03-20T01:28:37Z

IMPORTANT:

PR must be opened from your personal branch → dev
You must tag @Team-Deepiri/support-team
You must update Plaky to "Needs QA"
Never move a task to "Done" (Done = production release only)

Description

Briefly explain what this PR does and why.

Include:

Related Issue number
Data Ingestion Service Integration - Integrate with API Gateway and Synapse
Data Ingestion Service, API Gateway, Synapse, Database , Object Storage, Auth/User Context
Integration

Changes

Added StreamConsumerService to consume events from the platform-events Redis Stream using consumer group language--intelligence-consumers
Added streamRegistry.ts to register event handlers on startup, decoupling handler registration from the consumer loop
Added leaseEventHandler.ts, contractEventHandler.ts, and obligationEventHandler.ts to handle incoming stream events and update Prisma records accordingly

Testing

Start the stack and exec into the Redis container
Test - XADD platform-events * event lease-created timestamp 2026-03-20T00:00:00.000Z source test service test data "{"leaseId":"550e8400-e29b-41d4-a716-446655440000","leaseNumber":"L-001"}"
Check language intelligence logs: docker logs --tail 20 deepiri-language-intelligence-service-dev
Expected: [LeaseEventHandler] lease-created received { leaseId: '550e8400-e29b-41d4-a716-446655440000' }

Important Notes (Optional)

Known limitations:
Blockers:
CI/CD issues unrelated to this PR:
Dependencies required for testing:

Workflow Checklist (Required)

Branch is up to date with dev
PR is from your branch → dev (no longer directly into main)
PR title follows convention (feat:, fix:, refactor:, etc.)
Plaky feature/bug name included above
Tagged @Team-Deepiri/support-team
Plaky task moved to "Needs QA"

Review Requests

@Team-Deepiri/support-team

This reverts commit c60e326.

… into patrik_corri-dev

…submodule

jrb00013 · 2026-03-21T08:39:15Z


-# SINGLE atomic install
-RUN npm config set fetch-retries 5 \
- && npm config set fetch-retry-mintimeout 20000 \


Why were these taken out for each service in the Dockerfiles?

jrb00013 · 2026-03-21T08:40:50Z

-RUN chmod +x /usr/local/bin/load-k8s-env.sh /usr/local/bin/docker-entrypoint.sh
+COPY --chown=root:root platform-services/shared/scripts/load-k8s-env.sh /usr/local/bin/load-k8s-env.sh
+COPY --chown=root:root platform-services/shared/scripts/docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
+RUN sed -i 's/\r$//' /usr/local/bin/load-k8s-env.sh /usr/local/bin/docker-entrypoint.sh && \


Are you having linux build issues?

jrb00013 · 2026-03-21T08:43:59Z

            "agi-decisions",
-            "training-events"
+            "training-events",
+            "ingestion-events",


So you just set up the channel, but how does this ingest data?

All I see are dockerfile edits, small changes to references and the streaming client for the shared utils, which then is imported by the lang intelligence service which then creates client subscriptions to the channels, ensure that they match up.

@thatsnotrlght you need to be making some progress on the Redis stuff by now in order for @Patrik-Corri to start receiving data

Any dockerfile edits are because i cannot build the containers without those changes, so ignore those. The data ingestion files are all located in the language-intellegence-service submodule.
a98efcf

Every other file change can be ignored as they are only to help me build the docker containers.
You said i have to create a PR for both the submodule and platform correct?

Why is it that you are not able to build the containers? Do not put the sed command into the Dockerfile, you need to restore those changes and manually run the sed command on those files, then commit that state of the files. That will resolve the issues for everyone taking out those invisible windows chars, as well as it won't edit the Dockerfile. 2 birds with 1 stone.

You need to create a PR for both the submodule changes and the platform's submodule pointer as well as any platform file changes.

@Patrik-Corri

austinm2h35-sketch · 2026-03-21T20:19:36Z

@SeanSan06 Going to assign this one to you this week, there's another PR in language-intelligence-service related to it (#23). Plaky task associated with this is titled "Data Ingestion Service Integration - Integrate with API Gateway and Synapse", be sure to assign yourself as the engineer there and post your review comment on the task as well. If you need any help, let me know!

jrb00013 · 2026-03-22T05:27:59Z

@SeanSan06 @austinm2h35-sketch is busy helping me this week, give this to someone else. We had 3 more QAs join, we need to get them going and utilize them. Todd and Jeremy Chaves need help. @Todd838 @JC5467

austinm2h35-sketch · 2026-03-22T14:06:16Z

@jrb00013 Gotcha, thanks for the heads up! @JC5467 let this be your debut, see my comment above for procedural instruction. If you need help, let me know!

…ckerfiles Normalize load-k8s-env.sh, docker-entrypoint.sh, prisma-baseline.sh (and ops/k8s copies) to Unix LF. Drop RUN sed CRLF fixes from service Dockerfiles; chmod-only after COPY. Add .gitattributes eol=lf for shared/scripts and ops/k8s *.sh. Made-with: Cursor

…utes Made-with: Cursor

…RLF sed removed) Made-with: Cursor

jrb00013 · 2026-03-30T17:53:38Z

@thatsnotrlght and you are working together on this right

austinm2h35-sketch · 2026-03-30T19:04:40Z

@JC5467 Checking in on this, if you're stuck just reach out and I'll get you some help

austinm2h35-sketch · 2026-04-04T15:09:32Z

@Yoloholoknow I'm going to have you take over this testing, if you could. I'll tag you in the related one as well. If you need help just let me know. Thanks!

Yoloholoknow · 2026-04-04T17:49:39Z

@Patrik-Corri please resolve the conflicts before me and the team start reviewing and testing

jrb00013 · 2026-04-07T01:22:50Z

@Patrik-Corri use ai-conflict.py brother

…olution

Patrik-Corri added 9 commits March 7, 2026 20:40

fix: update Docker build for shared-utils across all backend services

d38a315

Refactoring scripts/dockerfiles

c60e326

Revert "Refactoring scripts/dockerfiles"

9e6499d

This reverts commit c60e326.

Merge branch 'main' of https://github.com/Team-Deepiri/deepiri-platform…

d0742a3

… into patrik_corri-dev

merge: incorporate main into patrik_corri-dev, keep realtime-gateway …

c0437c9

…submodule

data ingestion service

9ec47e1

fixed files

3b7798d

New Data Ingestion Service

fba31c3

feat: update language intelligence submodule reference

a98efcf

jrb00013 requested changes Mar 21, 2026

View reviewed changes

Patrik-Corri added 3 commits March 22, 2026 12:54

chore: enforce LF for shared/scripts and ops/k8s shell via .gitattrib…

04ef724

…utes Made-with: Cursor

chore: bump auth-service and external-bridge submodules (Dockerfile C…

6ae5db9

…RLF sed removed) Made-with: Cursor

jrb00013 changed the title ~~Data Ingestion Service~~ Implement Redis-based data ingestion service with event handlers and Synapse integration Apr 1, 2026

Add ingestion event types to shared-utils and update submodule refs

a2497d5

Patrik-Corri added 7 commits April 7, 2026 14:33

Update language-intelligence-service submodule ref after conflict res…

d2102b4

…olution

update api-gateway submodule pointer

91d5a3d

update auth-service submodule pointer

7869b5b

update external-bridge-service submodule pointer

0eb74bf

merge origin/dev into patrik_corri-dev

7ad5b88

remove shared-utils, clean submodule dirty state

737a38d

clean dirty submodules

9b89c21

merge origin/dev round 2 - resolve submodule conflicts

3b6190b

Conversation

Patrik-Corri commented Mar 20, 2026 • edited by JC5467 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Related

Testing

Important Notes (Optional)

Workflow Checklist (Required)

Review Requests

Uh oh!

jrb00013 Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

jrb00013 Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

jrb00013 Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

jrb00013 Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

Patrik-Corri Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

jrb00013 Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

jrb00013 Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

austinm2h35-sketch commented Mar 21, 2026

Uh oh!

jrb00013 commented Mar 22, 2026

Uh oh!

austinm2h35-sketch commented Mar 22, 2026

Uh oh!

jrb00013 commented Mar 30, 2026

Uh oh!

austinm2h35-sketch commented Mar 30, 2026

Uh oh!

austinm2h35-sketch commented Apr 4, 2026

Uh oh!

Yoloholoknow commented Apr 4, 2026

Uh oh!

jrb00013 commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Patrik-Corri commented Mar 20, 2026 •

edited by JC5467

Loading