Add semantic enrichment support for OpenSearch sink index creation by kkondaka · Pull Request #6771 · opensearch-project/data-prepper

kkondaka · 2026-04-20T23:04:56Z

Description

This PR adds support for creating OpenSearch indices with semantic enrichment via the AWS OpenSearch Service
and AOSS (OpenSearch Serverless) control plane APIs. When configured, Data Prepper will create indices with
semantic enrichment-enabled field mappings before the normal index setup, allowing OpenSearch to automatically
generate vector embeddings for specified text fields.

Problem
Currently, Data Prepper's OpenSearch sink creates indices using the standard OpenSearch REST API. However,
enabling
⧉ semantic enrichment https://docs.aws.amazon.com/opensearch-service/latest/developerguide/semantic-search.html
on index fields requires using the AWS control plane APIs (es:CreateIndex for managed domains,
OpenSearchServerless.CreateIndex for AOSS), which are not accessible through the standard OpenSearch client.
Users had to manually pre-create indices with semantic enrichment before running Data Prepper pipelines.

Solution
Introduced a new semantic_enrichment configuration block under aws settings that allows users to specify which
fields should have semantic enrichment enabled:

sink:
  - opensearch:
      aws:
        sts_role_arn: "arn:aws:iam::123456789012:role/MyRole"
        region: "us-west-2"
        semantic_enrichment:
          fields: ["title", "description"]
          language: "english"  # optional, defaults to "english"

How it works

During sink initialization, if semantic_enrichment.fields is configured and AWS SigV4 auth is enabled, the sink creates a SemanticEnrichmentIndexCreator
The creator auto-detects whether the target is a managed domain or serverless collection from the host URL
It builds an index schema with semantic_enrichment: { status: ENABLED, language_options: } on each specified field
The request is SigV4-signed and sent to the appropriate AWS control plane endpoint
If the index already exists (ConflictException / ResourceAlreadyExistsException), creation is silently skipped
Normal index setup via indexManager.setupIndex() proceeds afterward

Issues Resolved

Resolves #[Issue number to be closed when this PR is merged]

Check List

[ X] New functionality includes testing.
New functionality has a documentation issue. Please link to it in this PR.
- New functionality has javadoc added
[X ] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2026-04-20T23:18:11Z

✅ License Header Check Passed

All newly added files have proper license headers. Great work! 🎉

dlvenable · 2026-04-21T02:43:24Z

+        if (!hostname.contains(".aoss.")) {
+            throw new IllegalArgumentException(
+                    "Host does not appear to be an AOSS endpoint: " + hostname +
+                            ". Semantic enrichment via AOSS control plane requires a serverless collection.");


"control plane"

Also, "Serverless" should be capitalized.

dlvenable · 2026-04-21T02:43:51Z

+        // or vpc-{domain-name}-{random}.{region}.es.amazonaws.com
+        if (!hostname.contains(".es.amazonaws.com")) {
+            throw new IllegalArgumentException(
+                    "Host does not appear to be a managed OpenSearch domain: " + hostname +


"managed OpenSearch domain" -> "Amazon OpenSearch Service domain"

pyek-bot · 2026-04-22T18:21:04Z

+    static final String DEFAULT_LANGUAGE = "english";
+
+    @JsonProperty("fields")
+    private List<String> fields;


Each field can have it's own language. At the moment 2 options are present, multilingual and english.

pyek-bot · 2026-04-22T18:22:14Z

+    public void createIndex(final String indexName, final SemanticEnrichmentConfig semanticConfig) throws IOException {
+        if (serverless) {
+            createServerlessIndex(indexName, semanticConfig);
+        } else {


nit: else is redundant, can do an early return

pyek-bot · 2026-04-22T18:27:34Z

As @dlvenable mentioned, custom endpoints could cause an issue. Should the user have control to specify if it is an AOSS endpoint or AOS endpoint? This can resolve the issue.

Additionally, i'm not sure if silently failing is the right way. If the index fails to create, should it fail fast, so user can correct it. It can lead to unknown issues down the line if the mapping is not right. Maybe the index already exists and user can just provide it or we can check if index exists, if the mapping matches the one requested in the yaml.

Zhangxunmt · 2026-04-22T18:46:45Z

+    public String getLanguage() {
+        return language;
+    }


From the ASE doc, the language_options is set at each field in the create-index API so it can be different for different fiedls. But this config can only apply 1 language for all the fields. https://docs.aws.amazon.com/opensearch-service/latest/developerguide/opensearch-semantic-enrichment.html.
So is it better to define a LanguageConfig that includes both field and language, and use List instead of List fields?

@Zhangxunmt Maybe I should remove language option for now? What do you think?

Do you mean only support English as default?
I think defining a List of Map should work easily as below.

@JsonProperty("semantic_fields") private List<Map<String, String>> semanticFields

In the config

semantic_fields: - title_semantic: "english" - product_description: "multilingual"

Zhangxunmt · 2026-04-22T19:01:10Z

+            fieldMapping.put("semantic_enrichment", semanticEnrichment);
+            properties.put(field, fieldMapping);
+        }
+        return Map.of("mappings", Map.of("properties", properties));


I think this cannot support what's in the official API example that allows one field can be multilingual and another can be english, as mentioned above.

dlvenable · 2026-04-29T15:22:15Z

+    @JsonProperty("language")
+    private String language = DEFAULT_LANGUAGE;
+
+    @JsonProperty("collection_name")


I think we should move these new configurations into the general aws: block. We may have other features in the future that rely directly on AOS/AOSS. They would hit the same potential issues of custom domain names. Those features would then have duplicate configurations.

For clarity - just the domain_name and collection_name configurations would go into aws:. The others fit in this block.

@dlvenable aws: block has serverless: option, it doesn't make sense to put both domain_name and collection_name because if serverless: is true only collection_name makes sense, similarly when serverless: is false, only domain_name makes sense. I think we need some generic name

I propose generic resource_name: in aws: block

That name works for me.

pyek-bot · 2026-05-01T01:00:14Z

+public enum SemanticEnrichmentLanguage {
+    ENGLISH("english"),
+    MULTILINGUAL("multilingual");


Thanks for making the change! This does raise a question of what happens if additional language modes are supported.

Does it default to english?

yeah it would need to add the additional languages in the enum if new languages are supported in the future. It's not that flexible but using enum would avoid typo in the config.

Signed-off-by: Kondaka <krishkdk@amazon.com>

…dpoint for creating semantic enrichment enabled indexes Signed-off-by: Kondaka <krishkdk@amazon.com>

Signed-off-by: Kondaka <krishkdk@amazon.com>

…eparate class Signed-off-by: Kondaka <krishkdk@amazon.com>

dlvenable

@kkondaka ,

Overall this is good, but please update the logging terms to avoid "control plane" and "AWS OpenSearch" per my previous review.

Use "Amazon OpenSearch Serverless APIs" instead of "serverless control plane".
Use "Amazon OpenSearch Service APIs" instead of "managed domain control plane"
Use "Amazon OpenSearchService instead of "AWS OpenSearch"

For some reason I can't post comments on lines right now, but there are three occurrences I found.

Also the license header checks are still failing. See the comment it auto-created.

Signed-off-by: Kondaka <krishkdk@amazon.com>

kkondaka requested review from KarstenSchnitter, dinujoh, dlvenable, engechas, graytaylor0, oeyh, san81, sb2k16 and srikanthjg as code owners April 20, 2026 23:04

kkondaka force-pushed the ossink-ase branch from 130b842 to 5271f8f Compare April 20, 2026 23:17

dlvenable requested changes Apr 21, 2026

View reviewed changes

pyek-bot reviewed Apr 22, 2026

View reviewed changes

Zhangxunmt reviewed Apr 22, 2026

View reviewed changes

graytaylor0 previously approved these changes Apr 28, 2026

View reviewed changes

dlvenable requested changes Apr 29, 2026

View reviewed changes

kkondaka dismissed graytaylor0’s stale review via 8b5e24e April 30, 2026 05:55

kkondaka requested a review from divbok as a code owner April 30, 2026 05:55

pyek-bot reviewed May 1, 2026

View reviewed changes

kkondaka added 4 commits May 5, 2026 12:49

Add semantic enrichment support for OpenSearch sink index creation

2a754ae

Signed-off-by: Kondaka <krishkdk@amazon.com>

Fixed license headers

cd720b8

Signed-off-by: Kondaka <krishkdk@amazon.com>

Addressed review comments. Modified to use AWS SDK instead of http en…

62e0050

…dpoint for creating semantic enrichment enabled indexes Signed-off-by: Kondaka <krishkdk@amazon.com>

Addressed review comments

27c3754

Signed-off-by: Kondaka <krishkdk@amazon.com>

kkondaka force-pushed the ossink-ase branch from 8b5e24e to 27c3754 Compare May 5, 2026 19:49

Fixed failing end2end tests by creating a SemanticFieldMapping as a s…

20a8786

…eparate class Signed-off-by: Kondaka <krishkdk@amazon.com>

dlvenable requested changes May 6, 2026

View reviewed changes

kkondaka added 2 commits May 6, 2026 09:42

Fixed failing license checks. Addressed comments

2ec6226

Signed-off-by: Kondaka <krishkdk@amazon.com>

Fixed failing license checks. Addressed comments

7cecc72

Signed-off-by: Kondaka <krishkdk@amazon.com>

Conversation

kkondaka commented Apr 20, 2026

Description

Issues Resolved

Check List

Uh oh!

github-actions Bot commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ License Header Check Passed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pyek-bot commented Apr 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Zhangxunmt Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

github-actions Bot commented Apr 20, 2026 •

edited

Loading

Zhangxunmt Apr 29, 2026 •

edited

Loading