refactor: migrate elasticsearch to opensearch

[Junjiequan]

Description

THis PR migrates the search integration from Elasticsearch to OpenSearch.

Changes

Dependencies

• Removed @elastic/elasticsearch and @nestjs/elasticsearch
• Added @opensearch-project/opensearch ^3.5.1 JavaScript client compatible with opensearchproject/opensearch:3.5.0

Code Changes

• Renamed all references from Elasticsearch to OpenSearch

Configuration

• Removed hardcoded index settings and mappings. Configuration is now loaded from opensearchConfig.json if present; otherwise defaults from opensearchConfig.example.json are used

Search Integration

• Added opensearchQuery and opensearchFacet, used when OpenSearch is enabled and contains data
• New OpenSearch integration supports only $text queries, performing partial search on description and datasetName
• Restricted OpenSearch endpoints to admin users only. Search functionality is handled internally through the OpenSearch service

Data Synchronization

• Dataset synchronization from MongoDB to OpenSearch excludes scientificMetadata, history, and datasetlifecycle

Notes:

• opensearchQuery extracts the $text query, searches OpenSearch, retrieves dataset IDs up to the provided limit (default: 1000), and then applies additional filtering using MongoDB queries

• opensearchFacet extracts the $text query, searches OpenSearch, and retrieves results up to the index setting max_result_window. The example configuration sets this to 2000000, which may impact performance depending on server capacity and dataset volume

If performance issues arise with large result sets, alternative approaches such as pagination, scroll queries, or other optimizations can be considered as future improvements

Tests included

• Included for each change/fix?
• Passing?

Documentation

• swagger documentation updated (required for API changes)
• official documentation updated

official documentation info

[emigun]

I'm confused by the diff. It says a bunch of files are changed in folder elastic-search but the folder doesn't exist anymore when checking out this branch. When I search for lines like these, they don't exist anywhere either 🤔

[Junjiequan]

The folder is now renamed to opensearch

[emigun]

Yes I know. But the diff was incorrect. Now it says outdated and the problem is gone...

[minottic]

thanks a lot! I am not sure I see where this is used in the v4 datasets, isn't it only applied to v3 datasets?

I also don't fully understand the pletora of docker composes, why so many?

[Junjiequan]

@minottic
Opensearch was previously used by datasets v3 only so this PR only included changes for datasets v3. For V4 integration with opensearch I think its better to have new PR for that, I think this PR is too big already.
Docker composes needs to be cleaned up. I know docker-compose.api.yaml and docker-compose.local.yaml is in use but not sure about the others

[minottic]

Opensearch was previously used by datasets v3 only so this PR only included changes for datasets v3

ah ok, thanks. I was a bit puzzled because the tests are modified for the v4 controllers, but I did not see changes in the code that made me think opensearch was applied to v4. I agree v4 changes can come later if so

[minottic]

thanks a lot for the feature and for the patience!