Skip to content

[full-ci] feat: ocisdev-533 graph#12108

Merged
2403905 merged 6 commits into
masterfrom
feat/OCISDEV-533-graph
May 15, 2026
Merged

[full-ci] feat: ocisdev-533 graph#12108
2403905 merged 6 commits into
masterfrom
feat/OCISDEV-533-graph

Conversation

@2403905
Copy link
Copy Markdown
Contributor

@2403905 2403905 commented Mar 12, 2026

Description

Provide the separated vault storage that could be MFA-protected

Implementation approach:

Provide the dedicated storage-users and graph service to care only about vault storage.

  • Provide the new vault user storage with a dedicated VaultStorageProviderID mounted to "/vault/users" and "/vault/projects".
  • Teach the storage registry to associate the vault storage with the dedicated storage-users service.
  • Modify the graph service to force using StorageProviderID in a vault-mode. Run in addition the graph API serves the /vault prefix.
  • Run the dedicated storage-users service pointed to the vault.

Related reva PR owncloud/reva#559

How to run in a Docker

UPD: 13.04.2026

  • build the dev docker image
  • goto ./deployments/examples/ocis_full
  • in a .env uncomment the line KEYCLOAK=:keycloak.yml and VAULT_STORAGE=:vault-storage.yml
  • run the docker compose IDM_ADMIN_PASSWORD=admin DEMO_USERS=true OCIS_DOCKER_TAG=dev OCIS_MFA_ENABLED=true docker compose up -d

How to run locally

UPD: 18.03.2026 - No extra graph service needed
ocis main

WEB_ASSET_CORE_PATH=/Users/roman/projects/owncloud-extra/web/dist \
OCIS_MFA_ENABLED=true \
PROXY_CREATE_VAULT_HOME=true \
GRAPH_ENABLE_VAULT_MODE=true \
IDM_ADMIN_PASSWORD=admin IDM_CREATE_DEMO_USERS=true PROXY_ENABLE_BASIC_AUTH=true OCIS_LOG_LEVEL=info ./ocis/bin/ocis server

In a Keycloak setup set to trueOCIS_MFA_ENABLED
WEB_ASSET_CORE_PATH={path to web}/web/dist \

Vault storage-users

OCIS_LOG_LEVEL=debug \
STORAGE_USERS_ENABLE_VAULT_MODE=true \
STORAGE_USERS_SERVICE_NAME=storage-users-vault \
STORAGE_USERS_GRPC_ADDR=0.0.0.0:9170 \
STORAGE_USERS_HTTP_ADDR=0.0.0.0:9168 \
STORAGE_USERS_DATA_SERVER_URL=http://localhost:9168/data \
STORAGE_USERS_DEBUG_ADDR=0.0.0.0:9169 \
STORAGE_USERS_OCIS_ROOT=${HOME}/.ocis/storage/users-vault \
STORAGE_USERS_EVENTS_CONSUMER_GROUP=vault-dcfs \
./ocis/bin/ocis storage-users server
curl 'https://localhost:9200/vault/graph/v1beta1/drives' \
-H 'Accept: application/json' \
--data-raw '{"name":"vault Space"}' \
--insecure  -uadmin:admin

curl 'https://localhost:9200/vault/graph/v1beta1/me/drives?%24orderby=name+asc&%24filter=driveType+eq+project' \
-H 'Accept: application/json' \
--insecure  -uadmin:admin

curl 'https://localhost:9200/vault/graph/v1beta1/me/drives?%24orderby=name+asc&%24filter=driveType+eq+personal' \
-H 'Accept: application/json' \
--insecure  -uadmin:admin

FS:

~/.ocis/storage
$ tree  users*
users
├── indexes
│   ├── by-group-id
│   ├── by-type
│   │   └── personal.mpk
│   └── by-user-id
│       └── a032f2bd-fa5c-430b-a163-2c19f54190d0.mpk
├── spaces
│   └── a0
│       └── 32f2bd-fa5c-430b-a163-2c19f54190d0
│           └── nodes
│               └── a0
│                   └── 32
│                       └── f2
│                           └── bd
│                               ├── -fa5c-430b-a163-2c19f54190d0
│                               ├── -fa5c-430b-a163-2c19f54190d0.mlock
│                               └── -fa5c-430b-a163-2c19f54190d0.mpk
└── uploads
users-vault
├── indexes
│   ├── by-group-id
│   ├── by-type
│   │   └── personal.mpk
│   └── by-user-id
│       └── a032f2bd-fa5c-430b-a163-2c19f54190d0.mpk
├── spaces
│   └── a0
│       └── 32f2bd-fa5c-430b-a163-2c19f54190d0
│           └── nodes
│               └── a0
│                   └── 32
│                       └── f2
│                           └── bd
│                               ├── -fa5c-430b-a163-2c19f54190d0
│                               ├── -fa5c-430b-a163-2c19f54190d0.mlock
│                               └── -fa5c-430b-a163-2c19f54190d0.mpk
└── uploads

@update-docs
Copy link
Copy Markdown

update-docs Bot commented Mar 12, 2026

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

@2403905 2403905 requested review from jvillafanez and kobergj March 12, 2026 11:15
@2403905 2403905 force-pushed the feat/OCISDEV-533-graph branch from 96ce268 to c79f474 Compare March 12, 2026 11:31
Comment thread services/graph/pkg/config/config.go Outdated
@jvillafanez
Copy link
Copy Markdown
Member

There are a couple of things that seems weird to me:

  • I understand that we can setup a storage-user service in "vault mode" so it's only accessible / usable with MFA, but why graph?
    • The graph service can just check if the request is under MFA to do anything with the vault.
  • Using mount ids in the configuration isn't user-friendly. If possible, I'd drop them, otherwise they MUST be documented, including where such id can be found and / or provide a command to get the id.
    • Note that it's easy to make typos and setup a mount id that doesn't exist. And people makes mistakes and might use any random id found anywhere, so it might point to who-knows-what.

@2403905
Copy link
Copy Markdown
Contributor Author

2403905 commented Mar 12, 2026

There are a couple of things that seems weird to me:

  • I understand that we can setup a storage-user service in "vault mode" so it's only accessible / usable with MFA, but why graph?

Maybe we can improve it and use only one graph. Now I use the second one for enforcing the vault storage and MFA for all graph endpoints

  • Note that it's easy to make typos and setup a mount id that doesn't exist. And people makes mistakes and might use any random id found anywhere, so it might point to who-knows-what.

Ideally, we could try to add one more storageprovider in a config and get rid of the dedicated storage-users service.

"services": map[string]interface{}{
"storageprovider": map[string]interface{}{
"driver": cfg.Driver,
"drivers": StorageProviderDrivers(cfg),
"mount_id": cfg.MountID,
"expose_data_server": cfg.ExposeDataServer,
"data_server_url": cfg.DataServerURL,
"upload_expiration": cfg.UploadExpiration,
"events": map[string]interface{}{
"nats_address": cfg.Events.Addr,
"nats_clusterid": cfg.Events.ClusterID,
"tls_insecure": cfg.Events.TLSInsecure,
"tls_root_ca_cert": cfg.Events.TLSRootCaCertPath,
"nats_enable_tls": cfg.Events.EnableTLS,
"nats_username": cfg.Events.AuthUsername,
"nats_password": cfg.Events.AuthPassword,
},
},
},
"interceptors": map[string]interface{}{

Thank you.

@jvillafanez
Copy link
Copy Markdown
Member

Maybe we can improve it and use only one graph. Now I use the second one for enforcing the vault storage and MFA for all graph endpoints

MFA needs to be enforced in the vault storage. Technically, graph shouldn't need to enforce MFA; it can make the request and the request will fail. The fact that we want graph to check for MFA is mostly for convenience, to avoid making a request that we know it will fail.

In addition, whether the request is under MFA or not is information that should be part of the request, and should be propagated as part of the request. This is very similar to what is done with telemetry. Graph shouldn't need a configuration flag to know if it can access to the vault or if MFA is active.

@2403905 2403905 changed the title Feat/ocisdev 533 graph [full-ci] feat: ocisdev-533 graph Mar 14, 2026
@2403905 2403905 requested review from Copilot and mklos-kw March 16, 2026 14:33
@2403905 2403905 force-pushed the feat/OCISDEV-533-graph branch from df07882 to a9fa2c1 Compare March 16, 2026 14:35
@2403905 2403905 marked this pull request as ready for review March 16, 2026 14:35
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the vendored reva dependency and introduces “vault mode” support across gateway, storage-users, proxy, and graph, including storage scoping for events and storage space selection.

Changes:

  • Bump github.com/owncloud/reva/v2 vendor version and adapt code to upstream changes (events, storage registry filtering, gateway client acquisition).
  • Add vault-mode plumbing: vault storage provider IDs/constants, vault-specific space provider config, and passing storage_id via CS3 opaque to target the vault storage.
  • Add configurable event consumer group and storage-aware filtering for decomposedfs postprocessing events; add MFA middleware integration for graph routes.

Reviewed changes

Copilot reviewed 18 out of 34 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
vendor/modules.txt Updates vendored module version for reva/v2.
vendor/github.com/owncloud/reva/v2/pkg/utils/utils.go Adds vault storage provider/space IDs.
vendor/github.com/owncloud/reva/v2/pkg/storage/utils/decomposedfs/options/options.go Adds consumer_group event option + default.
vendor/github.com/owncloud/reva/v2/pkg/storage/utils/decomposedfs/decomposedfs.go Uses configurable consumer group; filters events by storage.
vendor/github.com/owncloud/reva/v2/pkg/storage/registry/spaces/spaces.go Filters vault spaces unless explicitly requested; supports storage_id filter.
vendor/github.com/owncloud/reva/v2/pkg/storage/cache/createpersonalspace.go Removes create-personal-space cache implementation.
vendor/github.com/owncloud/reva/v2/pkg/storage/cache/createhome.go Removes create-home cache implementation.
vendor/github.com/owncloud/reva/v2/pkg/storage/cache/cache.go Removes create-home/create-personal-space cache APIs.
vendor/github.com/owncloud/reva/v2/pkg/events/postprocessing.go Adds ResourceID to postprocessing events.
vendor/github.com/owncloud/reva/v2/internal/http/.../shares/spaces.go Simplifies provider client creation via pool directly.
vendor/github.com/owncloud/reva/v2/internal/grpc/services/storageprovider/storageprovider.go Ensures root-info IDs get storage provider IDs filled.
vendor/github.com/owncloud/reva/v2/internal/grpc/services/gateway/storageprovidercache.go Adjusts cache keying to include storage_id (opaque).
vendor/github.com/owncloud/reva/v2/internal/grpc/services/gateway/storageprovider.go Forwards storage_id via opaque and provider selection; removes some caching wrappers.
vendor/github.com/owncloud/reva/v2/internal/grpc/services/gateway/gateway.go Removes create-personal-space cache wiring.
services/storage-users/pkg/revaconfig/drivers.go Passes consumer_group to reva storage config.
services/storage-users/pkg/config/defaults/defaultconfig.go Sets MountID to vault provider ID when vault mode enabled.
services/storage-users/pkg/config/config.go Adds enable_vault_mode and consumer_group config fields.
services/proxy/pkg/middleware/create_home.go Adds short-lived in-process TTL cache; creates regular + vault homes via storage_id.
services/proxy/pkg/config/defaults/defaultconfig.go Adds proxy policy route for /vault/graph/.
services/postprocessing/pkg/postprocessing/postprocessing.go Propagates ResourceID into emitted events.
services/policies/pkg/service/event/service.go Propagates ResourceID into emitted events.
services/graph/pkg/service/v0/service.go Adds RequireMFA middleware and vault-mode MFA routing behavior.
services/graph/pkg/service/v0/graph_test.go Updates tests to exercise router (ServeHTTP) instead of direct handler calls.
services/graph/pkg/service/v0/drives.go Forces vault storage selection via storage_id in opaque; removes inline MFA checks.
services/graph/pkg/service/v0/driveitems.go Adds vault-mode filtering for personal root children.
services/graph/pkg/middleware/mfa.go New middleware to enforce MFA.
services/graph/pkg/config/service.go Adds env/yaml tags for service name.
services/graph/pkg/config/config.go Adds enable_vault_mode config field.
services/gateway/pkg/revaconfig/config.go Adds a dedicated vault spaces provider definition.
services/gateway/pkg/config/defaults/defaultconfig.go Removes create-home cache defaults.
services/gateway/pkg/config/config.go Removes create-home cache configuration fields.
go.mod / go.sum Updates reva/v2 dependency version checksums.
.gitignore Ignores .agents/ directory.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread services/proxy/pkg/middleware/create_home.go Outdated
Comment thread services/graph/pkg/service/v0/driveitems.go Outdated
Comment thread services/graph/pkg/config/config.go Outdated
Comment thread services/graph/pkg/config/service.go Outdated
Comment thread services/storage-users/pkg/config/config.go Outdated
Comment thread services/storage-users/pkg/config/config.go Outdated
@2403905
Copy link
Copy Markdown
Contributor Author

2403905 commented Mar 17, 2026

Maybe we can improve it and use only one graph. Now I use the second one for enforcing the vault storage and MFA for all graph endpoints

MFA needs to be enforced in the vault storage. Technically, graph shouldn't need to enforce MFA; it can make the request and the request will fail. The fact that we want graph to check for MFA is mostly for convenience, to avoid making a request that we know it will fail.

In addition, whether the request is under MFA or not is information that should be part of the request, and should be propagated as part of the request. This is very similar to what is done with telemetry. Graph shouldn't need a configuration flag to know if it can access to the vault or if MFA is active.

That is a good point. @jvillafanez
The idea is to implement the Vault that doesn't require MFA dependencies.
The Vault-graph should ensure that requests go to Vault storage by requiring that some requests, like CreateStorageSpace and GetAllDrives, use the Vault storage ID explicitly.
So, the Vault graph can work without MFA, ensuring the drives' separation.
Then I assumed that the Graph is a good place to force MFA for all of the routes under the /vault with minimal effort.
The cons of this we still need to take care of the ocdav api MFA protection.

We can get rid of the extra vault-graph instance to provide an additional router that could be above the MFA

@2403905
Copy link
Copy Markdown
Contributor Author

2403905 commented Mar 17, 2026

@jvillafanez @mklos-kw I made a commit that makes the graph able to handle the vault prefix. no extra service needed
d456f68

@2403905 2403905 force-pushed the feat/OCISDEV-533-graph branch 2 times, most recently from b53fb87 to 5fc45db Compare March 18, 2026 08:53
@mmattel
Copy link
Copy Markdown
Contributor

mmattel commented Mar 18, 2026

The envvar description text says the following:
"GRAPH_ENABLE_VAULT_MODE" desc:"Enable vault mode for the graph service running in addition to the regular graph service. Requires running the storage-users-vault additional service."

This part confuses me: Requires running the storage-users-vault additional service.
We have an additional service (which I cant identify) or just an instance of the storage-users service with a different config? (a config example (or reference) should be added e.g. in the storage-users readme)

The envvar text should be precised (or the missing service added).

@2403905 2403905 force-pushed the feat/OCISDEV-533-graph branch 3 times, most recently from 8fcac29 to f9c326c Compare March 19, 2026 18:16
@2403905
Copy link
Copy Markdown
Contributor Author

2403905 commented Mar 20, 2026

The envvar description text says the following: "GRAPH_ENABLE_VAULT_MODE" desc:"Enable vault mode for the graph service running in addition to the regular graph service. Requires running the storage-users-vault additional service."

This part confuses me: Requires running the storage-users-vault additional service. We have an additional service (which I cant identify) or just an instance of the storage-users service with a different config? (a config example (or reference) should be added e.g. in the storage-users readme)

The envvar text should be precised (or the missing service added).

For now, the Vault mode requires running the storage-users as an additional service with specific configuration

@mmattel
Copy link
Copy Markdown
Contributor

mmattel commented Mar 20, 2026

For now, the Vault mode requires running the storage-users as an additional service with specific configuration

OK, makes sense. Please rewrite the envvar text accordingly because the current one does not match.

@2403905 2403905 force-pushed the feat/OCISDEV-533-graph branch from 85d7fdb to 702c7c7 Compare March 23, 2026 09:54
Comment thread services/collaboration/pkg/connector/contentconnector.go Outdated
Comment thread deployments/examples/ocis_full/vault-storage.yml Outdated
Comment thread deployments/examples/ocis_full/vault-storage.yml Outdated
Comment thread services/gateway/pkg/config/config.go Outdated
Comment thread services/graph/pkg/config/config.go Outdated
Comment thread services/graph/pkg/config/config.go Outdated
Comment thread services/graph/pkg/config/service.go Outdated
Comment thread services/proxy/pkg/config/config.go Outdated
Comment thread services/storage-users/pkg/config/config.go Outdated
Comment thread services/storage-users/pkg/config/config.go Outdated
Comment thread services/gateway/pkg/config/config.go
@2403905 2403905 force-pushed the feat/OCISDEV-533-graph branch 2 times, most recently from c82ad90 to 81399ea Compare May 5, 2026 16:36
Copy link
Copy Markdown
Contributor

@kobergj kobergj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just small point. Rest looks 👍

Comment thread services/graph/pkg/middleware/auth.go Outdated
@2403905 2403905 force-pushed the feat/OCISDEV-533-graph branch 3 times, most recently from 3d5a280 to ba38ca0 Compare May 11, 2026 12:33
Comment thread changelog/unreleased/enhancement-vault-storage.md
@2403905 2403905 requested a review from kobergj May 11, 2026 19:08
@2403905 2403905 force-pushed the feat/OCISDEV-533-graph branch from b95df4d to feb0c8e Compare May 14, 2026 20:30
2403905 and others added 6 commits May 15, 2026 15:46
feat: Separate the storage-users and graph to handle

feat: move the space create cache

configurate the vault srorage Postprocessing

hide the assignment of the MountID to VaultStorageProviderID behind a flag to avoid the ID mismatch

configurate the proxy to create the vault home space

update the filter

Make the graph able to handle vault prefix. no extra service needed

apply the vault filter to sharedbyme and sharedwithme

added vault-storage docker

use squashed reva

fix the finishUpload event

restict webdav copy/move from the vault

feat: provide the mfa to webdav and grpc

feat: refactoring: provide the mfa to webdav and grpc

mfa collaboration

fix GetRootDriveChildren and more

fix: send the MFA header in HTTP clients from the collaboration service

provide the default storage user vault configuration

update create_home middleware

Update deployments/examples/ocis_full/vault-storage.yml

Co-authored-by: Martin <github@diemattels.at>

Update services/gateway/pkg/config/config.go

Co-authored-by: Martin <github@diemattels.at>

Update services/graph/pkg/config/config.go

Co-authored-by: Martin <github@diemattels.at>

Update services/graph/pkg/config/config.go

Co-authored-by: Martin <github@diemattels.at>

Update services/storage-users/pkg/config/config.go

Co-authored-by: Martin <github@diemattels.at>

Update services/proxy/pkg/config/config.go

Co-authored-by: Martin <github@diemattels.at>

Update services/storage-users/pkg/config/config.go

Co-authored-by: Martin <github@diemattels.at>

Update deployments/examples/ocis_full/vault-storage.yml

Co-authored-by: Martin <github@diemattels.at>

bump reva

deduplicate the requireMFA middeware on vault routes. fix AppendPlainToOpaque.

added the changlog and comments
Co-authored-by: Martin <github@diemattels.at>
@2403905 2403905 force-pushed the feat/OCISDEV-533-graph branch from cd35203 to 33d9d96 Compare May 15, 2026 13:53
@2403905 2403905 merged commit 39e3d33 into master May 15, 2026
107 of 109 checks passed
@2403905 2403905 deleted the feat/OCISDEV-533-graph branch May 15, 2026 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants