MyOpenCRE: allow for mapping CREs to a standard automatically

[northdpole]

MyOpenCRE is a nifty addition to the project that allows users to add their own mappings.
Using MyOpenCRE, users can download a CSV of all existing CREs, then map their own standard sections/subsections and finally re-upload for processing.

Mapping standards to CREs is a time consuming process. Instead we could use a bit of generative AI to do the mapping ourselves.
This feature would use a combination of AI techniques to derive highly accurate mappings between a CRE and the described information of the control of a Standard.

Stretch goal: If a mapping is not possible automatically or if a CRE does not exist for that mapping, the application should identify the controls for which a mapping could not be produced

Example Outcome: MyOpenCRE can map automatically the following standards:

• PCI-DSS
• DORA
• SOC2

Stretch goal example outcome: MyOpenCRE can partially map the OWASP AI Exchange and the top 10 for LLMs while identifying the controls that require human intervention or the release of new CREs.

[adityarra]

Hi @northdpole

I'm really excited about this project and eager to contribute to MyOpenCRE. My current undergrad project has to do with graphRAG models, which has given me insights into optimizing data retrieval and enhancing the efficiency of information processing. In addition to my work on graphRAG, I have a solid foundation in MERN(JavaScript, Next.js, React.js), Python/Flask and Kubernetes. I’m eager to collaborate with the team and help drive the project forward. I am submitting my proposal through GSOC.

Regards,
Adit

[FelixS90]

@northdpole @ashutosh-engineer Any updates on this one?

[ashutosh-engineer]

@northdpole @robvanderveer
I did a quick review of the codebase to understand how the CSV export/import flow for OpenCRE currently works. Here’s a short summary of what I found.

The CSV export/import functionality is fully implemented on the backend, including APIs, data processing, tests, and documentation. However, this functionality is not currently exposed through the web UI.

**************My Proposal for the issue ***************************
I explored this problem further and one possible approach is to first leverage the structured nature of security standards before introducing AI. Since controls follow predictable domains and patterns, a metadata + graph-based mapping system could provide an explainable and scalable baseline.

This could later be extended with AI for edge cases where deterministic mapping is insufficient. I’m happy to draft a concrete proposal if this direction aligns with the project.

[ashutosh-engineer]

@northdpole @robvanderveer

Why This Rule-Based Plan is Better Than Adding AI

A rule-based approach fits OpenCRE better because it already has structured data and expert-validated graph mappings. Rules are explainable, auditable, low-cost, and reuse human expertise directly, unlike opaque AI embeddings.

While reviewing the OpenCRE codebase, I found that mapping a standard to CREs currently works as follows:

1.Users download all CREs using GET /rest/v1/cre_csv
2.They open the CSV in Excel or Google Sheets
3.For each control in their standard, they manually search for relevant CREs
4.Mappings are added using a strict format (StandardName|name, id, hyperlink)
5.The modified CSV is uploaded using POST /rest/v1/cre_csv_import

Although the backend APIs for CSV export/import are fully implemented, there is no frontend UI to support this workflow. Users must rely on curl or similar tools. Additionally, the CSV structure itself is complex (hierarchical CRE columns like CRE 0–CRE 5), making it easy to introduce formatting errors.

Why This Does Not Scale

1.This approach becomes impractical as standards grow:
2.Mapping 100+ controls can take multiple days
3.There is no way to search or browse CREs while mapping
4.Small CSV formatting mistakes can break imports
5.Each mapper starts from scratch with no guidance
6.Existing mappings are not reused to help map new standards

Key Insight

OpenCRE already stores rich and valuable data that could assist mappers:

1.Parent/child CRE hierarchies
2.Relationships between CREs
3.Mappings between CREs and multiple standards

This information exists in the database and graph layer but is not surfaced to users during the mapping process. As a result, expert knowledge that already exists in OpenCRE is not reused. In effect, OpenCRE has the data needed to help mappers—but no mechanism to expose it.

Proposed Direction: Rule-Based Mapping Assistance

I propose adding a rule-based mapping assistance layer that helps users map controls more efficiently while keeping humans fully in control.

The system would generate suggestions, not automatic mappings, using existing OpenCRE data:

1.Structured metadata describing what each CRE covers .
2.Graph relationships between CREs and previously mapped standards.
3.Cross-standard reuse, where mappings from similar standards inform new ones.
4.Curated terminology, such as synonym dictionaries and common control patterns.

This is intentionally not an AI-first approach. The data in OpenCRE is already structured, curated, and expert-driven. A deterministic system allows suggestions to be explainable, predictable, and easy to maintain.

Architectural Overview

1. Structured CRE Metadata

CREs currently have free-form tags that are inconsistently applied. Introducing optional structured metadata (stored as JSON) would allow CREs to be categorized in a consistent way, such as by:

1.Security domain (authentication, cryptography, logging, etc.)
2.Lifecycle phase (design, implementation, testing)
3.Technology layer (network, application, data)

This metadata would be optional and backward-compatible. Existing CREs would continue to work unchanged, while maintainers could incrementally enrich CREs with better structure.

2. Leveraging the Existing Graph

OpenCRE already maintains a graph (via NetworkX) that captures relationships between CREs and their mappings to standards. What’s missing are higher-level queries that answer questions like:

1.“Which CREs are commonly used for authentication controls?”
2.“Which CREs were mapped by similar controls in other standards?”

This allows OpenCRE to reuse expert mapping work that has already been done. If multiple standards mapped similar controls to the same CRE, that becomes a strong signal worth suggesting to future mappers.

3. Deterministic Mapping Suggestions
   A lightweight mapping engine can combine multiple signals:

1.Keyword matching on control text
2.Synonym expansion for security terminology
3.Graph-based reuse of existing mappings
4.Recognition of common control patterns.

Each suggestion would include an explanation describing why it was suggested, such as:

1.“Matches authentication-related keywords”
2.“Used by similar controls in NIST 800-53”
3.“Part of the Access Control CRE hierarchy”
4.This keeps the system transparent and trustworthy.

4. Frontend to Expose Existing Capabilities

Although OpenCRE has strong backend support for CSV workflows, there is currently no UI to support mapping. A simple frontend page could:

1.Allow users to upload a file containing standard controls.
2.Display suggested CRE mappings with explanations.
3.Let users accept, reject, or manually search CREs

Export the final mappings back to CSV for import

1.This would make existing backend functionality accessible to non-technical users and significantly reduce friction.

Why a Rule-Based Approach Fits OpenCRE

1.This approach aligns well with OpenCRE’s goals and data model:
2.It is explainable—users understand why suggestions are made
3.It is deterministic—no hidden inference or model behavior
4.It is maintainable—no model training or external dependencies
5.It reuses expert work already present in the graph
6.It scales naturally as more standards are mapped

AI-based approaches can be explored later if needed, but starting with a deterministic system provides a strong, auditable foundation.

If this proposal turns out to be incomplete, I have also looked into possible AI-based approaches that could be Integrated. I’m open to feedback and iteration.

If you think this is a good direction, I’d appreciate it if you could assign this issue to me.

[ashutosh-engineer]

@northdpole Can i start working on the Issue???

[Subh24ai]

Hi @northdpole,

I've now reviewed the codebase in detail and wanted to share a more technical perspective.
I can see that embeddings are already generated and stored via PromptHandler.generate_embeddings_for() and db.add_embedding() — the Embeddings table exists in SQLite with vectors stored per CRE and Standard node. The missing piece for issue #585 is a find_similar_cres() function that performs cosine similarity between a new Standard section's embedding and all existing CRE embeddings.

My proposed approach:

1. When a user uploads a CSV of a new standard (e.g. PCI-DSS), each row becomes a Standard object with name, section, description
2. Generate an embedding for each Standard section using the existing PromptHandler
3. Run cosine similarity against all stored CRE embeddings via a new find_similar_cres(embedding, top_k) function in db.py
4. Return top-k CREs as candidate mappings with confidence scores
5. Stretch goal: flag low-confidence matches (below a threshold) for human review

I have production experience building a similar hybrid retrieval system (FAISS + BM25 + cross-encoder reranking) serving 15,000+ queries at 87% retrieval precision. I'd like to start with a PR implementing find_similar_cres() as a focused first contribution.

Does this direction align with your vision for the feature?

[Subh24ai]

Hello @northdpole

After reading prompt_client.py, I can see get_id_of_most_similar_cre_paginated() already returns a similarity score alongside the CRE ID. The missing piece is a suggest_cre_mappings() function in cre_main.py that iterates over uploaded Standard entries, generates embeddings for each section, calls the existing similarity function, and splits results into high-confidence mappings vs. low-confidence ones flagged for human review -- directly addressing the stretch goal.

I'd like to implement this as my first PR. Would you prefer this as a backend-only function first, or should it include a MyOpenCRE UI endpoint from the start?

[Subh24ai]

Hello @northdpole

I've submitted PR #753 which implements the foundational suggest_cre_mappings() function for this feature. Happy to iterate based on feedback.