Question about gene-to-image retrieval and prognosis heatmap scripts

[LeonardGreen]

Dear Multi-Embed team,

Thank you for sharing the code and this very interesting work. I have been trying to reproduce some of the interpretability analyses described in the paper, especially the gene-to-image retrieval analysis and the visualization of prognosis-related attention heatmaps. However, I could not find the corresponding scripts in the current GitHub repository.

Specifically, I am interested in:

1. The pipeline for projecting a gene signature, such as the EMT signature from EcoTyper, into the Multi-Embed space and retrieving pathology tiles based on cosine similarity.
2. The scripts used to generate WSI-level heatmaps for prognosis-related high-risk regions / attention scores.

Would it be possible to release these scripts, or provide some guidance on how to reproduce these analyses using the current codebase? Any example code or instructions would be greatly appreciated.

Thank you very much for your time and for making this resource available.

[Epoch1128]

Hi,

Thank you for your interest in Multi-Embed!

The protocol for gene-to-image retrieval generally follows these steps:

1. Obtain gene expression signatures from the EcoTyper platform or other sources (you can refer to this EMT example: epi_sig.csv).
2. Generate the molecular embedding for the expression signature using the Multi-Embed model pre-trained on the corresponding cancer type.
3. Extract the morphological embedding for each patch in a given slide using the same Multi-Embed model.
4. Calculate the cosine similarity between the patch-level morphological embeddings and the signature's molecular embedding.
5. Retrieve the top patches (e.g., top-1, 5, or 10) based on the highest similarity scores.

The source code for this specific pipeline is currently undergoing final cleanup, and we plan to release all application scripts in the near future.

For the scripts used to generate WSI-level heatmaps, please refer to the source code of CLAM and PORPOISE. We would like to express our gratitude to the Mahmood Lab for their outstanding contributions to the development of computational pathology toolboxes!