CADBUSI-Database

SQLite database manager that is designed to process breast ultrasound data from Mayo Clinic and store it in a structured format, making it easy to anonymize, manipulate, label, analyze, and prepare for ML research.

Setup

• Create a GCP AI Factory instance with a T4 GPU
• Clone repository: git clone https://github.com/Poofy1/CADBUSI-Database.git
• Go into dir: cd CADBUSI-Database
• Install requirements: pip install -r requirements.txt
• Create/Configure: ./config.py
• Obtain certificate ./src/dicom_downloader/CertEmulationCA.crt
• Obtain user privileges: gcloud auth application-default login --no-launch-browser

Configuration

All user parameters will be controlled from a ./config.py file, you will need to configure the following parameters:

CONFIG = {
    # Environment configuration
    "env": {
        "project_id": "your-project-id",
        "region": "us-central1",
        "topic_name": "dicom-processing-topic",
        "subscription_name": "dicom-processing-subscription",
        "my_service_account": "your-service-id",
        "service_account_identity": "your-service-account@your-project-id.iam.gserviceaccount.com"
    },
    
    # Cloud Run configuration
    "cloud_run": {
        "service": "pubsub-push-cloudrun",
        "version": "1.0",
        "ar": "your-artifact-registry",
        "ar_name": "pubsub-push-cloudrun",
        "target_tag": "us-central1-docker.pkg.dev/your-project-id/your-artifact-registry/pubsub-push-cloudrun:1.0",
        "vpc_shared": "your-shared-vpc-id",
        "vpc_name": "your-vpc-name"
    },
    
    # Storage configuration
    "storage": {
        "gcs_log": "gs://your-bucket-name/cloudbuild_log",
        "gcs_stage": "gs://your-bucket-name/cloudbuild_stage",
        "bucket_name": "your-bucket-name",
        "download_path": "Downloads",
    },
    
    "BUCKET": "your-bucket-name",
    "WINDIR": "D:/DATA/YOUR_PROJECT/",
    "DATABASE_DIR": "Databases/database_YYYY_MM_DD/", # Final location of the database.

    "LABELBOX_API_KEY": "your-labelbox-api-key",
    "PROJECT_ID": "your-labelbox-project-id",
    "LABELBOX_LABELS": "labelbox_data/",
    "TARGET_CASES": "/failed_cases.csv", # Directory of worst performing cases from training. Prepares these cases for instance labeling on Label Box.
    "VIDEO_SAMPLING": 0, # every nth frame, 0 turns off videos
    "DEBUG_DATA_RANGE": [], # Process a reduced set of dicom files (Ex: [0, 1000]).
}

Usage

The pipeline is operated through a single command-line interface in main.py, which provides several functions. For general purpose, you should perform these commands in this order:

python main.py --query [optional: --limit=N]
python main.py --deploy
python main.py --cleanup
python main.py --database

Querying Data

To query breast imaging data: python main.py --query [optional: --limit=N]

This will:

1. Run a query to retrieve breast imaging records
2. Filter and clean the radiology and pathology data
3. Create a final dataset for processing
4. Save results to query_data/endpoint_data.csv

Example with a limit:python main.py --query --limit=100

Query Diagram --query [optional: limit=N]

Downloading DICOM Files

The tool offers Cloud Run deployment for efficient DICOM downloads. Dicoms will appear in specified GCP bucket storage:

# Deploy the FastAPI service to Cloud Run and start dicom data download (REQUIRED)
python main.py --deploy

# Resend the download requests to the pre-deployed service (OPTIONAL)
python main.py --rerun 

# Clean up Cloud Run resources when finished (REQUIRED)
python main.py --cleanup

IMPORTANT: After python main.py --deploy finishes execution, that does not mean the data transfer is complete. The download requests have been sent to Cloud Run. Check the bucket storage to see when population is finished. Only then should you run python main.py --cleanup

Processing DICOM Files

To process the downloaded DICOM files into a complete database:

python main.py --database [optional: --skip-inpaint]

This will:

1. Generate encryption keys for safely anonymizing patient IDs
2. Deidentify DICOM files from CONFIG['storage']['download_path']
3. Process image files in the specified output directory in the destination bucket
4. [optional: --skip-inpaint] will skip the caliper removal process

Example:

python main.py --database

Data Pipeline

• CADBUSI-Database
• CADBUSI-Training