Skip to content

Add TIPSv2 Segmentation/Depth/Normal DPT Pytorch#22

Open
washingtonsk8 wants to merge 1 commit intogoogle-deepmind:mainfrom
washingtonsk8:main
Open

Add TIPSv2 Segmentation/Depth/Normal DPT Pytorch#22
washingtonsk8 wants to merge 1 commit intogoogle-deepmind:mainfrom
washingtonsk8:main

Conversation

@washingtonsk8
Copy link
Copy Markdown
Contributor

Summary

This PR introduces a new Colab-ready Jupyter notebook: TIPSv2_Segmentation_Depth_Normal_DPT_Pytorch.ipynb. This notebook demonstrates how to use TIPSv2 models for multiple dense prediction tasks—specifically Semantic Segmentation, Depth Estimation, and Surface Normals Prediction—using PyTorch and DPT (Dense Prediction Transformer) decoders.

Key Features Demonstrated in the Notebook

  • Multi-Task Inference: Runs three different dense prediction tasks on the same input image using specialized DPT heads.
  • Dynamic Model Selection: Supports loading various TIPSv2 variants (B, L, So, g) with appropriate handling of FFN layers (e.g., SwiGLU for g) and intermediate feature dimensions.
  • PyTorch Compatibility: Uses PyTorch implementations for the DPT decoders, making it easy for external users to integrate into their PyTorch workflows.
  • Dataset & Checkpoint Automation: Automates the download of required TIPSv2 checkpoints, DPT task-specific weights, and a sample from the NYU dataset for immediate testing.

Notebook Structure

  1. Setup: Installs dependencies and clones the tips repository.
  2. Download: Fetches model checkpoints and sample data.
  3. Configuration: Sets up model variants and paths.
  4. Model Definition & Loading: Imports DPT decoders and loads weights.
  5. Inference & Visualization: Runs the model on a sample NYU image and plots Input, Segmentation, Depth, and Surface Normals side-by-side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant