Cookify: Recipe Extraction from Cooking Videos

Cookify is a Python-based tool that extracts structured recipe information from cooking videos using computer vision, speech recognition, and natural language processing to identify ingredients, tools, cooking steps, and other recipe components.

📋 Table of Contents

• Features
• Recent Improvements
• System Requirements
• Quick Start
• Installation
• Usage
• Web Interface
• Output Format
• Project Structure
• Development
• Contributing
• License
• FAQ

✨ Features

• 🎥 Extract recipe information from cooking videos with ingredient quantities and units
• 🔧 Identify cooking tools used in the video
• 📝 Extract step-by-step instructions with timestamps and cooking actions
• 🍳 Recognize cooking techniques and temperatures
• 📤 Generate structured JSON output of complete recipes
• 🌐 Modern web interface for easy video upload and result visualization
• 🛡️ Robust error handling and graceful degradation
• 🔄 Multiple model fallback strategies for better reliability
• 📱 Responsive design that works on desktop and mobile devices
• ⬇️ Drag & drop upload for easy video processing

🚀 Recent Improvements

• Fixed directory structure: Eliminated nested cookify directory
• Enhanced error handling: Comprehensive error handling throughout the pipeline
• Improved model loading: Robust model loading with fallback strategies
• Better dependency management: Updated dependencies for Python 3.12 compatibility
• Working examples: Added functional example scripts
• Graceful degradation: System continues working even if optional dependencies are missing

💻 System Requirements

• Python 3.8 or higher (Python 3.12 supported)
• FFmpeg (for video and audio processing)
• CUDA-compatible GPU recommended for faster processing (but not required)

🚀 Quick Start

Cookify offers two ways to process videos:

Command Line Interface

# Clone and navigate to the project
git clone https://github.com/kapeleshh/cookify2.git
cd cookify2

# Create and activate virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -e .

# Download models
python -m src.utils.model_downloader

# Process a video
cookify path/to/cooking/video.mp4

Web Interface (Recommended)

# Clone and navigate to the project
git clone https://github.com/kapeleshh/cookify2.git
cd cookify2

# Create and activate virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -e .

# Download models
python -m src.utils.model_downloader

# Start the web server
python src/ui/app.py

Then open your browser to http://localhost:5000

📥 Installation

1. Clone the repository

git clone https://github.com/kapeleshh/cookify2.git
cd cookify2

2. Create a virtual environment (recommended)

Virtual environments help isolate project dependencies, preventing conflicts between different Python projects on your system.

Why use a virtual environment?

• Isolates project dependencies from system Python packages
• Ensures consistent environment across different machines
• Makes it easy to specify and install exact dependency versions
• Simplifies package management and project sharing

Creating a virtual environment

On Windows:

# Create a virtual environment named "venv"
python -m venv venv

# Activate the virtual environment
venv\Scripts\activate

# Your terminal prompt should change, indicating the environment is active
# (venv) C:\Users\username\cookify2>

On macOS/Linux:

# Create a virtual environment named "venv"
python3 -m venv venv

# Activate the virtual environment
source venv/bin/activate

# Your terminal prompt should change, indicating the environment is active
# (venv) username@hostname:~/cookify2$

Using other virtual environment tools

With Anaconda/Miniconda:

# Create a conda environment
conda create -n cookify python=3.10

# Activate the environment
conda activate cookify

With Pipenv:

# Install pipenv if you don't have it
pip install pipenv

# Create environment and install dependencies
pipenv install

Deactivating the virtual environment

When you're done working with Cookify, you can deactivate the virtual environment:

# For venv
deactivate

# For conda
conda deactivate

3. Install the package and dependencies

pip install -e .

This command installs the package in development mode with all its dependencies. Here's what it does:

• The -e flag stands for "editable" mode, which creates a link to the source code
• The . refers to the current directory (where setup.py is located)
• It installs all dependencies listed in the setup.py file
• Changes to the source code take effect immediately without reinstalling

Alternative: Using requirements.txt

You can also install dependencies using the requirements file:

pip install -r requirements.txt

What's the difference?

• pip install -e . installs the current project as an editable package plus all dependencies
• pip install -r requirements.txt only installs the dependencies listed in the requirements file
• Using -e . is recommended for development as it allows the cookify command to work from anywhere
• The requirements file is useful for deployment or when you don't need to modify the code

The dependencies include:

• OpenCV and FFmpeg for video processing
• PyTorch and YOLOv8 for object detection
• EasyOCR for text recognition
• Whisper for speech-to-text
• spaCy for NLP

4. Download pre-trained models

python -m src.utils.model_downloader

🛠️ Usage

Basic Usage

cookify path/to/cooking/video.mp4

This will process the video and save the extracted recipe as recipe.json in the current directory.

Advanced Options

cookify path/to/cooking/video.mp4 --output custom_output.json --verbose

For more options:

cookify --help

🌐 Web Interface

Cookify includes a modern web interface for easy video upload and recipe extraction.

Starting the Web Server

1. Ensure you've completed the installation steps above (virtual environment, dependencies)

2. Activate the virtual environment (if not already active):

   source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Start the Flask web server from the project root:

   python src/ui/app.py

   You should see output like:

   Initializing pipeline
   Pipeline initialized and optimized
   * Running on http://127.0.0.1:5000
   * Running on http://0.0.0.0:5000
   Press CTRL+C to quit
   

4. Open your web browser and navigate to:

   • http://127.0.0.1:5000 (localhost)
   • http://0.0.0.0:5000 (network access)

Using the Web Interface

Step 1: Upload a Video

• Drag and drop a cooking video onto the upload area, or
• Click "Browse Files" to select a video file
• Supported formats: MP4, AVI, MOV, MKV, WEBM (max 50MB)
• Wait for the upload to complete

Step 2: Processing

• The system will automatically process your video after upload
• A progress indicator shows the processing status
• Processing time depends on video length (typically 1-5 minutes)
• You'll see updates in real-time

Step 3: View Results

• Once complete, you'll be redirected to the results page
• The results page displays:
  • Recipe title and metadata (servings, total time)
  • Ingredients list with quantities and units
  • Step-by-step cooking instructions with timestamps
  • Cooking tools used
  • Interactive video player with clickable timestamps for each step

Step 4: Download or Share

• Click "Download Recipe (JSON)" to save the structured recipe data
• Use "Upload Another Video" to process additional videos
• Share the results page URL to share with others (while the server is running)

Web Interface Features

• Modern UI: Clean, responsive design built with Bootstrap 5
• Drag & Drop Upload: Easy video file upload with visual feedback
• Real-time Progress: Live progress updates during video processing
• Interactive Results: Click on cooking steps to jump to specific video timestamps
• Mobile Friendly: Responsive design works on desktop and mobile devices
• Error Handling: Clear error messages and graceful failure handling
• Upload Management: Automatic file organization with unique IDs
• JSON Export: Download recipe data in structured JSON format

Troubleshooting Web Interface

If you encounter issues starting the web server:

1. Check dependencies: Ensure Flask is installed:

   pip install flask

2. Verify Python path: Make sure you're running from the project root directory

3. Ensure models are downloaded: The pipeline requires model files. Run:

   python -m src.utils.model_downloader

4. Check port availability: If port 5000 is in use, modify the port in src/ui/app.py:

   app.run(debug=True, host='0.0.0.0', port=5001)  # Change port number

5. View logs: The terminal will show detailed logs of any errors during startup or processing

For more troubleshooting information, see the Troubleshooting Guide.

📊 Output Format

The extracted recipe is saved as a JSON file with the following structure:

{
  "title": "Recipe Title",
  "servings": "Number of servings",
  "ingredients": [
    {"name": "ingredient name", "qty": "quantity", "unit": "measurement unit"}
  ],
  "tools": ["tool1", "tool2"],
  "steps": [
    {
      "idx": "step number",
      "start": "timestamp start",
      "end": "timestamp end",
      "action": "cooking action",
      "objects": ["ingredients/tools involved"],
      "details": "additional instructions",
      "temp": "temperature (optional)",
      "duration": "cooking duration (optional)"
    }
  ]
}

📁 Project Structure

cookify2/
├── data/               # Data directory for input/output files
│   ├── input/          # Input videos
│   └── output/         # Output recipes and processed data
├── models/             # Pre-trained models
├── src/                # Source code
│   ├── preprocessing/  # Video preprocessing
│   ├── frame_analysis/ # Frame analysis (object detection, OCR)
│   ├── audio_analysis/ # Audio transcription and NLP
│   ├── integration/    # Multimodal integration
│   ├── recipe_extraction/ # Recipe structure extraction
│   ├── output_formatting/ # Output formatting
│   ├── utils/          # Utility functions
│   │   ├── config_loader.py  # Configuration management
│   │   ├── logger.py         # Enhanced logging system
│   │   ├── model_downloader.py # Model management
│   │   └── performance_optimizer.py # Performance optimization
│   ├── ui/             # Web interface
│   │   ├── app.py      # Flask web application
│   │   ├── templates/  # HTML templates
│   │   ├── uploads/    # Uploaded video files
│   │   └── results/    # Processing results
│   └── pipeline.py     # Main processing pipeline
├── tests/              # Unit tests
├── documentation/      # Project documentation
├── examples/           # Example scripts
├── main.py             # Main entry point (CLI)
├── requirements.txt    # Dependencies
├── requirements-dev.txt # Development dependencies
├── config.yaml         # Configuration file
└── setup.py            # Setup script

Key Files

• main.py - Command line interface entry point
• src/pipeline.py - Core processing pipeline for recipe extraction
• src/ui/app.py - Flask web application
• src/utils/config_loader.py - Configuration management
• src/utils/model_downloader.py - Model downloading and management

For a more detailed technical overview, see the Architecture Documentation.

👨‍💻 Development

Setting Up a Development Environment

# Clone the repository
git clone https://github.com/kapeleshh/cookify2.git
cd cookify2

# Create and activate a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install development dependencies
pip install -e .
pip install -r requirements-dev.txt

Running Tests

# Run all tests
pytest tests/

# Run tests with coverage report
pytest tests/ --cov=src

# Run specific test file
pytest tests/test_phase1.py

Project Documentation

The documentation/ directory contains detailed information about the project:

• Documentation Index
• Architecture Overview
• Preprocessing Phase
• Frame Analysis Phase
• Audio Analysis Phase
• Multimodal Integration
• Recipe Extraction
• Output Formatting
• Future Directions

Machine Learning Models

Cookify uses several pre-trained machine learning models:

• Object Detection: YOLOv8 for identifying food items and cooking tools
• Text Recognition: EasyOCR for recognizing text in video frames
• Speech Recognition: Whisper for transcribing audio to text
• Action Recognition: Custom model for identifying cooking actions

For more information on how the models work together, see the Multimodal Integration documentation.

🤝 Contributing

We welcome contributions to Cookify! Here's how to get started:

1. Fork the repository
2. Create a feature branch:

   git checkout -b feature/your-feature-name

3. Make your changes and commit them:

   git commit -m "Add some feature"

4. Push to the branch:

   git push origin feature/your-feature-name

5. Open a pull request

Coding Standards

• Follow PEP 8 style guidelines
• Write docstrings for all functions, classes, and modules
• Include type hints
• Ensure test coverage for new features

📄 License

MIT License

❓ FAQ

Does Cookify work with all cooking videos?

Cookify works best with clear, well-lit cooking videos that feature close-ups of ingredients and cooking actions. While it can handle a variety of formats, professional cooking videos typically yield the best results.

Do I need a GPU to run Cookify?

No, but a CUDA-compatible GPU will significantly speed up processing, especially for longer videos.

How accurate is the recipe extraction?

Accuracy depends on several factors including video quality, clarity of actions, and audio quality. In optimal conditions, Cookify can achieve over 90% accuracy for ingredients and 80% for cooking steps.

Can Cookify handle videos in languages other than English?

Currently, Cookify works best with English-language videos, but it can recognize ingredients and tools in any language. Audio transcription is optimized for English but can work with other languages with reduced accuracy.

---

Made with ❤️ by the Cookify team

Report Bug • Request Feature