Skip to content

Subdomain Scanner with OpenRouter API ( Choose your AI model ) and Free Grok API . First Subdomain scanner with AI integration

License

Notifications You must be signed in to change notification settings

bidhata/SubGrab

Repository files navigation

SubGrab - Advanced Subdomain Enumeration Tool

Python | MIT License | Cross-Platform

πŸš€ Next-Generation Subdomain Discovery with AI-Powered Intelligence

SubGrab is a high-performance, multi-threaded subdomain enumeration tool designed for security researchers, penetration testers, and bug bounty hunters. Enhanced with OpenRouter AI and Grok AI integration, it combines traditional reconnaissance techniques with advanced artificial intelligence to uncover subdomains that other tools might miss.


🌟 Why Choose SubGrab?

SubGrab stands out with its robust feature set and AI-driven capabilities, making it an essential tool for comprehensive subdomain discovery:

  • πŸ€– AI-Powered Discovery: Leverages OpenRouter API with multiple AI models for intelligent subdomain generation.
  • πŸ”„ Multi-Source Enumeration: Combines 25+ discovery techniques for maximum coverage.
  • ⚑ High Performance: Multi-threaded architecture with intelligent rate limiting.
  • 🎯 Comprehensive Coverage: Combines passive and active reconnaissance methods.
  • πŸ“Š Rich Output: Supports multiple export formats with detailed reporting.
  • πŸ–₯️ User-Friendly: Offers both CLI and GUI interfaces.
  • πŸ”’ Security-Focused: Includes built-in subdomain takeover detection.
  • πŸ“± Cross-Platform: Compatible with Windows, Linux, and macOS.

✨ Features

πŸ” Discovery Capabilities

  • Certificate Transparency Logs: Enhanced queries to crt.sh and CertSpotter with comprehensive parsing (4000+ certificates processed).
  • DNS Enumeration: Supports brute force, SRV records, and zone transfers.
  • Web Archives: Extracts subdomains from Wayback Machine, CommonCrawl, and other archives.
  • Search Engine Reconnaissance: Uses Google dorks and other search engines for indexed subdomains.
  • C99 Subdomain Finder: Automated scan retrieval from subdomainfinder.c99.nl with IP and Cloudflare status data.
  • Threat Intelligence Sources: AlienVault OTX, Anubis, ThreatCrowd, HackerTarget, Robtex, Sitedossier.
  • Premium APIs: BeVigil, BufferOver, C99.nl, Chaos, FullHunt, IntelX, Netlas, LeakIX, ZoomEye.
  • Additional Sources: FOFA, Hunter, Quake, WhoisXML, BuiltWith, Facebook Graph API.
  • Security APIs: Integrates with VirusTotal, SecurityTrails, Censys, and Shodan for enriched data.
  • GitHub Code Search: Analyzes code repositories for subdomain leaks.
  • Reverse DNS Lookups: Maps IPs to domains for additional insights.
  • πŸ€– AI-Powered Generation: Uses OpenRouter to intelligently generate subdomain candidates.

πŸ›‘οΈ Security Analysis

  • Subdomain Takeover Detection: Identifies vulnerabilities across 50+ services (e.g., AWS S3, GitHub Pages, Heroku).
  • SSH Service Detection: Scans for open SSH services on port 22.
  • HTTP/HTTPS Status Verification: Checks for live subdomains and their status codes.
  • Wildcard DNS Detection: Identifies wildcard DNS configurations to avoid false positives.
  • Port Scanning Integration: Detects common services (HTTP, HTTPS, FTP, SMTP) and their versions.

πŸ“Š Output & Reporting

  • Multiple Formats: Exports results in TXT, CSV, JSON, and HTML.
  • Interactive HTML Reports: Includes charts and statistics for easy analysis.
  • Real-time Progress Tracking: Displays live scan updates.
  • Detailed Vulnerability Reports: Highlights security findings like takeover risks.
  • Tool Compatibility: Exports results compatible with Nmap and Masscan.

βš™οΈ Advanced Options

  • Multi-threading: Configurable threads (1-200) for performance tuning.
  • Proxy Support: Supports HTTP, HTTPS, and SOCKS proxies for stealth and bypassing rate limits.
  • Custom Wordlists: Allows user-defined wordlists for DNS brute forcing.
  • Rate Limiting & Stealth Mode: Minimizes detection with random delays and throttling.
  • Custom DNS Servers: Supports user-specified DNS resolvers.
  • Timeout Configuration: Adjustable request timeouts for flexibility.

πŸ€– AI Integration

SubGrab's OpenRouter API integration enhances subdomain discovery with cutting-edge AI capabilities.

🧠 Supported AI Models

Model Provider Best For Performance Cost
Claude 3.5 Sonnet Anthropic Best overall ⭐⭐⭐⭐⭐ Medium
Claude 3 Haiku Anthropic Fast & cost-effective ⭐⭐⭐⭐ Low
Grok 3 xAI General use (recommended) ⭐⭐⭐⭐⭐ Low
Grok 4 xAI Complex pattern analysis ⭐⭐⭐⭐⭐ Medium
GPT-4o OpenAI High-quality analysis ⭐⭐⭐⭐⭐ High
GPT-4o Mini OpenAI Balanced performance ⭐⭐⭐⭐ Medium
Gemini Pro 1.5 Google Good alternative ⭐⭐⭐⭐ Medium
Llama 3.1 8B Meta Open source option ⭐⭐⭐ Low

🎯 AI Workflow

  1. Traditional Discovery: Gathers initial subdomains using passive and active methods.
  2. Pattern Analysis: AI analyzes discovered subdomains to identify naming patterns, numbering schemes, and environments.
  3. Intelligent Generation: Generates new subdomain candidates based on observed patterns (e.g., if api1 exists, suggests api2, api3).
  4. Quality Validation: Filters AI-generated candidates for relevance and likelihood before DNS testing.

🎯 AI Capabilities

  • πŸ“Š Pattern Recognition: Identifies naming conventions and organizational patterns.
  • 🧠 Intelligent Variations: Suggests logical extensions based on existing subdomains.
  • 🏒 Context Awareness: Considers organization type and industry-specific patterns.
  • πŸ” Quality Validation: Ensures high-probability subdomain candidates.
  • ⚑ Efficiency: Activates AI analysis only when sufficient data (3+ subdomains) is available.

πŸ› οΈ Installation

πŸ“¦ Method 1: Python Installation

# Clone the repository
git clone https://github.com/bidhata/SubGrab.git
cd SubGrab

# Install dependencies
pip install -r requirements.txt

# Run SubGrab
python subgrab.py example.com

πŸ’» Method 2: Windows Binaries (Recommended)

  1. Download the latest release from Releases.
  2. Extract the ZIP file.
  3. Double-click QuickStart.bat for an interactive menu, or run subgrab.exe example.com directly.

🐳 Method 3: Docker

# Build Docker image
docker build -t subgrab .

# Run with Docker
docker run -v $(pwd)/results:/app/results subgrab example.com

πŸš€ Quick Start

🎯 Basic Usage

# Simple subdomain enumeration
python subgrab.py example.com

# With AI enhancement (OpenRouter)
python subgrab.py example.com --openrouter-key sk-or-xxxxx

# Fast scan with multiple threads
python subgrab.py example.com --fast --threads 100

# Stealth mode with delays
python subgrab.py example.com --stealth

πŸ€– AI-Enhanced Scanning

# FREE: Using Grok AI (xAI) - Recommended for budget-conscious users
python subgrab.py example.com \
  --grok-key xai-xxxxx

# Grok with specific model
python subgrab.py example.com \
  --grok-key xai-xxxxx \
  --grok-model grok-3

# AI with Claude 3.5 Sonnet (OpenRouter)
python subgrab.py example.com \
  --openrouter-key sk-or-xxxxx \
  --openrouter-model anthropic/claude-3.5-sonnet

# Cost-effective with Claude Haiku
python subgrab.py example.com \
  --openrouter-key sk-or-xxxxx \
  --openrouter-model anthropic/claude-3-haiku

# Use BOTH Grok AND OpenRouter for maximum AI coverage
python subgrab.py example.com \
  --grok-key xai-xxxxx \
  --openrouter-key sk-or-xxxxx \
  --openrouter-model anthropic/claude-3.5-sonnet

# Maximum coverage with multiple APIs
python subgrab.py example.com \
  --grok-key xai-xxxxx \
  --openrouter-key sk-or-xxxxx \
  --shodan-key YOUR_SHODAN_KEY \
  --virustotal-key YOUR_VT_KEY

πŸ–₯️ GUI Usage

# Launch GUI
python subgrab_gui.py

# Or on Windows
subgrab_gui.exe

🎯 Enhanced Results

With the new sources, SubGrab can now discover significantly more subdomains:

  • Before: ~200-500 subdomains for typical domains
  • After: ~2000-22000+ subdomains for the same domains
  • Example: example.com went from 5 to 22,705 subdomains discovered!

πŸ“– Usage

πŸŽ›οΈ Command Line Options

usage: subgrab.py [-h] [--threads THREADS] [--timeout TIMEOUT] [--fast] [--stealth]
                  [--proxy-file PROXY_FILE] [--wordlist WORDLIST] [--nameservers NAMESERVERS]
                  [--shodan-key SHODAN_KEY] [--securitytrails-key SECURITYTRAILS_KEY]
                  [--virustotal-key VIRUSTOTAL_KEY] [--censys-id CENSYS_ID]
                  [--censys-secret CENSYS_SECRET] [--github-token GITHUB_TOKEN]
                  [--whoisxml-key WHOISXML_KEY]
                  [--openrouter-key OPENROUTER_KEY] [--openrouter-model OPENROUTER_MODEL]
                  [--grok-key GROK_KEY] [--grok-model GROK_MODEL]
                  domain

positional arguments:
  domain                Target domain to enumerate

optional arguments:
  -h, --help            Show this help message and exit
  -t, --threads THREADS Number of threads (default: 50)
  --timeout TIMEOUT     Request timeout in seconds (default: 30)
  --fast                Fast mode - skip intensive tasks
  --stealth             Enable stealth mode with random delays
  --proxy-file PROXY_FILE File containing proxy list
  --wordlist WORDLIST   Custom wordlist file
  --nameservers NAMESERVERS DNS nameservers to use

AI Integration:
  --openrouter-key OPENROUTER_KEY     OpenRouter API key
  --openrouter-model OPENROUTER_MODEL OpenRouter model (default: anthropic/claude-3.5-sonnet)
  --grok-key GROK_KEY                 xAI Grok API key (FREE tier available)
  --grok-model GROK_MODEL             Grok model (default: grok-3, also: grok-3-mini, grok-4, grok-4.1-fast)

API Keys:
  --shodan-key SHODAN_KEY             Shodan API key
  --securitytrails-key SECURITYTRAILS_KEY SecurityTrails API key
  --virustotal-key VIRUSTOTAL_KEY     VirusTotal API key
  --censys-id CENSYS_ID               Censys API ID
  --censys-secret CENSYS_SECRET       Censys API secret
  --github-token GITHUB_TOKEN         GitHub API token
  --whoisxml-key WHOISXML_KEY         WhoisXML API key (500 free credits)
  
Note: 25+ API sources supported! Use GUI for easy configuration of all API keys.

πŸ“ Usage Examples

🎯 Basic Enumeration

# Simple scan
python subgrab.py example.com

# With custom wordlist
python subgrab.py example.com --wordlist /path/to/wordlist.txt

# Using custom DNS servers
python subgrab.py example.com --nameservers 1.1.1.1 8.8.8.8

πŸ€– AI-Enhanced Scanning

# AI pattern analysis
python subgrab.py example.com --openrouter-key sk-or-xxxxx

# Specific AI model
python subgrab.py example.com \
  --openrouter-key sk-or-xxxxx \
  --openrouter-model openai/gpt-4o

⚑ Performance Optimization

# High-speed scanning
python subgrab.py example.com --threads 200 --timeout 10

# Stealth mode
python subgrab.py example.com --stealth --threads 10

πŸ”‘ Multiple API Keys

python subgrab.py example.com \
  --openrouter-key sk-or-xxxxx \
  --shodan-key YOUR_SHODAN_KEY \
  --virustotal-key YOUR_VT_KEY \
  --securitytrails-key YOUR_ST_KEY \
  --censys-id YOUR_CENSYS_ID \
  --censys-secret YOUR_CENSYS_SECRET \
  --github-token YOUR_GITHUB_TOKEN

πŸ”§ Configuration

πŸ€– AI API Setup

Grok (xAI) - FREE Option ⭐ Recommended

  1. Visit console.x.ai or x.ai/api.
  2. Sign up for a free account.
  3. Free credits available for new accounts.
  4. Generate an API key from the console.
  5. Use the key with --grok-key xai-xxxxx.

Why Grok?

  • πŸ’° Free credits for new accounts
  • πŸš€ High-quality AI from xAI
  • ⚑ Fast and efficient
  • πŸ”Œ OpenAI-compatible API
  • πŸ†“ Very affordable token pricing

OpenRouter Setup

  1. Visit openrouter.ai.
  2. Sign up and add $5-10 credits.
  3. Generate an API key.
  4. Use the key with --openrouter-key sk-or-xxxxx.

πŸ”‘ API Keys Configuration

Create api_keys.json (or use the GUI for easy management):

{
  "grok": "xai-xxxxx",
  "openrouter": "sk-or-xxxxx",
  "shodan": "your_shodan_key",
  "virustotal": "your_vt_key",
  "securitytrails": "your_st_key",
  "censys": {
    "id": "your_censys_id",
    "secret": "your_censys_secret"
  },
  "github": "your_github_token",
  "bevigil": "your_bevigil_key",
  "bufferover": "your_bufferover_key",
  "c99": "your_c99_key",
  "chaos": "your_chaos_key",
  "fullhunt": "your_fullhunt_key",
  "intelx": "your_intelx_key",
  "netlas": "your_netlas_key",
  "leakix": "your_leakix_key",
  "zoomeye": "your_zoomeye_key",
  "fofa": "your_fofa_key",
  "hunter": "your_hunter_key",
  "quake": "your_quake_key",
  "whoisxml": "your_whoisxml_key",
  "builtwith": "your_builtwith_key",
  "facebook": "your_facebook_token"
}

πŸ’‘ Tip: Use the GUI's "API Keys" tab for easy configuration with organized categories and direct links to get API keys!

πŸ“ Custom Wordlists

# Example wordlist (custom_subdomains.txt)
www
api
dev
staging
admin
mail
ftp

🌐 Proxy Configuration

# Proxy file format (proxies.txt)
http://proxy1:8080
https://user:pass@proxy2:3128
socks5://proxy3:1080

πŸ“Š Output Formats

πŸ“ Output Directory Structure

example.com_results/
β”œβ”€β”€ all_subdomains.txt          # Complete subdomain list
β”œβ”€β”€ active_subdomains.txt       # HTTP/HTTPS responsive
β”œβ”€β”€ inactive_subdomains.txt     # Non-responsive
β”œβ”€β”€ ssh_enabled.txt             # SSH service detected
β”œβ”€β”€ takeover_candidates.txt     # Potential takeovers
β”œβ”€β”€ scan_results.json           # Detailed JSON report
β”œβ”€β”€ scan_results.csv            # CSV format
└── report.html                 # Interactive HTML report

πŸ“„ Report Contents

πŸ“‹ Text Reports

  • Complete subdomain lists with status.
  • Categorized by response type.
  • Security findings (e.g., takeovers, SSH).
  • Statistics and summaries.

πŸ“Š JSON Report

{
  "scan_info": {
    "target": "example.com",
    "start_time": "2024-01-01T12:00:00Z",
    "duration": 120.5,
    "total_subdomains": 156,
    "active_subdomains": 89,
    "ai_generated": 23
  },
  "subdomains": [
    {
      "subdomain": "api.example.com",
      "status": "active",
      "ip": "192.168.1.1",
      "http_status": 200,
      "technologies": ["nginx", "cloudflare"],
      "discovery_method": "certificate_transparency"
    }
  ],
  "vulnerabilities": [
    {
      "subdomain": "old.example.com",
      "type": "subdomain_takeover",
      "service": "github.io",
      "confidence": "high"
    }
  ]
}

🌐 HTML Report

Interactive dashboard with:

  • πŸ“ˆ Statistics charts.
  • πŸ—ΊοΈ Discovery method breakdown.
  • πŸ” Searchable subdomain table.
  • ⚠️ Security findings.
  • πŸ“Š Response time analysis.

πŸŽ›οΈ API Integrations

πŸ”‘ Core Security APIs

Service Purpose Free Tier Rate Limit
OpenRouter AI subdomain generation ❌ Model dependent
Grok (xAI) AI subdomain generation βœ… Free credits Model dependent
Shodan Infrastructure discovery βœ… Limited 1 req/sec
VirusTotal Domain intelligence βœ… 4 req/min 4 req/min
SecurityTrails DNS history βœ… 50 req/month Varies
Censys Certificate data βœ… 250 req/month 0.2 req/sec
GitHub Code search βœ… 5000 req/hour 30 req/min

πŸ†“ Free Threat Intelligence Sources

Service Purpose Coverage Performance
AlienVault OTX Passive DNS data βœ… Free High
Anubis Subdomain database βœ… Free Very High
ThreatCrowd Community intelligence βœ… Free Medium
HackerTarget DNS reconnaissance βœ… Free Medium
Robtex DNS/IP intelligence βœ… Free Medium
Sitedossier Domain analysis βœ… Free Low

πŸ’Ž Premium API Sources

Service Purpose Pricing Coverage
BeVigil Mobile app security πŸ’° Paid Mobile-focused
BufferOver DNS data provider πŸ’° Paid High
C99.nl Multi-tool platform πŸ’° Paid Very High
Chaos ProjectDiscovery data πŸ’° Paid High
FullHunt Attack surface mgmt πŸ’° Paid High
IntelX Intelligence platform πŸ’° Paid Very High
Netlas Internet assets search πŸ’° Paid High
LeakIX Leak detection πŸ’° Paid Medium
ZoomEye Cyberspace search πŸ’° Paid High
FOFA Cyberspace assets πŸ’° Paid High
Hunter Threat intelligence πŸ’° Paid High
Quake Cyberspace mapping πŸ’° Paid High
WhoisXML Subdomain lookup βœ… 500 credits High
BuiltWith Technology profiler πŸ’° Paid Medium
Facebook Social platform API πŸ’° Paid Low

πŸ–₯️ GUI Interface

🎯 Features

  • πŸ“ Easy Configuration: Point-and-click setup with organized tabs.
  • πŸ“Š Real-time Progress: Live scan updates with detailed output.
  • πŸ”‘ Enhanced API Key Management: 25+ API sources organized in categories with direct "Get Key" links.
  • πŸ“ Result Management: Direct access to reports and results folder.
  • 🎨 Modern Interface: Clean, scrollable design with improved layout.
  • πŸ’Ύ Save/Load Configurations: Export and import API key configurations.
  • πŸ–±οΈ Mouse Wheel Scrolling: Smooth navigation through extensive API key lists.

πŸš€ Usage

python subgrab_gui.py
# Or on Windows
subgrab_gui.exe

πŸ’» Windows Binaries

πŸ“¦ Package Contents

  • subgrab.exe: Command-line version (213 MB).
  • subgrab_gui.exe: GUI version (29 MB).
  • QuickStart.bat, run_cli.bat, run_gui.bat: Easy launchers.
  • Complete documentation and examples.

πŸš€ Quick Start (Windows)

  1. Download the latest release ZIP.
  2. Extract to a folder.
  3. Run QuickStart.bat and choose GUI or CLI mode.

πŸ” Discovery Methods

🌐 Passive Methods

  • Enhanced Certificate Transparency: Comprehensive crt.sh parsing (4000+ certificates), CertSpotter integration.
  • C99 Subdomain Finder: Automated scan retrieval from subdomainfinder.c99.nl with IP and Cloudflare data.
  • Threat Intelligence: AlienVault OTX (500+ subdomains), Anubis (20000+ subdomains), ThreatCrowd, HackerTarget.
  • DNS Intelligence: Brute forcing, SRV records, zone transfers, reverse DNS, Robtex, Sitedossier.
  • Web Archives: Wayback Machine, CommonCrawl, and Archive.today for historical data.
  • Search Engines: Google, Bing, DuckDuckGo with advanced operators.
  • Premium APIs: BeVigil, BufferOver, C99.nl, Chaos, FullHunt, IntelX, Netlas, LeakIX, ZoomEye.
  • Additional Sources: FOFA, Hunter, Quake, WhoisXML, BuiltWith, Facebook Graph API.
  • Security APIs: VirusTotal, SecurityTrails, Censys, Shodan.
  • Code Repositories: GitHub and GitLab code search.

🎯 Active Methods

  • HTTP/HTTPS Probing: Status code verification, technology fingerprinting.
  • Port Scanning: SSH, HTTP, HTTPS, FTP, SMTP detection.

πŸ€– AI-Powered Methods

  • Intelligent Generation: Context-aware subdomain suggestions.
  • Pattern Analysis: Identifies naming conventions and environments.

πŸ›‘οΈ Security Features

🎯 Subdomain Takeover Detection

  • Supported Services: AWS S3, Azure, GitHub Pages, Heroku, Netlify, and 50+ others.
  • Detection Methods: DNS resolution, HTTP response analysis, SSL certificate validation.

πŸ”’ Security Best Practices

  • Authorized testing only.
  • Rate limiting and stealth mode for responsible scanning.
  • Proxy support for anonymity.
  • Comprehensive logging for audit trails.

πŸ“ˆ Performance

⚑ Benchmarks

Metric Traditional Tools SubGrab v1 SubGrab v2 Enhanced
Subdomains Found 200-300 400-600 2000-22000+
Discovery Sources 5-10 15+ 25+
Execution Time 5-10 min 3-7 min 5-15 min
False Positives 10-15% 5-8% 2-5%
Unique Discoveries Standard Enhanced Comprehensive
API Integrations 2-3 6 25+

Real Example: example.com discovery improved from 5 subdomains to 22,705 subdomains with enhanced sources!


πŸš€ Recent Enhancements (v2.1)

🎯 Major Improvements

  • WhoisXML Subdomain API: Comprehensive subdomain lookup with 500 free credits
  • AI Retry Logic: Automatic retry with exponential backoff for transient API errors (502/503/504)
  • Grok AI Integration: xAI Grok support for affordable AI-powered subdomain discovery
  • 25+ Discovery Sources: Expanded from 15 to 25+ enumeration techniques
  • C99 Subdomain Finder: Automated scan retrieval with IP and Cloudflare status data
  • Comprehensive Certificate Transparency: Processes 4000+ certificates with improved parsing
  • Free Threat Intelligence: AlienVault OTX, Anubis (20K+ subdomains), ThreatCrowd, HackerTarget
  • Premium API Support: 15+ premium sources including BeVigil, C99.nl, FullHunt, IntelX, Netlas
  • Enhanced GUI: Organized API key management with 25+ sources, scrollable interface, save/load configs

πŸ“Š Performance Improvements

  • 22,000+ Subdomains: Real example of example.com going from 5 to 22,705 subdomains
  • Intelligent Rate Limiting: Respectful crawling with automatic backoff
  • Comprehensive Coverage: Extensive subdomain extraction from multiple scan sources
  • Multiple Extraction Methods: Table parsing, regex extraction, JSON API calls

🎨 GUI Enhancements

  • Organized Categories: Core APIs, Premium Intelligence, Additional Sources
  • Direct Links: "Get Key" buttons for each API provider
  • Save/Load Configurations: Easy API key management
  • Scrollable Interface: Handles 25+ API sources efficiently
  • Professional Layout: Clean design with proper spacing and organization

🀝 Contributing

We welcome contributions from the security community! To contribute:

  1. Fork the repository.
  2. Create a feature branch.
  3. Make changes and test thoroughly.
  4. Submit a pull request with clear documentation.

See the Contributing section in the original documentation for detailed guidelines.


πŸ“„ License

This project is licensed under the MIT License. See the LICENSE file for details.

βš–οΈ Terms of Use

  • βœ… Use only for authorized testing.
  • βœ… Follow responsible disclosure practices.
  • ❌ Do not use for malicious purposes.
  • ❌ No warranty; use at your own risk.

πŸ“ž Contact & Support

Author: Krishnendu Paul
GitHub: bidhata/SubGrab
Support: Open an issue on GitHub or contact the author for assistance.

⭐ If SubGrab helped you, please give it a star! ⭐
Made with ❀️ for the Security Community.

About

Subdomain Scanner with OpenRouter API ( Choose your AI model ) and Free Grok API . First Subdomain scanner with AI integration

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages