Python Tools Documentation 2025

Last Updated: July 28, 2025 (v1.5.0)
Major Achievement: Complete interactive_utils migration eliminating EOF errors in CI/CD environments

Related Code Files:

enhanced_standard_arg_parser.py - Standardized argument parsing framework
common_utils.py - Common utilities and secure base classes
common_config.py - Unified configuration management
error_logger.py - Comprehensive error logging system
run_any_python_tool.sh - Universal wrapper with approval system

🎯 Executive Summary

The code-intelligence-toolkit provides a comprehensive suite of 80+ production-ready Python tools for advanced code analysis, refactoring, and development workflows. As of January 2025, ALL production tools have been successfully standardized with enhanced argument parsing, preflight validation, and enterprise-grade security.

🆕 Interactive Utils Module (July 2025)

Complete Non-Interactive Support Revolution

The new interactive_utils.py module provides unified handling for all interactive prompts:

from interactive_utils import (
    get_confirmation,      # Yes/no prompts
    get_multi_choice,      # Multiple choice (y/n/q)
    get_numbered_selection,# Numbered menus
    check_auto_yes_env,    # Environment variable support
    is_interactive,        # Auto-detect CI/CD environments
)

Key Features:

No More EOF Errors: Clear messages instead of crashes
Automatic Detection: CI/CD, pipes, no TTY environments
Configuration Hierarchy: Flags > env vars > .pytoolsrc > defaults
Multiple Prompt Types: Supports all common interaction patterns

Migrated Tools (v1.5.0):

text_undo.py - Numbered selections for restore
safe_file_manager.py - Risk-based confirmations
safegit.py - Multi-choice and typed confirmations
replace_text_v9.py - Large change confirmations
replace_text_ast_v3.py - Batch operation confirmations

🚀 Major Achievements in 2025

1. Complete Standardization Success

100% of production tools now use EnhancedArgumentParser
Consistent --help formatting across all tools
Standardized argument patterns: <tool> <target> [location_flags] [options]
Universal preflight validation system

2. Enhanced Argument Parser System

The new EnhancedArgumentParser extends Python's ArgumentParser with:

class EnhancedArgumentParser(ArgumentParser):
    """Enhanced argument parser with preflight checks and standardization"""
    
    def add_preflight_check(self, check_type, argument_name):
        """Add automatic validation before tool execution"""
        # Supported checks:
        # - file_exists: Verify file exists
        # - directory_exists: Verify directory exists
        # - readable: Check read permissions
        # - writable: Check write permissions
        # - python_syntax: Validate Python syntax
        # - java_syntax: Validate Java syntax

Key Features:

Automatic Validation: Files, directories, permissions checked before execution
Consistent Help Format: Usage, description, examples in every tool
Standard Flags: -v, -q, --json, --no-color work everywhere
Better Error Messages: Clear, actionable error reporting

3. Tool Categories and Capabilities

🔍 Search and Analysis (Enhanced)

find_text_v7.py V6: Enhanced with context lines, ± syntax, auto file-finding, block extraction, and standalone wholefile mode
find_references_rg.py: Multi-threaded with --threads support
cross_file_analysis_ast.py: Full AST-based dependency graphs

🧬 AST-Based Tools (Production Ready)

navigate_ast_v2.py: 100% accurate definition finding
semantic_diff_v3.py: Enterprise semantic diff with risk assessment
method_analyzer_ast_v2.py: Complete call flow analysis
show_structure_ast_v4.py: Hierarchical viewer with annotation filtering

📂 File Operations (Bulletproof V2 + Atomic)

safe_move.py: Interactive undo mode with atomic operations and retry logic
organize_files.py: Manifest-based operations with full reversibility and atomic writes
refactor_rename.py: Automation support with --yes flag and atomic file/symbol renaming
replace_text_v9.py: Atomic text replacement with intelligent retry mechanisms
replace_text_ast_v3.py: AST-based atomic replacements with rollback protection
unified_refactor.py: Universal refactoring interface with language-aware auto-detection (auto, python_ast, rope, java_scope, text_based)

🛠️ Infrastructure (Enterprise Grade)

common_config.py: Project-aware configuration via .pytoolsrc
error_logger.py: Automatic error capture to ~/.pytoolserrors/
analyze_errors.py: Error pattern analysis dashboard with --clear flag for log cleanup

🛡️ Git Safety (SafeGIT v2.0 - Now with Non-Interactive Support)

safegit.py: Comprehensive git safety wrapper preventing data loss with 37+ dangerous command patterns
safe_git_commands.py: Core safety analysis and backup operations
safegit_undo_stack.py: Multi-level undo system with atomic metadata capture
Core Features: Command interception, automatic backups, dry-run mode, reflog hints, atomic file operations
AI Protection: Prevents AI agents from executing destructive git commands (single rule enforcement)
Enhanced Safety: Typed confirmations, branch protection detection, recovery instructions
NEW v2.0 Features:
- Non-interactive mode for CI/CD (--yes, --force-yes, --non-interactive)
- Environment variable support (SAFEGIT_NONINTERACTIVE, SAFEGIT_ASSUME_YES)
- CI platform auto-detection (GitHub Actions, GitLab CI, Jenkins, Travis)
- Graduated safety levels (safe → medium → dangerous)
- Comprehensive automation logging

4. Security and Reliability Enhancements

Centralized Security Framework

All tools inherit from secure base classes providing:

Path traversal protection
Command injection prevention
Resource limit enforcement
Atomic file operations with rollback
Comprehensive audit trails

Atomic File Operations and Retry Logic

NEW in 2025: Enterprise-grade atomic operations across all file write tools

Core Features:

True Atomicity: All file writes use temporary files + atomic rename
Intelligent Retry Logic: Configurable retry attempts with exponential backoff
Failure Recovery: Automatic rollback on partial failures
Cross-Platform Safety: Works reliably on Windows, macOS, Linux
Process Safety: Protected against interruption and concurrent access

6 Enhanced Tools with Atomic Operations:

replace_text_v9.py - Atomic text replacement with retry logic
replace_text_ast_v3.py - AST-based atomic replacements
unified_refactor.py - Universal refactoring interface with language-aware auto-detection (replaces ast_refactor.py, ast_refactor_enhanced.py, java_scope_refactor.py)
refactor_rename.py - Atomic file/symbol renaming across files
safe_move.py - Enhanced atomic file moves
organize_files.py - Atomic batch file organization

Environment Variables for Customization:

# Retry configuration (applies to all atomic tools)
export ATOMIC_RETRY_ATTEMPTS=5      # Default: 3 attempts
export ATOMIC_RETRY_DELAY=2.0       # Default: 1.0 seconds base delay
export ATOMIC_RETRY_BACKOFF=1.5     # Default: 2.0 exponential factor
export ATOMIC_OPERATION_TIMEOUT=60  # Default: 30 seconds per operation

# Tool-specific overrides
export REPLACE_TEXT_RETRY_ATTEMPTS=5
export SAFE_MOVE_RETRY_DELAY=0.5
export REFACTOR_RENAME_TIMEOUT=120

5. Multi-Level Undo System (NEW)

SafeGIT-style protection extended to all text operations - Every edit is now reversible!

Core Components

text_operation_history.py: Core undo tracking system with atomic operations
text_undo.py: CLI tool for managing undo operations
Operation tracking: Automatic for all v9/v3/v2 tools

Key Features

Automatic Operation Tracking: Every text modification is recorded by default
Multi-Level Undo: Restore to any previous state in history
Atomic Storage: Thread-safe with file locking for concurrent safety
Smart Compression: Files >1KB automatically compressed with gzip
Recovery Scripts: Auto-generated shell scripts for manual recovery
Cross-Tool Support: Works across all editing tools in the toolkit

Integrated Tools with Undo Support

replace_text_v9.py: Full text replacement with undo (superset of v8)
replace_text_ast_v3.py: AST-based refactoring with undo (superset of v2)
unified_refactor_v2.py: Universal refactoring with undo
refactor_rename_v2.py: Batch renaming with undo
safe_file_manager_undo_wrapper.py: File operations with undo tracking

Usage Examples

# Basic replacement with automatic undo tracking
./run_any_python_tool.sh replace_text_v9.py "old" "new" file.py
# Output: [Undo ID: 1753732740927_91513] Use 'text_undo.py undo --operation 1753732740927_91513' to undo.

# View operation history
./run_any_python_tool.sh text_undo.py history
./run_any_python_tool.sh text_undo.py history --file script.py
./run_any_python_tool.sh text_undo.py history --since 2h

# Undo operations
./run_any_python_tool.sh text_undo.py undo --last
./run_any_python_tool.sh text_undo.py undo --operation 1753732740927_91513
./run_any_python_tool.sh text_undo.py undo --interactive

# System management
./run_any_python_tool.sh text_undo.py stats
./run_any_python_tool.sh text_undo.py clean  # Remove operations >30 days old

Configuration Options

# Disable undo for specific operations
./run_any_python_tool.sh replace_text_v9.py --no-undo "old" "new" file.py

# Add custom descriptions for better history
./run_any_python_tool.sh replace_text_v9.py --undo-description "Fix critical bug in auth" "broken" "fixed" *.py

# Environment variables
export TEXT_UNDO_MAX_HISTORY=10000      # Max operations to keep (default: 5000)
export TEXT_UNDO_MAX_AGE_DAYS=60        # Backup retention (default: 30)
export TEXT_UNDO_COMPRESSION=0          # Disable compression (default: 1)

Storage Structure

~/.text_operation_history/
├── operations.jsonl      # Operation log (append-only)
├── backups/             # Compressed/uncompressed file backups
├── metadata/            # Operation metadata
└── recovery_scripts/    # Auto-generated recovery scripts

Honest Compile Checking

Complete overhaul of compile validation:

Clear feedback: ✓ Compiles, ✗ Syntax Error, ✗ Cannot check
Support for Python (AST), Java (javac), JavaScript (node), TypeScript (tsc)
Default visibility (no verbose mode needed)
Configurable via --check-compile / --no-check-compile

5. Atomic Operations and Retry Logic Framework

Overview

The atomic operations framework provides enterprise-grade safety for all file write operations across the toolkit. Every file modification is guaranteed to be atomic (all-or-nothing) with intelligent retry logic to handle temporary failures.

Architecture

# Core atomic operation framework
class AtomicFileOperation:
    def __init__(self, retry_attempts=3, retry_delay=1.0, backoff_factor=2.0):
        self.retry_attempts = retry_attempts
        self.retry_delay = retry_delay
        self.backoff_factor = backoff_factor
    
    def execute(self, operation):
        """Execute operation with atomic guarantees and retry logic"""
        # 1. Create temporary file
        # 2. Perform operation on temp file  
        # 3. Validate result
        # 4. Atomic rename to target
        # 5. Retry on failure with exponential backoff

Enhanced Tools with Atomic Operations

replace_text_v9.py - Atomic Text Replacement

# Basic atomic replacement
./run_any_python_tool.sh replace_text_v9.py "oldText" "newText" file.java --atomic

# With custom retry settings
./run_any_python_tool.sh replace_text_v9.py "old" "new" src/ --atomic --retry-attempts 5 --retry-delay 2.0

# Environment variable configuration
export REPLACE_TEXT_RETRY_ATTEMPTS=3
export REPLACE_TEXT_RETRY_DELAY=1.5
export REPLACE_TEXT_ATOMIC_TIMEOUT=60
./run_any_python_tool.sh replace_text_v9.py "pattern" "replacement" files/ --atomic

replace_text_ast_v3.py - AST-Based Atomic Replacement

# Atomic AST-based replacement with scope awareness
./run_any_python_tool.sh replace_text_ast_v3.py --file MyClass.java --line 42 oldVar newVar --atomic

# With compilation verification
./run_any_python_tool.sh replace_text_ast_v3.py --file MyClass.java oldMethod newMethod --atomic --check-compile

# Custom retry configuration
export REPLACE_TEXT_AST_RETRY_ATTEMPTS=5
./run_any_python_tool.sh replace_text_ast_v3.py --file Complex.java complex_var simple_var --atomic

refactor_rename.py - Atomic Symbol Renaming

# Atomic multi-file symbol renaming
./run_any_python_tool.sh refactor_rename.py --replace oldSymbol newSymbol --in "src/**/*.java" --atomic

# With automation support for CI/CD
./run_any_python_tool.sh refactor_rename.py --replace oldVar newVar --in "**/*.py" --yes --atomic

# Custom timeout for large operations
export REFACTOR_RENAME_ATOMIC_TIMEOUT=300
./run_any_python_tool.sh refactor_rename.py --replace LongClassName ShortName --in "large_project/**/*.java" --atomic

unified_refactor.py - Universal Refactoring Interface (NEW)

Professional Features:

Language-Aware Auto-Detection: Automatically selects optimal backend (.java → java_scope, .py → python_ast)
Unified Diff Previews: See exact changes before applying with professional diff formatting
AST-Guided Rope Targeting: Smart offset calculation for precise symbol targeting
JSON Pipeline Integration: Read operations from JSON files or stdin for workflow automation
Cross-Language Support: Seamless refactoring across mixed Java/Python codebases
Multi-Engine Support: Choose optimal backend (python_ast, rope, java_scope, text_based) per task

# Auto-detection: Python files use python_ast backend automatically
./run_any_python_tool.sh unified_refactor.py rename oldFunc newFunc --file script.py --dry-run

# Auto-detection: Java files use java_scope backend automatically  
./run_any_python_tool.sh unified_refactor.py rename OldClass NewClass --file MyClass.java --dry-run

# Mixed codebase: each file uses optimal backend automatically
./run_any_python_tool.sh unified_refactor.py find calculateValue --scope src/  # Auto-detects per file

# JSON pipeline workflow - read operations from file
echo '[{"file":"script.py","old":"oldMethod","new":"newMethod","line":42}]' | \
./run_any_python_tool.sh unified_refactor.py rename --from-json -

# Rope backend with AST-guided offset calculation
./run_any_python_tool.sh unified_refactor.py rename old_var new_var --backend rope --file code.py --line 15

# Find all references to a symbol with JSON output
./run_any_python_tool.sh unified_refactor.py find calculateValue --scope src/ --json

# Analyze code structure with professional formatting
./run_any_python_tool.sh unified_refactor.py analyze --file MyClass.java --json

# Text-based fallback for any language with unified diff
./run_any_python_tool.sh unified_refactor.py rename oldVar newVar --backend text_based --file config.yml --dry-run

# Backend options: auto (default - language-aware), python_ast, rope, java_scope, text_based
# Commands: rename, find, analyze, rename-project
# Professional features: --dry-run, --from-json, --json, unified diffs, smart targeting

safe_move.py - Enhanced Atomic File Operations

# Atomic file move with rollback capability
./run_any_python_tool.sh safe_move.py move source.java destination.java --atomic

# Atomic copy with retry logic
./run_any_python_tool.sh safe_move.py copy important.java backup.java --atomic --retry-attempts 5

# Interactive undo with atomic operation history
./run_any_python_tool.sh safe_move.py undo --interactive --show-atomic-history

organize_files.py - Atomic Batch File Operations

# Atomic file organization with manifest
./run_any_python_tool.sh organize_files.py ~/Downloads --by-ext --atomic --create-manifest

# Atomic archiving with compression
./run_any_python_tool.sh organize_files.py ~/old_files --archive-by-date 90 --atomic

# Batch processing with atomic safety
export ORGANIZE_FILES_BATCH_SIZE=50
./run_any_python_tool.sh organize_files.py large_directory/ --atomic --progress

Environment Variables Reference

Global Atomic Settings

# Core retry configuration (affects all atomic tools)
export ATOMIC_RETRY_ATTEMPTS=3          # Number of retry attempts (default: 3)
export ATOMIC_RETRY_DELAY=1.0           # Base delay between retries in seconds (default: 1.0)
export ATOMIC_RETRY_BACKOFF=2.0         # Exponential backoff factor (default: 2.0)
export ATOMIC_OPERATION_TIMEOUT=30      # Timeout per operation in seconds (default: 30)
export ATOMIC_TEMP_DIR="/tmp/atomic"    # Temporary directory for atomic operations
export ATOMIC_OPERATION_LOG="~/.pytoolserrors/atomic/"  # Atomic operation log directory

Tool-Specific Overrides

# replace_text_v9.py specific settings
export REPLACE_TEXT_RETRY_ATTEMPTS=5
export REPLACE_TEXT_RETRY_DELAY=1.5
export REPLACE_TEXT_ATOMIC_TIMEOUT=60

# replace_text_ast_v3.py specific settings  
export REPLACE_TEXT_AST_RETRY_ATTEMPTS=3
export REPLACE_TEXT_AST_RETRY_DELAY=2.0
export REPLACE_TEXT_AST_ATOMIC_TIMEOUT=45

# refactor_rename.py specific settings
export REFACTOR_RENAME_RETRY_ATTEMPTS=5
export REFACTOR_RENAME_RETRY_DELAY=1.0
export REFACTOR_RENAME_ATOMIC_TIMEOUT=120

# safe_move.py specific settings
export SAFE_MOVE_RETRY_ATTEMPTS=3
export SAFE_MOVE_RETRY_DELAY=0.5
export SAFE_MOVE_ATOMIC_TIMEOUT=30

# organize_files.py specific settings
export ORGANIZE_FILES_RETRY_ATTEMPTS=3
export ORGANIZE_FILES_BATCH_SIZE=100
export ORGANIZE_FILES_ATOMIC_TIMEOUT=180

# unified_refactor.py specific settings
export UNIFIED_REFACTOR_RETRY_ATTEMPTS=5
export UNIFIED_REFACTOR_ATOMIC_TIMEOUT=90

Atomic Operation Features

Failure Recovery

Automatic Rollback: Failed operations automatically restore original state
Partial Failure Handling: Multi-file operations rollback completely on any failure
State Validation: Each step verified before proceeding to next
Progress Preservation: Successful operations logged for partial recovery

Monitoring and Debugging

# Monitor atomic operations in real-time
./run_any_python_tool.sh tool_name.py [args] --atomic --atomic-verbose

# View atomic operation history
./run_any_python_tool.sh analyze_errors.py --atomic-history --recent 10

# Debug atomic operation failures
./run_any_python_tool.sh tool_name.py [args] --atomic --debug-atomic

# Test atomic operations without executing
./run_any_python_tool.sh tool_name.py [args] --atomic --dry-run

Performance Characteristics

Minimal Overhead: ~5-10% performance impact for safety guarantees
Parallel Safety: Multiple atomic operations can run concurrently
Resource Efficient: Temporary files cleaned up automatically
Cross-Platform: Consistent behavior across Windows, macOS, Linux

6. Performance Optimizations

Multi-Threading Support

Key tools now support configurable thread counts:

# 4x faster analysis with 8 threads
./run_any_python_tool.sh dead_code_detector.py src/ --threads 8
./run_any_python_tool.sh find_references_rg.py method --threads 8
./run_any_python_tool.sh cross_file_analysis_ast.py --threads 8

Timeout Protection

Critical tools wrapped with configurable timeouts:

dead_code_detector.py: 60s default (via DEAD_CODE_TIMEOUT)
trace_calls_with_timeout.py: Protected tracing

6. Intelligent Documentation Generation (NEW)

doc_generator.py & doc_generator_enhanced.py

Revolutionary documentation generation combining code analysis with natural language understanding

Core Features:

Original Version (doc_generator.py): Integrated with data_flow_tracker_v2.py for intelligent analysis
Enhanced Version (doc_generator_enhanced.py): Full integration with 5 AST analysis tools
7 Documentation Styles: API docs, user guides, technical analysis, quick reference, tutorials, architecture, call graphs
5 Output Formats: Markdown, HTML, interactive HTML, reStructuredText, Python docstrings
Full Language Parity: Both Python and Java support all features including data flow analysis

Enhanced Version AST Tool Integration:

navigate_ast_v2.py - Precise symbol location and context
method_analyzer_ast_v2.py - Method call hierarchies and relationships
trace_calls_with_timeout.py - Execution path visualization
data_flow_tracker_v2.py - Variable dependencies and impact analysis
cross_file_analysis_ast.py - Import and module relationships
show_structure_ast.py - Hierarchical code organization

Interactive HTML Documentation:

# Generate interactive documentation with 6-tab analysis
./run_any_python_tool.sh doc_generator_enhanced.py \
  --class MyClass \
  --file MyClass.java \
  --style api-docs \
  --format interactive \
  --output MyClass_Interactive.html

Generated Interactive Tabs:

Overview: Quick summary and key metrics
Navigation: Symbol locations with AST context
Call Flow: Method relationships and execution paths
Data Flow: Variable dependencies and impact visualization
Structure: Code organization and hierarchy
Dependencies: Import and cross-file relationships

Documentation Styles:

# API Documentation - Technical reference with complete analysis
./run_any_python_tool.sh doc_generator_enhanced.py --function calculate --file calc.py --style api-docs

# User Guide - Friendly documentation for end users
./run_any_python_tool.sh doc_generator_enhanced.py --class UserManager --file auth.py --style user-guide

# Technical Analysis - Deep dive with complexity metrics
./run_any_python_tool.sh doc_generator_enhanced.py --module --file system.py --style technical --depth deep

# Architecture Overview - System design and structure
./run_any_python_tool.sh doc_generator_enhanced.py --module --file main.py --style architecture --format html

# Call Graph - Visual function relationships
./run_any_python_tool.sh doc_generator_enhanced.py --function main --file app.py --style call-graph

# Quick Reference - Concise API summary
./run_any_python_tool.sh doc_generator_enhanced.py --class API --file api.py --style quick-ref

# Tutorial - Step-by-step learning guide
./run_any_python_tool.sh doc_generator_enhanced.py --class Database --file db.py --style tutorial

Advanced Features:

Smart Caching: AST operations cached for performance
ANSI Code Stripping: Clean HTML output from tool results
Graceful Degradation: Individual tool failures don't break generation
Java Support: Full parity with Python including data flow analysis
Template System: Custom Jinja2 templates for branding

Performance Characteristics:

Small files (< 500 lines): ~1-2 seconds for all styles
Medium files (500-2000 lines): ~3-5 seconds with deep analysis
Large files (> 2000 lines): ~5-10 seconds, use surface depth for speed
Interactive HTML: Additional ~2-3 seconds for tab generation

🆕 Interactive Utils Module (July 2025)

Unified Non-Interactive Mode Support

All Python tools now use a shared interactive_utils.py module that provides consistent handling of interactive prompts and non-interactive environments.

Key Features:

Automatic Detection: Detects CI/CD, pipes, no TTY, and non-interactive environments
Clear Error Messages: No more EOF crashes - helpful instructions instead
Configuration Support: Respects .pytoolsrc settings for project defaults
Multiple Prompt Types: Yes/no, typed phrases, numbered selections, multi-choice

Configuration Methods (Priority Order):

Command-line flags: --yes, --non-interactive (highest)
Environment variables: {TOOL}_ASSUME_YES=1
Tool-specific .pytoolsrc section: [tool_name]
Global .pytoolsrc defaults: [defaults]
Hard-coded defaults (lowest)

Example Usage:

# Command flag
./run_any_python_tool.sh text_undo.py undo --last --yes

# Environment variable
export TEXT_UNDO_ASSUME_YES=1
./run_any_python_tool.sh text_undo.py undo --last

# .pytoolsrc configuration
[text_undo]
assume_yes = true

# Global non-interactive mode
export PYTOOLSRC_NON_INTERACTIVE=1

Error Messages:

# Clear, actionable guidance
ERROR: Interactive confirmation required but running in non-interactive mode.
       Use --yes flag to skip confirmation
       Or set TEXT_UNDO_ASSUME_YES=1 environment variable
       Or set 'assume_yes = true' in .pytoolsrc [text_undo] section

Tools with Full Support:

text_undo.py - Complete with numbered selections
safe_file_manager.py - Risk-based confirmations
safegit.py - Typed confirmations
All new tools use this module

📋 Configuration System

.pytoolsrc Configuration

Project-specific defaults with hierarchical loading:

# .pytoolsrc example
[DEFAULT]
ast_context = true          # Enable AST context by default
check_compile = true        # Enable compile checking
verbose = false            # Quiet mode default
threads = 4                # Default thread count

[find_text]
context_lines = 3          # Default -C value

[smart_ls]
max_items = 50            # Default --max value

Configuration Priority

Command-line arguments (highest)
.pytoolsrc in current directory
.pytoolsrc in project root
Tool defaults (lowest)

🔧 Tool Usage Examples

Enhanced Search with Context and Auto-Finding

# V5 features: Context line display with ± syntax
./run_any_python_tool.sh find_text_v7.py "TODO" ±10                    # 10 lines before/after
./run_any_python_tool.sh find_text_v7.py "error" --file MyClass.java ±5  # With specific file

# Auto file finding (v5)
./run_any_python_tool.sh find_text_v7.py "processData" --file DataManager.java  # Finds file automatically
./run_any_python_tool.sh find_text_v7.py "calculate" --file Calculator.java --auto-find

# Extract line ranges for piping
./run_any_python_tool.sh find_text_v7.py "TODO" --extract-ranges       # Output: file.java:100±5
./run_any_python_tool.sh find_text_v7.py "TODO" --extract-ranges --merge-ranges  # Merges overlapping

# Traditional context options still work
./run_any_python_tool.sh find_text_v7.py "pattern" -C 5                # 5 lines of context
./run_any_python_tool.sh find_text_v7.py "pattern" -A 3 -B 2           # 3 after, 2 before

# Extract methods containing patterns (v4 features retained)
./run_any_python_tool.sh find_text_v7.py "TODO" --extract-method
./run_any_python_tool.sh find_text_v7.py "deprecated" --extract-method-alllines

# Search with AST context (shows class → method hierarchy)
./run_any_python_tool.sh find_text_v7.py "calculateValue" --file DataBook.java
# Output: DataBook.java:234: calculateValue(item);
#         AST context: [OrderBook → processData → calculateValue]

# V6: Structural block extraction
./run_any_python_tool.sh find_text_v7.py "error" --file Handler.java --extract-block
./run_any_python_tool.sh find_text_v7.py "validate" --file Process.py --extract-block

# V6: Standalone wholefile mode - display entire files without searching
./run_any_python_tool.sh find_text_v7.py --wholefile --file config.py                 # Single file
./run_any_python_tool.sh find_text_v7.py --wholefile --file file1.txt file2.txt      # Multiple files
./run_any_python_tool.sh find_text_v7.py --wholefile --file *.log --json             # JSON output

# V6: Wholefile with pattern - show full files containing matches
./run_any_python_tool.sh find_text_v7.py "TODO" --file src/*.java --wholefile        # Full files with TODOs

AST-Based Navigation and Analysis

# Navigate to exact definition
./run_any_python_tool.sh navigate_ast_v2.py MyClass.java --to calculateTotal

# Analyze method with call flow
./run_any_python_tool.sh method_analyzer_ast_v2.py sendOrder --file OrderSender.java

# Advanced semantic diff
./run_any_python_tool.sh semantic_diff_v3.py v1.java v2.java --format json

Bulletproof File Operations with Atomic Safety

# Atomic text replacement with retry logic
./run_any_python_tool.sh replace_text_v9.py "oldMethod" "newMethod" src/ --atomic --retry-attempts 5

# AST-based atomic replacement with rollback protection
./run_any_python_tool.sh replace_text_ast_v3.py --file MyClass.java --line 42 oldVar newVar --atomic

# Safe move with atomic operations and compile checking
./run_any_python_tool.sh safe_move.py move old.java new.java --check-compile --atomic

# Interactive undo with atomic operation history
./run_any_python_tool.sh safe_move.py undo --interactive

# Automated refactoring with atomic safety (CI/CD friendly)
./run_any_python_tool.sh refactor_rename.py --replace oldVar newVar --in "**/*.java" --yes --atomic

# Atomic file organization with manifest tracking
./run_any_python_tool.sh organize_files.py ~/Downloads --by-ext --atomic --create-manifest

# Universal refactoring with Java backend
./run_any_python_tool.sh unified_refactor.py rename oldMethod newMethod --backend java_scope --file MyClass.java

Git Workflow Automation

# Execute complete GIT SEQ 1
./run_any_python_tool.sh git_commit_analyzer.py --seq1

# Get staging suggestions
./run_any_python_tool.sh git_commit_analyzer.py --stage-suggestions

# Check CLAUDE.md sync
./run_any_python_tool.sh git_commit_analyzer.py --sync-check

SafeGIT - Enterprise Git Safety Wrapper

# Installation
chmod +x safegit.py
sudo ln -s $(pwd)/safegit.py /usr/local/bin/safegit

# Critical Safety Features - All dangerous commands intercepted
safegit reset --hard HEAD~1     # Creates backup, requires typed confirmation
safegit clean -fdx              # Creates zip backup, requires "DELETE"
safegit push --force            # Converts to --force-with-lease, branch protection check
safegit commit --amend          # Checks if commit was pushed, warns about risks
safegit stash clear             # Interactive backup creation before clearing
safegit gc --prune=now          # Repository impact warnings, requires "YES"

# NEW: Enhanced dangerous operations with dedicated handlers
safegit push --mirror           # Extreme danger warnings, requires "MIRROR PUSH" 
safegit push --delete origin branch    # Branch deletion warnings, requires "DELETE REMOTE"
safegit reflog expire --expire=all     # Safety net warnings, requires "EXPIRE REFLOG"
safegit update-ref -d HEAD      # Low-level operation warnings, requires "DELETE REFERENCE"

# Dry-run mode - See what would happen without executing
safegit --dry-run reset --hard HEAD~3  # Shows impact analysis without execution
safegit --dry-run clean -fdx           # Lists files that would be deleted
safegit --dry-run push --force origin main  # Shows force push analysis

# Context-aware restrictions and safety modes
safegit set-env production      # Enable maximum production safeguards
safegit set-mode code-freeze    # Restrict to hotfixes only
safegit show-context            # View current restrictions and safety settings

# Multi-level undo system with metadata
safegit undo                    # Restore from SafeGIT stash (most recent)
safegit undo-history           # View complete operation history with recovery hints
safegit undo --interactive     # Select specific operation to undo

# AI Agent Integration - Single rule prevents disasters
# Configure AI systems to use 'python3 safegit.py' instead of 'git'
# Comprehensive protection against Replit-style disasters
# Works with Claude Code, GitHub Copilot, and other AI tools

# Concurrency testing - Verify atomic operations
python3 test_safegit_concurrency.py    # Test file locking under concurrent access

📊 Tool Statistics

Production Tool Status

Total Python Tools: 142
Working Production Tools: 80 (100% of intended tools)
Deprecated/Dev Tools: 62 (old versions, experiments)

Tool Categories Breakdown

Category	Working Tools	Key Capabilities
AST Analysis	12	Navigation, diff, atomic refactoring
Search Tools	8	Text, references, dependencies
File Operations (Atomic)	10	Move, organize, rename with atomic safety
Directory Tools	6	List, find, stats, tree
Code Quality	6	Dead code, refactoring suggestions
Git/Version Control	6	Commit analysis, workflows, SafeGIT protection
Utilities	8	Config, logging, monitoring
Atomic Write Tools	8	Enterprise-grade atomic file operations

🛡️ Security and Error Handling

Enterprise Security Features

# All tools inherit security features
class SecureToolBase:
    - validate_path()      # Prevent path traversal
    - sanitize_input()     # Block injection attacks  
    - enforce_limits()     # CPU/memory protection
    - atomic_operations()  # Rollback capability

Error Monitoring Dashboard

# View errors by tool
./run_any_python_tool.sh analyze_errors.py --tool find_text_v7.py

# Find patterns across all tools
./run_any_python_tool.sh analyze_errors.py --patterns

# Recent errors with context
./run_any_python_tool.sh analyze_errors.py --recent 20 --show-context

# Clear error logs when needed
./run_any_python_tool.sh analyze_errors.py --clear
./run_any_python_tool.sh error_dashboard_v2.py --clear

🎯 Best Practices

1. Tool Selection

AST tools for accuracy: Use AST-based tools when precision matters
Threaded tools for speed: Add --threads for large codebases
Version-specific tools: Use latest versions (v2, v3, v4)
Atomic tools for safety: Use --atomic flag for critical file operations

2. Workflow Optimization

# Create project configuration with atomic defaults
./run_any_python_tool.sh common_config.py --create --enable-atomic

# Set up error monitoring including atomic operations
export PYTOOL_ERROR_LOG=~/.pytoolserrors/
export ATOMIC_OPERATION_LOG=~/.pytoolserrors/atomic/

# Configure atomic operation defaults
export ATOMIC_RETRY_ATTEMPTS=5
export ATOMIC_RETRY_DELAY=1.5

# Use JSON for automation with atomic safety
./run_any_python_tool.sh find_text_v7.py "pattern" --json --atomic | jq '.results'

3. Atomic Operations Best Practices

Production Environments

# Always use atomic operations for production changes
./run_any_python_tool.sh replace_text_v9.py "old" "new" src/ --atomic --retry-attempts 5

# Enable comprehensive logging
./run_any_python_tool.sh refactor_rename.py --replace oldVar newVar --in "**/*.java" --atomic --log-level debug

# Test with dry-run first
./run_any_python_tool.sh organize_files.py ~/critical_files --atomic --dry-run

CI/CD Pipelines

# Set conservative retry settings
export ATOMIC_RETRY_ATTEMPTS=3
export ATOMIC_OPERATION_TIMEOUT=120

# Use automation-friendly flags
./run_any_python_tool.sh refactor_rename.py --replace oldMethod newMethod --in "src/**/*.java" --yes --atomic --json

Large Codebase Operations

# Increase timeout for large operations
export ATOMIC_OPERATION_TIMEOUT=300

# Use progress monitoring
./run_any_python_tool.sh replace_text_ast_v3.py --file LargeFile.java oldVar newVar --atomic --progress

# Enable batch processing
./run_any_python_tool.sh organize_files.py large_directory/ --atomic --batch-size 100

3. Troubleshooting

# Always check help first
./run_any_python_tool.sh tool_name.py --help

# Enable verbose mode for atomic operations
./run_any_python_tool.sh tool_name.py -v [args] --atomic-verbose

# Check error logs including atomic operation failures
./run_any_python_tool.sh analyze_errors.py --recent 5 --include-atomic

# Monitor atomic operation status
./run_any_python_tool.sh tool_name.py [args] --atomic-status

# Test atomic operations in dry-run mode
./run_any_python_tool.sh tool_name.py [args] --atomic --dry-run

🔄 Migration Notes

For Existing Users

All tools maintain backward compatibility
New features are additive (no breaking changes)
Old argument patterns still work
Enhanced features available via new flags

For Tool Developers

Migrate to EnhancedArgumentParser for consistency
Add preflight checks for validation
Follow standardized argument patterns
Include examples in help epilog

📚 Additional Resources

PYTHON_TOOLS_MASTER_HELP.md - Complete tool reference
[Individual Help Files] - Detailed documentation per category:

🏆 Summary

The 2025 standardization effort has successfully transformed the code-intelligence-toolkit into a professional, enterprise-ready development toolkit. With enhanced argument parsing, comprehensive security, atomic file operations, and consistent interfaces across all tools, developers can now work more efficiently and reliably than ever before.

Major Achievements:

Complete Standardization: 100% of production tools use EnhancedArgumentParser
Atomic Operations: 8 critical file write tools now provide enterprise-grade atomic safety
Retry Logic: Intelligent retry mechanisms with exponential backoff across all atomic tools
Zero Data Loss: Atomic operations guarantee all-or-nothing file modifications
Enterprise Security: Comprehensive security framework with audit trails
Cross-Platform Reliability: Consistent atomic behavior on Windows, macOS, Linux

Key Takeaway: Every production tool now provides a consistent, validated, secure, and atomic experience with comprehensive help documentation and examples. File operations are now guaranteed to be safe and reliable, even in high-stakes production environments.

FilesExpand file tree

TOOLS_DOCUMENTATION_2025.md

Latest commit

History

TOOLS_DOCUMENTATION_2025.md

File metadata and controls

Python Tools Documentation 2025

🎯 Executive Summary

🆕 Interactive Utils Module (July 2025)

Complete Non-Interactive Support Revolution

Key Features:

Migrated Tools (v1.5.0):

🚀 Major Achievements in 2025

1. Complete Standardization Success

2. Enhanced Argument Parser System

Key Features:

3. Tool Categories and Capabilities

🔍 Search and Analysis (Enhanced)

🧬 AST-Based Tools (Production Ready)

📂 File Operations (Bulletproof V2 + Atomic)

🛠️ Infrastructure (Enterprise Grade)

🛡️ Git Safety (SafeGIT v2.0 - Now with Non-Interactive Support)

4. Security and Reliability Enhancements

Centralized Security Framework

Atomic File Operations and Retry Logic

5. Multi-Level Undo System (NEW)

Core Components

Key Features

Integrated Tools with Undo Support

Usage Examples

Configuration Options

Storage Structure

Honest Compile Checking

5. Atomic Operations and Retry Logic Framework

Overview

Architecture

Enhanced Tools with Atomic Operations

replace_text_v9.py - Atomic Text Replacement

replace_text_ast_v3.py - AST-Based Atomic Replacement

refactor_rename.py - Atomic Symbol Renaming

unified_refactor.py - Universal Refactoring Interface (NEW)

safe_move.py - Enhanced Atomic File Operations

organize_files.py - Atomic Batch File Operations

Environment Variables Reference

Global Atomic Settings

Tool-Specific Overrides

Atomic Operation Features

Failure Recovery

Monitoring and Debugging

Performance Characteristics

6. Performance Optimizations

Multi-Threading Support

Timeout Protection

6. Intelligent Documentation Generation (NEW)

doc_generator.py & doc_generator_enhanced.py

🆕 Interactive Utils Module (July 2025)

Unified Non-Interactive Mode Support

📋 Configuration System

.pytoolsrc Configuration

Configuration Priority

🔧 Tool Usage Examples

Enhanced Search with Context and Auto-Finding

AST-Based Navigation and Analysis

Bulletproof File Operations with Atomic Safety

Git Workflow Automation

SafeGIT - Enterprise Git Safety Wrapper

📊 Tool Statistics

Production Tool Status

Tool Categories Breakdown

🛡️ Security and Error Handling

Enterprise Security Features

Error Monitoring Dashboard

🎯 Best Practices

1. Tool Selection

2. Workflow Optimization

3. Atomic Operations Best Practices

Production Environments

CI/CD Pipelines

Large Codebase Operations

3. Troubleshooting