Email Queue Extension - Developer Documentation

Architecture Overview

The Email Queue Extension is built using a modular architecture that separates concerns and provides extensibility:

CRM/Emailqueue/
├── BAO/           # Business Access Objects (data layer)
├── Form/          # Form controllers (presentation layer)
├── Page/          # Page controllers (presentation layer)
├── Mailer/        # Custom mailer implementations
├── Utils/         # Utility classes
└── Config.php     # Configuration management

api/v3/            # API endpoints
templates/         # Smarty templates
xml/              # Menu and route definitions
tests/            # Unit and integration tests
scripts/          # Deployment and maintenance scripts

Core Components

1. Queue Mailer (CRM_Emailqueue_Mailer_QueueMailer)

The core component that intercepts emails and queues them instead of sending immediately.

Key Methods:

send() - Queues email instead of sending
parseRecipients() - Handles various recipient formats
getHtmlBody() / getTextBody() - Extracts email content

2. Queue BAO (CRM_Emailqueue_BAO_Queue)

Handles all database operations for the email queue.

Key Methods:

addToQueue() - Add email to queue
searchEmails() - Advanced search functionality
getEmailPreview() - Get email for preview
processQueue() - Process queued emails
bulkAction() - Bulk operations

3. Configuration (CRM_Emailqueue_Config)

Centralized configuration management with constants and helper methods.

Features:

Default settings management
Validation functions
Environment-specific configuration

4. Error Handling (CRM_Emailqueue_Utils_ErrorHandler)

Comprehensive error handling and logging system.

Features:

Multiple log levels
Database logging
Critical error alerts
Exception handling

Database Schema

email_queue Table

CREATE TABLE email_queue (
  id INT AUTO_INCREMENT PRIMARY KEY,
  to_email VARCHAR(255) NOT NULL,
  subject TEXT,
  from_email VARCHAR(255),
  reply_to VARCHAR(255),
  cc TEXT,
  bcc TEXT,
  body_html LONGTEXT,
  body_text LONGTEXT,
  headers TEXT,
  created_date DATETIME NOT NULL,
  scheduled_date DATETIME NULL,
  sent_date DATETIME NULL,
  status ENUM('pending', 'processing', 'sent', 'failed', 'cancelled') DEFAULT 'pending',
  priority TINYINT DEFAULT 3,
  retry_count INT DEFAULT 0,
  max_retries INT DEFAULT 3,
  error_message TEXT,
  validation_score TINYINT UNSIGNED NULL,
  validation_warnings TEXT NULL,
  tracking_code VARCHAR(64) NULL,

  INDEX idx_status (status),
  INDEX idx_scheduled (scheduled_date),
  INDEX idx_priority (priority),
  INDEX idx_created (created_date),
  INDEX idx_status_priority_created (status, priority, created_date),
  INDEX idx_validation_score (validation_score),
  INDEX idx_tracking_code (tracking_code)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

email_queue_log Table

CREATE TABLE email_queue_log (
  id INT AUTO_INCREMENT PRIMARY KEY,
  queue_id INT NOT NULL,
  action VARCHAR(50) NOT NULL,
  message TEXT,
  created_date DATETIME NOT NULL,

  FOREIGN KEY (queue_id) REFERENCES email_queue(id) ON DELETE CASCADE,
  INDEX idx_queue_id (queue_id),
  INDEX idx_action (action),
  INDEX idx_created (created_date)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

API Reference

Core APIs

Search Emails

$result = civicrm_api3('Emailqueue', 'search', [
  'to_email' => 'user@example.com',
  'status' => ['pending', 'failed'],
  'date_from' => '2025-07-01',
  'limit' => 50,
  'order_by' => 'created_date',
  'order_dir' => 'DESC'
]);

Preview Email

$email = civicrm_api3('Emailqueue', 'preview', [
  'id' => 12345
]);

Bulk Actions

$result = civicrm_api3('Emailqueue', 'bulkaction', [
  'action' => 'cancel',
  'email_ids' => [123, 124, 125]
]);

Administrative APIs

Health Check

$health = civicrm_api3('EmailqueueAdmin', 'healthcheck');

System Metrics

$metrics = civicrm_api3('EmailqueueAdmin', 'getmetrics');

Cleanup Operations

$result = civicrm_api3('EmailqueueAdmin', 'cleanup', [
  'sent_retention_days' => 90,
  'cleanup_failed' => true
]);

Hooks and Extension Points

Email Processing Hooks

/**
 * Modify email before queuing
 */
function mymodule_civicrm_emailqueue_pre_queue(&$emailData) {
  // Modify email data before queuing
  $emailData['priority'] = 1; // High priority
}

/**
 * Process email after queuing
 */
function mymodule_civicrm_emailqueue_post_queue($emailId, $emailData) {
  // Log or track queued email
}

/**
 * Modify email before sending
 */
function mymodule_civicrm_emailqueue_pre_send(&$emailData) {
  // Last-minute modifications before sending
  return true; // Return false to cancel sending
}

/**
 * Process email after sending
 */
function mymodule_civicrm_emailqueue_post_send($emailId, $result) {
  // Track sent email or update external systems
}

Custom Validation

/**
 * Add custom email validation
 */
function mymodule_civicrm_emailqueue_validate($email, &$validation) {
  // Add custom validation logic
  if (strpos($email, 'blocked-domain.com') !== false) {
    $validation['errors'][] = 'Domain is blocked';
  }
}

Custom Mailer

/**
 * Provide custom mailer for specific emails
 */
function mymodule_civicrm_emailqueue_get_mailer($emailData, &$mailer) {
  if ($emailData['priority'] == 1) {
    // Use premium mailer for high priority
    $mailer = new MyCustomMailer();
  }
}

Development Setup

Prerequisites

PHP 7.2+
MySQL 5.7+ or MariaDB 10.3+
CiviCRM 5.50+
PDO MySQL extension
CV (CiviCRM CLI tool) - recommended

Installation for Development

Clone Repository

cd /path/to/civicrm/ext
git clone [repository-url] com.yourorg.emailqueue

Install Dependencies

cd com.yourorg.emailqueue
composer install --dev

Setup Test Database

mysql -u root -p
CREATE DATABASE emailqueue_test;
GRANT ALL ON emailqueue_test.* TO 'civicrm_test'@'localhost';

Configure for Testing

cp config/samples/database.conf.sample config/database.test.conf
# Edit test configuration

Install Extension

cv ext:install com.yourorg.emailqueue

Running Tests

# Unit tests
phpunit tests/phpunit/

# Integration tests
phpunit tests/integration/

# API tests
phpunit tests/api/

# All tests
phpunit

Code Standards

Follow CiviCRM coding standards:

# Check code standards
phpcs --standard=CiviCRM src/

# Fix code standards
phpcbf --standard=CiviCRM src/

Debugging

Enable debug mode:

// In civicrm.settings.php or via API
Civi::settings()->set('emailqueue_log_level', 'debug');

Check logs:

CiviCRM logs: civicrm/ConfigAndLog/
Extension logs: Database email_queue_log table
Error logs: Server error logs

Testing Framework

Unit Test Example

class CRM_Emailqueue_BAO_QueueTest extends CiviUnitTestCase {

  public function testAddToQueue() {
    $emailData = [
      'to_email' => 'test@example.com',
      'subject' => 'Test Email',
      'body_html' => '<p>Test content</p>',
      'status' => 'pending',
      'priority' => 3
    ];

    $queueId = CRM_Emailqueue_BAO_Queue::addToQueue($emailData);

    $this->assertIsInt($queueId);
    $this->assertGreaterThan(0, $queueId);
  }
}

Integration Test Example

class CRM_Emailqueue_IntegrationTest extends CiviIntegrationTestCase {

  public function testEmailQueueing() {
    // Enable extension
    Civi::settings()->set('emailqueue_enabled', true);

    // Send test email
    $result = civicrm_api3('Email', 'send', [
      'to' => 'test@example.com',
      'subject' => 'Test Email',
      'html' => '<p>Test content</p>'
    ]);

    // Verify email was queued
    $queued = civicrm_api3('Emailqueue', 'search', [
      'to_email' => 'test@example.com',
      'status' => 'pending'
    ]);

    $this->assertEquals(1, $queued['count']);
  }
}

Performance Optimization

Database Optimization

Indexes

Composite indexes for common queries
Regular index maintenance
Query optimization

Partitioning (for large datasets)

ALTER TABLE email_queue PARTITION BY RANGE (YEAR(created_date)) (
  PARTITION p2024 VALUES LESS THAN (2025),
  PARTITION p2025 VALUES LESS THAN (2026),
  PARTITION p_future VALUES LESS THAN MAXVALUE
);

Archiving

Move old emails to archive tables
Implement data retention policies
Regular cleanup procedures

Application Optimization

Batch Processing

Optimize batch sizes
Implement efficient querying
Use prepared statements

Caching

Cache configuration settings
Cache filter options
Use Redis/Memcached for session data

Connection Pooling

Reuse database connections
Implement connection pooling
Monitor connection usage

Security Considerations

Data Protection

Email Content

Encrypt sensitive content
Sanitize HTML content
Validate email addresses

Database Security

Use dedicated database user
Limit database permissions
Enable SSL connections

Access Control

Implement role-based access
Log all administrative actions
Monitor for suspicious activity

Input Validation

// Example validation
function validateEmailData($data) {
  $errors = [];

  // Validate email address
  if (!filter_var($data['to_email'], FILTER_VALIDATE_EMAIL)) {
    $errors[] = 'Invalid email address';
  }

  // Check email size
  $size = strlen($data['body_html']) + strlen($data['body_text']);
  if ($size > CRM_Emailqueue_Config::getEmailSizeLimit()) {
    $errors[] = 'Email too large';
  }

  // Sanitize HTML content
  $data['body_html'] = CRM_Emailqueue_Utils_Email::sanitizeHtmlContent($data['body_html']);

  return ['valid' => empty($errors), 'errors' => $errors, 'data' => $data];
}

Deployment Strategies

Production Deployment

Blue-Green Deployment

Maintain two identical environments
Switch traffic after testing
Quick rollback capability

Rolling Updates

Update servers one by one
Monitor during deployment
Gradual traffic migration

Database Migrations

Test migrations on staging
Backup before deployment
Monitor performance impact

Monitoring

Application Monitoring

Queue size monitoring
Processing rate tracking
Error rate monitoring

Infrastructure Monitoring

Database performance
Server resources
Network connectivity

Alerting

Critical error alerts
Performance degradation
Capacity warnings

Contributing

Development Workflow

Feature Development

Create feature branch
Write tests first (TDD)
Implement feature
Update documentation

Code Review

Submit pull request
Peer review process
Address feedback
Final approval

Release Process

Version bumping
Changelog updates
Tag releases
Deploy to production

Guidelines

Follow PSR-12 coding standards
Write comprehensive tests
Document all public APIs
Maintain backward compatibility
Update user documentation

Troubleshooting

Common Issues

Database Connection Issues

Check credentials
Verify network connectivity
Check firewall settings
Review SSL configuration

Performance Issues

Analyze slow queries
Check index usage
Monitor resource usage
Optimize batch sizes

Email Delivery Issues

Verify SMTP configuration
Check email content
Review bounce handling
Monitor sending limits

Debug Tools

# Check system status
cv api3 EmailqueueAdmin.getstatus

# Run health check
cv api3 EmailqueueAdmin.healthcheck

# Get error logs
cv api3 EmailqueueAdmin.geterrorlogs limit=20

# Test system
cv api3 EmailqueueAdmin.testsystem

Resources

License

This extension is licensed under AGPL-3.0. See LICENSE file for details.

FilesExpand file tree

Developer.md

Latest commit

History