Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Dependencies
node_modules/
package-lock.json
yarn.lock

# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*

# Runtime data
pids
*.pid
*.seed
*.pid.lock

# Coverage directory
coverage/
.nyc_output/

# Editor directories and files
.vscode/
.idea/
*.swp
*.swo
*~
.DS_Store

# Temporary files
tmp/
temp/
*.tmp

# Build output
dist/
build/
160 changes: 159 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1 +1,159 @@
# testRepo
# testRepo

Job Handler with Deduplication Support

## Overview

This repository provides a `handleSendToJob` function that handles sending jobs to a processing queue with built-in deduplication support. It prevents duplicate jobs from being processed by tracking job IDs.

## Features

- **Deduplication**: Automatically prevents duplicate jobs from being sent based on job ID
- **Error Handling**: Comprehensive error handling for invalid inputs and send failures
- **Tracking**: Utilities to check processed job count and status
- **Simple API**: Easy-to-use function interface with clear return values

## Installation

```bash
npm install
```

## Usage

### Basic Usage

```javascript
const { handleSendToJob } = require('./jobHandler');

// Define a function that sends jobs to your queue/API
function sendToQueue(job) {
// Your implementation here (e.g., HTTP request, queue push, etc.)
console.log('Sending job:', job.id);
return { success: true };
}

// Send a job
const result = handleSendToJob(
{ id: 'job123', data: { task: 'process-data' } },
sendToQueue
);

console.log(result);
// Output: { success: true, status: 'sent', message: 'Job job123 sent successfully', jobId: 'job123', result: {...} }
```

### Deduplication in Action

```javascript
const { handleSendToJob } = require('./jobHandler');

function sendToQueue(job) {
console.log('Processing job:', job.id);
return { processed: true };
}

// First attempt - succeeds
const result1 = handleSendToJob({ id: 'job123', data: 'test' }, sendToQueue);
console.log(result1.success); // true
console.log(result1.status); // 'sent'

// Second attempt with same ID - fails (duplicate)
const result2 = handleSendToJob({ id: 'job123', data: 'test' }, sendToQueue);
console.log(result2.success); // false
console.log(result2.status); // 'duplicate'
console.log(result2.message); // 'Job with ID job123 has already been processed'
```

### Utility Functions

```javascript
const {
handleSendToJob,
clearProcessedJobs,
getProcessedJobCount,
hasJobBeenProcessed,
} = require('./jobHandler');

// Check if a job has been processed
if (hasJobBeenProcessed('job123')) {
console.log('Job already processed');
}

// Get count of processed jobs
console.log('Processed jobs:', getProcessedJobCount());

// Clear the deduplication cache (useful for testing or reset)
clearProcessedJobs();
```

## API Reference

### `handleSendToJob(job, sendFunction)`

Sends a job to the processing queue with deduplication.

**Parameters:**
- `job` (Object): The job object to process
- `id` (string, required): Unique identifier for the job
- `data` (any): Job data/payload
- `sendFunction` (Function): Function to send the job (e.g., to queue/API)

**Returns:**
- Object with the following properties:
- `success` (boolean): Whether the operation succeeded
- `status` (string): Status code ('sent', 'duplicate', or 'error')
- `message` (string): Human-readable message
- `jobId` (string): The job ID
- `result` (any): Result from the send function (if successful)
- `error` (Error): Error object (if failed)

### `clearProcessedJobs()`

Clears the deduplication cache. Useful for testing or when you want to reset tracking.

### `getProcessedJobCount()`

Returns the number of jobs that have been processed.

**Returns:** `number`

### `hasJobBeenProcessed(jobId)`

Checks if a job ID has been processed.

**Parameters:**
- `jobId` (string): The job ID to check

**Returns:** `boolean`

## Testing

Run the test suite:

```bash
npm test
```

## Implementation Details

The deduplication mechanism uses a `Set` to track processed job IDs in memory. This provides:
- O(1) lookup time for checking duplicates
- Efficient memory usage
- Simple and reliable deduplication

**Note:** The deduplication cache is stored in memory and will be cleared when the process restarts. For persistent deduplication across restarts, consider using a database or external cache (Redis, etc.).

## Error Handling

The function handles various error scenarios:
- Invalid job object (null, undefined, or not an object)
- Missing job ID
- Missing or invalid send function
- Exceptions thrown by the send function

All errors return a structured response with `success: false` and descriptive error messages.

## License

ISC
65 changes: 65 additions & 0 deletions demo.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
/**
* Visual demonstration of deduplication in action
*/

const {
handleSendToJob,
clearProcessedJobs,
getProcessedJobCount,
} = require('./jobHandler');

// Mock queue sender
function sendToQueue(job) {
return { queued: true, timestamp: Date.now() };
}

console.log('\n╔════════════════════════════════════════════════════════════╗');
console.log('β•‘ handleSendToJob - Deduplication Demonstration β•‘');
console.log('β•šβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•\n');

clearProcessedJobs();

// Simulate receiving jobs (some duplicates)
const incomingJobs = [
{ id: 'order-101', data: { amount: 150.00 } },
{ id: 'order-102', data: { amount: 200.00 } },
{ id: 'order-101', data: { amount: 150.00 } }, // DUPLICATE
{ id: 'order-103', data: { amount: 75.50 } },
{ id: 'order-102', data: { amount: 200.00 } }, // DUPLICATE
{ id: 'order-104', data: { amount: 300.00 } },
{ id: 'order-101', data: { amount: 150.00 } }, // DUPLICATE
{ id: 'order-105', data: { amount: 125.00 } },
];

console.log('Processing incoming jobs...\n');

const stats = {
sent: 0,
duplicates: 0,
errors: 0,
};

incomingJobs.forEach((job, index) => {
const result = handleSendToJob(job, sendToQueue);

const icon = result.success ? 'βœ“' : 'βœ—';
const statusColor = result.status === 'sent' ? '🟒' :
result.status === 'duplicate' ? '🟑' : 'πŸ”΄';

console.log(`${index + 1}. ${statusColor} Job ${job.id}: ${icon} ${result.status.toUpperCase()}`);

if (result.status === 'sent') stats.sent++;
else if (result.status === 'duplicate') stats.duplicates++;
else stats.errors++;
});

console.log('\n' + '─'.repeat(60));
console.log('\nπŸ“Š Summary:');
console.log(` Total jobs attempted: ${incomingJobs.length}`);
console.log(` Successfully sent: ${stats.sent}`);
console.log(` Duplicates blocked: ${stats.duplicates}`);
console.log(` Errors: ${stats.errors}`);
console.log(` Unique jobs tracked: ${getProcessedJobCount()}`);

console.log('\nπŸ’‘ Deduplication prevented ' + stats.duplicates + ' duplicate job(s) from being processed!');
console.log('\n' + '═'.repeat(60) + '\n');
Loading