Chaitanya - Mailchimp Replacement - Email Announcement System #1945

csk731 · 2025-12-07T05:14:36Z

Description

This PR implements a comprehensive Email Announcement System for One Community to create, manage, and send email announcements to HGN users and external subscribers. The system includes reusable email templates with variable substitution, batch processing with retry logic, email outbox management.

Related PRs (if any):

To test this backend PR you need to checkout the #4492 frontend PR.

Frontend PR: OneCommunityGlobal/HighestGoodNetworkApp#4492

Main changes explained:

1. Email Template Management System

Created EmailTemplate model (src/models/emailTemplate.js) - Stores reusable email templates with:
- Template name (unique, case-insensitive)
- Subject and HTML content with variable placeholders ({{variableName}})
- Declared variables with types (text, url, number, textarea, image)
- Creator/updater tracking and timestamps
Created EmailTemplateService (src/services/announcements/emails/emailTemplateService.js) - Provides:
- Template CRUD operations with validation
- Variable validation and usage checking
- Template rendering with variable substitution
- HTML sanitization for security
- Preview functionality with test variable values
Created EmailTemplateController (src/controllers/emailTemplateController.js) - REST API endpoints:
- GET /api/email-templates - List all templates with search/sort
- GET /api/email-templates/:id - Get template by ID
- POST /api/email-templates - Create new template
- PUT /api/email-templates/:id - Update template
- DELETE /api/email-templates/:id - Delete template
- POST /api/email-templates/:id/preview - Preview template with variables

2. Email Announcement System

Created Email model (src/models/email.js) - Parent record tracking:
- Email subject and HTML content
- Status lifecycle (PENDING → SENDING → SENT/PROCESSED/FAILED)
- Creator tracking and timing fields
Created EmailBatch model (src/models/emailBatch.js) - Child records for batch processing:
- Groups recipients into batches (100 recipients per batch by default)
- Tracks status per batch (PENDING → SENDING → SENT/FAILED)
- Records attempt counts, error details, and send responses
- Supports TO, CC, and BCC email types
Created EmailService (src/services/announcements/emails/emailService.js) - Parent email management:
- Email creation with validation
- Status management and tracking
- Stuck email recovery on server restart
Created EmailBatchService (src/services/announcements/emails/emailBatchService.js) - Batch processing:
- Creates email batches by chunking recipients
- Validates recipient limits (max 2000 per request)
- Manages batch status transitions
- Syncs parent email status based on batch statuses
- Handles retry logic for failed batches

3. Email Sending Infrastructure

Created EmailSendingService (src/services/announcements/emails/emailSendingService.js) - SMTP integration:
- Gmail API integration using OAuth2 authentication
- Automatic token refresh for each send
- Exponential backoff retry logic (3 attempts by default)
- Comprehensive error handling and validation
- Configurable sender name from environment variable (ANNOUNCEMENT_EMAIL_SENDER_NAME)
Created EmailProcessor (src/services/announcements/emails/emailProcessor.js) - Queue-based processing:
- In-memory queue for sequential email processing
- Processes emails asynchronously without blocking requests
- Concurrent batch processing with rate limiting
- Configurable concurrency (3 batches) and delays
- Automatic recovery of stuck emails/batches on startup
- Sequential processing to prevent overwhelming Gmail API

4. Email Management Controllers

Created EmailController (src/controllers/emailController.js) - Core email operations:
- POST /api/send-emails - Send email to specific recipients (max 2000)
- POST /api/broadcast-emails - Broadcast to all active HGN users and confirmed subscribers
- POST /api/resend-email - Resend previously sent email
  - Options: 'all', 'specific', or 'same' recipients
- POST /api/retry-email/:emailId - Retry failed email batches
- POST /api/process-pending-and-stuck-emails - Manual trigger for processing
- Subscription management endpoints for HGN users and non-HGN subscribers
Created EmailOutboxController (src/controllers/emailOutboxController.js) - Outbox view:
- GET /api/emails - List all sent emails
- GET /api/emails/:emailId - Get email details with batch status

5. Configuration and Validation

Created emailConfig.js (src/config/emailConfig.js) - Centralized configuration:
- Email status enums (PENDING, SENDING, SENT, PROCESSED, FAILED)
- Batch status enums
- Configurable limits (recipients, HTML size, subject length)
- Batch processing settings (batch size, concurrency, delays)
- Retry configuration
Created email validation utilities - Validates:
- Email address format
- HTML content size limits (1MB max)
- Subject length limits (200 characters)
- Template variable usage
- Recipient count limits

Key Features:

Reusable Email Templates: Create templates with variables that can be reused for multiple sends
Batch Processing: Automatically splits large recipient lists into manageable batches
Rate Limiting: Configurable delays and concurrency to respect Gmail API limits
Retry Logic: Automatic retry with exponential backoff for failed sends
Status Tracking: Comprehensive status tracking at both email and batch levels
Resend Functionality: Resend emails to different audiences
Retry Functionality: Retry failed email batches separately from resending
Subscription Management: Manage email subscriptions for both HGN users and external subscribers
Stuck Email Recovery: Automatic recovery of emails stuck in SENDING status on server restart

How to test:

Check into current branch
Run npm install to install dependencies
Ensure required environment variables are set:
- ANNOUNCEMENT_EMAIL - Gmail account email for sending
- ANNOUNCEMENT_EMAIL_SENDER_NAME - Sender display name
- ANNOUNCEMENT_EMAIL_CLIENT_ID - OAuth2 client ID
- ANNOUNCEMENT_EMAIL_CLIENT_SECRET - OAuth2 client secret
- ANNOUNCEMENT_EMAIL_CLIENT_REDIRECT_URI - OAuth2 redirect URI
- ANNOUNCEMENT_EMAIL_REFRESH_TOKEN - OAuth2 refresh token
Run npm run dev to start the backend server
Follow frontend PR steps to test the email announcement feature

Screenshots or videos of changes:

(Screenshots/videos would be provided by frontend PR #4492)

Note:

Follow this video to setup Gmail API with OAuth2 : https://www.youtube.com/watch?v=-rcRf7yswfM
The system uses Gmail API with OAuth2 authentication. Ensure all OAuth2 credentials are properly configured.
Email processing is asynchronous and queue-based to prevent blocking HTTP requests.

…hments before sending email refactor(emailController): enhance email handling and subscription logic; improve error handling and HTML formatting feat(emailController): enhance email sending logic to support multiple recipients with BCC; improve response messages feat(routes): add email template router to API for enhanced email management feat(emailTemplateController): enhance getAllEmailTemplates with pagination, sorting, and field selection; add updated_by field to model feat(emailController, emailTemplateController): implement email batch processing for sending and managing emails feat(emailBatch): add retry functionality for failed batch items and update routes

…nce email batch auditing features - Added email announcement job processor to server initialization. - Introduced new audit trail endpoints in emailBatchController for tracking email and batch actions. - Updated email batch item status logic to reflect QUEUED state instead of PENDING. - Removed deprecated emailBatchDashboardController and related routes. - Enhanced email batch model to support comprehensive auditing and recipient management.

- Added EmailBatch and Email models for managing email batches and their recipients. - Introduced EmailBatchService and EmailService for handling batch creation, status updates, and email sending. - Implemented EmailProcessor for processing email batches with retry logic and status tracking. - Enhanced email announcement job processor to handle batch processing at scheduled intervals. - Updated email validation utilities for improved recipient handling and HTML content validation. - Added new endpoints in emailBatchController for managing email records and batch operations.

- Added permission checks for viewing emails, email details, and email audits in emailBatchController. - Implemented permission validation for sending emails, resending, and managing email subscriptions in emailController. - Updated emailTemplateController to enforce permission checks for creating, updating, and deleting email templates. - Refactored routes to replace deprecated sendEmailToAll with sendEmailToSubscribers. - Enhanced test cases to ensure proper handling of permission-related responses.

…zation - Removed redundant cron job initialization and logging from EmailAnnouncementJobProcessor. - Updated EmailProcessor to include additional final email status check for PROCESSED, enhancing email state validation.

…s and models

…emplates

…ions - Deleted the announcementEmailJob and associated configurations from emailJobConfig. - Removed emailBatchController and its routes, streamlining email processing logic. - Updated email models and services to reflect changes in email status management. - Refactored emailController to utilize new email processing methods and improved error handling. - Enhanced email validation utilities for better recipient management and HTML content checks.

… tracking

…ling - Added functionality to process pending and stuck emails on server startup after database connection. - Updated email configuration to increase maximum recipients per request and refined batch processing settings. - Refactored email processing to queue emails for non-blocking, sequential handling instead of immediate processing. - Introduced new methods for managing stuck emails and batches, ensuring robust error handling and status updates. - Enhanced email validation utilities for better normalization of email fields.

…ervice - Updated emailController and emailTemplateController to utilize EmailTemplateService for template rendering and variable validation. - Introduced TEMPLATE_VARIABLE_TYPES in emailConfig for better management of variable types. - Removed the deprecated TemplateRenderingService, streamlining the email template handling process.

… management - Removed deprecated templateId from Email model to simplify email structure. - Updated email subscription list schema to enforce lowercase and trim email entries for consistency. - Introduced new methods for retrying failed emails and processing pending emails, improving error handling and user experience. - Enhanced email validation and normalization processes across controllers to ensure consistent data handling. - Refactored routes to improve clarity and organization of email-related endpoints.

- Enhanced validation for email subscriptions, including checks for missing or invalid email fields. - Improved error handling for subscription confirmation and removal processes, ensuring appropriate status codes are returned. - Updated positive test cases to reflect changes in subscription management and email normalization. - Removed deprecated sendEmailToAll functionality, streamlining email operations.

Anusha-Gali

Hi Chaitanya,

I have reviewed the PR locally and have mentioned by comments in the frontend PR: OneCommunityGlobal/HighestGoodNetworkApp#4492 (review) , in case any of them correlate to backend, please do recheck. Thank you

csk731 · 2025-12-08T16:19:39Z

Hi Chaitanya,

I have reviewed the PR locally and have mentioned by comments in the frontend PR: OneCommunityGlobal/HighestGoodNetworkApp#4492 (review) , in case any of them correlate to backend, please do recheck. Thank you

Hi @Anusha-Gali ,

Thanks for taking the time to test the feature. I have gone through your feedback. There is one issue that correlates with the backend - currently, our system do not have the ability to detect emails that are not delivered to recipients, as this needs additional operational complexity. This feature behaves exactly like how we send an email via Gmail/Outlook. If delivery fails, we will be notified later by Gmail, as shown in your screenshots (e.g., “Address not found”). It’s essentially a “spray and pray” approach.

To provide more context, this is how the feature currently works:

User sends an email.
Based on the number of recipients, the email is split into batches.
A job picks up the email and its batches and processes them each at a time.
If a batch is successfully sent to the Gmail APIs, we treat the batch as sent. If the batch fails due to any internal issues, we treat it as failed.
Once all batches are processed, we determine the email status as follows:
- All batches sent -> email successful
- All batches failed -> email failed
- Some batches sent/failed -> email processed

Also, I verified that the retry/resend functionality works as expected from the backend perspective. However, it seems like there may be some misalignment between the frontend API endpoints and the API contract.

VijayAnirudh

A test case is failing! Backend is working fine! Please mention the details that you wanted to include on the env file in the description.

Anusha-Gali

Hi Chaitanya,

I have re-reviewed your PR and left comments on frontend (mainly point 2 - i hope it is what you had explained to me in your previous comment) : OneCommunityGlobal/HighestGoodNetworkApp#4492 (review)

csk731 added 15 commits October 28, 2025 14:24

feat(email): refactor email processing and enhance job management

34a2746

feat(email): enhance documentation and validation in email controller…

28cdf56

…s and models

refactor(email): remove media validation from email controllers and t…

6297b20

…emplates

feat(email): enhance email batch status management with attempt count…

271cb8e

… tracking

feat(email): add sender name configuration and improve logging

5ab8a2d

one-community added the High Priority - Please Review First This is an important PR we'd like to get merged as soon as possible label Dec 7, 2025

one-community mentioned this pull request Dec 7, 2025

Apoorva - Redo of email management UI improvements PR#4670 OneCommunityGlobal/HighestGoodNetworkApp#4492

Open

Anusha-Gali requested changes Dec 8, 2025

View reviewed changes

VijayAnirudh requested changes Dec 12, 2025

View reviewed changes

Anusha-Gali approved these changes Dec 18, 2025

View reviewed changes

one-community mentioned this pull request Jan 7, 2026

Apoorva - Email management UI improvements OneCommunityGlobal/HighestGoodNetworkApp#4670

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Chaitanya - Mailchimp Replacement - Email Announcement System #1945

Chaitanya - Mailchimp Replacement - Email Announcement System #1945

Uh oh!

csk731 commented Dec 7, 2025 •

edited

Loading

Uh oh!

Anusha-Gali left a comment

Uh oh!

csk731 commented Dec 8, 2025

Uh oh!

VijayAnirudh left a comment

Uh oh!

Anusha-Gali left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Chaitanya - Mailchimp Replacement - Email Announcement System #1945

Are you sure you want to change the base?

Chaitanya - Mailchimp Replacement - Email Announcement System #1945

Uh oh!

Conversation

csk731 commented Dec 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related PRs (if any):

Main changes explained:

1. Email Template Management System

2. Email Announcement System

3. Email Sending Infrastructure

4. Email Management Controllers

5. Configuration and Validation

Key Features:

How to test:

Screenshots or videos of changes:

Note:

Uh oh!

Anusha-Gali left a comment

Choose a reason for hiding this comment

Uh oh!

csk731 commented Dec 8, 2025

Uh oh!

VijayAnirudh left a comment

Choose a reason for hiding this comment

Uh oh!

Anusha-Gali left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

csk731 commented Dec 7, 2025 •

edited

Loading