This PR introduces a new batch billing system to streamline and optimize billing operations. Instead of processing each billing operation individually, the system aggregates them into batches for improved efficiency and reduced system load.

Key Changes

Batch Billing Implementation

• New Module:
  • A dedicated batch_billing.ts file has been added to handle all batch billing operations.
• Batch Aggregation:
  • Billing operations are stored in a Redis list.
  • Operations are grouped by team_id, subscription_id, and is_extract.
  • Total credits are summed up for each group, allowing billing to be processed collectively.
• Locking Mechanism:
  • A Redis-based lock ensures that only one batch is processed at a time, preventing concurrent processing conflicts.
• Periodic Processing:
  • A timer (setInterval) periodically checks the Redis queue and triggers batch processing.
  • Immediate processing is also triggered when the number of queued operations reaches the defined batch size.

Queue and Job Management

• Billing Queue Enhancements:
  • A new billing queue is integrated into the system to manage billing-related jobs.
  • Jobs can trigger batch processing (process-batch) or individual billing operations (bill_team), which are then queued for batch processing.
• Immediate Billing Trigger:
  • Functions such as addBillingBatchJob and triggerImmediateBillingProcess are introduced to ensure timely processing of billing operations.

Worker Integration

• Worker Updates:
  • Worker logic is updated to process both index and billing jobs.
  • Billing jobs extend their lock intervals during processing to ensure stability.
  • Detailed logging and error capture via Sentry are implemented for robust error handling.
• Seamless Integration:
  • The billing worker is integrated into the existing queue-worker system, ensuring smooth operation alongside other job types.

Deprecation of Direct Billing

• Modified supaBillTeam:
  • The direct billing function in credit_billing.ts is deprecated.
  • Billing operations are now forwarded to the batch billing system via the new queueing mechanism.
• Optimized Flow:
  • This centralizes billing operations, ensuring consistency and reducing redundant code.

Additional Enhancements

• Improved Logging:
  • Detailed logs capture key details such as team IDs, total credits billed, and operation counts.
• Queue Service Updates:
  • The billing queue is configured with appropriate job options to ensure proper management.
• Worker Resilience:
  • Extended job locks and robust error handling improve system reliability during high load.

Benefits

• Efficiency:
  Batching reduces the overhead of processing many individual billing operations.
• Reliability:
  The locking mechanism prevents race conditions by ensuring single-batch processing.
• Scalability:
  The system is better equipped to handle high volumes of billing operations by grouping them together.