🎯Aim

Add the configuration necessary to run the Celery task queue.

📕Context

It is quite common for web applications to need to offload work to a background task running in order to block the regular request -> response cycle. Time consuming tasks like sending emails or processing data are places on a queue that workers can pick up and process in their own time.

The de facto standard (although not the only option) within the Python eco-system is Celery. It is robust, battle tested and extremely flexible. With that flexibility comes some things that can catch you out. As such a known good configuration for our template project is a good idea to distil what we have learnt in previous projects, and avoid those gotchas.

Whilst there is a common need it isn't universal. As such it is also an opportunity to start developing variants for our template project. The base template project should contain only that which is needed by all projects. Variants should provide addition functionality.

Tooling to allow for variants be automatically fully integrated into a project would be neat but is not a hard requirements. A starting position of simply having default configuration and documentation on how to add the variant would be sufficient at this time.

📝Relevant resources/doc's/people

• Celery Documentation
• Django Celery Beat, celery can run periodic tasks, the django-celery-beat package allows you to use Django to manage that schedule which can be handy
• Equivalent Variant in the Rails Template. Useful to see how they have dealt with the variant concept in the rails world.

✅Acceptance Criteria

• The concept of a Variant is added to the template project
• A new task_runner variant based on Celery is added
• The variant provides:
  • A known good set of configuration to ensure common gotchas are avoided
  • Sufficient documentation to explain the configuration choices and to guide a developer to add it an instance of the template project
  • Configuration to use Redis as the broker
  • Provides the ability to run a worker locally so that the development environment matches as closely as possible the production environment

🎈 Stretch Goals

• Add django-celery-beat in order to allow the configuration of task schedules in the Django database
• Add some tooling around the variants that allow the automation of adding the variant to an existing project.