Siddheshwari Bankar

Task Queues System using Redis and Docker Compose

What is this project about?

Ever wondered how big websites handle background tasks like sending emails or processing large amounts of data without slowing down the user experience? In this report, I’ll explain how I built a system that handles these tasks in the background, using two tools: Redis and Docker Compose. This project implements a task queue system with Redis as the broker, enabling smooth task management and processing. By incorporating Docker Compose, the system becomes more scalable and easier to deploy, offering a practical example of distributed systems in action.

The Problem: Handling Long Tasks

Imagine you're using a website that needs to send you an email. If the website tried to send the email while you waited, it would make the site feel slow. What if the task takes several seconds or more? It’s not fun to stare at a loading screen, right? To solve this, we need a way to send the email in the background while letting the website respond quickly. This is where a task queue comes in! Instead of doing the hard work right away, the website can hand off the task to a worker in the background. Then the worker does the heavy lifting while the website stays fast and responsive.

The Solution: Redis Task Queue

To make this happen, I built a system where:

1. The website adds a task (like sending an email) to a queue.
2. Redis stores that task in memory, waiting for a worker to process it.
3. Workers (other programs running in the background) check Redis and pick up tasks to complete them.
4. Once done, the result can be checked through the website.

Let’s break down the main pieces of this system:

What is Redis?

Redis is a super-fast, in-memory database that can store data. In this case, it stores tasks in a queue. Think of Redis like a to-do list where tasks are added by the website, and workers come by to pick up tasks one by one to finish.

What is Docker Compose?

Docker Compose helps us manage different parts of the system in separate containers. Containers are like mini virtual computers that run different programs. Here, we use Docker Compose to run the website, Redis, and workers separately but make sure they can talk to each other.

How It Works: Step-by-Step

1. Task Creation (The Website)

I built a simple website using Flask (a Python web framework) where users can add new tasks. When a user performs an action (like clicking a button), it sends a task to Redis to be handled in the background.

2. The Task Queue (Redis)

Redis holds onto the tasks that need to be done, like a waiting room for tasks. Each task sits in line until a worker picks it up to work on it.

3. Worker Processes

These are small programs running in the background. They constantly check Redis to see if there’s a new task waiting. If there is, they grab it, complete the work (like sending an email), and move on to the next task. These workers run in their own containers using Docker Compose.

4. Fetching Results

After the worker finishes a task, the website can check Redis to see the task’s status and show you the result, like "Task completed!" or "Still processing…".

Tech Stack

• Redis: For storing and managing tasks in the queue.
• Docker Compose: To manage all the different parts (the website, Redis, workers) in containers so they can work together smoothly.
• Python: The programming language behind all of this.
• RQ (Redis Queue): A Python library that lets us easily queue tasks in Redis.
• Flask: For building the website that adds tasks and fetches results.

Conclusion

Building this system taught me a lot about task management and scaling in web applications. By using Redis for task queues and Docker Compose for containerization, I created a system that can handle tasks in the background without affecting the main website’s performance. This concept is widely used in modern web applications to make sure users get a fast, smooth experience.

Implementation