Moving Big Number Of Messages Between SQS Queues

Moving Big Number Of Messages Between SQS Queues


8 min read

- Dead-letter queue is used to send undeliverable messages to a dead-letter queue. Sometimes, for example, if there’s a bug in the worker code, you can configure SQS to send such problematic messages to a dead-letter queue (DLQ), where you can inspect them in isolation and work out what went wrong.

- Once we’ve found the problem in the worker, fixed the bug and deployed a new version, we want to send all the messages from the DLQ back to the original input queue, so they can be processed by the updated worker. There’s no way to do this in SQS directly, so we’ve written a script to do it for us.

- The automation script moves SQS messages between queues with proper way to avoid impact to other services which using the same queue. This post also provides CDK code to create SQS queues to test and the python script to generate a bunch of messages.

Alt Text

What’s In This Document

🚀 Create SQS queues using CDK

1. Init CDK project

⚡ $ cdk init -l python
Applying project template app for python

# Welcome to your CDK Python project!

This is a blank project for Python development with CDK.

The `cdk.json` file tells the CDK Toolkit how to execute your app.

✅ All done!

2. Create CDK stacks

from aws_cdk import (
    aws_sqs as sqs

class SqsStackStack(core.Stack):

    def __init__(self, scope: core.Construct, construct_id: str, env, **kwargs) -> None:
        super().__init__(scope, construct_id, env=env, **kwargs)

        # Create my-queue
        my_dl_queue = sqs.Queue(self, id="SQSTestDLQueueMovingMsg", queue_name="my_queue_dl_test")
        my_queue = sqs.Queue(self, id="SQSTestMovingMsg", queue_name="my_queue_test",
                             dead_letter_queue=sqs.DeadLetterQueue(max_receive_count=1, queue=my_dl_queue)
  • Deploy stacks
⚡ $ cdk ls

⚡ $ cdk deploy 
sqs-stack: deploying...

 ✅  sqs-stack

Stack ARN:

Alt Text Alt Text

🚀 Generate 100 Messages to Dead-letter Queue For Testing

import boto3
import uuid

def gen_msg():
    msg = list()
    for i in range(0, 100):
        body = f"Hello {i}"
        msg.append({"Body": body})
        if len(msg) > 9:
            yield msg
            msg = list()

sqs_client = boto3.client("sqs", region_name='ap-northeast-2')
sqs_url = ''
while True:
    for message_batch in gen_msg():
            Entries=[{"Id": str(uuid.uuid4()), "MessageBody": message["Body"]} for message in message_batch]
  • Run
    ⚡ $ python
    Alt Text

🚀 Move All Messages At Once

#!/usr/bin/env python
Move all the messages from one SQS queue to another.

Usage: --src-url=<SRC_QUEUE_URL> --dst-url=<DST_QUEUE_URL> --max-msg=<MAX_MSG_PROCESS> -h | --help
import argparse
import itertools
import sys
import uuid
import boto3

from pprint import pprint

def parse_args():
    """Parse command-line arguments."""
    parser = argparse.ArgumentParser(
        description="Move all the messages from one SQS queue to another."

    parser.add_argument("-s", "--src-url", required=True, help="Queue to read messages from")
    parser.add_argument("-d", "--dst-url", required=True, help="Queue to move messages to")
    parser.add_argument("-m", "--max-msg", required=False, type=int,
                        help="Max number of messages to process, no limit if not specify."
                             "Use this to limit the message processed to avoid pending message from others")
    parser.add_argument("-r", "--region", required=True, help="Region of the SQS queue, assume src and dest SQS"
                                                              "is in the same region")
    return parser.parse_args()

def get_messages_from_queue(sqs_client, queue_url, max_nr_msg=0):
    """Generates messages from an SQS queue.

    Note: this continues to generate messages until the queue is empty.
    Every message on the queue will be deleted.

    :param queue_url: URL of the SQS queue to read.
    :param sqs_client: boto3 sqs client to connect to AWS SQS
    msg_processed = 0

    while True:
        resp = sqs_client.receive_message(
            QueueUrl=queue_url, AttributeNames=["All"], MaxNumberOfMessages=10

        entries = []
            msg_to_copy = resp["Messages"]
            if max_nr_msg > 0:
                if msg_processed >= max_nr_msg or msg_to_copy == 0:
                elif (len(msg_to_copy) + msg_processed) > max_nr_msg:
                    msg_to_copy = resp["Messages"][0: (max_nr_msg - msg_processed)]
            msg_processed += len(msg_to_copy)
        except KeyError as err:
            """ Don't return, need to delete messages """
            print(f'found no messages')
            msg_to_copy = list()

        for msg in msg_to_copy:
                entries.append({"Id": msg["MessageId"], "ReceiptHandle": msg["ReceiptHandle"]})
                yield msg
            except KeyError:
                print('failed parsing key')
        if entries:
            resp = sqs_client.delete_message_batch(QueueUrl=queue_url, Entries=entries)
            if len(resp["Successful"]) != len(entries):
                raise RuntimeError(f"Failed to delete messages: entries={entries!r} resp={resp!r}")

def chunked_iterable(iterable, *, size):
    Read ``iterable`` in chunks of size ``size``.
    it = iter(iterable)
    while True:
        chunk = tuple(itertools.islice(it, size))
        if not chunk:
        yield chunk

if __name__ == "__main__":
    args = parse_args()

    src_queue_url = args.src_url
    dst_queue_url = args.dst_url
    region = args.region
    max_msg = args.max_msg
    if not max_msg:
        max_msg = 0

    if src_queue_url == dst_queue_url:
        sys.exit("Source and destination queues cannot be the same.")

    sqs_client = boto3.client("sqs", region_name=region)

    messages = get_messages_from_queue(sqs_client, queue_url=src_queue_url, max_nr_msg=max_msg)

    # The SendMessageBatch API supports sending up to ten messages at once.
    for message_batch in chunked_iterable(messages, size=10):
        print(f"Writing {len(message_batch):2d} messages to {dst_queue_url}")
            Entries=[{"Id": str(uuid.uuid4()), "MessageBody": message["Body"]} for message in message_batch]
  • Run
⚡ $ python -s -d -r ap-northeast-2 
Writing 10 messages to
Writing 10 messages to
Writing 10 messages to
Writing 10 messages to
Writing 10 messages to
Writing 10 messages to
Writing 10 messages to
Writing 10 messages to
Writing 10 messages to
Writing 10 messages to
found no messages, Done!

🚀 Move Half Messages To Avoid Workload

⚡ $ python -s -d -r ap-northeast-2 -m 50
Writing 10 messages to
Writing 10 messages to
Writing 10 messages to
Writing 10 messages to
Writing 10 messages to

Alt Text

🚀 Destroy Test Queues

⚡ $ cdk destroy 
Are you sure you want to delete: sqs-stack (y/n)? y
sqs-stack: destroying...
11:31:35 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::Stack | sqs-stack
11:31:35 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::Stack | sqs-stack
11:32:38 PM | DELETE_IN_PROGRESS   | AWS::SQS::Queue    | SQSTestDLQueueMovingMsg
 ✅  sqs-stack: destroyed

Blog · Github · Web · Linkedin · Group · Page · Twitter