Skip to main content
AWS · lesson 15 of 19

Beginner

4

Intermediate

13

Advanced

2

Also part of AWS

8
Open the AWS page →
AWS intermediate Lesson 15 of 19

SQS: FIFO Queues, Message Groups, and Deduplication

Send the same message body twice to a FIFO queue and watch content-based deduplication deliver it only once, then interleave two message groups and confirm each group's own order is preserved — against the moto emulator.

By SkillByExample · Updated

aws sqs messaging

A FIFO queue (name ending in .fifo) adds two guarantees a standard SQS queue doesn’t make: deduplication of identical sends, and strict ordering within a message group. Both run for real here against moto.

Content-Based Deduplication

With ContentBasedDeduplication enabled, two sends with the same body — no explicit dedup ID needed — are treated as one message.

#!/usr/bin/env bash
set -e
aws() { MSYS_NO_PATHCONV=1 docker run --rm --network aws15-lab -e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_DEFAULT_REGION=us-east-1 amazon/aws-cli:2.39.5 "$@" --endpoint-url http://aws15-moto:5000; }

Q_URL=$(aws sqs create-queue --queue-name orders.fifo \
  --attributes '{"FifoQueue":"true","ContentBasedDeduplication":"true"}' \
  --query 'QueueUrl' --output text)
echo "queue: $Q_URL"

aws sqs send-message --queue-url "$Q_URL" --message-group-id customer-alice \
  --message-body '{"orderId":"o-1"}' --query 'MessageId' --output text
aws sqs send-message --queue-url "$Q_URL" --message-group-id customer-alice \
  --message-body '{"orderId":"o-1"}' --query 'MessageId' --output text

echo "messages actually in the queue:"
aws sqs receive-message --queue-url "$Q_URL" --max-number-of-messages 10 --query 'length(Messages)'
$ docker network create aws15-lab
5d5fce82bf55588a17d87b489deb91158f419aeca96d154afe00ae14a28368c9
$ docker run -d --name aws15-moto --network aws15-lab motoserver/moto:5.2.5
6b2ed2f759976e2cc141a62ba79b924765abd8e2dfbe03c8501cf85a8b80680d
$ until docker run --rm --network aws15-lab curlimages/curl:8.11.1 -s -o /dev/null http://aws15-moto:5000; do sleep 0.3; done
$ bash dedup.sh
queue: http://aws15-moto:5000/123456789012/orders.fifo
715bb9d0-f6c6-4636-810a-8af8df2e9119
5e64663d-fc6c-4e11-9c1f-474f8418e018
messages actually in the queue:
1

Two SendMessage calls, each returning its own MessageId, but exactly one message ends up receivable — the second send was accepted, not rejected, but deduplicated against the first within the (default 5-minute) dedup interval, so it never results in a second delivery.

Ordering Within a Message Group

Two message groups, alice and bob, interleaved on the same queue — each should keep its own send order even though the groups are mixed together.

#!/usr/bin/env bash
set -e
aws() { MSYS_NO_PATHCONV=1 docker run --rm --network aws15-lab -e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_DEFAULT_REGION=us-east-1 amazon/aws-cli:2.39.5 "$@" --endpoint-url http://aws15-moto:5000; }

Q_URL=$(aws sqs create-queue --queue-name groups.fifo \
  --attributes '{"FifoQueue":"true","ContentBasedDeduplication":"true"}' \
  --query 'QueueUrl' --output text)

for i in 1 2 3; do
  aws sqs send-message --queue-url "$Q_URL" --message-group-id alice --message-body "alice-$i" >/dev/null
  aws sqs send-message --queue-url "$Q_URL" --message-group-id bob --message-body "bob-$i" >/dev/null
done

aws sqs receive-message --queue-url "$Q_URL" --max-number-of-messages 10 \
  --attribute-names MessageGroupId --query 'Messages[].[Attributes.MessageGroupId,Body]' --output text
$ bash groups.sh
alice	alice-1
bob	bob-1
alice	alice-2
bob	bob-2
alice	alice-3
bob	bob-3

Sends alternated alice, bob, alice, bob, … but each group’s own three messages still come back in send order (alice-1, alice-2, alice-3 and bob-1, bob-2, bob-3, each internally in order) — MessageGroupId orders within itself, with no promise about how two different groups interleave relative to each other.

Practice

1. Send the same body again after changing even one character, and confirm it's treated as a new, undeduplicated message. 
#!/usr/bin/env bash
set -e
aws() { MSYS_NO_PATHCONV=1 docker run --rm --network aws15-lab -e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_DEFAULT_REGION=us-east-1 amazon/aws-cli:2.39.5 "$@" --endpoint-url http://aws15-moto:5000; }

Q_URL="http://aws15-moto:5000/123456789012/orders.fifo"
aws sqs send-message --queue-url "$Q_URL" --message-group-id customer-carol \
  --message-body '{"orderId":"o-2"}' >/dev/null

aws sqs receive-message --queue-url "$Q_URL" --max-number-of-messages 10 --query 'length(Messages)'
$ bash practice1.sh
1

One new message is receivable — o-2’s body hashes differently from o-1’s, so content-based deduplication has nothing to match it against; a single changed character is enough.

2. Create a second FIFO queue without ContentBasedDeduplication, and confirm sending a message with no explicit MessageDeduplicationId is rejected outright. 
#!/usr/bin/env bash
aws() { MSYS_NO_PATHCONV=1 docker run --rm --network aws15-lab -e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_DEFAULT_REGION=us-east-1 amazon/aws-cli:2.39.5 "$@" --endpoint-url http://aws15-moto:5000; }

aws sqs create-queue --queue-name strict.fifo --attributes '{"FifoQueue":"true"}' >/dev/null
aws sqs send-message --queue-url http://aws15-moto:5000/123456789012/strict.fifo \
  --message-group-id g1 --message-body '{"x":1}'
$ bash practice2.sh

aws: [ERROR]: An error occurred (InvalidParameterValue) when calling the SendMessage operation: The queue should either have ContentBasedDeduplication enabled or MessageDeduplicationId provided explicitly

Rejected with InvalidParameterValue — ContentBasedDeduplication is what lets a FIFO queue hash the body automatically; without it, a caller has to pass MessageDeduplicationId itself on every send, and the service refuses to guess.

3. Try FifoQueue=true on a queue name that doesn't end in .fifo, and read moto's validation error. 
aws() { MSYS_NO_PATHCONV=1 docker run --rm --network aws15-lab -e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_DEFAULT_REGION=us-east-1 amazon/aws-cli:2.39.5 "$@" --endpoint-url http://aws15-moto:5000; }

aws sqs create-queue --queue-name orders-not-fifo --attributes '{"FifoQueue":"true"}'
$ bash practice3.sh

aws: [ERROR]: An error occurred (InvalidParameterValue) when calling the CreateQueue operation: Queue name must end in .fifo for FIFO queues

Rejected — the .fifo suffix is enforced as part of CreateQueue’s own validation, not just a documentation convention; the API refuses to create a mismatched queue rather than silently ignoring FifoQueue=true.

4. Receive with MaxNumberOfMessages=1 three times with no DeleteMessage in between, then delete the in-flight message and confirm the group unblocks. 
#!/usr/bin/env bash
set -e
aws() { MSYS_NO_PATHCONV=1 docker run --rm --network aws15-lab -e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_DEFAULT_REGION=us-east-1 amazon/aws-cli:2.39.5 "$@" --endpoint-url http://aws15-moto:5000; }

Q_URL=$(aws sqs create-queue --queue-name onebyone.fifo \
  --attributes '{"FifoQueue":"true","ContentBasedDeduplication":"true"}' \
  --query 'QueueUrl' --output text)

for i in 1 2 3; do
  aws sqs send-message --queue-url "$Q_URL" --message-group-id carol --message-body "carol-$i" >/dev/null
done

echo "receive 1 (saves its receipt handle, does not delete yet):"
IFS=$'\t' read -r BODY RECEIPT <<< "$(aws sqs receive-message --queue-url "$Q_URL" --max-number-of-messages 1 \
  --query 'Messages[0].[Body,ReceiptHandle]' --output text)"
echo "$BODY"

echo "receive 2 (carol-1 still in flight, undeleted):"
aws sqs receive-message --queue-url "$Q_URL" --max-number-of-messages 1 --query 'Messages[0].Body' --output text

echo "receive 3 (same):"
aws sqs receive-message --queue-url "$Q_URL" --max-number-of-messages 1 --query 'Messages[0].Body' --output text

echo "now delete carol-1 using the saved receipt handle:"
aws sqs delete-message --queue-url "$Q_URL" --receipt-handle "$RECEIPT"

echo "receive 4 (after the delete):"
aws sqs receive-message --queue-url "$Q_URL" --max-number-of-messages 1 --query 'Messages[0].Body' --output text
$ bash practice4.sh
receive 1 (saves its receipt handle, does not delete yet):
carol-1
receive 2 (carol-1 still in flight, undeleted):
None
receive 3 (same):
None
now delete carol-1 using the saved receipt handle:
receive 4 (after the delete):
carol-2

Receives 2 and 3 both come back empty (None) even though carol-2 and carol-3 are sitting in the queue — a FIFO queue won’t hand a group’s next message to anyone while an earlier one from that same group is still in flight (received but neither deleted nor timed out). That’s the mechanism that makes the ordering guarantee real: it would be meaningless if carol-2 could be picked up before anyone finished (or gave up on) carol-1. The moment carol-1 is deleted, the block lifts and receive 4 hands out carol-2 — confirming it was specifically about the undeleted message, not a permanent stall. A standard (non-FIFO) queue has no such restriction.

Cleanup

$ docker rm -f aws15-moto
aws15-moto
$ docker network rm aws15-lab
aws15-lab

Next: SNS fan-out to multiple SQS queues, with a filter policy routing different messages to different subscribers.

Frequently Asked Questions

What does ContentBasedDeduplication actually hash to decide two sends are duplicates?
A SHA-256 of the message body. Two SendMessage calls with byte-for-byte identical bodies, to the same FIFO queue, within the same 5-minute deduplication interval, are treated as the same message — only the first is ever delivered. Without ContentBasedDeduplication enabled, a caller must supply its own MessageDeduplicationId explicitly instead of relying on the body hash.
Does a duplicate SendMessage call fail, or does it silently succeed without enqueueing anything?
It succeeds — SendMessage still returns a MessageId for a deduplicated call, it just never results in a second delivery, exactly as this lesson's own receive-count check shows. On moto, that second MessageId differs from the original send's. Real SQS's exact MessageId value for a deduplicated send isn't something this lesson independently confirms either way, so treat 'succeeds without double delivery' as established and the specific MessageId behavior as moto's own, not a verified real-AWS fact.
Why does the queue name have to end in .fifo?
SQS uses the .fifo suffix as the signal that a queue is a FIFO queue at creation time. CreateQueue rejects FifoQueue=true on a name that doesn't end in .fifo — moto enforces this half, and this lesson's Practice 3 captures the real rejection. The reverse (a name that already ends in .fifo but omits FifoQueue=true) is also rejected on real AWS, since the suffix and the attribute have to agree either way; moto, in a quick check against 5.2.5, didn't enforce that direction, accepting the queue instead. Either way, every queue URL in this lesson ends in .fifo because its FifoQueue=true was always set explicitly, never left to the suffix alone.
What does MessageGroupId guarantee that an ordinary FIFO send doesn't?
Strict ordering within that specific group only — messages sharing a MessageGroupId are always delivered in the order they were sent, but messages in different groups on the same queue carry no ordering guarantee relative to each other. This lesson's interleaved alice/bob groups show exactly that: each group's own sequence stays intact while the two groups interleave with each other.