Skip to main content
AWS · lesson 16 of 19

Beginner

4

Intermediate

13

Advanced

2

Also part of AWS

8
Open the AWS page →
AWS intermediate Lesson 16 of 19

SNS: Fan-Out to SQS and Filter Policies

Fan one SNS topic out to two SQS queues, give one of them a FilterPolicy, and watch it get only the messages that match — while the unfiltered queue gets everything — against the moto emulator.

By SkillByExample · Updated

aws sns sqs messaging

An SNS topic can fan one message out to many independent subscribers — here, two SQS queues — and a FilterPolicy lets one subscriber opt into only a subset of what gets published, based on message attributes. Both run for real against moto.

One Topic, Two Queues, One Filtered

shipping’s FilterPolicy only admits messages whose kind attribute is "physical"; billing has no filter, so it gets everything.

#!/usr/bin/env bash
set -e
aws() { MSYS_NO_PATHCONV=1 docker run --rm --network aws16-lab -e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_DEFAULT_REGION=us-east-1 amazon/aws-cli:2.39.5 "$@" --endpoint-url http://aws16-moto:5000; }

TOPIC_ARN=$(aws sns create-topic --name orders --query 'TopicArn' --output text)
echo "topic: $TOPIC_ARN"

BILLING_URL=$(aws sqs create-queue --queue-name billing --query 'QueueUrl' --output text)
SHIPPING_URL=$(aws sqs create-queue --queue-name shipping --query 'QueueUrl' --output text)
BILLING_ARN=$(aws sqs get-queue-attributes --queue-url "$BILLING_URL" --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)
SHIPPING_ARN=$(aws sqs get-queue-attributes --queue-url "$SHIPPING_URL" --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)

aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol sqs --notification-endpoint "$BILLING_ARN" >/dev/null
SHIPPING_SUB=$(aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol sqs --notification-endpoint "$SHIPPING_ARN" --query 'SubscriptionArn' --output text)

aws sns set-subscription-attributes --subscription-arn "$SHIPPING_SUB" \
  --attribute-name FilterPolicy --attribute-value '{"kind":["physical"]}'
echo "filter policy set on shipping"
$ docker network create aws16-lab
0abb125fe01aa3ea318d2924e8084c994254576a0958e517bb604e65c66089be
$ docker run -d --name aws16-moto --network aws16-lab motoserver/moto:5.2.5
b791599705f4c3e63091004163912d531a69f1a33fd37b57eee03e4ff5e3f84a
$ until docker run --rm --network aws16-lab curlimages/curl:8.11.1 -s -o /dev/null http://aws16-moto:5000; do sleep 0.3; done
$ bash setup.sh
topic: arn:aws:sns:us-east-1:123456789012:orders
filter policy set on shipping

moto applies a new FilterPolicy instantly, so the next script can publish right after setting one; on real AWS a filter-policy change can take up to 15 minutes to fully propagate, so publishing immediately afterward (here, and again in Practice 2) would be racy against a real account.

A helper, reused in every script below, drains a queue by receiving and immediately deleting every message in it — the same receive-then-delete a real consumer does, which also keeps each of this lesson’s steps starting from a clean queue instead of re-finding an earlier step’s leftovers.

#!/usr/bin/env bash
# Receives and deletes one message at a time (never two in the same call) so that a
# multi-line pretty-printed Body can't be mistaken for more than one message — bash's
# parameter expansion below splits on the *last* tab regardless of embedded newlines,
# unlike a line-oriented `read`, which would misparse a Body containing real newlines.
drain() {
  local url="$1"
  while true; do
    local out
    out=$(aws sqs receive-message --queue-url "$url" --max-number-of-messages 1 \
      --query 'Messages[0].[Body,ReceiptHandle]' --output text)
    if [ "$out" = "None" ] || [ -z "$out" ]; then break; fi
    local receipt="${out##*$'\t'}"
    local body="${out%$'\t'*}"
    # Signature/SigningCertURL/UnsubscribeURL/Timestamp are real but different on every
    # run; drop just those lines (pretty JSON puts one field per line) so the displayed
    # envelope is deterministic without touching the fields this lesson is actually about.
    # Side effect: whichever of these was the JSON object's last field leaves a trailing
    # comma behind on the line before it, so the printed envelope below isn't strictly
    # valid JSON any more -- fine for reading, not something to feed back into a parser.
    echo "$body" | grep -vE '"(Signature|SigningCertURL|UnsubscribeURL|Timestamp)":'
    aws sqs delete-message --queue-url "$url" --receipt-handle "$receipt" >/dev/null
  done
}
#!/usr/bin/env bash
set -e
aws() { MSYS_NO_PATHCONV=1 docker run --rm --network aws16-lab -e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_DEFAULT_REGION=us-east-1 amazon/aws-cli:2.39.5 "$@" --endpoint-url http://aws16-moto:5000; }
source drain.sh

TOPIC_ARN="arn:aws:sns:us-east-1:123456789012:orders"
BILLING_URL="http://aws16-moto:5000/123456789012/billing"
SHIPPING_URL="http://aws16-moto:5000/123456789012/shipping"

aws sns publish --topic-arn "$TOPIC_ARN" --message '{"orderId":"o-1"}' \
  --message-attributes '{"kind":{"DataType":"String","StringValue":"ebook"}}' >/dev/null
aws sns publish --topic-arn "$TOPIC_ARN" --message '{"orderId":"o-2"}' \
  --message-attributes '{"kind":{"DataType":"String","StringValue":"physical"}}' >/dev/null

echo "billing (both copies, as SQS actually received them):"
drain "$BILLING_URL"

echo "shipping:"
drain "$SHIPPING_URL"
$ bash publish_and_receive.sh
billing (both copies, as SQS actually received them):
{
  "Message": "{\"orderId\":\"o-1\"}",
  "MessageAttributes": {
    "kind": {
      "Type": "String",
      "Value": "ebook"
    }
  },
  "MessageId": "bd5d1410-44a8-4957-8d3d-c27d5f535c22",
  "SignatureVersion": "2",
  "TopicArn": "arn:aws:sns:us-east-1:123456789012:orders",
  "Type": "Notification",
}
{
  "Message": "{\"orderId\":\"o-2\"}",
  "MessageAttributes": {
    "kind": {
      "Type": "String",
      "Value": "physical"
    }
  },
  "MessageId": "0a542efe-cc1d-43c9-ad82-c60c6a943e12",
  "SignatureVersion": "2",
  "TopicArn": "arn:aws:sns:us-east-1:123456789012:orders",
  "Type": "Notification",
}
shipping:
{
  "Message": "{\"orderId\":\"o-2\"}",
  "MessageAttributes": {
    "kind": {
      "Type": "String",
      "Value": "physical"
    }
  },
  "MessageId": "0a542efe-cc1d-43c9-ad82-c60c6a943e12",
  "SignatureVersion": "2",
  "TopicArn": "arn:aws:sns:us-east-1:123456789012:orders",
  "Type": "Notification",
}

Both publishes went to the same topic; billing (no filter) got both messages, while shipping (filtered on kind: physical) got only one — SNS itself decided each subscriber’s own delivery list from the exact same two Publish calls. The sample messages also show what SQS actually stores by default: not the plain {"orderId":"o-1"} that was published, but an SNS envelope wrapping it — Message holds the original body as an escaped JSON string, alongside SNS’s own MessageId and TopicArn (the envelope’s Signature, SigningCertURL, UnsubscribeURL, and Timestamp fields are real too, just stripped from what’s shown above for determinism — see drain.sh). RawMessageDelivery (Practice 3) strips the whole envelope away. This is moto’s own rendering of the envelope; a real SNS/SQS pair’s JSON carries the same fields but isn’t guaranteed to match this exact key order or spacing.

Practice

1. Publish a message with no kind attribute at all and confirm billing still gets it while shipping doesn't. 
#!/usr/bin/env bash
set -e
aws() { MSYS_NO_PATHCONV=1 docker run --rm --network aws16-lab -e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_DEFAULT_REGION=us-east-1 amazon/aws-cli:2.39.5 "$@" --endpoint-url http://aws16-moto:5000; }
source drain.sh

TOPIC_ARN="arn:aws:sns:us-east-1:123456789012:orders"
BILLING_URL="http://aws16-moto:5000/123456789012/billing"
SHIPPING_URL="http://aws16-moto:5000/123456789012/shipping"

aws sns publish --topic-arn "$TOPIC_ARN" --message '{"orderId":"o-3"}' >/dev/null

echo "billing:"
drain "$BILLING_URL"
echo "shipping (expect nothing printed below):"
drain "$SHIPPING_URL"
$ bash practice1.sh
billing:
{
  "Message": "{\"orderId\":\"o-3\"}",
  "MessageId": "20e52dbf-1d3a-4596-9b72-f278f6331bb8",
  "SignatureVersion": "2",
  "TopicArn": "arn:aws:sns:us-east-1:123456789012:orders",
  "Type": "Notification",
}
shipping (expect nothing printed below):

billing gets o-3’s envelope since it has no filter at all. shipping prints nothing — its FilterPolicy requires a kind attribute equal to "physical", and a message with no kind attribute present can never match that, the same way a missing field can’t equal any specific value.

2. Widen shipping's FilterPolicy to match either "physical" or "heavy", and confirm a "heavy" message now gets through too. 
#!/usr/bin/env bash
set -e
aws() { MSYS_NO_PATHCONV=1 docker run --rm --network aws16-lab -e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_DEFAULT_REGION=us-east-1 amazon/aws-cli:2.39.5 "$@" --endpoint-url http://aws16-moto:5000; }
source drain.sh

TOPIC_ARN="arn:aws:sns:us-east-1:123456789012:orders"
BILLING_URL="http://aws16-moto:5000/123456789012/billing"
SHIPPING_URL="http://aws16-moto:5000/123456789012/shipping"
SHIPPING_SUB=$(aws sns list-subscriptions-by-topic --topic-arn "$TOPIC_ARN" \
  --query "Subscriptions[?Endpoint=='arn:aws:sqs:us-east-1:123456789012:shipping'].SubscriptionArn" --output text)

aws sns set-subscription-attributes --subscription-arn "$SHIPPING_SUB" \
  --attribute-name FilterPolicy --attribute-value '{"kind":["physical","heavy"]}'

aws sns publish --topic-arn "$TOPIC_ARN" --message '{"orderId":"o-4"}' \
  --message-attributes '{"kind":{"DataType":"String","StringValue":"heavy"}}' >/dev/null

echo "shipping:"
drain "$SHIPPING_URL"
drain "$BILLING_URL" >/dev/null
$ bash practice2.sh
shipping:
{
  "Message": "{\"orderId\":\"o-4\"}",
  "MessageAttributes": {
    "kind": {
      "Type": "String",
      "Value": "heavy"
    }
  },
  "MessageId": "e08e332c-5d66-43c1-b0a7-cb00d0c5c9ab",
  "SignatureVersion": "2",
  "TopicArn": "arn:aws:sns:us-east-1:123456789012:orders",
  "Type": "Notification",
}

A FilterPolicy value array is an OR, not an AND — {"kind":["physical","heavy"]} matches either value, so widening the array to add "heavy" is enough to let a differently-tagged message through with no change to how messages are published. billing’s own copy of o-4 is drained too (silently, since billing having no filter is already established) so it doesn’t carry over into the next task.

3. Enable RawMessageDelivery on billing's subscription and compare the next message's shape to the enveloped ones above. 
#!/usr/bin/env bash
set -e
aws() { MSYS_NO_PATHCONV=1 docker run --rm --network aws16-lab -e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_DEFAULT_REGION=us-east-1 amazon/aws-cli:2.39.5 "$@" --endpoint-url http://aws16-moto:5000; }
source drain.sh

TOPIC_ARN="arn:aws:sns:us-east-1:123456789012:orders"
BILLING_URL="http://aws16-moto:5000/123456789012/billing"
BILLING_SUB=$(aws sns list-subscriptions-by-topic --topic-arn "$TOPIC_ARN" \
  --query "Subscriptions[?Endpoint=='arn:aws:sqs:us-east-1:123456789012:billing'].SubscriptionArn" --output text)

aws sns set-subscription-attributes --subscription-arn "$BILLING_SUB" \
  --attribute-name RawMessageDelivery --attribute-value "true"

aws sns publish --topic-arn "$TOPIC_ARN" --message '{"orderId":"o-5"}' >/dev/null
drain "$BILLING_URL"
$ bash practice3.sh
{"orderId":"o-5"}

The body is now exactly {"orderId":"o-5"} — no Message/Signature/TopicArn envelope at all, unlike every receive earlier in this lesson. RawMessageDelivery trades the envelope’s built-in authenticity fields for a body a consumer can parse directly as the original payload.

4. Unsubscribe shipping from the topic and confirm a new published message never reaches it. 
#!/usr/bin/env bash
set -e
aws() { MSYS_NO_PATHCONV=1 docker run --rm --network aws16-lab -e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_DEFAULT_REGION=us-east-1 amazon/aws-cli:2.39.5 "$@" --endpoint-url http://aws16-moto:5000; }
source drain.sh

TOPIC_ARN="arn:aws:sns:us-east-1:123456789012:orders"
BILLING_URL="http://aws16-moto:5000/123456789012/billing"
SHIPPING_URL="http://aws16-moto:5000/123456789012/shipping"
SHIPPING_SUB=$(aws sns list-subscriptions-by-topic --topic-arn "$TOPIC_ARN" \
  --query "Subscriptions[?Endpoint=='arn:aws:sqs:us-east-1:123456789012:shipping'].SubscriptionArn" --output text)

aws sns unsubscribe --subscription-arn "$SHIPPING_SUB"
aws sns publish --topic-arn "$TOPIC_ARN" --message '{"orderId":"o-6"}' \
  --message-attributes '{"kind":{"DataType":"String","StringValue":"physical"}}' >/dev/null

echo "shipping after unsubscribe (expect nothing printed below):"
drain "$SHIPPING_URL"
echo "done"
drain "$BILLING_URL" >/dev/null
$ bash practice4.sh
shipping after unsubscribe (expect nothing printed below):
done

Nothing prints between the two echo lines — even a message that would have matched shipping’s old filter never arrives, because Unsubscribe removes the subscription from the topic entirely; there’s no subscription left for Publish to evaluate a filter against.

Cleanup

$ docker rm -f aws16-moto
aws16-moto
$ docker network rm aws16-lab
aws16-lab

Next: wiring an Auto Scaling Group behind an Application Load Balancer, and what the emulator can’t show about either.

Frequently Asked Questions

What does SNS fan-out to SQS actually mean?
One SNS Publish call delivers an independent copy of the message to every subscription on that topic — here, both the billing and shipping queues. Each subscriber gets its own copy and processes it on its own schedule; a slow or down shipping consumer has no effect on billing's copy, which is the core reason to fan out through SNS rather than have one service call two others directly.
How does a FilterPolicy decide whether a subscriber gets a message?
By default it matches against the message's MessageAttributes, not the message body — this lesson's FilterPolicy {"kind":["physical"]} only lets a message through to that subscription if its kind attribute's value is exactly "physical", with SNS never looking inside the body itself. SNS also supports a FilterPolicyScope of MessageBody, which does match against the body's own JSON instead; this lesson only exercises the (more common) MessageAttributes scope, which is also the default.
Does a FilterPolicy change what the topic publishes, or just what each subscriber receives?
Only what each subscriber receives. Publish always succeeds the same way regardless of any subscription's FilterPolicy; filtering happens per-subscription at delivery time, which is why this lesson's billing queue (no filter) gets both messages while shipping (filtered) gets only one, from the exact same two Publish calls.
What is RawMessageDelivery, and why does the default SQS message look so different from what was published?
By default, SNS wraps a published message in its own envelope (Message, MessageAttributes, Signature, Timestamp, TopicArn, and more) before delivering it to SQS — useful for verifying authenticity, but extra parsing for a consumer that only wants the original body. RawMessageDelivery=true strips that envelope so the SQS message body is exactly the original Publish call's Message, with none of SNS's own metadata.
Would this lesson's scripts work unmodified against a real AWS account instead of moto?
Not quite: a real SQS queue needs an access policy explicitly allowing sns.amazonaws.com to call sqs:SendMessage on it, scoped with an aws:SourceArn condition equal to the topic's ARN — without it, SNS's deliveries to that queue silently fail (the queue just never receives anything, with no error on the Publish call itself). moto doesn't enforce this at all, which is why these scripts work here with no queue policy set; add one before trying this against a real account.