DynamoDB RSS Feed: Query Newest Items and Build XML

Short answer: To build an RSS feed from Amazon DynamoDB, design the table so that the newest published items can be read with a Query and not a Scan: use a key or an index that groups items and sorts them by publication time, read them in descending order with a small limit, convert each item to an RSS entry with escaped text, and serve the XML with the RSS content type and cache headers. Keep drafts and future items out with a status check, give every item a permanent identifier and let a cache absorb repeated requests from readers and auto-posters such as PostRSS.

DynamoDB is a managed key-value and document database that is popular for serverless applications. It is fast and scales without servers to manage, but it rewards careful design: you cannot ask it any question you like, only the questions your keys and indexes were built to answer. That matters for an RSS feed, which always asks the same thing: give me the newest published items, in order. If the table was designed only to fetch an article by its identifier, answering this question tempts developers to run a Scan, which reads the entire table and becomes slower and more expensive as the content grows. This guide shows how to model the data for a feed, how to query it in the right way, how to turn the result into valid XML and how to serve it with caching. Service limits and console names change, so verify details in the current documentation.

Why a Scan is the wrong tool

A Scan reads every item in the table, then filters. You pay for everything it reads, not for what it returns, and the time grows with the table. For a blog with a few hundred articles the first version may seem fine, and it becomes a problem later, exactly when traffic grows. A Query, in contrast, reads only the items that share a partition key and can use the sort key to return them in order, with a limit. A feed should always use a Query. If your first design cannot answer the feed question with a Query, change the design now, while the table is small, and not after the feed is connected to many tools.

Model the table for the feed

Start with the access patterns. You need to fetch one article by its slug for the web page, and you need the newest published articles for the feed. A common solution is a table with the slug as the primary key and a global secondary index that groups items by a feed key and sorts them by publication time.

With this design the feed query asks the index for the partition main, in descending order of the sort key, with a limit of thirty items. One caution: if every article has the same index partition value, all writes and reads go to one partition. For a feed that is small and read mostly this is acceptable, and caching reduces the load, but a very large or very busy site can spread items across several partitions, for example one per month, and query the current month and the previous one. Another option is to write only published items into the index and leave drafts without the index attributes, because items that lack the keys of an index do not appear in it, which keeps drafts out automatically.

Query the newest items

The query reads from the index in descending order. The status condition is applied as a filter, and the publication time is limited to now so that scheduled items stay hidden.

import boto3
from boto3.dynamodb.conditions import Key, Attr
from datetime import datetime, timezone

table = boto3.resource("dynamodb").Table("posts")

def latest(limit=30):
    now = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
    resp = table.query(
        IndexName="feed-index",
        KeyConditionExpression=Key("feed").eq("main") & Key("published_at").lte(now),
        FilterExpression=Attr("status").eq("published"),
        ScanIndexForward=False,
        Limit=limit,
    )
    return resp["Items"]

A filter expression is applied after the items are read, and the limit counts the items read before filtering. If many drafts sit between published items, the query might return fewer items than requested. Keep drafts out of the index, as described above, or read in a loop until you have enough items. Test with a mix of drafts and published items to make sure the result has the number you expect.

Turn the items into XML

The conversion is the same as for any database: build each item with the title, the full address, the identifier, the date and the description, escape every value and wrap the items in a channel.

from email.utils import format_datetime
from xml.sax.saxutils import escape

SITE = "https://example.com"

def to_rss(items):
    out = []
    for p in items:
        link = SITE + "/" + p["slug"] + "/"
        when = datetime.strptime(p["published_at"], "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc)
        out.append(
            "<item><title>" + escape(p["title"]) + "</title>"
            "<link>" + escape(link) + "</link>"
            "<guid isPermaLink='true'>" + escape(link) + "</guid>"
            "<pubDate>" + format_datetime(when) + "</pubDate>"
            "<description>" + escape(p["summary"]) + "</description></item>"
        )
    return ("<?xml version='1.0' encoding='UTF-8'?><rss version='2.0'><channel>"
            "<title>Example blog</title><link>" + SITE + "</link>"
            "<description>News from Example</description>" + "".join(out) +
            "</channel></rss>")

DynamoDB returns numbers as a special decimal type, so convert them before using them in text if your feed includes any. Strings need no conversion. If you store HTML in the item, keep it out of the description or wrap it correctly in a content element.

Serve the feed and cache it

The feed can be served by a function behind an API gateway or a function URL, by a container or by a static file written to object storage after each publication. The last option is the cheapest and the simplest to cache: write the XML file when an article is published, and let readers fetch the file. If you generate the feed on request, return the RSS content type with a UTF-8 charset and a short cache lifetime, and put a content delivery network in front. The patterns are the same as in the guides to an AWS Lambda feed and a Cloudflare Workers feed. A lifetime of a few minutes means that a thousand requests cost one read.

Control costs and capacity

A query of thirty small items consumes a small number of read units, and with caching the monthly cost of a feed is tiny. Still, check the mode of the table. On-demand capacity suits unpredictable traffic and is simple for a small site. Provisioned capacity can be cheaper for steady traffic but needs attention. Whatever you choose, set an alarm on read usage, so that a faulty client that requests the feed without a cache does not surprise you. Keep each item small by storing long article bodies in object storage and keeping only the summary in the item that the feed reads.

Support category feeds

As the site grows, readers and auto-posters ask for separate feeds for separate topics. With DynamoDB there are two sensible approaches. The first is to use the category as part of the index partition value, such as the feed name news or tutorials, and to write each published article into the index once for each feed it belongs to. The second is to add a second index keyed by category and sorted by time. Both answer the question newest items in a category with a Query. Choose the first when you have a few fixed feeds and the second when categories are many and change often. Keep the identifiers identical across feeds, so an item that appears in two of them is recognized as the same article by any tool.

Keep the feed working during failures

A database call can fail or slow down, and a feed that returns an error page for a few minutes can make tools report an invalid feed. Protect the endpoint with a short timeout, retry once, and fall back to the last good copy kept in a cache or in object storage if the query fails. Return that older copy with a normal status, because a temporary error status usually makes readers retry harder. Log every failure with a request identifier, and set an alarm on repeated errors so you hear about the problem before your readers do. A feed that is slightly stale is almost always better than a feed that is missing.

Handle updates and deletions

When an article is edited, update the item but keep the slug and the publication time, so that tools do not treat it as a new entry. When an article is unpublished, change its status or remove the index attributes, then purge the cache. Do not reuse a slug for a different article. If a title changes, the feed will carry the new title in the same item, and readers that already saw the old one will usually ignore the change. Keep a record of removed slugs and redirect them so that old links still work.

Test before connecting a tool

1. Insert a published item, a draft and an item with a future date, and confirm that only the first appears.
2. Add a title with an ampersand, a quotation mark and an accented letter, and confirm that it is displayed correctly.
3. Paste the address into the W3C Feed Validation Service and fix every error.
4. Check that the items are sorted newest first and that the identifiers stay constant between requests.
5. Request the feed twice and confirm that the second response is served from the cache.

Common mistakes to avoid

• Using a Scan. It reads the whole table on every request.
• Filtering drafts after the limit. The feed may return too few items.
• Sorting by a text date in many formats. Use one ISO format in UTC.
• Changing slugs. Tools will see old articles as new ones.
• No cache. Every request becomes a database read.
• Large items. Keep full bodies out of the feed item.

How PostRSS works with a feed from DynamoDB

PostRSS has been publishing RSS and Atom feeds to social networks since 2014. Paste the address of your feed, connect your accounts and choose how each post is built; PostRSS can check the feed as often as every minute and publishes each new item with its real image and link to the networks you select among its 66 supported networks. A cached feed with stable identifiers is ideal, because each article is posted once. Add UTM parameters to the posted links to see which network brings readers. See the PostRSS features page for the options and the pricing page for plan limits.

Powiązane materiały

• AWS Lambda RSS feed
• Cloudflare Workers RSS feed
• RSS feed validation before you auto-post

The bottom line

A DynamoDB feed is simple when the table is designed for it. Use an index that groups published items and sorts them by time, query in descending order with a small limit, keep drafts out of the index, escape every value, give each item a permanent identifier and cache the result. Test with drafts, future items and special characters. Then connect the address to an auto-poster, so each new article reaches your audience on social networks without further work.

FAQ

Should an RSS feed use a Scan on DynamoDB?

No. A Scan reads the whole table. Design an index so that a Query returns the newest published items in order with a small limit.

How do I keep drafts out of the feed?

Add the index attributes only when an article is published, or filter by status. Test with a draft to confirm that it never appears.

What should the item identifier be?

Use something permanent, such as the slug or the full address. Never use the title or the date, because both can change.

How often should the table be read?

Rarely. Cache the feed for a few minutes or write it to a file after each publication, so many requests cost one read.

Can PostRSS read a feed built from DynamoDB?

Yes. PostRSS reads any valid RSS or Atom address that is reachable without a login, however the data is stored.