NATS JetStream RSS Feed: Rebuild on Publish Events

Short answer: With NATS JetStream you can keep an RSS feed fresh by publishing a small message to a stream whenever an article goes live, reading it with a durable pull consumer, rebuilding the feed file from the current database state and acknowledging the message only after the file has been written. Use message identifiers to remove duplicates, a maximum delivery count for failures and a slow timer as a safety net. The result is an ordinary feed address that PostRSS can read and post to your social networks.

NATS is known for being small, fast and simple to run. Core NATS is a fire-and-forget messaging system, which is perfect for many tasks but not for a feed that must never miss a publication. JetStream, the persistence layer built into the NATS server, adds streams, stored messages, consumers that remember their position and explicit acknowledgements. That is exactly what a feed builder needs. This guide shows how to model the stream, publish events safely, write a consumer that survives restarts, make the rebuild idempotent and monitor the pipeline. The code samples use Python with the nats-py client, but the concepts transfer to Go, Node.js and the other official clients.

Why JetStream and not plain NATS

Plain publish and subscribe in core NATS delivers a message only to subscribers that are connected at that moment. If the feed builder restarts during a deployment and an article is published in that window, the message is gone. For a feed, that means a missing item until the next manual rebuild. JetStream stores messages in a stream on disk or in memory and lets a named consumer resume from where it stopped. The publisher gets an acknowledgement that the message was stored, and the consumer acknowledges each message after processing. Together these two acknowledgements give you at-least-once delivery from the moment an article is saved to the moment the feed file changes.

Design the stream

A stream captures messages for one or more subjects. For a content site, a single subject family is enough, for example articles.published এবং articles.updated, captured together by one stream with a wildcard subject. Choose file storage so that messages survive a server restart, set a retention limit by age or count so that the stream does not grow forever and decide how many replicas you need. A three-node cluster with three replicas is common in production, while a single node with file storage may be enough for a small site.

import asyncio, nats

async def setup():
    nc = await nats.connect("nats://nats:4222")
    js = nc.jetstream()
    await js.add_stream(
        name="ARTICLES",
        subjects=["articles.>"],
        storage="file",
        max_age=7 * 24 * 3600,
    )
    await nc.close()

asyncio.run(setup())

Seven days of retention gives a consumer that was offline over a long weekend enough history to catch up, without keeping data you will never need.

Publish with a message identifier

The publisher should send a tiny payload and set the Nats-Msg-Id header. JetStream uses that header to detect duplicates within a window of time, so a retry after a network error does not create a second message. Build the identifier from the article and its version, for example the article number and the update timestamp.

async def announce(js, article_id, updated_at):
    payload = f'{{"article_id": {article_id}}}'.encode()
    ack = await js.publish(
        "articles.published",
        payload,
        headers={"Nats-Msg-Id": f"{article_id}-{updated_at}"},
    )
    return ack.seq

The returned acknowledgement carries the stream sequence number. If the call raises an error, treat the publication as not delivered and retry; because of the identifier, a retry is safe. As with every event-driven design, publish after the database transaction commits, or write the event to an outbox table in the same transaction and let a relay send it. Otherwise the consumer may rebuild the feed before the article is visible.

Create a durable pull consumer

A durable consumer has a name, and the server remembers its position under that name. A pull consumer asks for messages in batches when it is ready, which gives you natural flow control: a slow rebuild simply fetches fewer messages per minute. Configure explicit acknowledgements, a reasonable acknowledgement wait and a maximum number of deliveries.

from nats.js.api import ConsumerConfig, AckPolicy

async def consume():
    nc = await nats.connect("nats://nats:4222")
    js = nc.jetstream()
    sub = await js.pull_subscribe(
        "articles.published",
        durable="feed-builder",
        config=ConsumerConfig(
            ack_policy=AckPolicy.EXPLICIT,
            ack_wait=60,
            max_deliver=5,
        ),
    )
    while True:
        try:
            msgs = await sub.fetch(20, timeout=5)
        except nats.errors.TimeoutError:
            continue
        try:
            rebuild_feed()
        except Exception:
            for m in msgs:
                await m.nak(delay=30)
            continue
        for m in msgs:
            await m.ack()

Notice the order: rebuild first, acknowledge afterwards. If the process dies in between, the server waits for the acknowledgement time, then delivers the messages again, and the rebuild runs once more. Because the rebuild is safe to repeat, that is fine.

Batching is free idempotence

The loop above fetches up to twenty messages and rebuilds once for the whole batch. This is a natural debounce. If an editor publishes a dozen articles in a minute, you do not need a dozen rebuilds, because each rebuild reads the full current state anyway. One rebuild per fetch is enough, followed by acknowledgements for every message in the batch. If the rebuild fails, a negative acknowledgement with a delay returns the messages to the queue after thirty seconds instead of hammering the database.

Make the rebuild idempotent

Ignore the message body when you rebuild. Query the database for the newest published articles, render the whole document and replace the file. The function does not care how many times it runs or in which order the messages arrived. Filter on the published status and on a publication date that is not in the future, so that drafts and scheduled articles cannot appear in a public feed even if a stale or malformed message arrives.

Write the file atomically

Write the feed to a temporary file in the target directory and rename it over the final name. A rename on the same filesystem is atomic, so a reader sees either the old complete file or the new complete file. If the feed lives in object storage, upload it in a single request so that the object is replaced as a whole.

import os, tempfile

def write_atomic(path, xml):
    fd, tmp = tempfile.mkstemp(dir=os.path.dirname(path), suffix=".tmp")
    with os.fdopen(fd, "w", encoding="utf-8") as handle:
        handle.write(xml)
    os.chmod(tmp, 0o644)
    os.replace(tmp, path)

Handle failures and poison messages

The maximum delivery setting protects the consumer from a message that can never succeed. After five attempts the server stops delivering it and publishes an advisory event. Subscribe to the advisory subject, log the stream sequence of the failed message and alert a person. Because the rebuild ignores the body, true poison messages are rare, but a bug in the rendering code for one article can still cause repeated failures, so guard each item with its own error handling and skip the one that fails.

Keep a timer as a safety net

A message can be lost in ways that no acknowledgement protects: a producer that crashed after the commit but before the publish, a stream that was recreated, a consumer that was deleted by mistake. Add a slow timer, for example every thirty minutes, that calls the same rebuild function. It costs almost nothing and guarantees the feed catches up. Events are the fast path, and the timer is the guarantee.

Monitor the consumer and the feed

NATS exposes consumer information such as the number of pending messages, the number of messages waiting for acknowledgement and the number of redeliveries. Watch those numbers. A growing pending count means the consumer is stuck or too slow, and a high redelivery count means the rebuild fails or takes longer than the acknowledgement wait. Add two application metrics as well: the age of the feed file and the age of its newest item. Alert when the file is older than your tolerance, for example fifteen minutes on an active site.

Secure the connection

Use authentication and TLS between your services and the NATS servers, and give each service only the permissions it needs. The publisher may publish to the article subjects and the feed builder may consume from the stream and nothing else. Keep the database account of the consumer read-only. Never put drafts, private notes or personal data into the messages, and remember that the public part of the system is the feed file only.

Test the whole path

Publish a staging article and measure the seconds until the new item appears at the public address. Stop the consumer, publish two articles, restart it and confirm that it catches up. Send the same message twice with the same identifier and confirm that only one is stored. Then validate the feed with a standard validator and check the title, link, publication date and image of the new item. Repeat the test after upgrades of the NATS server or the client library.

Serve several feeds from one stream

Most sites need more than one feed: the whole blog, one per category, perhaps one for products. You do not need a stream for each. Add the category to the subject, for example a subject per category, and create one durable consumer per feed with a filter on the subjects it cares about. Each consumer keeps its own position and rebuilds only its own file. A new feed then costs a consumer definition and a rebuild function, and the publisher does not change at all. This is one of the quiet advantages of putting events in a stream: the producer does not need to know who listens.

Why not simply poll the database

A fair question is whether all of this beats a query every minute. For many sites it does not, and a timer alone is a perfectly good design. The stream pays off when freshness matters, when the rebuild is expensive or when several systems already react to publications. If you already run NATS, adding the feed builder is a small step. If you do not, start with the timer and add events later; the idempotent rebuild you wrote for the timer is the same function the consumer will call.

Common mistakes to avoid

• Using core NATS subscriptions and losing messages during restarts.
• Acknowledging the message before the feed file has been written.
• Skipping the message identifier and creating duplicates on retries.
• Publishing before the database commit.
• Leaving the delivery count unlimited so that one failure blocks progress.
• Trusting events alone and having no timer to repair a missed publication.

How PostRSS works with an event-driven feed

PostRSS reads any valid RSS or Atom address that is reachable without a login. It checks the address, finds items it has not posted before and publishes them, with image and link, to the networks you connect. Your NATS consumer keeps the file fresh within seconds of publication, so the next check finds the new article. The features page lists the networks, keyword filters and scheduling options, and the pricing page shows the plans.

সম্পর্কিত পাঠ্য

• Cloudflare Workers RSS feed
• AWS Lambda RSS feed
• RSS vs webhooks vs API polling

The bottom line

NATS JetStream gives a feed builder durable messages, remembered positions and explicit acknowledgements with very little operational weight. Publish a tiny event with a message identifier after the commit, fetch it with a durable pull consumer, rebuild the whole feed idempotently, write atomically, acknowledge only after success, limit redeliveries and keep a timer as a safety net. Then connect the address to an auto-poster, and each new article follows to your social networks without manual work.

FAQ

Do I need JetStream to build an RSS feed?

No. A scheduled job is enough for most sites. JetStream is a good fit if you already run NATS and want durable events with acknowledgements.

What is the difference between core NATS and JetStream?

Core NATS delivers only to subscribers that are connected at that moment. JetStream stores messages and lets durable consumers resume from their last position.

How does JetStream avoid duplicate messages?

The publisher sets a message identifier header, and the stream ignores a second message with the same identifier inside its duplicate window.

When should the consumer acknowledge a message?

Only after the feed file has been written successfully. If the process crashes earlier, the message is delivered again and the idempotent rebuild repeats safely.

Can PostRSS read a feed built from NATS events?

Yes. PostRSS reads any valid RSS or Atom address that is reachable without a login, no matter how the file is produced.