rss2masto

rss2masto

A Go library for publishing RSS/Atom feed items as Mastodon posts. Designed to handle hundreds or thousands of feeds concurrently, with built-in scheduling, deduplication via Redis, and ETag-based conditional fetching.

Features

• Concurrent processing of any number of RSS/Atom feeds using goroutines
• Per-feed scheduler with configurable check interval
• Deduplication via Redis — each item is posted exactly once
• ETag / If-None-Match support — unchanged feeds skip parsing entirely
• HTML sanitization and automatic post truncation to instance character limit
• Hashtag generation from feed item categories or URL patterns
• Text replacement rules per feed
• Post visibility control (public, unlisted, private)
• Automatic language detection from feed metadata
• Follower count tracking per Mastodon account
• Optional state persistence to feed.yaml

Requirements

• Go 1.25+
• Redis (used for deduplication and caching) — optional.
  If Redis is unavailable, an in-process TinyLFU cache is used as a fallback. Deduplication will not persist across restarts in that case.

Installation

go get -u github.com/glaydus/rss2masto

Or add the import and run go mod tidy:

import "github.com/glaydus/rss2masto"

Quick start

package main

import (
    "log"
    "time"

    "github.com/glaydus/rss2masto"
)

func main() {
    fm, err := rss2masto.NewFeedsMonitor()
    if err != nil {
        log.Fatalln(err)
    }

    // Run once
    fm.Start()

    // Or run on a ticker (e.g. every minute)
    ticker := time.NewTicker(time.Minute)
    for range ticker.C {
        fm.Start()
    }
}

NewFeedsMonitor reads feed.yaml from the current directory. Each call to Start processes all feeds whose scheduler counter has reached its configured interval.

Configuration — feed.yaml

instance:
  url: https://mastodon.example        # Mastodon instance base URL (HTTPS required)
  lang: en                             # default post language (ISO 639-1)
  timezone: Europe/Warsaw              # timezone for timestamps (IANA format)
  limit:                               # max characters per post; auto-detected from instance if empty
  save: false                          # persist last_run timestamps back to feed.yaml after each run

  feed:
    - name: My Tech Blog               # display name (used as log prefix and idempotency key prefix)
      url: https://example.com/rss     # RSS or Atom feed URL (single URL or a list of fallback URLs)
      token: <MASTODON_API_TOKEN>      # Mastodon access token for this account
      interval: 10                     # check every N scheduler ticks (e.g. 10 = every 10 minutes if ticker is 1 min)
      visibility: public               # public | unlisted | private
      prefix: Tech                     # optional hashtag prefix added to every generated tag
      hashtag:                         # static hashtag always added to every post from this feed
      hashlink:                        # regex with one capture group — extracts hashtag from item URL
      replace_from:                    # regex applied to post description
      replace_to:                      # replacement string (used with replace_from)
      replace_link:                    # regex applied to item link — all matches are removed from the URL

    - name: Another Feed
      url: https://another.example/feed.xml
      token: <ANOTHER_TOKEN>
      interval: 30
      visibility: unlisted

    - name: Feed With Fallbacks
      url:                             # list of URLs — first is primary, rest are fallbacks tried in order
        - https://primary.example/feed.xml
        - https://mirror.example/feed.xml
      token: <YET_ANOTHER_TOKEN>
      interval: 60
      visibility: public

Field reference

Redis

Redis is used for two purposes:

1. Deduplication — an idempotency key (<feed_prefix>:<item_hash>) is stored after each successful post. Items already in Redis are skipped on subsequent runs.
2. Caching — a local TinyLFU cache (backed by go-redis/cache) reduces Redis round-trips for hot keys.

The same idempotency key is also sent to the Mastodon API as the Idempotency-Key request header on every post. This provides a second layer of duplicate protection — if the same request is submitted more than once within 1 hour (e.g. due to a retry), the Mastodon instance will return the original status instead of creating a duplicate.

The Redis connection is configured via the REDIS_HOST environment variable:

export REDIS_HOST=localhost:6379

The value is passed directly to redis.ParseURL, so full Redis URLs are also accepted:

export REDIS_HOST=redis://:password@localhost:6379/0

Hash dictionary

When hashtags are extracted from item links via hashlink, the raw URL segment is looked up in an optional dictionary before being used as a hashtag. This lets you map slugs that would otherwise be unusable (contain hyphens, lack diacritics, etc.) to proper hashtag forms.

The dictionary is loaded from ./hashdict.txt at startup. The path can be changed by setting rss2masto.HashDictFile before calling NewFeedsMonitor. The dictionary can also be reloaded at runtime without restarting by calling rss2masto.ReloadHashDict(data).

File format

# lines starting with # are comments
krakow=Kraków
hokej-na-lodzie=HokejNaLodzie
zuzel=Żużel

Each line is key=value. The key is the raw string extracted from the URL; the value is the hashtag that will be used instead. If a key is not found in the dictionary, the original string is used (with title-casing applied).

Runtime reload

data, err := os.ReadFile("hashdict.txt")
if err == nil {
    rss2masto.ReloadHashDict(data)
}

Passing nil re-reads the file at HashDictFile:

rss2masto.ReloadHashDict(nil)

Scheduler

Start() is designed to be called repeatedly on a fixed ticker. Each feed has an interval field that acts as a divisor — a feed with interval: 10 is only processed on every 10th call to Start(). This lets you run a single tight ticker (e.g. every minute) while checking different feeds at different frequencies without managing multiple goroutines externally.

ticker: 1 minute
feed A interval: 5  → checked every  5 minutes
feed B interval: 15 → checked every 15 minutes
feed C interval: 60 → checked every  1 hour

Start() is safe to call concurrently — a built-in atomic guard prevents overlapping runs.

Scaling

The library is designed to handle large numbers of feeds efficiently:

• All feeds within a single Start() call are processed in parallel via goroutines.
• HTTP fetching uses fasthttp with connection pooling and DNS caching.
• ETag support means unchanged feeds generate zero parsing overhead.
• Redis connection pool is pre-configured for high concurrency (20 connections, 5 idle minimum).

There is no hard limit on the number of feeds. Practical limits depend on available Redis connections, network bandwidth, and Mastodon API rate limits per token.

Environment variables

License

MIT