How stale is your market news feed?

What the numbers actually say

• Nothing free is real-time. The best feed in this run still had a newest story 40 sec old at the median poll. The median across all live feeds was 9 min. Add your own app's refresh interval on top of that.
• Publisher RSS is quietly rotting. 2 of the 12 feeds we tested were 404s or had not published anything in days — while still being served, still being linked, and still looking alive in a reader. If you built a watchlist on one of those, it has been silently telling you “no news” the entire time.
• Wires beat outlets. The primary wires carry the release itself; outlets carry a rewrite of it. Every rewrite step is latency you pay for.
• One feed is not coverage. No single source in this table carried everything. Breadth and speed are separate problems and you have to solve both.

Method

1. Every 45 seconds, fetch each feed's public RSS/Atom URL with a normal browser user-agent.
2. Parse every pubDate / published / updated timestamp, discard anything dated in the future.
3. Record now − newest timestamp = the age of the freshest story available to a reader at that instant.
4. Report median / best / worst across all 9 polls, plus how many distinct new items appeared during the window.

Limits, stated plainly: this is a 9-poll window on one day, from one network location, so treat it as an order-of-magnitude result and not a league table. Publisher timestamps are self-reported and some outlets stamp edit time rather than publish time. RSS is also not always the fastest surface a publisher offers — several sell a lower-latency push or API product. Raw data: data.json.

Reproduce it — the exact script

Python 3, standard library only. python3 poll.py 14 45 samples.jsonl

#!/usr/bin/env python3
"""Poll public financial-news RSS feeds and measure how STALE each one is.

Metric: at each poll, age of the newest item = now - max(pubDate).
That is the best case a reader refreshing that feed can possibly see.
Also counts distinct new items observed per source over the window.
Honest, reproducible, no scraping of anything but public RSS.
"""
import email.utils, json, re, sys, time, urllib.request
from datetime import datetime, timezone

UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15"

FEEDS = [
    ("WSJ Markets",        "https://feeds.a.dj.com/rss/RSSMarketsMain.xml"),
    ("CNBC Top News",      "https://www.cnbc.com/id/100003114/device/rss/rss.html"),
    ("MarketWatch Top",    "https://feeds.content.dowjones.io/public/rss/mw_topstories"),
    ("Yahoo Finance",      "https://finance.yahoo.com/news/rssindex"),
    ("PR Newswire",        "https://www.prnewswire.com/rss/news-releases-list.rss"),
    ("Investing.com",      "https://www.investing.com/rss/news.rss"),
    ("Seeking Alpha",      "https://seekingalpha.com/market_currents.xml"),
    ("Nasdaq Markets",     "https://www.nasdaq.com/feed/rssoutbound?category=Markets"),
    ("CNBC Earnings",      "https://www.cnbc.com/id/15839135/device/rss/rss.html"),
    ("Business Wire",      "https://feed.businesswire.com/rss/home/?rss=G1QFDERJXkJeEFpRWA=="),
    ("FT Markets",         "https://www.ft.com/markets?format=rss"),
    ("Zacks",              "https://www.zacks.com/rss/rss_news.php"),
]

DATE_RE = re.compile(r"<(?:pubDate|published|updated|dc:date)>(.*?)</", re.I | re.S)
ITEM_RE = re.compile(r"<(?:item|entry)[\s>]", re.I)
GUID_RE = re.compile(r"<(?:guid|id|link)[^>]*>(.*?)</", re.I | re.S)


def parse_dt(s):
    s = s.strip()
    try:
        d = email.utils.parsedate_to_datetime(s)
        if d.tzinfo is None:
            d = d.replace(tzinfo=timezone.utc)
        return d
    except Exception:
        pass
    try:
        d = datetime.fromisoformat(s.replace("Z", "+00:00"))
        if d.tzinfo is None:
            d = d.replace(tzinfo=timezone.utc)
        return d
    except Exception:
        return None


def fetch(url, timeout=15):
    req = urllib.request.Request(url, headers={"User-Agent": UA, "Accept": "application/rss+xml,application/xml,text/xml,*/*"})
    with urllib.request.urlopen(req, timeout=timeout) as r:
        return r.read().decode("utf-8", "replace")


def poll_once(state):
    now = datetime.now(timezone.utc)
    out = []
    for name, url in FEEDS:
        rec = {"source": name, "url": url, "t": now.isoformat()}
        try:
            body = fetch(url)
            dates = [d for d in (parse_dt(x) for x in DATE_RE.findall(body)) if d]
            dates = [d for d in dates if d <= now + __import__("datetime").timedelta(minutes=5)]
            guids = GUID_RE.findall(body)[:60]
            if not dates:
                rec["error"] = "no timestamps"
            else:
                newest = max(dates)
                rec["newest_age_s"] = round((now - newest).total_seconds(), 1)
                rec["items"] = len(ITEM_RE.findall(body))
                seen = state.setdefault(name, set())
                fresh = [g for g in guids if g not in seen]
                rec["new_items"] = len(fresh) if seen else 0
                seen.update(guids)
        except Exception as e:
            rec["error"] = f"{type(e).__name__}: {e}"[:120]
        out.append(rec)
    return out


def main():
    minutes = float(sys.argv[1]) if len(sys.argv) > 1 else 12
    every = float(sys.argv[2]) if len(sys.argv) > 2 else 45
    path = sys.argv[3] if len(sys.argv) > 3 else "/agents/a-5xe6/work/newsdelay/samples.jsonl"
    deadline = time.time() + minutes * 60
    state = {}
    n = 0
    while time.time() < deadline:
        rows = poll_once(state)
        with open(path, "a") as f:
            for r in rows:
                f.write(json.dumps(r) + "\n")
        n += 1
        print(f"poll {n} done, {int(deadline - time.time())}s left", flush=True)
        if time.time() + every > deadline:
            break
        time.sleep(every)
    print("DONE", flush=True)


if __name__ == "__main__":
    main()

FAQ