How Do I Fetch All Old Items on an RSS Feed?

Question

RSS/Atom feeds does not allow for historic information to be retrieved. It is up to the publisher of the feed to provide it if they want such as in the blogger or wordpress examples you gave above.

The only reason that Google Reader has more information is that it remembered it from when it came up the first time.

There is some information on something like this talked about as an extension to the ATOM protocol, but I don't know if it is actually implemented anywhere.

As the other replies here mentioned, a feed may not provide archival data but historical items may be available from another source.

Archive.org’s Wayback Machine has an API to access historical content, including RSS feeds (if their bots have downloaded it). I’ve created the web tool Backfeed that uses this API to regenerate a feed containing concatenated historical items. If you'd like to discuss the implementation in detail please get in touch.

In my experience with RSS, the feed is compiled by the last X items where X is a variable. Certain Feeds may have the full list, but for bandwidth sake most places are likely limiting to just the last few items.

The likely answer for google reader having the old info, is that it is storing it on its side for users later.

Further to what David Dean said the RSS/Atom feeds will only contain what the publisher of the feed has up at that moment and someone would need to be actively collecting this informaton in order to have any historical information. Basically Google Reader was doing this for free and when you interacted with it you could retrieve this stored informaton from the google database servers.

Now that they have retired the service, to my knowledge you have two choices. You either have to start collection of this information from your feeds of interest and store the data using XML or some such, or you could pay for this data from one of the companies who sell this type of archived feed information.

I hope this information helps somebody.

Seán

Another potential solution that might not have been available when the question was originally asked and shouldn't require any specific service.

1. Find the URL of the RSS feed you want and use waybackpack to get the archived urls for that feed.
2. Use FeedReader or a similar library to pull down the archived RSS feed.
3. Take the URLs from each feed and scrape them as you wish. If you're going way back in time it's possible there might be some dead links.

All previous answers more or less relied on existing services to still have a copy of that feed or the feed engine to be able to provide older items dynamically.

There's though another, admittedly pro-active and rather theoretical way to do so: Let your feedreader use a caching proxy which semantically understands RSS and/or Atom feeds and caches them on a per-item base up to as many items as you configure.

If the feedreader doesn't poll feeds regularily, the proxy could fetch known feeds time-based on its own to not miss an item in highly volatile feeds like the one from User Friendly which has only one item and changes every day (or at least used to do so). Hence if the feedreadere.g. crashed or lost network connection while you are away for a few days, you might loose items in your feedreader's cache. Having the proxy to fetch those feeds regularily (e.g. from a data center instead from at home or on a server instead of a laptop) allows you to easily run the feedreader only then and when without loosing items which were posted after your feedreader fetched feeds the last time but rotated out again before you fetch them the next time.

I call that concept a Semantic Feed Proxy and I've implemented a proof of concept implementation called sfp. It's though not much more than a proof of concept and I haven't developed it further. (So I'd be happy about hints to projects with similar ideas or purposes. :-)