*UNOFFICIAL* Frontier Forums Developer Posts RSS feed

[Athan]

A couple of weeks ago I finally got around to writing the necessary code to scrape 'developer' posts from the Frontier Forums and produce an RSS feed from that data. I then updated it for the upgrade to the current version of the forums. It all seems to be working OK (high forum load today aside), so I thought it might be time to publicise the existence of the feed.

You can find details about the feed on https://ed.miggy.org/devposts.html (now present in my forum signature). As that page says... if you have an RSS Extension or your browser otherwise handles RSS feeds then you should see an icon in your URL/Address bar. Else use the URL in that page to add the feed to your reader by hand.

Any feedback or concerns about this RSS feed can be addressed to me in this thread, through PM, or to the email address: edrss@miggy.org (which is included in the feed details).

Also, kudos to Bewilderbeest for his dev post tracker which prodded me to actually complete work on this project.

ChangeLog
2014-10-30 ~22:15 UTC - Fixed an issue where I was only picking up replies from devs, and not the post for when they started a new thread. All posts since midnight today re-imported to be sure.

eh, who am I kidding, just read the changelog on the main site https://ed.miggy.org/devposts.html

 

[Snake Man]

Thank you, this is very useful, much appreciated.

 

[Arbanax]

Great idea and all seems to be working well, thanks for sharing.

Ab

 

[Xzanron]

I think this is fantastic. The engineer in me though thinks this ought to be done on the forum side to reduce load. Until FD get around to that.. I'll use yours

Edit - Feedly tells me there are already 5 readers!

 

[Athan]

I think this is fantastic. The engineer in me though thinks this ought to be done on the forum side to reduce load. Until FD get around to that.. I'll use yours

Edit - Feedly tells me there are already 5 readers!

Oh I agree. I even waited on announcing this to see if there was such a feed in these new forums. But when I saw the RSS icons back on the forum section listings, but still no hint of such a dev tracker feed I decided it was time to let the world know I had an alternative .

 

[Peter Powers]

btw, is there any news on some official RSS feeds for the whole forums?

 

[Xeelix]

What a great idea, thanks for making this!

 

[brumster]

Have some rep.. err. oh...

 

[rootsrat]

This is extremely useful, thanks!

 

[mojonaut]

I'm using this with Firefox, when Firefox is first run I get a list of the Dev posts, but it doesn't seem to be updating when new posts are made, or when it does its several hours later after the Devs post.

eg compared with results returned from http://www.altairvista.com/frontier.html

not sure if its an issue with the Forum or Firefox, anyone else having this issue?

 

[Larrow]

+1 Rep
Excellent idea and something i was hoping FD would have done with the forum upgrade. Now i can delete my FD dev contacts list as i no longer have to sit down everyday and go through them one by one to see any official responses.

 

[jojje]

Kudos for doing this. Was exactly the idea I had for a weekend project to get a data source backend for a client UI.
One question though, what historical data do you have?

If you have more than just the last day's worth of data, would you mind sharing that?

I had a previous discussion with Andrew Sayers on reuse of scraped data, but his project didn't rely on much historical information, so I'm keen to understand whether or not you're also just retaining the last couple of days worth of dev posts, or if you have a full history.
It would be great to have a common source for this data so we don't need to have nthteen scrapers, one for each project, putting additional load on the already overloaded forum site.

 

[Athan]

I'm using this with Firefox, when Firefox is first run I get a list of the Dev posts, but it doesn't seem to be updating when new posts are made, or when it does its several hours later after the Devs post.

eg compared with results returned from http://www.altairvista.com/frontier.html

not sure if its an issue with the Forum or Firefox, anyone else having this issue?

What I see in my Tiny Tiny RSS instance this morning matches that page, and indeed I actually had no collector errors overnight (the load problems on the forums since upgrade have caused some issues). That and given the feed passes http://feedvalidator.org/ checks makes me think this is an issue with your reader.

 

[Athan]

Kudos for doing this. Was exactly the idea I had for a weekend project to get a data source backend for a client UI.
One question though, what historical data do you have?

If you have more than just the last day's worth of data, would you mind sharing that?

I had a previous discussion with Andrew Sayers on reuse of scraped data, but his project didn't rely on much historical information, so I'm keen to understand whether or not you're also just retaining the last couple of days worth of dev posts, or if you have a full history.
It would be great to have a common source for this data so we don't need to have nthteen scrapers, one for each project, putting additional load on the already overloaded forum site.

I'm retaining a full history from when I started collecting. The first post I have in the live DB is:

2014-10-10 11:52:00 | showthread.php?p=857694#post857694 | It's the semi major axis in the left panel. ...     | showthread.php?t=48602 | Does the System Map show distance from Star?                              | Beta Discussion Forum

Currently the RSS feed only has the 100 most recent posts in it at the time of generation (which is set to the time of the most recent post).

I was initially going to have it scrape every page of post histoy (on a member profile, not the whole forums!) from the old forums, but it would have been a bit of extra work to make it page through, so instead it seeded with just what was on the first page at the time. Each collection run, per forum member it checks, keeps going until it sees a post it has seen before, then stops, but again with the assumption that the first page will contain this. So if a dev goes on a REALLY fast posting spree and/or my collector has sufficient downtime I might miss posts. If that ever does become an issue I'll make it follow the "More Activity" link as needed.

 

[Colonel Kenney]

Very handy thanks for doing this +1 rep

 

[jippyuk]

Was getting tonnes of dev posts in the rss feed you provided (thanks so much for that!) yesterday but today its not given me any notifications... it still working?

 

[Athan]

Was getting tonnes of dev posts in the rss feed you provided (thanks so much for that!) yesterday but today its not given me any notifications... it still working?

Yes, it's still working. I use it myself as I mentioned above (why else would I have created it?). The most recent post showing in my tt-rss instance is from 15:33 UTC today.

For anyone that's having issues please specify exactly how you're using the feed. I might end up asking you to hit another page so I know your IP and useragent to then go check the access.log for hits on the RSS feed to see if I can say "you're just not even attempting to download it again", or the like.

 

[mojonaut]

What I see in my Tiny Tiny RSS instance this morning matches that page, and indeed I actually had no collector errors overnight (the load problems on the forums since upgrade have caused some issues). That and given the feed passes http://feedvalidator.org/ checks makes me think this is an issue with your reader.

Strange

As I type this the latest post in the RSS feed is 5:39pm by Ed Lewis, I setup the feed in Internet Explorer and it returns the same ie Ed's 5:39pm post is the latest.

However http://www.altairvista.com/frontier.html lists Ashley Barleys 7:52pm post as the newest but it doesn't appear in either Firefox or IE's rss feed.

 

[Athan]

Strange

As I type this the latest post in the RSS feed is 5:39pm by Ed Lewis, I setup the feed in Internet Explorer and it returns the same ie Ed's 5:39pm post is the latest.

However http://www.altairvista.com/frontier.html lists Ashley Barleys 7:52pm post as the newest but it doesn't appear in either Firefox or IE's rss feed.

Aha! You're quite right, I was only looking for "forum_post"s when I should be looking for "forum_thread"s and maybe some other things as well. I'll get it updated ASAP (and may well purge the last couple of days of posts on every ID to make sure the next run picks up everything).

 

[Athan]

OK, the "not picking up when a dev starts a new thread" issue has been fixed. Live RSS feed now has Ashley's "Minor Update to 3.02 incoming!" post, and any others it might have missed today.