Page 1 of 5 12345 LastLast
Results 1 to 15 of 64

Thread: *UNOFFICIAL* Frontier Forums Developer Posts RSS feed

  1. #1

    *UNOFFICIAL* Frontier Forums Developer Posts RSS feed

    A couple of weeks ago I finally got around to writing the necessary code to scrape 'developer' posts from the Frontier Forums and produce an RSS feed from that data. I then updated it for the upgrade to the current version of the forums. It all seems to be working OK (high forum load today aside), so I thought it might be time to publicise the existence of the feed.

    You can find details about the feed on https://ed.miggy.org/devposts.html (now present in my forum signature). As that page says... if you have an RSS Extension or your browser otherwise handles RSS feeds then you should see an icon in your URL/Address bar. Else use the URL in that page to add the feed to your reader by hand.

    Any feedback or concerns about this RSS feed can be addressed to me in this thread, through PM, or to the email address: edrss@miggy.org (which is included in the feed details).

    Also, kudos to Bewilderbeest for his dev post tracker which prodded me to actually complete work on this project.

    ChangeLog
    2014-10-30 ~22:15 UTC - Fixed an issue where I was only picking up replies from devs, and not the post for when they started a new thread. All posts since midnight today re-imported to be sure.

    eh, who am I kidding, just read the changelog on the main site https://ed.miggy.org/devposts.html

  2. #2
    Thank you, this is very useful, much appreciated.
    PMC Tactical
    Operation Flashpoint, Armed Assault, ArmA 2, ArmA 3 and Falcon 4 Theaters.
    Addons/Mods, Tutorials and more.
    http://tactical.nekromantix.com/
    Elite Dangerous videos from PMC youtube channel.

  3. #3
    Great idea and all seems to be working well, thanks for sharing.

    Ab
    http://i205.photobucket.com/albums/bb74/Steeple_jack/8498379e-441b-43db-8412-cdea5105f8b9_zpsfd040db4.png

  4. #4
    I think this is fantastic. The engineer in me though thinks this ought to be done on the forum side to reduce load. Until FD get around to that.. I'll use yours

    Edit - Feedly tells me there are already 5 readers!

  5. #5
    Originally Posted by Xzanron View Post (Source)
    I think this is fantastic. The engineer in me though thinks this ought to be done on the forum side to reduce load. Until FD get around to that.. I'll use yours

    Edit - Feedly tells me there are already 5 readers!
    Oh I agree. I even waited on announcing this to see if there was such a feed in these new forums. But when I saw the RSS icons back on the forum section listings, but still no hint of such a dev tracker feed I decided it was time to let the world know I had an alternative .

  6. #6
    btw, is there any news on some official RSS feeds for the whole forums?

  7. #7
    What a great idea, thanks for making this!

  8. #8

  9. #9

  10. #10
    I'm using this with Firefox, when Firefox is first run I get a list of the Dev posts, but it doesn't seem to be updating when new posts are made, or when it does its several hours later after the Devs post.

    eg compared with results returned from http://www.altairvista.com/frontier.html

    not sure if its an issue with the Forum or Firefox, anyone else having this issue?

  11. #11
    +1 Rep
    Excellent idea and something i was hoping FD would have done with the forum upgrade. Now i can delete my FD dev contacts list as i no longer have to sit down everyday and go through them one by one to see any official responses.

  12. #12

    Thumbs up

    Kudos for doing this. Was exactly the idea I had for a weekend project to get a data source backend for a client UI.
    One question though, what historical data do you have?

    If you have more than just the last day's worth of data, would you mind sharing that?

    I had a previous discussion with Andrew Sayers on reuse of scraped data, but his project didn't rely on much historical information, so I'm keen to understand whether or not you're also just retaining the last couple of days worth of dev posts, or if you have a full history.
    It would be great to have a common source for this data so we don't need to have nthteen scrapers, one for each project, putting additional load on the already overloaded forum site.

  13. #13
    Originally Posted by mojonaut View Post (Source)
    I'm using this with Firefox, when Firefox is first run I get a list of the Dev posts, but it doesn't seem to be updating when new posts are made, or when it does its several hours later after the Devs post.

    eg compared with results returned from http://www.altairvista.com/frontier.html

    not sure if its an issue with the Forum or Firefox, anyone else having this issue?
    What I see in my Tiny Tiny RSS instance this morning matches that page, and indeed I actually had no collector errors overnight (the load problems on the forums since upgrade have caused some issues). That and given the feed passes http://feedvalidator.org/ checks makes me think this is an issue with your reader.

  14. #14
    Originally Posted by jojje View Post (Source)
    Kudos for doing this. Was exactly the idea I had for a weekend project to get a data source backend for a client UI.
    One question though, what historical data do you have?

    If you have more than just the last day's worth of data, would you mind sharing that?

    I had a previous discussion with Andrew Sayers on reuse of scraped data, but his project didn't rely on much historical information, so I'm keen to understand whether or not you're also just retaining the last couple of days worth of dev posts, or if you have a full history.
    It would be great to have a common source for this data so we don't need to have nthteen scrapers, one for each project, putting additional load on the already overloaded forum site.
    I'm retaining a full history from when I started collecting. The first post I have in the live DB is:

    Code:
    2014-10-10 11:52:00 | showthread.php?p=857694#post857694 | It's the semi major axis in the left panel. ...     | showthread.php?t=48602 | Does the System Map show distance from Star?                              | Beta Discussion Forum
    Currently the RSS feed only has the 100 most recent posts in it at the time of generation (which is set to the time of the most recent post).

    I was initially going to have it scrape every page of post histoy (on a member profile, not the whole forums!) from the old forums, but it would have been a bit of extra work to make it page through, so instead it seeded with just what was on the first page at the time. Each collection run, per forum member it checks, keeps going until it sees a post it has seen before, then stops, but again with the assumption that the first page will contain this. So if a dev goes on a REALLY fast posting spree and/or my collector has sufficient downtime I might miss posts. If that ever does become an issue I'll make it follow the "More Activity" link as needed.

  15. #15
    Very handy thanks for doing this +1 rep

Page 1 of 5 12345 LastLast