Page 157 of 189 FirstFirst ... 147153154155156157158159160161167 ... LastLast
Results 2,341 to 2,355 of 2831

Thread: TradeDangerous: power-user trade optimizer

  1. #2341
    Originally Posted by Avi0013 View Post (Source)
    It's back online!

    I wonder if there is any way we can help maddavo make it more resilient or have a backup server.
    Someone a few pages back in this forum suggested that we need to get a US and EU mirror of Maddavo’s site. I have no idea how that is done though. Is there anyone technical on the forum could give us a clue how this could be achieved? Would the mirror site still work if Maddavo’s goes down or would the problem be replicated?

  2. #2342
    Originally Posted by CMDR Ibizan View Post (Source)
    Someone a few pages back in this forum suggested that we need to get a US and EU mirror of Maddavo’s site. I have no idea how that is done though. Is there anyone technical on the forum could give us a clue how this could be achieved? Would the mirror site still work if Maddavo’s goes down or would the problem be replicated?
    Would not be easier to move to a JSON format from other dumps?

  3. #2343
    Originally Posted by AnthorNet View Post (Source)
    Would not be easier to move to a JSON format from other dumps?
    Quite possibly Anthornet but I have no idea how to do that!

  4. #2344
    Neither do I, but I'd be interested in helping get a mirror/backup off of the ground if possible. What is entailed in either mirroring or haveing similar functionality using JSONS or whatnot? My programming skills are limited to statistical programming; I have no experience or skills with websites or internet communication

  5. #2345
    It's stuck again

  6. #2346
    It seems that Oliver (kfsone), bernd (gazelle), and Dave (maddavo) have all moved on to bigger and better things, and G-d bless them for it! Is there anyone who uses Trade Dangerous who would be able to figure out how the imports work (perhaps based on the edapi plugin) and who could engineer a way for us to use the master EDDB database (which is constantly being updated by those of us running EDMC or EDDiscovery and the like) instead of maddavo's database? As EDDB is updated in realtime, it's more accurate than maddavo, I reckon, but they may want us (or whomever) to cretae a semi-static databse which only pulls maybe once an hour so as to minimize traffic. I don't know Perhaps there is a charitable soul in EDDB or EDSM etc who would be willing to help us out? Is there a developers forum on this or another related database which at which we can ask?

  7. #2347

    edapi plugin

    hi all,

    is it not possible in a easy way to use this plugin ?
    (Avi0013 also mentioned it many times)
    the data is already there !

    i still love TD (or would love it even more again)
    if only one gifted person could do a little programming,
    to make it again a little more usable (speed especially),
    i would love this person too, forever

    greetings and thank you
    (beyond all things)
    nepo

  8. #2348
    I have not abandoned TD. I'm just really lazy to change/add things if I don't need them. I'm only using TD for my private dataset and for that the EDAPI plugin ist enough.

    Let's see what's needed to make an EDDB importer:
    • Download listings.csv (~141,439 kb)
    • Where is that commodity_id?
      • Integrate commodity_id form EDDB into TD
      • Sync commodities.json (~99 kb)
    • Ok but station_id is still missing.
      • Integrate station_id form EDDB into TD
      • Sync stations.jsonl (~112,158 kb)
      • But the stations have only an system_id?
        • Integrate system_id form EDDB into TD
        • Sync systems_populated.jsonl (~21,040 kb)
    • Now we are ready to import the prices from EDDB.


    As you can see, that's some work to do. And in the worst case you'll have to download ~274,736 kb every day from EDDB. Don't know if themroc (EDDB) would like that.

  9. #2349
    Originally Posted by gazelle View Post (Source)
    I have not abandoned TD. I'm just really lazy to change/add things if I don't need them. I'm only using TD for my private dataset and for that the EDAPI plugin ist enough.

    Let's see what's needed to make an EDDB importer:
    • Download listings.csv (~141,439 kb)
    • Where is that commodity_id?
      • Integrate commodity_id form EDDB into TD
      • Sync commodities.json (~99 kb)
    • Ok but station_id is still missing.
      • Integrate station_id form EDDB into TD
      • Sync stations.jsonl (~112,158 kb)
      • But the stations have only an system_id?
        • Integrate system_id form EDDB into TD
        • Sync systems_populated.jsonl (~21,040 kb)
    • Now we are ready to import the prices from EDDB.


    As you can see, that's some work to do. And in the worst case you'll have to download ~274,736 kb every day from EDDB. Don't know if themroc (EDDB) would like that.
    Since Maddavo's website is now defunct, what wold be involved in taking over that functionality from him ad providing a replacement site?

  10. #2350
    Originally Posted by MarkAusten View Post (Source)
    Since Maddavo's website is now defunct, what wold be involved in taking over that functionality from him ad providing a replacement site?
    Maddavo made the market share by himself and never shared the code (with me), so I can't tell you what would be needed. Abstractly speaking you would need:
    • 24/7 server
    • EDDN listener
    • some Database
    • processing program

  11. #2351
    Hi, gazelle. Still no thought about Oliver's approved changes to the penalty?

  12. #2352
    Originally Posted by Avi0013 View Post (Source)
    Hi, gazelle. Still no thought about Oliver's approved changes to the penalty?
    I'm still not happy about it. Try it out in Excel/Calc for different values. I was thinking about a sigmoid function like:
    Code:
    # maybe make A and B parameters
    A = 4000 # 50% value for ls==A if lsPenalty==1
    B = 500 # smaller = more rectangular, higher more S shaped, >1
    
    # this for each ls value, 0 <= lsPenalty <= 1
    ls = dstStation.lsFromStar
    val = math.exp((A-ls)/B)
    multiplier = 1 - lsPenalty/(1 + val)
    /edit: formula

  13. #2353
    Originally Posted by gazelle View Post (Source)
    Maddavo made the market share by himself and never shared the code (with me), so I can't tell you what would be needed. Abstractly speaking you would need:
    • 24/7 server
    • EDDN listener
    • some Database
    • processing program
    Anyone want to look into the TCE-Relay python program and see if that can be used as a replacement plugin (with appropriate modifications, of course)?

    https://github.com/flattermann/TCE-Relay

  14. #2354
    Okay, so, this is about 95% finished, in that I haven't gotten the RareItem import coded yet, but I've spent the last few days working on a plugin that does half the job Maddavo's site used to do:

    When the plugin is run (depending on the chosen options), it will download the needed daily dumps from eddb and process them to create all the tables in the TradeDangerous database, including market data, and will generate a .prices file from the listings.csv

    This won't get the most up-to-the-minute market data, since the eddb dumps happen once per day, it gets everything but the data submitted since the latest dump, which means not more than 24 hours after the submission.
    It does check to see if the data already entered is newer than the data from eddb, so it won't overwrite newer data gathered by the user (i.e. with "trade.py import -P edapi -O eddn") if that's more recent.

    I'm working on a user-side eddn listen server next, so that this plugin and the listen server combined fully replace Maddavo's site.

    It makes a few changes to the database, removing AUTOINCREMENT from most of the tables and adding a cost column to the Upgrade table, which means on first run it has to completely erase the database and rebuild it from scratch. I did not make the decision to do this lightly: AUTOINCREMENT was removed so that the TDDB entries would match with EDDB, not have a random id based on at what point the entry was added, which simplifies adding entries that reference entries in other tables (like Stations reference the id of the System they're in, for example).

    For some reason, trade.py throws an error trying to compile the help for the plugin, so these are the options:

    'item': "Regenerate Categories and Items using latest commodities.json dump.",
    'system': "Regenerate Systems using latest system-populated.jsonl dump.",
    'station': "Regenerate Stations using latest stations.jsonl dump. (implies '-O system')",
    'ship': "Regenerate Ships using latest coriolis.io json dump.",
    'shipvend': "Regenerate ShipVendors using latest stations.jsonl dump. (implies '-O system,station,ship')",
    'upgrade': "Regenerate Upgrades using latest modules.json dump.",
    'upvend': "Regenerate UpgradeVendors using latest stations.jsonl dump. (implies '-O system,station,upgrade')",
    'listings': "Update market data using latest listings.csv dump. (implies '-O item,system,station')",
    'all': "Update everything with latest dumpfiles. (Regenerates all tables)",
    'clean': "Erase entire database and rebuild from empty. (Regenerates all tables.)",

    If you want to try it out, place 'eddblink_plug.py' in the plugins folder of your TD install and run it.
    On first run it will run with '-O clean' regardless of the options passed to it, because of the changes it needs to make to the database.
    Thereafter, it is recommended to run with '-O listings', as the Ship, ShipVendor, Upgrade, and UpgradeVendor tables don't need to be updated under normal circumstances.

    If anyone has any improvements to make the thing run faster, I'd appreciate it.

    If anyone is willing to mirror the eddb files so as not to stress the eddb server, that'd be awesome, let me know the URL and I'll update the plugin to use that instead.

    I've included the .prices file that it generated as well as the updated Category, Item, Station, and System .csv files, so anyone that wants can have a market update that's current as of approximately 2018-04-21 03:00:00 UTC (The approximate time of the latest dump as of this writing.)

    The plugin and the market update can both be found in my Google Drive:
    eddblink_plug.py
    Market Update 2018-04-21.7z

  15. #2355
    Wow eyeonus, this was unexpected. I was beginning to think that TD was on its last legs. Many thanks for doing this. I'll give it a try tomorrow when I'm back at my pc.