Page 173 of 209 FirstFirst ... 168171172173174175178 ... LastLast
Results 2,581 to 2,595 of 3130

Thread: TradeDangerous: power-user trade optimizer

  1. #2581

    RUNNING TRADE DANGEROUS WITH EDDBLINK PLUGIN CLIENT/SERVER

    Eyeonus & I have been working (and playing) pretty hard over the past couple of days to bring you this latest version of the EDDBLink plugin.

    The net result is that the TD community can, once again, have up to the minute pricing data from the EDDN Network.

    I know most users have been following along with this process, but I wanted to make a summary here, so it's clear how to install and run this software. Bearing in mind it's a power user's package, I'm not going into minute detail of the install process. If anyone does struggle, do feel free to post.

    Firstly you will need to install Python.
    If you are running Linux, likely you already have it - though possibly the wrong version. If anyone gets stuck with Linux, pm me. I had enough trouble myself that I wouldn't want to try and explain it here.

    If you are running windows, download the latest release from:
    https://www.python.org/downloads/windows/
    At time of writing this is 3.6.5

    Once you have it installed, open up a command prompt or a shell and install required module requests using pip[1]
    Code:
    pip install requests
    Next download and unpack the Trade Dangerous software from github. We are using the bgol fork from:
    https://bitbucket.org/bgol/tradedangerous
    Your choice of the main or horizons branch.

    You now need to download the eddblink plugin from:
    https://github.com/eyeonus/EDDBlink
    The file eddblink_plug needs to be placed into the plugins folder of your TD tree.

    In order to generate trades, you need to get a hold of the latest data. This will always be available on my website[2] or failing that, the plugin will fail back to downloading the latest data from EDDB[3].
    You firstly need to run the command
    Code:
    trade.py import -P eddblink -O clean
    This will make some required changes to the database then return back to the command prompt, instructing you to run the same command a second time. On the second run, all the data will be downloaded and processed into the TD database.
    You are now ready to generate trades. There is a fine manual at the bgol link above which tells you how to do this, with plenty of examples.

    To keep your data up to date, run
    Code:
    trade.py import -P eddblink -O all
    This will check for and download only those data files which have changed since you last ran an import. Other plugin options are documented on the EDDBLink page as above. Another common choice is to use -O all,skipvend to save a little time by not regenerating the ship and module vendors.

    To ensure you have up to the minute pricing data, the server does two things.
    1: Downloads all the nightly data files from EDDB. This includes systems, stations, etc as well as the listings.csv which contains all pricing information held by EDDB
    2: Listens to the data stream from EDDN and every few minutes builds a new file listings-live.csv which contains only those prices received since the last listings.csv was retrieved from EDDB
    Those two files together make up the totality of the pricing data. If you run with -O all (as above) more than once a day, chances are you will only get an updated listings-live.csv on the second or subsequent data imports.

    Though you can if you wish run a local copy of Eyeonus' listener, if you are using the plugin as above, you really shouldn't need to.

    [1] Eyeonus says this shouldn't be required, but I've had to install this module on both windows and Linux with Python 3.6.5, feel free to try it without.
    [2] If my website is down, please pm me as something has broken and I need to fix it!
    [3] EDDB generates files once a day, so this is unlikely to be quite as up to date, but better than nothing.

  2. #2582
    Excellent news Tromador. Very many thanks to you and Eyeonus for your work on this. KFSone would be proud of you!

  3. #2583
    Found a couple of potential problems. Both of them are in the plugin, one of them only affects the listener, both are fixed in the 0.27 update.

    Immediate download is recommended.

    Plugin version: 0.27
    Listener commit: 56e854c

  4. #2584
    Some corrections/additions:
    Originally Posted by Tromador View Post (Source)
    Firstly you will need to install Python.
    Python 3, specifically. A lot of people have Python 2 installed already, whether they know it or not, and that's not a compatible version.
    Python 3.6.5 is the most recent version of Python 3, but Python 2 is still in active development as well, which is why I thought it pertinent to mention this.
    Once you have it installed, open up a command prompt or a shell and install required module requests using pip[1]
    Code:
    pip install requests
    If you don't manually install the requests module, TD will ask you if it can install it for you. This is why I say this step isn't needed.
    You firstly need to run the command
    Code:
    trade.py import -P eddblink -O clean
    This will make some required changes to the database then return back to the command prompt, instructing you to run the same command a second time. On the second run, all the data will be downloaded and processed into the TD database.
    Not anymore. The plugin no longer needs to be run a second time for the changes it makes to take effect, it now reloads the changed things on first run.
    This means a single run is all that is needed.

    Also, it is highly recommended, especially for slower computers, to run with the option 'skipvend' whenever doing a plugin import, even when doing 'clean'. (i.e., 'trade.py import -P eddblink -O clean,skipvend')

    Lastly, Tromador, I imagine you're probably still tweaking things server end, but how often do you currently have it set to do the live listings export? If you have it at the default 5 minutes, I think you can knock it down from there, since the export is much faster now than it was when I set the default value. (Also you may want to update the user facing page, "http://elite.ripz.org/", to reflect that it is no longer "nothing more than a few files mirrored from EDDB".)

  5. #2585
    Originally Posted by eyeonus View Post (Source)
    Found a couple of potential problems. Both of them are in the plugin, one of them only affects the listener, both are fixed in the 0.27 update.

    Immediate download is recommended.

    Plugin version: 0.27
    Listener commit: 56e854c
    Server updated & reloaded.

  6. #2586
    Originally Posted by eyeonus View Post (Source)
    Lastly, Tromador, I imagine you're probably still tweaking things server end, but how often do you currently have it set to do the live listings export? If you have it at the default 5 minutes, I think you can knock it down from there, since the export is much faster now than it was when I set the default value. (Also you may want to update the user facing page, "http://elite.ripz.org/", to reflect that it is no longer "nothing more than a few files mirrored from EDDB".)
    I think updates every 5 mins or so is sufficient, don't you? I doubt anyone is going to have major trauma if their data is 10 minutes old.

    And I will work on that page, but functionality is more important.

  7. #2587
    Originally Posted by Tromador View Post (Source)
    I think updates every 5 mins or so is sufficient, don't you? I doubt anyone is going to have major trauma if their data is 10 minutes old.

    And I will work on that page, but functionality is more important.
    Probably. Just saying you can.

    Agreed. You'll get to it when you get to it.

  8. #2588
    Originally Posted by eyeonus View Post (Source)
    Tromador found another bug in the plugin. The listings weren't getting updated. Apparently you have to TELL SQLite to commit, otherwise it just scraps all the transactions you just did.

    v0.26.2 uploaded to fix that particular headache.
    Just updated both files, and called the listener expecting to have it run the dump first. Got the following message:

    Code:
    F:\Elite\TD>python eddblink_listener.py
    Press CTRL-C at any time to quit gracefully.
    EDDB update available, waiting for busy signal acknowledgement before proceeding.
    Message processor acknowledging busy signal.
    Busy signal acknowledged, performing EDDB dump update.
    NOTE: Downloading file: 'modules.json'.
    NOTE: Requesting http://elite.ripz.org/files/modules.json
    NOTE: Downloaded 292.3KB of gziped data   2.4MB/s
    NOTE: Processing Upgrades: Start time = 2018-06-12 21:24:00.099153
    NOTE: Finished processing Upgrades. End time = 2018-06-12 21:24:00.109127
    NOTE: Downloading file: 'index.json'.
    NOTE: Requesting https://raw.githubusercontent.com/EDCD/coriolis-data/master/dist/index.json
    NOTE: Downloaded   0.6MB of gziped data  24.9MB/s
    NOTE: Processing Ships: Start time = 2018-06-12 21:24:00.278673
    NOTE: Finished processing Ships. End time = 2018-06-12 21:24:00.286652
    NOTE: Downloading file: 'systems_populated.jsonl'.
    NOTE: Requesting http://elite.ripz.org/files/systems_populated.jsonl
    NOTE: Downloaded  20.5MB of gziped data   7.9MB/s
    NOTE: Processing Systems: Start time = 2018-06-12 21:24:03.640226
    NOTE: Finished processing Systems. End time = 2018-06-12 21:24:05.474340
    NOTE: Downloading file: 'stations.jsonl'.
    NOTE: Requesting http://elite.ripz.org/files/stations.jsonl
    NOTE: Downloaded 113.4MB of gziped data  19.0MB/s
    NOTE: Processing Stations, this may take a bit: Start time = 2018-06-12 21:24:12.180977
    NOTE: Finished processing Stations. End time = 2018-06-12 21:24:20.171168
    NOTE: Downloading file: 'commodities.json'.
    NOTE: Requesting http://elite.ripz.org/files/commodities.json
    NOTE: Downloaded  99.3KB of gziped data  97.3MB/s
    NOTE: Processing Categories and Items: Start time = 2018-06-12 21:24:20.904218
    NOTE: Finished processing Categories and Items. End time = 2018-06-12 21:24:20.909204
    NOTE: F:\Elite\TD\data\Category.csv exported.
    NOTE: F:\Elite\TD\data\Item.csv exported.
    NOTE: F:\Elite\TD\data\Ship.csv exported.
    NOTE: F:\Elite\TD\data\Station.csv exported.
    NOTE: F:\Elite\TD\data\System.csv exported.
    NOTE: F:\Elite\TD\data\Upgrade.csv exported.
    NOTE: Downloading file: 'listings.csv'.
    NOTE: Requesting http://elite.ripz.org/files/listings.csv
    NOTE: Downloaded 149.5MB of gziped data  10.1MB/s
    NOTE: Processing market data from listings.csv: Start time = 2018-06-12 21:24:36.959043
    Exception in thread Thread-2:
    Traceback (most recent call last):
      File "C:\Program Files\Python36\lib\threading.py", line 916, in _bootstrap_inner
        self.run()
      File "C:\Program Files\Python36\lib\threading.py", line 864, in run
        self._target(*self._args, **self._kwargs)
      File "eddblink_listener.py", line 358, in check_update
        trade.main(('trade.py','import','-P','eddblink','-O',options))
      File "F:\Elite\TD\trade.py", line 77, in main
        results = cmdenv.run(tdb)
      File "F:\Elite\TD\commands\commandenv.py", line 81, in run
        return self._cmd.run(results, self, tdb)
      File "F:\Elite\TD\commands\import_cmd.py", line 124, in run
        if not plugin.run():
      File "F:\Elite\TD\plugins\eddblink_plug.py", line 822, in run
        self.importListings(self.listingsPath)
      File "F:\Elite\TD\plugins\eddblink_plug.py", line 590, in importListings
        if int(listing['collected_at']) == updated and listing_file == LISTINGS:
    NameError: name 'listing_file' is not defined

  9. #2589
    Originally Posted by Tromador View Post (Source)
    Next download and unpack the Trade Dangerous software from github. We are using the bgol fork from:
    https://bitbucket.org/bgol/tradedangerous
    Your choice of the main or horizons branch.
    If issues #14 and #15 concern you, please take a look at this pull request which addresses the negative scoring issue. I understand Bernd has his own thoughts on the matter; I haven't compared them yet. I know, I should.

  10. #2590
    Arg. I'm always doing that. It should be listings_file, not listing_file. I keep forgetting that 's'. Fixed and uploaded, same version number, same release, new commit base.

  11. #2591
    Originally Posted by eyeonus View Post (Source)
    Arg. I'm always doing that. It should be listings_file, not listing_file. I keep forgetting that 's'. Fixed and uploaded, same version number, same release, new commit base.
    What did you say about beta testers…

  12. #2592
    Originally Posted by Avi0013 View Post (Source)
    [If issues #14 and #15 concern you... ...Bernd...
    It's in the back of my mind. I have sent a message to Bernd to see if he is even interested in TD any more. How we move forward with it will very much depend on any response.

    So, I know you are there and have some ideas for improvements, but please be a little patient until we know where we stand.

  13. #2593
    Patience? How DARE you demand patience after giving up so much of your spare time and capacity for our free use. What do you think you deserve, GRATITUDE??!!


    Well, ackshually, yes 8-)

    Thanks!

  14. #2594
    Originally Posted by eyeonus View Post (Source)
    Arg. I'm always doing that. It should be listings_file, not listing_file. I keep forgetting that 's'. Fixed and uploaded, same version number, same release, new commit base.
    Unfortunately, I was stuck in bed with my illness, so I've only just now fixed this. Normal service will resumed momentarily.

  15. #2595
    I've been considering the idea of writing a drop-in replacement "maddavo_plug.py" that would act as a translator/intercept. That is, running the maddavo plugin would run the EDDBlink plugin, and the options passed to the maddavo plugin would get translated to equivalent options for EDDBlink. Thoughts?