Release Trade Dangerous (Est. 2015) Power user's highly configurable trade optimizer
- Thread starter Tromador
- Start date
I'm not seeing quite that improvement but there is certainly a noticeable increase in processing speed.
Fantastic indeed.
Agreed
Thanks for the update, I'll remove the option from TDH2.
Latest release (TD): v9.5.3
For those of you paying attention, yes, TD is moving to using version numbers.
Thanks almost entirely to the efforts of kmpm, TD is on its way to be an install-able package via pip. We're still working out some of the kinks, so don't try to install it or upgrade past v9.5.3 until further notice.
EDIT:
Latest release (TD): v10.0.2
Okay, unless you are using TDHelper2, it's okay to upgrade at this point. If you are using the listener, which you shouldn't be unless you're Tromador, you need to update the listener as well.
(If you're using TDHelper2, you can update, but you can't install it with pip until Mark says so. You'll have to still download the source and overwrite the old TD wherever you put it then.)
Keep in mind that the old way of simply overwriting the old TD install with the new one will still work, so if you want to upgrade but don't want to do a pip install, you don't have to. Just be sure to delete everything in your td install folder except the data folder before you drop the new version of TD in there.
v9.5.3 and v10.0.2 are functionally identical, so there's no need to upgrade at this point as there are no bug-fixes or new features.
(That is, other than now being able to install it with pip.)
You can install TD v10.0.2 via pip with 'pip install tradedangerous'
Running TD is exactly like before, except if you install it, you run it from anywhere you like with 'trade', (not from only the folder it's in with 'trade.py',) as in 'trade import -P eddblink -O listings,skipvend'.
Unless you set an environment variable, the database will be created in the same folder that the "trade" command is run in. (So if you're in 'C:\', you'll suddenly have a 'C:\data' folder, and if you're in '/home/username/', you'll suddenly have a '/home/username/data' folder.)
If TD v10.0.2+ is being used, and TD_DATA is not set, the location of the data and tmp folders will be wherever TDH2 is installed. Either move your current data folder to where TDH2 is installed or set TD_DATA to that folder if you want to keep the data from before the upgrade.
Setting an environment variable called 'TD_DATA' will cause TD to store all the data in the folder set by 'TD_DATA', eg. '$env:TD_DATA='C:\td-data"' or 'export TD_DATA="/var/lib/td-data"'. I will probably look into also being able to pass it in at run time as well.
The tmp folder acts in this way as well, and has the env-var 'TD_TMP'
Mark, this means you can check to see if tradedangerous is installed and use that, as well as offering to install it if it isn't, and offering to upgrade it if a new version is available, if you would like to do such a thing.
It also means that TDH will need to use 'trade <command>' if peeps are using the pip-installed module rather than '<td install>/trade.py <command>'.
In case anyone's interested, here's the pypi page: https://pypi.org/project/tradedangerous
EDIT:
Okay, here's a cool thing: put the data folder in the TDH2 install folder, and put the 'trade.py' from 10.0.2+ in there as well (just that one file, nothing else from TD), install TD with pip, point TDH2 to the trade.py in its own folder, and voila! It works. No update to TDH2 required, which is pretty cool. That little bit of awesome ought to help Mark out tremendously.
For those of you paying attention, yes, TD is moving to using version numbers.
Thanks almost entirely to the efforts of kmpm, TD is on its way to be an install-able package via pip. We're still working out some of the kinks, so don't try to install it or upgrade past v9.5.3 until further notice.
EDIT:
Latest release (TD): v10.0.2
Okay, unless you are using TDHelper2, it's okay to upgrade at this point. If you are using the listener, which you shouldn't be unless you're Tromador, you need to update the listener as well.
(If you're using TDHelper2, you can update, but you can't install it with pip until Mark says so. You'll have to still download the source and overwrite the old TD wherever you put it then.)
Keep in mind that the old way of simply overwriting the old TD install with the new one will still work, so if you want to upgrade but don't want to do a pip install, you don't have to. Just be sure to delete everything in your td install folder except the data folder before you drop the new version of TD in there.
v9.5.3 and v10.0.2 are functionally identical, so there's no need to upgrade at this point as there are no bug-fixes or new features.
(That is, other than now being able to install it with pip.)
You can install TD v10.0.2 via pip with 'pip install tradedangerous'
Running TD is exactly like before, except if you install it, you run it from anywhere you like with 'trade', (not from only the folder it's in with 'trade.py',) as in 'trade import -P eddblink -O listings,skipvend'.
Unless you set an environment variable, the database will be created in the same folder that the "trade" command is run in. (So if you're in 'C:\', you'll suddenly have a 'C:\data' folder, and if you're in '/home/username/', you'll suddenly have a '/home/username/data' folder.)
If TD v10.0.2+ is being used, and TD_DATA is not set, the location of the data and tmp folders will be wherever TDH2 is installed. Either move your current data folder to where TDH2 is installed or set TD_DATA to that folder if you want to keep the data from before the upgrade.
Setting an environment variable called 'TD_DATA' will cause TD to store all the data in the folder set by 'TD_DATA', eg. '$env:TD_DATA='C:\td-data"' or 'export TD_DATA="/var/lib/td-data"'. I will probably look into also being able to pass it in at run time as well.
The tmp folder acts in this way as well, and has the env-var 'TD_TMP'
Mark, this means you can check to see if tradedangerous is installed and use that, as well as offering to install it if it isn't, and offering to upgrade it if a new version is available, if you would like to do such a thing.
It also means that TDH will need to use 'trade <command>' if peeps are using the pip-installed module rather than '<td install>/trade.py <command>'.
In case anyone's interested, here's the pypi page: https://pypi.org/project/tradedangerous
EDIT:
Okay, here's a cool thing: put the data folder in the TDH2 install folder, and put the 'trade.py' from 10.0.2+ in there as well (just that one file, nothing else from TD), install TD with pip, point TDH2 to the trade.py in its own folder, and voila! It works. No update to TDH2 required, which is pretty cool. That little bit of awesome ought to help Mark out tremendously.
Last edited:
Awesome indeed! I've just published an update for couple of bug fixes so I'll have a look at this great new feature and get THD2 working with that.
Are TD_DATA and TD_TMP set up when using pip install? If not, how should TDH2 check for a pip installed TD? Likewise, how should TDH2 check the current TD install version number and whether there is an updated version of TD available?
Thanks.
No, they have to be set manually. The process for doing so is shown in the two examples in my post: the first one is for windows, the second for the other two.
By default, it saves '/data/' and '/tmp/' in the cwd, which for TDH2 is wherever TDH2 is installed. For example, my install is located at "G:\Elite Dangerous Programs\Trade Dangerous Helper", so when TDH runs TD, TD creates the folders "G:\Elite Dangerous Programs\Trade Dangerous Helper\data" and "G:\Elite Dangerous Programs\Trade Dangerous Helper\tmp", because the cwd for TDH2 is the TDH2 install location.
As to how to check, it uses pip, you can Google it. I'm at work now so no time to explain myself right now.
Thanks. I had figured out the CWD part but I am more concerned about determining if TD has been installed using pip. If it has then TDH2 has to work in a slightly different manner than if it has not, as you correctly point out. I can't use the fact that the data fielder might be in the TDH2 install folder since if you update TD to the latest version by downloading it and overwriting the existing files, that is by not using pip, then the data folder is created in the TDH2 install order in this situation since it is the CWD.
[EDIT]So, I can use pip show <package name> to find out if it is installed and which version, then I have to query https://pypi.python.org/pypi/{PKG_NAME}/json to get the list of available versions. Onwards and upwards!
As a hack, if you include just the trade.py file in TDH2, and try a trade command with it, it'll work if the tradedangerous package is installed, and will error if it isn't. That's how I'm currently running it with v2.1.0.4.
Looks to me like you're on the right path to do it the right way.
As long as you updated when I sent you the email, there's no need for concern. The recent spate of updates don't change anything from a purely functional standpoint. The only update to the listener just fixes the importing to work with the new pip-installable TD, and the updates to TD just make it pip-installable.
I didn't - because it was an optimisation and not a data change and the server runs like lightning anyway.
That said, I think I will set up the PIP method on the new virtual host then I can test and switch it over in DNS later - TD can have the honour of being the first live service on the upgraded server.
Less a question and more a clarification -
People running on Linux (like me, or rather the server) will likely be running TD as an unprivileged user, but in turn need certain modules installed into the system.
Now, normally pip will run and resolve any dependencies and install the requested module, but when we want different things in different places, it gets a little more complicated.
From this, can I correcfly deduce that TD requires chardet, urllib3, idna, certifi & requests to be installed? I realise many Python releases may have one or more of these by default, however CentOS (even CentOS 7) is a bit strange when it comes to Python 3 and needs a bit of extra help (though nowhere near as much as the CentOS 6 that the server currently runs upon). Don't worry about any other errors I may have got, I'll handle those separately, just need the requested information for now.
People running on Linux (like me, or rather the server) will likely be running TD as an unprivileged user, but in turn need certain modules installed into the system.
Now, normally pip will run and resolve any dependencies and install the requested module, but when we want different things in different places, it gets a little more complicated.
Code:
Collecting tradedangerous
Downloading https://files.pythonhosted.org/packages/fe/c0/268e641cfb032f980244f548a3424f324d9224b2a5c9fe31400884ad2a71/tradedangerous-10.0.2-py3-none-any.whl (220kB)
100% |████████████████████████████████| 225kB 1.8MB/s
Collecting requests (from tradedangerous)
Downloading https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl (57kB)
100% |████████████████████████████████| 61kB 4.3MB/s
Collecting chardet<3.1.0,>=3.0.2 (from requests->tradedangerous)
Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
100% |████████████████████████████████| 143kB 2.4MB/s
Collecting urllib3<1.25,>=1.21.1 (from requests->tradedangerous)
Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
100% |████████████████████████████████| 122kB 3.6MB/s
Collecting idna<2.9,>=2.5 (from requests->tradedangerous)
Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
100% |████████████████████████████████| 61kB 4.3MB/s
Collecting certifi>=2017.4.17 (from requests->tradedangerous)
Downloading https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl (154kB)
100% |████████████████████████████████| 163kB 2.8MB/s
Installing collected packages: chardet, urllib3, idna, certifi, requests, tradedangerous
...bomb out... fail... etc... error messages galore.From this, can I correcfly deduce that TD requires chardet, urllib3, idna, certifi & requests to be installed? I realise many Python releases may have one or more of these by default, however CentOS (even CentOS 7) is a bit strange when it comes to Python 3 and needs a bit of extra help (though nowhere near as much as the CentOS 6 that the server currently runs upon). Don't worry about any other errors I may have got, I'll handle those separately, just need the requested information for now.
The only thing listed as a requirement in TD's install script is requests. Those others are probably requirements either of requests or of one of requests' requirements, but I can't say for certain. What happens if you install requests, and then install TD?
EDIT: pytest-runner and pytest are also required for setup and testing, respectively, but I don't think that's an issue in this case.
Code:
setup.py:
...
install_requires=["requests"],
setup_requires=["pytest-runner"],
tests_require=["pytest"],
...
Last edited:
Ok good. That appears to be the case.
Code:
[elite@quoth ~]$ trade
#/home/elite/.local/bin/trade: TradeDangerous provides a set of trade database facilities for Elite:Dangerous.And I can take it from here. Note that my maximum (currently) version of Python is 3.6 so unless the repositories get updated, anything requiring an higher version will cause me issues.
Last edited:
Installed and set up, but when I try to run just TD (without the listener for now), it vomits up and says it can't find TradeDangerous.sql
Edit: The templates folder, present on Github, is not present in my pip installed TD distribution (ie, there is no copy at all of Added.csv, RareItem.csv or TradeDangerous.sql). I've tried creating these files in the same place in the heirarchy, but no dice. I'll try copying in TradeDangerous.sql to where it thinks it wants it, but still.
Edit: The templates folder, present on Github, is not present in my pip installed TD distribution (ie, there is no copy at all of Added.csv, RareItem.csv or TradeDangerous.sql). I've tried creating these files in the same place in the heirarchy, but no dice. I'll try copying in TradeDangerous.sql to where it thinks it wants it, but still.
Code:
[elite@quoth ~]$ echo $TD_DATA
/home/elite/tradedangerous/tddata
[elite@quoth ~]$ echo $TD_TMP
/home/elite/tradedangerous/tdtmp
[elite@quoth ~]$ trade import -P eddblink -O clean
NOTE: Rebuilding cache file: this may take a few moments.
Traceback (most recent call last):
File "/home/elite/.local/bin/trade", line 10, in <module>
sys.exit(main())
File "/home/elite/.local/lib/python3.6/site-packages/tradedangerous/cli.py", line 70, in main
trade(argv)
File "/home/elite/.local/lib/python3.6/site-packages/tradedangerous/cli.py", line 126, in trade
results = cmdenv.run(tdb)
File "/home/elite/.local/lib/python3.6/site-packages/tradedangerous/commands/commandenv.py", line 81, in run
return self._cmd.run(results, self, tdb)
File "/home/elite/.local/lib/python3.6/site-packages/tradedangerous/commands/import_cmd.py", line 122, in run
if not plugin.run():
File "/home/elite/.local/lib/python3.6/site-packages/tradedangerous/plugins/eddblink_plug.py", line 878, in run
tdb.reloadCache()
File "/home/elite/.local/lib/python3.6/site-packages/tradedangerous/tradedb.py", line 758, in reloadCache
cache.buildCache(self, self.tdenv)
File "/home/elite/.local/lib/python3.6/site-packages/tradedangerous/cache.py", line 912, in buildCache
with sqlPath.open('rU', encoding='utf-8') as sqlFile:
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pathlib.py", line 1161, in open
opener=self._opener)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pathlib.py", line 1015, in _opener
return self._accessor.open(self, flags, mode)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pathlib.py", line 387, in wrapped
return strfunc(str(pathobj), *args)
FileNotFoundError: [Errno 2] No such file or directory: '/home/elite/tradedangerous/tddata/TradeDangerous.sql'
Last edited:
It's supposed to be in the data folder. Hmm.. I'll look into this later when I have the time.