Release Trade Dangerous (Est. 2015) Power user's highly configurable trade optimizer

Tromador

Tromador
Somewhat apologies for the recent server being erratic. The guys in the office (my former workplace, who kindly provide me a free server hosting - http://www.aoc-uk.com) have been doing some upgrades on the kit. This is good as I have a slightly better solution, bit more RAM, bit faster CPU. It turns out, however, that TD does not like live migrations and whilst the server chugs along with 99.9% of its processes unaffected, TD falls over and has to be restarted. So, I've asked them to let me know when they fiddle with it.

Moving forward, I'm going to do a bit of testing of Centos Stream 9 and that will likely end up being the underlying platform at some point Soon™.
 

marcuscav

marcuscav
Tromador said:
Somewhat apologies for the recent server being erratic. The guys in the office (my former workplace, who kindly provide me a free server hosting - http://www.aoc-uk.com) have been doing some upgrades on the kit. This is good as I have a slightly better solution, bit more RAM, bit faster CPU. It turns out, however, that TD does not like live migrations and whilst the server chugs along with 99.9% of its processes unaffected, TD falls over and has to be restarted. So, I've asked them to let me know when they fiddle with it.

Moving forward, I'm going to do a bit of testing of Centos Stream 9 and that will likely end up being the underlying platform at some point Soon™.
Click to expand...
Tromodor, if folks are still running into problems with the SSL CERTIFICATE_VERIFY_FAILED error during eddblink imports, I've found the Windows 11 solution. Let me know if ya'll are still in need of it.
 

Tromador

Tromador
marcuscav said:
Tromodor, if folks are still running into problems with the SSL CERTIFICATE_VERIFY_FAILED error during eddblink imports, I've found the Windows 11 solution. Let me know if ya'll are still in need of it.
Click to expand...
I have no idea, but nobody is reporting a problem and I want to avoid bypassing the security if at all possible. The certificate is perfectly valid. If there is still a problem the best we can do is raise a ticket with python support.
 

marcuscav

marcuscav
Based on reports involving other Python projects, it looks like it's a problem with how the Python package is installed on systems with multiple user permissions. The solution I found involved downloading and installing locally the legitimate certificates of the server eddblink uses (Let's Encrypt). By downloading and installing the .DER certificates for Let's Encrypt's R1, R2, and R3, I was able to run the import -P eddblink without any error.
 

eyeonus

E
Apologies to everyone, I'm certain I had subscribed to this thread, but I haven't been getting notifications in my email and I totally forgot about this thing.

In any case, I've updated TD, thought I'd share.

It's not an essential update, all I've done is move the GUI stuff out of the trade.py
For those of you that only use the CLI, this means you shouldn't have to worry about having tk installed anymore, since only the GUI-specific stuff (, that is, tradegui.py and tradedangerous/gui.py,) ever uses it or imports anything from it.

Speaking of, launching the TD GUI using >trade gui no longer works with this update, the GUI has its own entry point: >tradegui. Note the lack of space.

Next step is to remove the appJar dependency, which I'm working on now.

Ultimately, the goal is to be feature-complete with TDH, using TD internal features (tdrc, form files, ...).
 
Last edited:

CMDR Ibizan

C
The servers gone off again, Tromador. I thought initially that eyeonus (welcome back!) had broken it with his update but elite.tromador.com shows no updated files coming through since 30th May. Can you give it your normal remedy, please?
 

ShadowGar

Banned
ShadowGar
Is this the same error as the certificate error because I get this on first install and import.
Traceback (most recent call last):
File "C:\Program Files\Python36\Lib\site-packages\tradedangerous\trade.py", line 44, in <module>
cli.main(sys.argv)
File "C:\Program Files\Python36\lib\site-packages\tradedangerous\cli.py", line 70, in main
trade(argv)
File "C:\Program Files\Python36\lib\site-packages\tradedangerous\cli.py", line 125, in trade
results = cmdenv.run(tdb)
File "C:\Program Files\Python36\lib\site-packages\tradedangerous\commands\commandenv.py", line 83, in run
return self._cmd.run(results, self, tdb)
File "C:\Program Files\Python36\lib\site-packages\tradedangerous\commands\import_cmd.py", line 126, in run
if not plugin.run():
File "C:\Program Files\Python36\lib\site-packages\tradedangerous\plugins\eddblink_plug.py", line 1098, in run
self.importCommodities()
File "C:\Program Files\Python36\lib\site-packages\tradedangerous\plugins\eddblink_plug.py", line 668, in importCommodities
edcd_csv = request.urlopen(edcd_source)
File "C:\Program Files\Python36\lib\urllib\request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "C:\Program Files\Python36\lib\urllib\request.py", line 526, in open
response = self._open(req, data)
File "C:\Program Files\Python36\lib\urllib\request.py", line 544, in _open
'_open', req)
File "C:\Program Files\Python36\lib\urllib\request.py", line 504, in _call_chain
result = func(*args)
File "C:\Program Files\Python36\lib\urllib\request.py", line 1361, in https_open
context=self._context, check_hostname=self._check_hostname)
File "C:\Program Files\Python36\lib\urllib\request.py", line 1320, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [WinError 10054] An existing connection was forcibly closed by the remote host>
Click to expand...
 

Tromador

Tromador
Nope, I am getting a completely different traceback:

Code: 
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/elite/.local/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/home/elite/.local/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "./eddblink_listener.py", line 374, in check_update
    trade.main(('trade.py', 'import', '-P', 'eddblink', '-O', options))
  File "/home/elite/.local/lib/python3.7/site-packages/tradedangerous/cli.py", line 70, in main
    trade(argv)
  File "/home/elite/.local/lib/python3.7/site-packages/tradedangerous/cli.py", line 125, in trade
    results = cmdenv.run(tdb)
  File "/home/elite/.local/lib/python3.7/site-packages/tradedangerous/commands/commandenv.py", line 83, in run
    return self._cmd.run(results, self, tdb)
  File "/home/elite/.local/lib/python3.7/site-packages/tradedangerous/commands/import_cmd.py", line 126, in run
    if not plugin.run():
  File "/home/elite/.local/lib/python3.7/site-packages/tradedangerous/plugins/eddblink_plug.py", line 1109, in run
    self.importListings(self.listingsPath)
  File "/home/elite/.local/lib/python3.7/site-packages/tradedangerous/plugins/eddblink_plug.py", line 884, in importListings
    updated = timegm(datetime.datetime.strptime(result[0], '%Y-%m-%d %H:%M:%S').timetuple())
  File "/home/elite/.local/lib/python3.7/_strptime.py", line 577, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/home/elite/.local/lib/python3.7/_strptime.py", line 362, in _strptime
    data_string[found.end():])
ValueError: unconverted data remains: .733

These kinds of errors generally (iirc) come from wierdness in the data which causes TD to barf.

Hopefully Eyeonus will see this. I will ping him elsewhere also.
 

Tromador

Tromador
We are currently up and running. I had to clean the database, so most of the day's new data won't be imported until tomorrow morning. New data from this moment will be reflected in the listings-new.csv files which your clients will download normally.
 

Tromador

Tromador
Recently we've been having a wierd problem with server stopping with a variety of error messages, such as the one I posted above.

I've done some testing with Eyeonus and this turns out to be due to the occasional (but not consistent) presence of microseconds in the timestamp and TD barfing, because it doesn't expect to see them, indeed 99% (give or take a 1%) of entries don't have the microsecond portion of the timestamp present.

Although this has mostly caused problems with the server, there's no good reason it couldn't affect clients as well, the server has much heavier load on the db than any of you will have, so is more susceptible to hitting particularly situational bugs. Thus, we recommend everyone immediately update their version of TD and have pushed a new version with the required fixes.
 

Scotty113

S
Everything working great with the new update. Ran a couple of decent routes. Tried to run an listings update and this is what was returned. Im using TD helper

Code: 
Command line:  import -P eddblink -O listings
NOTE: Checking for update to 'systems_populated.jsonl'.
WARNING: Problem with download:
URL: https://elite.tromador.com/files/systems_populated.jsonl
Error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)>
WARNING: Problem with download (fallback enabled):
URL: https://eddb.io/archive/v6/systems_populated.jsonl
Error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)>
NOTE: Checking for update to 'stations.jsonl'.
WARNING: Problem with download (fallback enabled):
URL: https://eddb.io/archive/v6/stations.jsonl
Error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)>
NOTE: Checking for update to 'commodities.json'.
WARNING: Problem with download (fallback enabled):
URL: https://eddb.io/archive/v6/commodities.json
Error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)>
NOTE: Checking for update to 'listings.csv'.
WARNING: Problem with download (fallback enabled):
URL: https://eddb.io/archive/v6/listings.csv
Error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)>
NOTE: Import completed.
 

eyeonus

E
Scotty113 said:
Everything working great with the new update. Ran a couple of decent routes. Tried to run an listings update and this is what was returned. Im using TD helper

Code: 
Command line:  import -P eddblink -O listings
NOTE: Checking for update to 'systems_populated.jsonl'.
WARNING: Problem with download:
URL: https://elite.tromador.com/files/systems_populated.jsonl
Error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)>
WARNING: Problem with download (fallback enabled):
URL: https://eddb.io/archive/v6/systems_populated.jsonl
Error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)>
NOTE: Checking for update to 'stations.jsonl'.
WARNING: Problem with download (fallback enabled):
URL: https://eddb.io/archive/v6/stations.jsonl
Error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)>
NOTE: Checking for update to 'commodities.json'.
WARNING: Problem with download (fallback enabled):
URL: https://eddb.io/archive/v6/commodities.json
Error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)>
NOTE: Checking for update to 'listings.csv'.
WARNING: Problem with download (fallback enabled):
URL: https://eddb.io/archive/v6/listings.csv
Error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)>
NOTE: Import completed.
Click to expand...

This isn't something we can fix, unfortunately, but see these for some help with the problem:

Tromador said:
Unfortunately the SSL problem is caused by something in Python and not in my website. We have fully tested that browsers can access the site and checked that the SSL certificate is valid.
The weirdness is that I cannot replicate the problem on my PC, it. all works fine for me, so it can't be an issue with Python's own root certificate store or I would have the same problem. I have also checked to make sure that there is no difference between the certificate store installed on my PC and on Scottie's PC and they are indeed identical.
I am guessing, given that the exact same error comes up when TD attempts to connect to fallback that Python for some reason cannot read its own certificate store. And now we progress into something wierd in the windows permissions, even though the file looks fine to me, with the folder tree also having permissions which seem sane.
Scottie did have some kind of per application system permissions set which I had never seen before, maybe that is the ultimate cause. He didn't install windows and I am wondering if the shop did something unusual which then has a knock on effect to how Python works.
In short, it's something I can't fix, because everything looks fine, so it's very deeply lost in the Python install or I think more likely the interaction between Python and Windows.
Click to expand...

marcuscav said:
Tromodor, if folks are still running into problems with the SSL CERTIFICATE_VERIFY_FAILED error during eddblink imports, I've found the Windows 11 solution. Let me know if ya'll are still in need of it.
Click to expand...

marcuscav said:
Based on reports involving other Python projects, it looks like it's a problem with how the Python package is installed on systems with multiple user permissions. The solution I found involved downloading and installing locally the legitimate certificates of the server eddblink uses (Let's Encrypt). By downloading and installing the .DER certificates for Let's Encrypt's R1, R2, and R3, I was able to run the import -P eddblink without any error.
Click to expand...
 

MarkAusten

MarkAusten
JarangMandi said:
For what it's worth, I've forked the TDH repository on GitHub and have fixed a number of issues - including crashes when the CMDR button is clicked, and the regular update reminders that have been reported.

I don't want to tread on @MarkAusten's toes even though he's said that he's no longer able to maintain it, and I have tried to contact him to ask whether he's OK for me to pick it up and develop further. Whilst the licence is permissive in that respect, I feel it only polite! Yet to hear back, so if anyone has a route to contact him please let me know. As such, there are no pre-compiled executables available at the moment, but the source code is there for the taking.
Click to expand...
Sorry to be so late to the party on this issue, for some reason I've not been getting messages either from the forum or from GitHub. Not sure why.

I have absolutely no problem with someone else taking over the TDH development as I still have no way to maintain it. Windows is not something I develop in these days except for my day job and they will not permit me to use their laptop for anything other than company projects.

I did try a Window VM on my ageing iMac but it was so painfully slow as to be impossible to use.

So, whoever wants to take over has my blessing.
 

Tromador

Tromador
Another quick update:
You may have noticed TD fallen over again. After we finished our testing for the problem, I did a full reinstall to clean up all the test code and get a clean install of TD on the server. Sadly, it looks like the fix had not propagated to the local github mirror and my server downloaded the old broken version.

I've once again updated the server and manually checked we have the correct code, which we do, so إِنْ شَاءَ ٱللَّٰهُ this won't be a problem again.
 
You must log in or register to reply here.
Back
Top Bottom