Release Trade Dangerous (Est. 2015) Power user's highly configurable trade optimizer

eyeonus · Nov 19, 2019

nepomuk said:
Sorry guys, next problem with me. I wanted to import the market data from stations I had flown to into the database
( e.g. c:\trade\trade.py import "c:\trade\Minmar.Bayliss Landing.2019-11-19T17.29.09.prices" ).
The command is executed but the following error message is displayed

Code:

C:\trade>trade.py import "c:\trade\Fomalhaut.Wargnerport.2019-11-19T16.46.54.prices" NOTE: Rebuilding cache file: this may take a few moments. *** INTERNAL ERROR: NOT NULL constraint failed: RareItem.station_id CSV File: C:\trade\data\RareItem.csv:2 SQL Query: INSERT OR REPLACE INTO RareItem (station_id,category_id,name,cost,max_allocation,illegal,suppressed) VALUES((SELECT Station.station_id FROM Station INNER JOIN System USING(system_id) WHERE System.name = ? AND Station.name = ?),(SELECT Category.category_id FROM Category WHERE Category.name = ?),?,?,?,?,?) Params: ['16 LYRAE', 'Budrys Ring', 'Legal Drugs', 'Lyrae Weed', '148', '10', 'N', 'N']

What am I doing wrong?

Follow this guide to correctly setup TD. Make sure you understand the difference between a pip installation and a portable installation.
https://github.com/eyeonus/Trade-Dangerous/wiki/Setup-Guide

The following section is especially relevant to what you're trying to do:
https://github.com/eyeonus/Trade-Dangerous/wiki/Setup-Guide#other-data-sources

You may also want to look at the documentation on the EDMC_batch import plugin, as it is designed to import multiple .prices files at once.
https://github.com/eyeonus/Trade-Dangerous/wiki/Plugin-Options#edmc_batch

nepomuk · Nov 19, 2019

I installed TD again according to instructions. I also updated the PIP version afterwards (I got a hint).
Command trade.py import --plug=eddblink --opt=clean works. The EDMC_batch import works also
trade.py import -P edmc_batch -O folder=C:\EDMC
...
Hmmm, very strange ...
Earlier I got some bugs (see spoiler), now it seems to work.

Code:

C:\trade>pip install tradedangerous
Requirement already satisfied: tradedangerous in c:\python38\lib\site-packages (10.6.1)
Requirement already satisfied: appJar in c:\python38\lib\site-packages (from tradedangerous) (0.94.0)
Requirement already satisfied: requests in c:\python38\lib\site-packages (from tradedangerous) (2.22.0)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\python38\lib\site-packages (from requests->tradedangerous) (1.25.7)
Requirement already satisfied: certifi>=2017.4.17 in c:\python38\lib\site-packages (from requests->tradedangerous) (2019.9.11)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\python38\lib\site-packages (from requests->tradedangerous) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in c:\python38\lib\site-packages (from requests->tradedangerous) (2.8)
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

C:\trade>phyton -m pip install --upgrade pip
Der Befehl "phyton" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.

C:\trade>python -m pip install --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl (1.4MB)
     |████████████████████████████████| 1.4MB 819kB/s
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-19.3.1

C:\trade>pip install tradedangerous
Requirement already satisfied: tradedangerous in c:\python38\lib\site-packages (10.6.1)
Requirement already satisfied: appJar in c:\python38\lib\site-packages (from tradedangerous) (0.94.0)
Requirement already satisfied: requests in c:\python38\lib\site-packages (from tradedangerous) (2.22.0)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\python38\lib\site-packages (from requests->tradedangerous) (1.25.7)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\python38\lib\site-packages (from requests->tradedangerous) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in c:\python38\lib\site-packages (from requests->tradedangerous) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in c:\python38\lib\site-packages (from requests->tradedangerous) (2019.9.11)

C:\trade>trade.py import "c:\trade\\LAWD 96.Wiener Prospect.2019-11-19T19.02.40.prices"
NOTE: Rebuilding cache file: this may take a few moments.
*** INTERNAL ERROR: NOT NULL constraint failed: RareItem.station_id
CSV File: C:\trade\data\RareItem.csv:2
SQL Query: INSERT OR REPLACE INTO RareItem (station_id,category_id,name,cost,max_allocation,illegal,suppressed) VALUES((SELECT Station.station_id FROM Station INNER JOIN System USING(system_id) WHERE System.name = ? AND Station.name = ?),(SELECT Category.category_id FROM Category WHERE Category.name = ?),?,?,?,?,?)
Params: ['16 LYRAE', 'Budrys Ring', 'Legal Drugs', 'Lyrae Weed', '148', '10', 'N', 'N']


C:\trade>python
Python 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> ^Z


C:\trade>

eyeonus · Nov 19, 2019

Read through the Getting Started and Other Data Sources sections:

Code:
TD does not come with a pre-built database and will not work without first building one. To do this, run the following command, to ensure you have the latest available data.

Code:

$ trade.py import -P eddblink -O all,solo

The 'all' option downloads all the data that's needed for TD to work correctly, while the 'solo' option prevents the EDDBlink plugin from downloading any crowd-sourced market data. Refer to the EDDBLink documentation for more details on how to use EDDBlink.

nepomuk · Nov 19, 2019

eyeonus said:
Read through the Getting Started and Other Data Sources sections:

Thanks for today. I'll take another look tomorrow. Has changed a little.

nepomuk · Nov 26, 2019

TD works well. Thank you.
I still have a little problem with TDHelper (3.0.0.2) when I update the Cmdr profile. I get the following error:

Informationen über das Aufrufen von JIT-Debuggen
anstelle dieses Dialogfelds finden Sie am Ende dieser Meldung.

** Ausnahmetext **
System.ArgumentOutOfRangeException: Der Wert 1800 ist für Value ungültig. Value sollte zwischen 'Minimum' und 'Maximum' liegen.
Parametername: Value
bei System.Windows.Forms.NumericUpDown.set_Value(Decimal value)
bei TDHelper.MainForm.SetShipList(Boolean refreshList)
bei TDHelper.MainForm.UpdateCommanderAndShipDetails()
bei TDHelper.MainForm.BackgroundWorker7_RunWorkerCompleted(Object sender, RunWorkerCompletedEventArgs e)
bei System.ComponentModel.BackgroundWorker.OnRunWorkerCompleted(RunWorkerCompletedEventArgs e)
bei System.ComponentModel.BackgroundWorker.AsyncOperationCompleted(Object arg)

** Geladene Assemblys **
mscorlib
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.4018.0 built by: NET48REL1LAST_C.
CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll.
----------------------------------------
TDHelper
Assembly-Version: 3.0.0.2.
Win32-Version: 3.0.0.2.
CodeBase: file:///C:/trade/TDHelper.exe.
----------------------------------------
System.Windows.Forms
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.4001.0 built by: NET48REL1LAST_C.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll.
----------------------------------------
System
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.4001.0 built by: NET48REL1LAST_C.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll.
----------------------------------------
System.Drawing
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.3752.0 built by: NET48REL1.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll.
----------------------------------------
System.Data.SQLite
Assembly-Version: 1.0.109.0.
Win32-Version: 1.0.109.0.
CodeBase: file:///C:/trade/System.Data.SQLite.DLL.
----------------------------------------
System.Data
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.3752.0 built by: NET48REL1.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_32/System.Data/v4.0_4.0.0.0__b77a5c561934e089/System.Data.dll.
----------------------------------------
System.Core
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.4018.0 built by: NET48REL1LAST_C.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll.
----------------------------------------
System.Transactions
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.3752.0 built by: NET48REL1.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_32/System.Transactions/v4.0_4.0.0.0__b77a5c561934e089/System.Transactions.dll.
----------------------------------------
System.Configuration
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.3752.0 built by: NET48REL1.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll.
----------------------------------------
System.Xml
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.3752.0 built by: NET48REL1.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll.
----------------------------------------
Accessibility
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.3752.0 built by: NET48REL1.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/Accessibility/v4.0_4.0.0.0__b03f5f7f11d50a3a/Accessibility.dll.
----------------------------------------
System.Windows.Forms.resources
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.3752.0 built by: NET48REL1.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms.resources/v4.0_4.0.0.0_de_b77a5c561934e089/System.Windows.Forms.resources.dll.
----------------------------------------
System.Deployment
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.3752.0 built by: NET48REL1.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Deployment/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Deployment.dll.
----------------------------------------
System.Deployment.resources
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.3752.0 built by: NET48REL1.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Deployment.resources/v4.0_4.0.0.0_de_b03f5f7f11d50a3a/System.Deployment.resources.dll.
----------------------------------------
SharpConfig
Assembly-Version: 3.2.8.0.
Win32-Version: 3.2.8.0.
CodeBase: file:///C:/trade/SharpConfig.DLL.
----------------------------------------
System.EnterpriseServices
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.3752.0 built by: NET48REL1.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_32/System.EnterpriseServices/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.EnterpriseServices.dll.
----------------------------------------
System.Numerics
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.3752.0 built by: NET48REL1.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Numerics/v4.0_4.0.0.0__b77a5c561934e089/System.Numerics.dll.
----------------------------------------
System.Data.resources
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.3752.0 built by: NET48REL1.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Data.resources/v4.0_4.0.0.0_de_b77a5c561934e089/System.Data.resources.dll.
----------------------------------------
System.Xml.Linq
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.3752.0 built by: NET48REL1.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Xml.Linq/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.Linq.dll.
----------------------------------------
System.Data.DataSetExtensions
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.3752.0 built by: NET48REL1.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Data.DataSetExtensions/v4.0_4.0.0.0__b77a5c561934e089/System.Data.DataSetExtensions.dll.
----------------------------------------
Newtonsoft.Json
Assembly-Version: 11.0.0.0.
Win32-Version: 11.0.2.21924.
CodeBase: file:///C:/trade/Newtonsoft.Json.DLL.
----------------------------------------
mscorlib.resources
Assembly-Version: 4.0.0.0.
Win32-Version: 4.8.3752.0 built by: NET48REL1.
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/mscorlib.resources/v4.0_4.0.0.0_de_b77a5c561934e089/mscorlib.resources.dll.
----------------------------------------

** JIT-Debuggen **
Um das JIT-Debuggen (Just-In-Time) zu aktivieren, muss in der
Konfigurationsdatei der Anwendung oder des Computers
(machine.config) der jitDebugging-Wert im Abschnitt system.windows.forms festgelegt werden.
Die Anwendung muss mit aktiviertem Debuggen kompiliert werden.

Zum Beispiel:

<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>

Wenn das JIT-Debuggen aktiviert ist, werden alle nicht behandelten
Ausnahmen an den JIT-Debugger gesendet, der auf dem
Computer registriert ist, und nicht in diesem Dialogfeld behandelt.

That's not further bad. By pressing the button "Next" TDHelper runs with the new values.
Can I adjust a setting ?
But if it's too big a circumstance, it doesn't have to be changed. As written above TDH still works if you ignore the error.

Greetings nepomuk

MarkAusten · Nov 28, 2019

I think the problem will be that in German, the decimal separator is a ',' whereas in Britain it is a '.'. Although it looks like you are entering 18 ly, in German this is 18,00. Since the TDH and TD are British/Ameerican written, the number is expected to be 18.00 and the comma is removed by TDH code and the result is 1800 and that is too large a number for the laden LY range.

I will try to work out a better way of dealing with this.

nepomuk · Nov 28, 2019

Yes, the explanation fits, in my opinion.
But that's not bad, just a small blemish. I can deal with it.
Thanks for the info !

Tromador · Dec 13, 2019

Taken a longish break and logged back in today.

What are currently thought to be good values for LS Penalty, Prune Score and Prune Hops?

Mark, is there a good reason why TDH won't allow me values higher than 20k in Max LS?

Also Mark, from DBMaint --> Analyze

Code:

Command line:  import -P eddblink -O ANALYZE
PLUGIN ERROR: Unknown plugin option: 'ANALYZE'.

Valid options for this plugin:
  all, clean, fallback, force, item, listings, progbar, ship, shipvend, skipvend, solo, station, system, upgrade, upvend.

Use '--opt=help' for details.

Unsure if it actually did the ANALYZE; on the db or not. It might have done, then wandered off into lala land throwing the error.

Tromador · Dec 13, 2019

Has something gone off the rails whilst I've been away?

trade run --fr="OR Delphini/Lichtenberg Station" --to="Shinrarta Dezhra/Jameson Memorial" --cap=182 --ins=22649012 --cr=1198537048 --ly=36.98 --empty=41.13 --jum=6 --start=2 --age=2 --unique --planetary=YN? --pad=L -vv --progress

isn't so hard is it? Currently into the multiple hours and counting to process.

Edit: Possibly because I had --jum and --hops confused in my head and created a nightmare of possible destinations. So never mind

CMDR Ibizan · Dec 14, 2019

Hi Tromador. The LS Penalty that the boffins came up with earlier was 12.5. I always use that along with prune hops 2 and prune score 90. They seem to give quite good trade runs for me.

Incidentally Mark, I am also getting the same error as Tromador with running the analyze from DBMaint.

eyeonus · Dec 15, 2019

12.5 is the default penalty, by the way.

Tromador · Dec 16, 2019

eyeonus said:
12.5 is the default penalty, by the way.

Yeah, I couldn't remember if we built it in as a default or not. Good to know.

eyeonus · Dec 22, 2019

Tromador, a user just brought to my attention that the listings.csv is hanging on download. I just went to the web host and tried to download it that way, and my browser has been "Waiting for elite.tromador.com..." for about 20 minutes now.

I'm not sure if it's a problem with the file itself or with the server, but it might be time for a reboot. The stations.jsonl file also hung once, but I can't reproduce it right now, and I haven't had any trouble with any of the other files trying multiple times.

Tromador · Dec 22, 2019

eyeonus said:
Tromador, a user just brought to my attention that the listings.csv is hanging on download. I just went to the web host and tried to download it that way, and my browser has been "Waiting for elite.tromador.com..." for about 20 minutes now.

I'm not sure if it's a problem with the file itself or with the server, but it might be time for a reboot. The stations.jsonl file also hung once, but I can't reproduce it right now, and I haven't had any trouble with any of the other files trying multiple times.

Cannot reproduce. TD simply downloads the file via HTTP, so to test try downloading from http://elite.tromador.com/ to test just the webserver. Files are being served fine for me. The server was rebooted a couple of days ago, to facilitate a change to the central backup server, but other than that hasn't been messed with and the webserver has been solid for a long time.

On principle I've restarted apache, but I am more of the opinion that there is/was a network glitch, either local to the user or to the physical location of the server.

CMDR Ibizan · Dec 22, 2019

My TD seems to get stuck at commodoties.json now!

CMDR Ibizan · Dec 22, 2019

Using the fallback option for the database update seems to be working as normal.

eyeonus · Dec 22, 2019

Tromador said:
Cannot reproduce. TD simply downloads the file via HTTP, so to test try downloading from http://elite.tromador.com/ to test just the webserver. Files are being served fine for me. The server was rebooted a couple of days ago, to facilitate a change to the central backup server, but other than that hasn't been messed with and the webserver has been solid for a long time.

On principle I've restarted apache, but I am more of the opinion that there is/was a network glitch, either local to the user or to the physical location of the server.

Interesting. I tried downloading direct from the webhost myself and it hung for me as well. I'll try again and see if it's still happening for me.

eyeonus · Dec 22, 2019

Tried downloading listings.csv direct from the webhost on Chrome, it's been sitting at 21.3MB (~10%) downloaded for a couple minutes now. Maybe it's a bandwidth thing?

eyeonus · Dec 22, 2019

Cancelled it and tried again, this time it got to 96.1MB (~50%) before getting stuck.

Tromador · Dec 22, 2019

Again guys, I cannot reproduce. I can say it's not for lack of bandwidth either. The server is hosted in a datacentre with plenty of Internet.

I think somewhere between there and "elsewhere" the Internet is being glitchy. Hopefully it will clear up sooner rather than later. happens, but as the server isn't the source of the fault, I can't do much.