I think you may need to update the trade.bat to use --plug=maddavo rather than --maddavo. At least I did and I thought I refreshed from master.
In-Development TradeDangerous: power-user trade optimizer
- Thread starter kfsone
- Start date
If you're thinking on switching to C++ to make the GUI for example then Qt would be great, you can deploy to windows, linux and even mac (but I'm sure you already knew about that).
Edit: downloaded last from bitbucket:
What does the "data from X and Y" mean? is it like "data FROM-STATION x TO-STATION y" type of thing?
Edit: downloaded last from bitbucket:
Code:
Load from DUAMTA/Polyakov Station:
532 x Indium 5,407cr each, 2,876,524cr total, data from 2hrs and 5days
Last edited:
Hi folks,
I have just started using TradeDangerous and I am very impressed - fantastic piece of work (to the point where I am digging my Python and Jython stuff out after several years and thinking about messing about with the UI). I am running everything locally (ie not using Maddavos or other crowdsourced data except for a single download of the Systems and Stations .csv files when I began), using my own .prices imports. This was working great for the past few days, until this morning when I noticed something odd. I have the following data for two systems, each in their own .prices file which I import into the larger TradeDangerous.prices file. (this is a snippet of the larger .prices files for both systems which serve to illustrate the problem). As you can see below, for items that have both sell and buy prices, I have been entering 50M for the buy price where no demand is known. If I try to generate a standard run for either system, (they are right next to each other, and it should be profitable in this and other similar minerals, and I have lots of money and a small cargo hold), I get the unable to generate run/no prices in the database error:
@ HIP 18918/Johnson Ring
+ Minerals
Lepidolite 474 496 50M 1557M
@ HIP 17621/Fleming Enterprise
+ Minerals
Lepidolite 594 0 823L -
What is interesting is when I look at the TradeDangerous.prices file, these systems (again a snippet shown here) have been recorded as:
@ HIP 17621/Fleming Enterprise
+ Minerals
Lepidolite 594 0 823L - 2014-12-29 13:43:25
@ HIP 18918/Johnson Ring
+ Minerals
Lepidolite 474 496 ? 1557M 2014-12-29 15:09:20
It would seem that my fake demand number of 50M is not being accepted, and being replaced with a ?, which TD interprets as ignore if I understand correctly. I also tried to use the 50? instead of the 50M in my two .prices files for those systems which I was importing, as the documentation suggests, and it was replaced with a ? as well. So when I try to find a route, it is not matching the 496 buy price at Johnson Ring with the 594 sell price at Fleming Enterprise. It is not unique to these two stations, every station I have entered (around 10 or so as I have just started using TD) seems to have this problem. Is this a bug, or simply something I am doing wrong? I tried rebuilding the cache with no luck, and I even backed up TradeDangerous.prices and then edited it manually to replace the ? with 50M, rebuilt the cache, and it still would not generate a run. The final kicker, if I ask it to 'buy' or 'sell' Lepidolite or a similar commodity after doing the manual edit of the TD .prices file, it finds and lists it correctly in both cases, but still will not generate a run.
Other than this instance, it seems to be working great, and at this point I am not sure if it is just me or if others have noticed this before? I checked the issues list on sourcefourge and did not see it, and looked through the most recent forum postings, but not all of them, so thought I would post it here. Apologies if this is something someone has already brought up, and thanks for all of your work on this tool.
I have just started using TradeDangerous and I am very impressed - fantastic piece of work (to the point where I am digging my Python and Jython stuff out after several years and thinking about messing about with the UI). I am running everything locally (ie not using Maddavos or other crowdsourced data except for a single download of the Systems and Stations .csv files when I began), using my own .prices imports. This was working great for the past few days, until this morning when I noticed something odd. I have the following data for two systems, each in their own .prices file which I import into the larger TradeDangerous.prices file. (this is a snippet of the larger .prices files for both systems which serve to illustrate the problem). As you can see below, for items that have both sell and buy prices, I have been entering 50M for the buy price where no demand is known. If I try to generate a standard run for either system, (they are right next to each other, and it should be profitable in this and other similar minerals, and I have lots of money and a small cargo hold), I get the unable to generate run/no prices in the database error:
@ HIP 18918/Johnson Ring
+ Minerals
Lepidolite 474 496 50M 1557M
@ HIP 17621/Fleming Enterprise
+ Minerals
Lepidolite 594 0 823L -
What is interesting is when I look at the TradeDangerous.prices file, these systems (again a snippet shown here) have been recorded as:
@ HIP 17621/Fleming Enterprise
+ Minerals
Lepidolite 594 0 823L - 2014-12-29 13:43:25
@ HIP 18918/Johnson Ring
+ Minerals
Lepidolite 474 496 ? 1557M 2014-12-29 15:09:20
It would seem that my fake demand number of 50M is not being accepted, and being replaced with a ?, which TD interprets as ignore if I understand correctly. I also tried to use the 50? instead of the 50M in my two .prices files for those systems which I was importing, as the documentation suggests, and it was replaced with a ? as well. So when I try to find a route, it is not matching the 496 buy price at Johnson Ring with the 594 sell price at Fleming Enterprise. It is not unique to these two stations, every station I have entered (around 10 or so as I have just started using TD) seems to have this problem. Is this a bug, or simply something I am doing wrong? I tried rebuilding the cache with no luck, and I even backed up TradeDangerous.prices and then edited it manually to replace the ? with 50M, rebuilt the cache, and it still would not generate a run. The final kicker, if I ask it to 'buy' or 'sell' Lepidolite or a similar commodity after doing the manual edit of the TD .prices file, it finds and lists it correctly in both cases, but still will not generate a run.
Other than this instance, it seems to be working great, and at this point I am not sure if it is just me or if others have noticed this before? I checked the issues list on sourcefourge and did not see it, and looked through the most recent forum postings, but not all of them, so thought I would post it here. Apologies if this is something someone has already brought up, and thanks for all of your work on this tool.
Last edited:
I see the database is already 4.8M. Are there already files for the past 48 hrs or the past 6 days to import?
Also, I have a somewhat stupid question.
Do we really need to know when an item is in stock that there are 3447631 units available or 3447632?
As long as there's say more than 10000, it shouldn't really matter?
Also, I have a somewhat stupid question.
Do we really need to know when an item is in stock that there are 3447631 units available or 3447632?
As long as there's say more than 10000, it shouldn't really matter?
I agree that the lack of a mature and industry-accepted GUI library a disadvantage. My usual workaround is to use the great http and templating packages and simply create a HTML5 page as a UI to be shown in the browser. I consider the deployment too be much simpler than that of a Python program, as you normally only deploy a single statically linked executable for any platform and do not need to care for certain dependencies. Setting up a development environment for Windows is comparable to Python: Installing the language tools and libraries from a MSI (C:\go vs. c:\Python34), setting up the PATH environment variable and running a command ('go build' vs. 'python trade.py')
That's one of the reasons why I went with a daemon instead of a CLI tool: I kept all price data in memory all the time. Back in Beta the memory consumption was never a problem, I still need to test it with the latest amount of data.
Which "the database"? If you are referring to maddavo's .prices file, he provides both a "full" and a "48 hours" file. If you use the maddavo plugin (e.g.
Code:
trade.py import --plug=maddavoaqeuttinger said:
I never do - I just replace anything about 10-15k with "?". The values you're seeing are probably from people using a scraping tool to obtain their data, or the occasional OCD person
-Oliver
- - - - - Additional Content Posted / Auto Merge - - - - -
Ok, I'll take a look again. Every time I've tried to work with someone's Go project it's gotten me eyeball deep in dependencies, otherwise I'd probably have started TD in Go instead of Python.
fawick said:
It was about the time I got to worrying about such things that I discovered that Python's parallelism is a non-starter. I mean, I knew of the GIL, but I just couldn't comprehend that it would mean a parallel python program will at best run slightly slower than a single threaded version. I guess it's easy to forget that Guido started Python as far back as 91.
I'll have to investigate this later.
"?" translates to (units=-1, level=-1) which means "unknown". "-" translates to (units=0, level=0) which means "ignore". TD doesn't currently have a way to represent the absence of a demand value or the demand value "0L" (i'm working on that but it's a lower priority right now).
If you want to investigate this further yourself, try running with various levels of debug ("-wwww" is spammy) and have it display a larger number of routes (e.g. --routes 30) to see if that trade is just not /winning/.
The translation from "50M" to "?" is happening because the demand column used to be useless, and last night I sold a bunch of stuff to a station that claimed it had no demand (bought from elsewhere). So an auto-conversion was added basically to discourage people from having to collect it.
I'm curious about what you were saying about maintaining your own .prices files. Are you using the trade.py "import" sub-command? This loads data from a file, puts it into the cache and then re-exports the cache to the .prices file.
Since it sounds like you may be doing something novel, be aware that in the next major version we're going to be making the ".db" file authoritative (and thus not routinely deleting/dropping/wiping it). A side effect of this will be a move away from the ".prices" file and probably a move towards adding json for exchanging data. Support for the .prices format will remain (not least because its useful for the text-editor thing
- - - - - Additional Content Posted / Auto Merge - - - - -
Probably not recently enough
- - - - - Additional Content Posted / Auto Merge - - - - -
Exactly it - the left one is how old the seller's data is and the right one is how old the buyer's is. It used to just show one date and people were a bit confused by that. Now it'll only show one date if the two are within a half hour or so of each other.
That can be done one of two ways:
I can do this one of two ways:
1. navigate to each via in the list and then finally to the destination,
2. take the list and find you the best path that visits all of the stations specified (known as the "travelling salesman" problem),
I'd been assuming people wanted #2 but I just realized I can always make that an option later. So I just checked in "--via" for nav.
No problem, many thanks for the quick response. I'll try to play around some more with it tomorrow with the debug on. I probably wasn't clear when I wrote what I did with the .prices files. Basically I am keeping one per system I visit, and importing them when I create or update them into the main TradeDangerous.prices and db. I've moved on from those systems but have encountered similar "problems" elsewhere, basically anywhere that a commodity is offered for both sell and buy. The workaround so far has been to enter a fake buy demand value, and at least then you can compare commodities when you use "buy" and "sell", it just does not work for a run. I'll try it with debug tomorrow and play around a bit and see what I can find. As a former Java developer in a past life, I know just enough Python to be dangerous (get it? dangerous. heh heh). I may eventually move away from the text files and try to use the gui, but like doing everything CLI first to get used to the tool. It may have had something to do with the number of hops also, although I tried everything from 1 to 5, or the fact that I just clued in tonite about the --via flag, although I tried that with the other route with a similar situation (both buy and sell values for the same commodity) this evening and got the same error.
JSON looks interesting, have not used it. At least you stayed away from XML, parsing it would have slowed things down tremendously I suspect, unless the parsers have improved in the last few years.
Cheers!
JSON looks interesting, have not used it. At least you stayed away from XML, parsing it would have slowed things down tremendously I suspect, unless the parsers have improved in the last few years.
Cheers!
I'll have to investigate this later.
"?" translates to (units=-1, level=-1) which means "unknown". "-" translates to (units=0, level=0) which means "ignore". TD doesn't currently have a way to represent the absence of a demand value or the demand value "0L" (i'm working on that but it's a lower priority right now).
If you want to investigate this further yourself, try running with various levels of debug ("-wwww" is spammy) and have it display a larger number of routes (e.g. --routes 30) to see if that trade is just not /winning/.
The translation from "50M" to "?" is happening because the demand column used to be useless, and last night I sold a bunch of stuff to a station that claimed it had no demand (bought from elsewhere). So an auto-conversion was added basically to discourage people from having to collect it.
I'm curious about what you were saying about maintaining your own .prices files. Are you using the trade.py "import" sub-command? This loads data from a file, puts it into the cache and then re-exports the cache to the .prices file.
Since it sounds like you may be doing something novel, be aware that in the next major version we're going to be making the ".db" file authoritative (and thus not routinely deleting/dropping/wiping it). A side effect of this will be a move away from the ".prices" file and probably a move towards adding json for exchanging data. Support for the .prices format will remain (not least because its useful for the text-editor thing
Last edited:
I am seeing an error relating to a DELETED station. This then prevents cache rebuild and proper functioning until the station is removed from stations.csv by hand. I wonder what causes this and if there is anything that can be done to make TD more tolerant of whatever is causing this error.
Code:
D:\Gadgets\TradeDangerous>trade.py import --plug=maddavo --option=stncsv --option=syscsv
Connecting to server: http://www.davek.com.au/td/System.csv
data/System.csv: 1,394,416/1,394,416 bytes | 56.32KB/s | 100.00%
Connecting to server: http://www.davek.com.au/td/station.asp
data/Station.csv: 90,817/90,817 bytes | 44.03KB/s | 100.00%
NOTE: Stale/missing local copy, downloading full .prices file.
Connecting to server: http://www.davek.com.au/td/prices.asp
import.prices: 4,953,297/4,953,297 bytes | 78.76KB/s | 100.00%
NOTE: Rebuilding cache file: this may take a moment
D:\Gadgets\TradeDangerous\trade.py: Station.csv:2280 ERROR Station 'Julian Gateway' is marked as DELETED and should not be used.That's not some random error, that's an explicit message from TD that it was given bad data. Zeta Aquilae / Julian Gateway doesn't exist, but apparently someone re-uploaded it to Maddavo's site.
- - - - - Additional Content Posted / Auto Merge - - - - -
Why not just put "?", which means "unknown"?
If you could upload your system / station / prices file someplace, I can take a look. I'm not able to reproduce the exact situation but I think that's probably because you're speaking english and I'm speaking td-python
If I can see the data and recreate it, I can also figure out what ambiguity I've introduced -Oliver
No problem, many thanks for the quick response. I'll try to play around some more with it tomorrow with the debug on. I probably wasn't clear when I wrote what I did with the .prices files. Basically I am keeping one per system I visit, and importing them when I create or update them into the main TradeDangerous.prices and db. I've moved on from those systems but have encountered similar "problems" elsewhere, basically anywhere that a commodity is offered for both sell and buy. The workaround so far has been to enter a fake buy demand value, and at least then you can compare commodities when you use "buy" and "sell", it just does not work for a run. I'll try it with debug tomorrow and play around a bit and see what I can find. As a former Java developer in a past life, I know just enough Python to be dangerous (get it? dangerous. heh heh). I may eventually move away from the text files and try to use the gui, but like doing everything CLI first to get used to the tool. It may have had something to do with the number of hops also, although I tried everything from 1 to 5, or the fact that I just clued in tonite about the --via flag, although I tried that with the other route with a similar situation (both buy and sell values for the same commodity) this evening and got the same error.
JSON looks interesting, have not used it. At least you stayed away from XML, parsing it would have slowed things down tremendously I suspect, unless the parsers have improved in the last few years.
Cheers!
So I did the following:
Code:
$ mv data data.bak
$ mkdir data
$ cp data.bak/TradeDangerous.sql data
$ cp data.bak/*.csv data
$ head -1 data.bak/Station.csv >data/Station.csv
$ head -1 data.bak/ShipVendor.csv >data/ShipVendor.csv
$ trade.py buildcache -f -iThis creates a clean, blank cache with no stations. Then I added your two test stations:
Code:
$ misc/add-station.py HIP 17621/Fleming Enterprise
$ misc/add-station.py HIP 18918/Johnson RingThen I imported your price data:
Code:
$ trade.py import -
@ HIP 17621/Fleming Enterprise
+ Minerals
Lepidolite 594 0 823L - 2014-12-29 13:43:25
@ HIP 18918/Johnson Ring
+ Minerals
Lepidolite 474 496 ? 1557M 2014-12-29 15:09:20
<ctrl+z>[you need the latest code for the - thing to work]
Then I checked I had the data:
Code:
$ trade.py buy lepid
Station Cost Stock Age/days StnLs Pad
--------------------------------------------------------------------
HIP 18918/Johnson Ring 496 1,557 0.60 ? ?
$ trade.py sell lepid
Station Cost Age/days StnLs Pad
---------------------------------------------------------------
HIP 17621/Fleming Enterprise 594 0.66 ? ?
HIP 18918/Johnson Ring 474 0.60 ? ?So the last thing is to try a hop:
Code:
$ tdrun johnsonring --to flemingent --hops 1
$ trade.py run -vv --ly 11.18 --empty 17.33 --cap 100 --jumps 6 --cr 923572 --from "johnsonring" --to flemingent --hops 1
HIP 18918/Johnson Ring -> HIP 17621/Fleming Enterprise (score: 9800.000000)
Load from HIP 18918/Johnson Ring:
100 x Lepidolite 496cr each, 49,600cr total, data from 14hrs and 15hrs
Jump HIP 18918 -> APIAN YI -> HIP 19232 -> GLIESE 9143 -> GEIRSEMI -> DABAKAJA -> HIP 17621
Unload at HIP 17621/Fleming Enterprise => Gain 9,800cr (98cr/ton) => 933,372cr
----------------------------------------------------------------------------
Finish at HIP 17621/Fleming Enterprise gaining 9,800cr => est 933,372cr totalHowever if you were to ask it to try and do two hops in this configuration, it would fail, because there's only one profitable trade (from Johnson to Fleming).
I'm making some changes to the output given when a route doesn't work, including dumping the best route it found at the last hop it could make.
I've tweaked the way that TD responds if it runs out of viable (profitable) destinations part way through a run calculation.
The most obvious case is the pure and simple impossible route:
now produces: Note that it shows you how far it did get...
It's also intended to cover cases where there's a hole or a gap that it can't cross with a profit, or - for example - if you ask it to go A->B->A but there's nothing at B that A wants to buy.
But I need your feedback to make sure it doesn't negatively impact other cases I wasn't thinking about. For instance, after implementing, I realized it would work rather nicely with "--unique":
There's a discussion thread in the group, or you can post issues on the issue tracker.
The most obvious case is the pure and simple impossible route:
Code:
trade.py run --cap 10 --cr 1000000 --ly 10 --jumps 1 --from whiteorb --to sol --hops 3now produces: Note that it shows you how far it did get...
Code:
############################################################################
SORRY: Could not find any routes that reached SOL at hop #3
You may need to add more hops to your route.
############################################################################
AIABIKO/White Orbital -> AIABIKO/Maxwell Orbital
AIABIKO/White Orbital: 10 x Marine Equipment,
CHEMAKU/Crampton Port: 10 x Tea,
AIABIKO/Maxwell Orbital +15,140crIt's also intended to cover cases where there's a hole or a gap that it can't cross with a profit, or - for example - if you ask it to go A->B->A but there's nothing at B that A wants to buy.
But I need your feedback to make sure it doesn't negatively impact other cases I wasn't thinking about. For instance, after implementing, I realized it would work rather nicely with "--unique":
Code:
$ trade.py run --cap 10 --cr 1000000 --ly 1 --jumps 1 --from aiabiko --hops 10 --unique
############################################################################
SORRY: Could not find routes beyond hop #5
############################################################################
AIABIKO/White Orbital -> AIABIKO/Maxwell Orbital
AIABIKO/White Orbital: 10 x Marine Equipment,
AIABIKO/Haisheng Port: 10 x Fish,
AIABIKO/Haber Dock: 10 x Marine Equipment,
AIABIKO/Jett Ring: 10 x Fruit And Vegetables,
AIABIKO/Maxwell Orbital +15,350crThere's a discussion thread in the group, or you can post issues on the issue tracker.
Understood. Obviously I can remove it from stations.csv and initiate a cache rebuild, but if TD simply will not run with this bad data, might it not be better to have TD discard the data and build cache without it, perhaps then issuing the error as a warning, rather than stopping dead.
Or to put it another way, if it's deleted and should not be used, then can TD take its own advice and simply not use it.
Last edited:
Sorry for the poor explanation, the problem isn't just that it doesn't know about it, but that it explicitly knows it's not supposed to exist, and it's occuring in the .csv file. The problem here is that maddavo's filters allowed it back in so that anyone who downloaded his .csv at that point has a deleted station.
You could post what error you get instead of just "It doesn't work." so someone might help you to figure it out.
What OS are you using, what python version, ...
How do i check if import has worked? should there be a message that station was updated?
First i f10 LHS 3447/Dalton Gateway commodities
then use EliteOCR i export to EDDN (no idea why, is it in use yet) then i export to a import.prices file
then run
from above it looks like the import of LHS 3447/Dalton Gateway is 2days old and the import did not go in. What am i doing wrong?
First i f10 LHS 3447/Dalton Gateway commodities
then use EliteOCR i export to EDDN (no idea why, is it in use yet) then i export to a import.prices file
then run
trade.py import "C:\Program Files Games\Frontier\eliteOCR\OCRSystemData\import.prices" <-ignore the space FD forum layout does this
trade.py run -vv --from "lhs 3447/dalton" --ly-per 10.1 --jumps-per 1 --capacity 8 --credits 1000 --insurance 0 --margin 0.02 --hops 1
LHS 3447/Dalton Gateway -> YAKABUGAI/Levi-Montalcini City:
Load from LHS 3447/Dalton Gateway:
8 x Grain 105cr each, 840cr total (2days) <-------------------------------------2days? is this how old data is for system/station
Jump LHS 3447 -> YAKABUGAI
Unload at YAKABUGAI/Levi-Montalcini City (509ls) => Gain 1,600cr (200cr/ton) => 2,600cr
----------------------------------------------------------------------------
Finish at YAKABUGAI/Levi-Montalcini City gaining 1,600cr => est 2,600cr total
trade.py run -vv --from "lhs 3447/dalton" --ly-per 10.1 --jumps-per 1 --capacity 8 --credits 1000 --insurance 0 --margin 0.02 --hops 1
LHS 3447/Dalton Gateway -> YAKABUGAI/Levi-Montalcini City:
Load from LHS 3447/Dalton Gateway:
8 x Grain 105cr each, 840cr total (2days) <-------------------------------------2days? is this how old data is for system/station
Jump LHS 3447 -> YAKABUGAI
Unload at YAKABUGAI/Levi-Montalcini City (509ls) => Gain 1,600cr (200cr/ton) => 2,600cr
----------------------------------------------------------------------------
Finish at YAKABUGAI/Levi-Montalcini City gaining 1,600cr => est 2,600cr total
Last edited: