In-Development TradeDangerous: power-user trade optimizer

[ShadowGar]

There are some systems in Brookes' file that does not have a corresponding entry in the sqlite database yet (I used the one from Smacker's fork).

For my own use I added a fake station to these systems, named "Unknown station XXX", with no other meaningful data. It's useful to know if a station is there for route planning, my starchart uses this information.

I don't use Trade Dangerous, but if you want me to add this data, I can create a fork to add it to TradeDangerous.sql.

All systems from Brookes was added. Now initially I only added stations that had markets. I was more concerned with getting a complete picture of total markets then adding in stations without. Inside the data folder you will see a txt document. Those are the systems without stations that was in Brookes list.

There was also other systems that had stations that were completely bugged and prevented entry into the system. Those were ticketed to FD. FD has not responded but they may have been fixed after these last couple patches.

Now the the community has released calculated stars, we have been adding them. But adding non-market stations will be a huge process. Some systems have up to 5+ stations and the mass majority of them have no market or functional mission board. They are just floating gas stations. But if someone wants to compile a list, I would be more then happy to add them.


I would recommend using the main repository only. Using a fork may not provide you with all of the current updates since we have 4,5? of us working on it.

 

[wolverine2710]

All systems from Brookes was added. Now initially I only added stations that had markets. I was more concerned with getting a complete picture of total markets then adding in stations without. Inside the data folder you will see a txt document. Those are the systems without stations that was in Brookes list.

Indeed all systems from the Sys-that-need-work.txt file are in TD v4.2.1 (Oct 15/2014). I just checked them all.

There was also other systems that had stations that were completely bugged and prevented entry into the system. Those were ticketed to FD. FD has not responded but they may have been fixed after these last couple patches.

Afaik that were the Wredguia star systems. I've been crowd sourcing 8+ of them in 2.0.5 and 2.0.6 so they seem to be fine. Actually all systems I've visited so far seem to be fine.

Now the the community has released calculated stars, we have been adding them. But adding non-market stations will be a huge process.

This is the "What is the most efficient way to crowdsource the 3D sytem coordinates for Beta 2 ?" thread. Atm in smackers fork there are about 397/398 star systems. They have to be merged into the kfsone master - pull request made by Smacker today. Atm we have multiple programs which can correctly calculate 3D coordinates for a system by taken distances from that star to 4-9 reference points. The REAL crowdsourcing effort can in principle already start but we try to automize things a bit so that for example volunteers know which systems already have been done. If its automated I think the rest of the 570-398=172 systems can be done in a couple of days if enough volunteers participate.

Some systems have up to 5+ stations and the mass majority of them have no market or functional mission board. They are just floating gas stations.

Indeed most stations/platform at the other end of the Galactic Pill don't have a functioning Bulletin Board nor a working uinversal catography system. Been flying around there without a way to make money...

But if someone wants to compile a list, I would be more then happy to add them.

With the 'compile a list' you mean coords found by the crowd sourcing effort or do you mean something else with 'compile a list'?

I would recommend using the main repository only. Using a fork may not provide you with all of the current updates since we have 4,5? of us working on it.

This is were you lost me. I know you have write access to the repository - for updating the tradedangerous.sql file, database. Not a git guy but is a fork not required to make a pull request? Or can a pull request be done on the cloned master repository of kfsone? So far new coords have been posted in the crowd sourcing thread and its being processed by you and Smacker I believe. Do you suggest we change that?

Sorry for the long post but miscommunication is very easy and its really really difficult to get things streamlined. Thats why I quoted you on every point. Hopefully you are able to answer my questions.

 

[Biteketkergetek]

Now the the community has released calculated stars, we have been adding them. But adding non-market stations will be a huge process. Some systems have up to 5+ stations and the mass majority of them have no market or functional mission board. They are just floating gas stations. But if someone wants to compile a list, I would be more then happy to add them.

Route planning only needs gas stations, markets are irrelevant. That is why I made this workaround. An alternative would have been a new boolean CanRefuel column in the System table.

I would recommend using the main repository only. Using a fork may not provide you with all of the current updates since we have 4,5? of us working on it.

Generally I would agree, but if one cares about the basic System and Station/Outpost info only, the fork with most frequent updates in this regard may be better. A fork also needs to merge upstream changes occasionally anyway.

This is only one single file we are talking about, so I could really "use" all of the forks if I wanted to. Maybe I shouldn't, have spent too much time on my project already.

 

[ShadowGar]

This is were you lost me. I know you have write access to the repository - for updating the tradedangerous.sql file, database. Not a git guy but is a fork not required to make a pull request? Or can a pull request be done on the cloned master repository of kfsone? So far new coords have been posted in the crowd sourcing thread and its being processed by you and Smacker I believe. Do you suggest we change that?

Sorry for the long post but miscommunication is very easy and its really really difficult to get things streamlined. Thats why I quoted you on every point. Hopefully you are able to answer my questions.

As far as I understand, the fork of the main branch can request a pull. The owner or anyone with approval rights can then approve the request to have it pulled into the main repository and update the main.

Also once the main repository updates, it sends a request for all forks to update as well. Which is why it is good practice to update from the main before adding your code and then requesting a pull. Someone else might have done the work your attempting to do now.

For streamlining purposes, it would be wise to send specific data to one person to input. But then you run into hoping that the person has time to do it.What would help speed up the process is if we had a list that consisted of something like this...

For stars

INSERT INTO "System" VALUES(9,'Bolg',-7.90625,34.71875,2.125,'2014-08-26 15:22:38');

For stations

INSERT INTO "Station" VALUES(17,'Moxon''s Mojo',9,0.0);

If you look at the stars code the "VALUES(9" identifies the star number and the station references that number "Moxon''s Mojo',9" in the station code to tie it to the system.

Just look at the main repository and find the last number used, and make your list and make a ticket in the main repository with the data. Then others can see what you've done and add theirs in with different numbers.
That would at least help us in respect to trade dangerous. Its much easier for us to see a ticket then search all the forum posts.

Generally I would agree, but if one cares about the basic System and Station/Outpost info only, the fork with most frequent updates in this regard may be better. A fork also needs to merge upstream changes occasionally anyway.

This is only one single file we are talking about, so I could really "use" all of the forks if I wanted to. Maybe I shouldn't, have spent too much time on my project already.

You could do that. But the main repository is never more than a day or two behind. Kfsone keeps up on it. Also once someone requests a pull, all of us can review the code before submission. Downloading from a fork that only had one pair of hands and eyes reviewing code may be buggier.

At this point, everyone that has forks of the main is either not doing anything or submitting to the main. I'm not seeing a fork off into something else.

 

[ShadowGar]

<snip -double post->

 

[kfsone]

Hello guys.

C:\td>trade.py run --sh hauler --fr chan --cr 20000
Traceback (most recent call last):
  File "C:\td\trade.py", line 936, in <module>
    main()
  File "C:\td\trade.py", line 925, in main
    tdb = TradeDB(debug=args.debug, dbFilename=args.db)
  File "C:\td\tradedb.py", line 344, in __init__
    self.reloadCache()
  File "C:\td\tradedb.py", line 411, in reloadCache
    data.buildcache.buildCache(dbPath=self.dbPath, sqlPath=self.sqlPath, pricesPath=self.pricesPath)
  File "C:\td\data\buildcache.py", line 156, in buildCache
    processPricesFile(tempDB, pricesPath)
  File "C:\td\data\buildcache.py", line 118, in processPricesFile
    for price in priceLineNegotiator(pricesFile, db, debug):
  File "C:\td\data\buildcache.py", line 99, in priceLineNegotiator
    itemID = itemsByName["{}:{}".format(categoryID, itemName)] if qualityItemWithCategory else itemsByName[itemName]
KeyError: 'Battle Weapons'

Anyone able to give me a hand?

Hi, Vixus; please get the latest version of TD (the one I just pushed) and let me know if you still have this problem. I've normalized some of the error handling so that errors like these should have better user feedback (in this case, it should be more succinct - a one liner - and yet more descriptive).

 

[kfsone]

I'm tinkering with a web interface passing args to trade.py in the background. I have managed to get trade.py to output the -h help and the run -h help on a web page so I know that python runs it OK.

So, bear in mind that Python is an eminently hackable language, and TradeDangerous is designed to be modular so that your best bet might actually be to write your own Python module that "import"s TD.

from tradedb import *
tdb = TradeDB()

# Count how much stuff is in the Price table in the DB:
if verbose:
  sqlDb = tdb.getDB()
  cursor = sqlDb.execute("SELECT COUNT(*) FROM Price")
  firstRow = cursor.fetchone()
  count = firstRow[0]
else:
  count = tdb.getDB().execute("SELECT COUNT(*) FROM Price").fetchone()[0]

print("DB loaded. There are {} prices in the DB.".format(count))

To this end, even the top-level trade code is actually done in such a way that you can "import trade". Look at the end of "trade.py" (the if __name__ == "main" bit). Basically, trade.py is it's own wrapper...

All that said, it looks like the code has a problem accessing the "data" sub-directory, which means it's either not executing from that directory or the permissions on that directory and its contents differ from elsewhere.

Please be sure to note which version you are using, if possible.

 

[kfsone]

Looks a fantastic tool. Is there any chance of embedding it into a GUI for ease of use?

Edit. Btw, love the use of the X52 MFD screen.

I've never been great at UIs (http://web.archive.org/web/20071015121411/http://kfs.org/~oliver/617/, http://web.archive.org/web/19981212012359/http://about.warbirds.org/, http://web.archive.org/web/20020803094040/http://oliver.kfs.org/daoc/, http://web.archive.org/web/20020701055226/http://www.kfs.org/arctest/, http://web.archive.org/web/20020704121346/http://www.kfs.org/memtest/ plus I'm kind of to blame for that ****ty "Voice Comms" dialog that Windows XP inherited from Battlefield Communicator which was based on a prototype I knocked up for BFC).

I'm more of a systems programmer, so ... as I said in the original blurb, I've built this so that people who are good at building front-ends can leverage the back-end code to power their tools; that's part of why this is open source.

 

[kfsone]

As a player with very little programming experience, am finding it difficult to work out how to update prices.

I tried running something like

trade.py update "chango" --editor "C:\Program Files (x86)\Notepad++\notepad++.exe" --all

Hi, SelectThis.

I've added built-in support for Notepad++ in the latest (v4.2.2) that I've just pushed now.

  trade.py update chango --vim

This should open notepad++ with a list of items at Chango, organized by categories. It is a reflection of the Market as seen in the game - so the first number column matches the first number column on the screen in-game - which is the price the station pays for the item. The second column is the price you pay the station for the item.

Delete lines that aren't listed at the station in-game, and if any items are shown in a different order for that station in-game, change the order in the list.

Where no price is shown in-game (e.g items the station doesn't sell) leave the value as 0.

HTH.

-Oliver

 

[kfsone]

That would be great. Hopefully "the One" is able to add it to TOTR when he has time. That was tongue in cheek ;-) In the crowd source effort we currently don't have a masterlist of system-coords - which is a headache. I'm hoping that tradedangerous.sql can be our reference till that happens. Also haven't checked but Harbinger yesterday updated his coords so that they are in 1/32LY grid. Have you taken that update into account?

I'm really greatful to you guys for crunching this stuff - I probably need to look at opening up admin rights to the repository or something. It's not my intent to throw grunt work anyone's way - I'm just in the middle of onboarding at a new job right now.

That's making it very difficult for me to keep track of the star/system stuff right now, let alone do it myself, so I really appreciate the work being done by folks here. Since folks like Smacker are actually doing work, I may need to add him or some other volunteer to the repository so they can keep stuff up to date.

 

[kfsone]

Perfecto.

Not quite sure about how TD works. Would it be possible just to fetch your tradedangerous.sql file and run sqllite3 on it or would that destroy data in the database - like prices info?

I know I can just remove the .db file and when running TD it will recreate it. Does is then also transfer the data in the .prices file to the database. As in effectively I've a new database with your coords and my prices?

The ".sql" and ".prices" files are basically blueprints for the database. Every time you start TD it checks those two files against the ".db" file. If the ".sql" or ".prices" files have been touched more recently than the ".db" it deletes the .db and rebuilds it.

So, your .prices file is a human-readable version of the price data, and you can just edit that rather than going through "trade.py update", if you want.

But if you run the ".sql" file through sqlite, it will generate a .db file without any price data (sqlite knows nothing about .prices) and I'm not sure, tbh, what will happen then. TD will think the .db file is newer than the .prices file and ignore it, and so your db will remain price free. If you run "update" at that point, you'll cause TD to re-generate the .prices file based on the .db file and you'll wind up losing everything.

 

[kfsone]

The actual prices are stored in prices.py I believe.

No, prices.py is just code for handling the .prices file (e.g. the function "priceLineNegotiator", because if you can't have fun naming your boring, dry, functions, you should probably work for Microsoft).

data/TradeDangerous.prices

is the human readable price data. It basically looks like a long list of the data you see in update (they're the same format, update just creates a temporary file with one station's data for you to edit and then merges your edit back in)

 

[kfsone]

Thank you guys for the feedback.
Git intimidates me a little bit, I'm just used to centralized repos like CVS/SVN ^^

Thumbs up for being such an active bunch

You can avoid the extreme gitishness of git by using Atlassian's "Source Tree".

http://www.sourcetreeapp.com/

 

[kfsone]

You can just copy the new tradedangerous.sql into your data directory (or edit your existing one) and the next call to trade.py will recreate the database and reimport your price file. The price file ist keept up to date which every "update" command you have done.

If a system or a station is renamend you had prices for then you also have to manually correct the names in the price file first otherwise the import will result in an error.

Really need to address that - keep a history of old names or something.

 

[kfsone]

There are some systems in Brookes' file that does not have a corresponding entry in the sqlite database yet (I used the one from Smacker's fork).

For my own use I added a fake station to these systems, named "Unknown station XXX", with no other meaningful data. It's useful to know if a station is there for route planning, my starchart uses this information.

I don't use Trade Dangerous, but if you want me to add this data, I can create a fork to add it to TradeDangerous.sql.

Hmm - TD shouldn't require a station in a system for the system to work, I changed that somewhere between B1 and B2, I thought.

In B1 versions of TD, I did require a station for every system, and to cope with that I made it automatically add a capitalized, whitespace- and punctuation-stripped star name, e.g. "i Bootis" would become "*IBOOTIS", "72 T'Pau" would become "*72TPAU".

I am actively trying to keep up with pull-requests from people who are forking the repos to make changes, and I'm trying also to give credit in the readme etc.

-Oliver

 

[kfsone]

Route planning only needs gas stations, markets are irrelevant. That is why I made this workaround. An alternative would have been a new boolean CanRefuel column in the System table.

I'd like to see all the stations added, so that we can eventually include refueling stops into extended routes; I've created a task to evaluate the value in adding flags to track what capabilities a station has, but that's probably redundant - you can, in theory, tell that a station doesn't have a market by doing "SELECT COUNT(*) FROM Price WHERE station_id = ?" and testing for non-zero.


https://bitbucket.org/kfsone/tradedangerous/issue/28/evaluate-adding-station-capabilities-flags

-Oliver

 

[kfsone]

As far as I understand, the fork of the main branch can request a pull. The owner or anyone with approval rights can then approve the request to have it pulled into the main repository and update the main.

Also once the main repository updates, it sends a request for all forks to update as well. Which is why it is good practice to update from the main before adding your code and then requesting a pull. Someone else might have done the work your attempting to do now.

For streamlining purposes, it would be wise to send specific data to one person to input. But then you run into hoping that the person has time to do it.What would help speed up the process is if we had a list that consisted of something like this...

For stars

INSERT INTO "System" VALUES(9,'Bolg',-7.90625,34.71875,2.125,'2014-08-26 15:22:38');

For stations

INSERT INTO "Station" VALUES(17,'Moxon''s Mojo',9,0.0);

If you look at the stars code the "VALUES(9" identifies the star number and the station references that number "Moxon''s Mojo',9" in the station code to tie it to the system.

Just look at the main repository and find the last number used, and make your list and make a ticket in the main repository with the data. Then others can see what you've done and add theirs in with different numbers.
That would at least help us in respect to trade dangerous. Its much easier for us to see a ticket then search all the forum posts.



You could do that. But the main repository is never more than a day or two behind. Kfsone keeps up on it. Also once someone requests a pull, all of us can review the code before submission. Downloading from a fork that only had one pair of hands and eyes reviewing code may be buggier.

At this point, everyone that has forks of the main is either not doing anything or submitting to the main. I'm not seeing a fork off into something else.

I'm still a rookie at handling push/pull requests. For the most part it's easy, and I'm largely not clear on the etiquette. For instance, if someone submits a pull request but when I get to it it is way behind master, am I being a if I decline and say "rebase, dude"? (there's a good chance that if they simply resync with master any merge conflicts will go away).

I'll also look into adding at least one more person to the admin status on the repos, I won't name names because I don't want to pressure them

-Oliver

 

[kfsone]

Smacker has a pull request with the latest systems data, but I have to eyeball it a little closer; I'm drawing wife agro and if I don't want to be KOS, I've got to scoot. I'll look at getting that merged asap tomorrow. Sorry, folks.

-Oliver

 

[Biteketkergetek]

Hmm - TD shouldn't require a station in a system for the system to work, I changed that somewhere between B1 and B2, I thought.

In B1 versions of TD, I did require a station for every system, and to cope with that I made it automatically add a capitalized, whitespace- and punctuation-stripped star name, e.g. "i Bootis" would become "*IBOOTIS", "72 T'Pau" would become "*72TPAU".

I am actively trying to keep up with pull-requests from people who are forking the repos to make changes, and I'm trying also to give credit in the readme etc.

-Oliver

Sorry, I was talking about my own project (online version here). It does not require stations being defined either, but uses this information, if available. Earlier I had only a bool value in my tables. I believe route planning cannot be done correctly without this info, otherwise the planner could return a route with many uninhabited systems in a row and no refuel points in between, so the ship would definitely run out of fuel.

That is the theory, I guess in practice it will not happen for two reasons: 1. the bubble is densely populated, 2. fuel scoops will be added at some point. When fuel scoops are going to be implemented, route planning will need to know, whether a system has a gas giant (or whatever necessary for refueling)

 

[Biteketkergetek]

I'm still a rookie at handling push/pull requests. For the most part it's easy, and I'm largely not clear on the etiquette. For instance, if someone submits a pull request but when I get to it it is way behind master, am I being a if I decline and say "rebase, dude"? (there's a good chance that if they simply resync with master any merge conflicts will go away).

I'll also look into adding at least one more person to the admin status on the repos, I won't name names because I don't want to pressure them

-Oliver

I think it is somewhat rude to submit a pull request that is way behind master. The person making a pull request is usually in a better position to make the rebase, since they know better what they need in the first place, so asking/having them to do it is in the interest of both parties. Perhaps they will think you are a moron for a second or two, but will do the rebase anyway. I think that is what Linus Torvalds said in a Google Talk about Git.