In-Development TradeDangerous: power-user trade optimizer

kfsone · Sep 22, 2014

I've just pushed version 3.9 which now includes a "nav" command, and is also the first step in working towards more customizable output (I'm going to use Python's format system to allow you to precisely customize what gets displayed, so that it's very easy for 3rd party tools to get the exact output they want).

NOTE: This version changes how the "--detail" command is used on the command line - I made it a common argument (used by any command) which means it has to go infront of the command now.

Before:

trade.py run --detail --from chango --to aulin --ship type6 --cr 10000

After:

trade.py --detail run --from chango --to aulin --ship type6 --cr 10000

-Oliver

Codec · Sep 22, 2014

Could you add an option (and probably db support) for time stamps, so you could see what the timelines of the data is.
Something like
>-> At ITHACA/Hume Depot, Buy: 100 x Indite (@1616cr as of 20:31/Sept-21),
or similar.

wolverine2710 · Sep 22, 2014

kfsone said:
I've just pushed version 3.9 which now includes a "nav" command, and is also the first step in working towards more customizable output (I'm going to use Python's format system to allow you to precisely customize what gets displayed, so that it's very easy for 3rd party tools to get the exact output they want).

NOTE: This version changes how the "--detail" command is used on the command line - I made it a common argument (used by any command) which means it has to go infront of the command now.

Before:

trade.py run --detail --from chango --to aulin --ship type6 --cr 10000

After:

trade.py --detail run --from chango --to aulin --ship type6 --cr 10000

-Oliver

Just back from holiday , finally I have my 'precious' back - internet access. Just noticed that last night there was an update. Great. Implementing the "Python's format system" in a future version is great and makes using TD as an engine by other authors much more feasible.

Are you considering adding additional CLI options to TD for output in JSON and XML format? That would give standardized output which other tools could parse. Or would that not be useful or feasible?

kfsone · Sep 22, 2014

Codec said:
Could you add an option (and probably db support) for time stamps, so you could see what the timelines of the data is.
Something like
>-> At ITHACA/Hume Depot, Buy: 100 x Indite (@1616cr as of 20:31/Sept-21),
or similar.

Read my mind - I actually added timestamps a while ago. I was thinking about exactly while flying last night as well as making it so that if two routes have the same profit (within "margin") it should choose the one with the more recent price.

I'm also thinking of making "--routes N --checklist" work by showing you each of the routes and then letting you pick one.

kfsone · Sep 22, 2014

wolverine2710 said:
Just back from holiday , finally I have my 'precious' back - internet access. Just noticed that last night there was an update. Great. Implementing the "Python's format system" in a future version is great and makes using TD as an engine by other authors much more feasible.

Are you considering adding additional CLI options to TD for output in JSON and XML format? That would give standardized output which other tools could parse. Or would that not be useful or feasible?

Yes, the one you just mentioned.

- Oliver

wolverine2710 · Sep 23, 2014

I've searched this thread but can't find back what I remembered reading here, so no quote this time. You mentioned somebody rewrote your code in another language and that that person was using it in its own program.

Has that commander already published his/her program?
Could you tell me what the url for the thread of that program is?

wolverine2710 · Sep 23, 2014

kfsone said:
You don't have to have EMDN-Tap running to use TD. EMDN-Tap is just a way to update the prices in the local database.

Technically, TD gets all of it's data from "data/TradeDangerous.prices" which is a human-readable file (also human editable).

The "local database" is a machine-readable form of the data. If you change the .prices file and then run trade.py, it rebuilds the db file - this is an efficiency optimization, a "cache". Open "TradeDangerous.prices" in an editor, fly the game, dock at a station, compare the entry for the station in .prices against the game, make your changes, save, and trade.py will pick up the changes next time you run it.

If you instead go to a web-based API, you're going to experience a lot of hurt unless that API essentially does all the work for you.

The problem that TD is trying to solve is officially described as an "NP Problem" [http://en.wikipedia.org/wiki/NP_(complexity)]. Most of the other profit calculators are approximating a result set by using a tweak of the rules.

For example: You have a Lakon Type 9, 228 capacity, 20,000 credits at StationA. The most profitable possible trade is ItemA StationA->StationB for 8000cr each profiting 1100cr/ton. The least profitable trade is ItemZ StationA->StationC for 85cr each profiting 55cr/ton.

Some calculators only bother to check highest profitability. ItemA wins, meaning you can make a maximum of 2200cr.

Others will try and fill up your hold with other trades you can make StationA->StationB, but after buying the gold you only have 4000cr left. The problem with this approach is that the individually most profitable items are usually the most expensive. So they perhaps boost you to 2800cr for a run.

You can comfortably afford a full load of ItemZ. 228 x 55cr profit = 12540kcr profit; that's 10k more than the obvious choices.

I also documented here another more complex example of how TD works harder for your earnings

In order to do this efficiently, you need to have a lot of data to form constraints with early on. Fundamentally, one of the things that makes a problem NP is that for some portion of the problem where you can't know what data can be discarded in advance - you *have* to do a computation with it to *prove* that it's not required: If I ask you to give me the largest number in this list (1,2,3,4,5) that's easy, but if I ask you which number in this list uses the most pixels (1,2,3,4,5) in helvetica you can't answer that without doing some work.

If you try and use TD from a web-based API you're going to have to pull all of the data from the API in order to start forming constraints; you can't specify which stations you want data for in advance.

You could probably cache the data locally and only refresh it periodically, but you're still going to be hammering the API.

If you're ok with that, you can probably work that way, although I think the guy providing the API might feel different about that fairly quickly

I've re read your post a few times. Perhaps my post was not clear enough or Im missing something here. For me up to date price info is crucial. With Slopey's BPC the prices are always up to date in his central database - fed by EMDN. With TD I need to run emdn-tap 24/7 to make sure all prices are up to date. This does waste band with at Andreas site. Ofc all the data is not really wasted because I could use for example dbvisualizer which has support to create nice graphs to visualize data in the database.

What I suggested was a subcommand which would grab the latest prices from the web-API and stores that in the local database - just like emdn-tap does. I could run that command when testing ED say every 5 or 10 minutes. I don't think the author would mind. In fact he encourages users to use it. At least that is how I interpret this question and his response. A snippet from the last part: "Will also be working on providing a caching layer for the API results to speed up access and provide better scalability - for those wondering, please feel free to hammer away at the thing. If it falls over I will fix it; if the site obviously slows down or suffers because of your requests consider slowing your request rate."

Edit: I made a post in elite metric thread where I asked him if he would mind if TD would use the web-api. His reply was positive. He has usage guidelines on the API webpage. He also has created snapshots which are updated every 15 minutes. This can also be used to feed TD, if you choose to implement it. They are about 24 KB small.

IxForres · Sep 23, 2014

As mentioned above I've added snapshots (updated every 15 minutes, and feel free to poll that frequently) to Elite Metrics. http://elite-metrics.com/pages/api/ has the gory details. Hammer away - the site's main purpose, first and foremost, is to provide data for other tools.

Codec · Sep 24, 2014

How do you go about using aggregate APIs?
Suppose I want to list all stations and then for each station what it has on offer - is there a way to do that rather than iteratively with this API?

IxForres · Sep 24, 2014

Codec said:
How do you go about using aggregate APIs?
Suppose I want to list all stations and then for each station what it has on offer - is there a way to do that rather than iteratively with this API?

No, other than using the snapshot. With 50 stations that approach might work, for more (and there'll be more) the periodic snapshots are a more scalable approach.

http://forums.frontier.co.uk/showthread.php?t=39671 might be a better place to discuss Elite Metrics, though, rather than derailing this fine thread

kfsone · Sep 24, 2014

wolverine2710 said:
I've re read your post a few times. Perhaps my post was not clear enough or Im missing something here. For me up to date price info is crucial. With Slopey's BPC the prices are always up to date in his central database - fed by EMDN. With TD I need to run emdn-tap 24/7 to make sure all prices are up to date. This does waste band with at Andreas site. Ofc all the data is not really wasted because I could use for example dbvisualizer which has support to create nice graphs to visualize data in the database.

What I suggested was a subcommand which would grab the latest prices from the web-API and stores that in the local database - just like emdn-tap does. I could run that command when testing ED say every 5 or 10 minutes. I don't think the author would mind. In fact he encourages users to use it. At least that is how I interpret this question and his response. A snippet from the last part: "Will also be working on providing a caching layer for the API results to speed up access and provide better scalability - for those wondering, please feel free to hammer away at the thing. If it falls over I will fix it; if the site obviously slows down or suffers because of your requests consider slowing your request rate."

Edit: I made a post in elite metric thread where I asked him if he would mind if TD would use the web-api. His reply was positive. He has usage guidelines on the API webpage. He also has created snapshots which are updated every 15 minutes. This can also be used to feed TD, if you choose to implement it. They are about 24 KB small.

The TradeDB module looks for two files: data/TradeDangerous.db and data/TradeDangerous.prices.

If the DB file is missing or has a "last modified" time stamp older than the ".prices" file, then it reads the .prices file and rebuilds the .db files.

If you like, you can use a text editor to edit the .prices file by hand, this is what it looks like:

Code:

# Source for TradeDangerous' price database.

@ ACIHAUT/Cuffey Plant
   + Chemicals
      Hydrogen Fuels             47     48   2014-09-24 02:34:15 demand       0L0 stock  645104L3
      Mineral Oil               198      0   2014-09-24 02:34:15 demand   48237L3 stock       0L0
      Explosives                147    163   2014-09-24 02:34:15 demand       0L0 stock   63298L1
   + Consumer Items
      Dom. Appliances           610      0   2014-09-24 02:34:15 demand    4803L3 stock       0L0
      Consumer Tech            7559      0   2014-09-24 02:34:15 demand    1643L3 stock       0L0
      Clothing                  307      0   2014-09-24 02:34:15 demand    1232L2 stock       0L0
   + Drugs
      Beer                      143      0   2014-09-24 02:34:15 demand   19394L2 stock       0L0

Or, you can write a little bit of code that updates the .db file or the .prices file for you. That's all emdn-tap is.

The .db is sqlite so you can use any language that has an SQLite driver. The .prices file is human readable text, so you can use any language you like.

kfsone · Sep 24, 2014

Codec said:
How do you go about using aggregate APIs?
Suppose I want to list all stations and then for each station what it has on offer - is there a way to do that rather than iteratively with this API?

I'm not 100% sure I get what you mean.

For this specific example you could just look at data/TradeDangerous.prices - it's a human readable text file. If you edit it, TD will regenerate the DB using the values in that file.

But you can also just directly access the SQLite database that TD uses as a cache. Grab the latest version of TD, specifically data/TradeDangerous.sql, and I've added a couple of views, vForSale and vForSaleOrdered, to help/demonstrate.

Code:

c:\dev\trade> sqlite3 data\tradedangerous.db
sqlite> .mode col
sqlite> .width 16
sqlite> .headers on
sqlite> SELECT * FROM vForSaleOrdered WHERE system = 'LHS 2819';
system            station          category    item        asking      stock       stock_level  paying      demand      demand_level  age
----------------  ---------------  ----------  ----------  ----------  ----------  -----------  ----------  ----------  ------------  ----------
LHS 2819          Tasaki Freeport  Chemicals   Explosives  0           0           0            291         4911        2             66372
LHS 2819          Tasaki Freeport  Chemicals   Hydrogen F  48          28175       3            47          0           0             66372
LHS 2819          Tasaki Freeport  Consumer I  Clothing    143         123         3            129         0           1             70494
LHS 2819          Tasaki Freeport  Consumer I  Consumer T  6050        22          3            5981        0           1             66372
LHS 2819          Tasaki Freeport  Consumer I  Dom. Appli  0           0           0            511         41          1             66372
LHS 2819          Tasaki Freeport  Drugs       Beer        0           0           0            144         1505        2             66372
LHS 2819          Tasaki Freeport  Drugs       Liquor      148         410         3            111         0           1             66372
LHS 2819          Tasaki Freeport  Drugs       Narcotics   0           0           0            144         90          2             66372
LHS 2819          Tasaki Freeport  Drugs       Tobacco     4191        229         3            4141        0           1             66372
LHS 2819          Tasaki Freeport  Drugs       Wine        0           0           0            235         70          2             66372
LHS 2819          Tasaki Freeport  Foods       Animal Mea  939         69          3            900         0           1             66372
LHS 2819          Tasaki Freeport  Foods       Coffee      0           0           0            1355        56          2             66372
LHS 2819          Tasaki Freeport  Foods       Fish        393         323         3            364         0           1             66372
LHS 2819          Tasaki Freeport  Foods       Food Cartr  0           0           0            103         480         2             66372
LHS 2819          Tasaki Freeport  Foods       Fruit and   0           0           0            280         21          1             66372
LHS 2819          Tasaki Freeport  Foods       Grain       0           0           0            182         646         2             66372
LHS 2819          Tasaki Freeport  Foods       Synthetic   0           0           0            235         278         2             66372
LHS 2819          Tasaki Freeport  Foods       Tea         1101        166         3            1055        0           1             66372
LHS 2819          Tasaki Freeport  Machinery   Mineral Ex  0           0           0            613         611         2             66372
LHS 2819          Tasaki Freeport  Medicines   Basic Medi  0           0           0            308         239         2             66372
LHS 2819          Tasaki Freeport  Medicines   Performanc  0           0           0            7161        260         2             66372
LHS 2819          Tasaki Freeport  Minerals    Bertrandit  1878        18757       3            1830        0           0             66372
LHS 2819          Tasaki Freeport  Minerals    Gallite     1383        29099       3            1346        0           0             66372
LHS 2819          Tasaki Freeport  Technology  Bioreducin  0           0           0            1026        1295        2             66372
LHS 2819          Tasaki Freeport  Technology  H.E. Suits  0           0           0            261         1044        2             66372
LHS 2819          Tasaki Freeport  Waste       Biowaste    16          73          1            10          0           0             66372
LHS 2819          Tasaki Freeport  Weapons     Non-Lethal  0           0           0            1725        54          1             66372
LHS 2819          Tasaki Freeport  Weapons     Personal W  3700        1682        2            3611        0           0             66372
LHS 2819          Tasaki Freeport  Weapons     Reactive A  1588        573         3            1546        0           1             66372

But programatically, the TradeDB module doesn't load a master price view, if that's what you're asking. It deliberately doesn't bother loading prices items that don't represent a useful price point, and it doesn't build an in-memory representation of "all the items available at this station with their cost and price"; it builds a trading representation of the data whereby the prices are organized by station[destination][item] -> Trade() where a Trade object has the cost to buy an item (i.e. the cost at station) and the potential gain.

If you want to know why, try timing how long TD takes to calculate 20 hops in a Type 6 starting with 42 credits and not specifying start or destination

(trade.py run --ship=type6 --cr=42 --hops=24; also, 3.2 million credits in 20 hops is pretty sweet)

The closest you could get with TradeDB right now is something like:

Code:

from tradedb import *

# so locale separators work:
import locale
locale.set_locale(locale.LC_ALL, '')

def age(seconds):
    seconds = int(seconds)
    if seconds <= 120: return str(seconds) + 's'
    minutes = int(seconds / 60)
    if minutes <= 120: return str(minutes) + 'm'
    hours = int(minutes / 60)
    if hours <= 72: return str(hours) + 'h'
    return int(hours / 24) + 'D'

tdb = TradeDB()
for system in tdb.systems():
    if system.stations:
        print('* SYSTEM: {}'.format(system.name()))
        for station in system.stations:
            print(' @ STATION: {}'.format(station.name()))
            for (dest, trades) in station.tradingWith.items():
                print('  + TRADE ROUTE: {}->{}'.format(station.name(), dest.name()))
                for trade in trades:
                    print('   {:.<30s} buy {:<7n} sell {:<7n} gain {:<7n} [src:{}, dst:{}]'.format(trade.name(), trade.costCr, trade.costCr + trade.gainCr, trade.gainCr, age(trade.srcAge), age(trade.dstAge)))
                print()

Outputs like this:

Code:

* SYSTEM: ACIHAUT
 @ STATION: ACIHAUT/Cuffey Plant
  + TRADE ROUTE: ACIHAUT/Cuffey Plant->G 239-25/Bresnik Mine
   Explosives.................... buy     163cr sell     342cr gain     179cr src 13m dst  2h

  + TRADE ROUTE: ACIHAUT/Cuffey Plant->ASELLUS PRIMUS/Beagle 2 Landing
   Gallite....................... buy   1,392cr sell   2,236cr gain     844cr src 13m dst  5m
   Gallium....................... buy   4,984cr sell   5,764cr gain     780cr src 13m dst  5m
   Lithium....................... buy   1,129cr sell   1,888cr gain     759cr src 13m dst  5m
   Beryllium..................... buy   8,209cr sell   8,854cr gain     645cr src 13m dst  5m
   Indium........................ buy   5,805cr sell   6,308cr gain     503cr src 13m dst  5m
   Uranium....................... buy   2,382cr sell   2,845cr gain     463cr src 13m dst  5m
   Hydrogen Fuels................ buy      48cr sell      50cr gain       2cr src 13m dst  5m

  + TRADE ROUTE: ACIHAUT/Cuffey Plant->LHS 5287/Mcarthur's Reach
   Explosives.................... buy     163cr sell     290cr gain     127cr src 13m dst 20h

  + TRADE ROUTE: ACIHAUT/Cuffey Plant->WYRD/Vonarburg Co-operative
   Biowaste...................... buy      18cr sell      85cr gain      67cr src 13m dst 11m
   Hydrogen Fuels................ buy      48cr sell      50cr gain       2cr src 13m dst 11m

  + TRADE ROUTE: ACIHAUT/Cuffey Plant->MAGEC/Xiaoguan Hub
   Biowaste...................... buy      18cr sell      85cr gain      67cr src 13m dst 64m
   Hydrogen Fuels................ buy      48cr sell      50cr gain       2cr src 13m dst 64m

  + TRADE ROUTE: ACIHAUT/Cuffey Plant->LFT 992/Szulkin Mines
   Explosives.................... buy     163cr sell     291cr gain     128cr src 13m dst 51h

  + TRADE ROUTE: ACIHAUT/Cuffey Plant->DAHAN/Dahan Gateway
   Gallite....................... buy   1,392cr sell   2,263cr gain     871cr src 13m dst  2m
   Bertrandite................... buy   1,868cr sell   2,727cr gain     859cr src 13m dst  2m
   Lepidolite.................... buy     323cr sell     702cr gain     379cr src 13m dst  2m

  + TRADE ROUTE: ACIHAUT/Cuffey Plant->BOLG/Moxon's Mojo
   Beryllium..................... buy   8,209cr sell   9,343cr gain   1,134cr src 13m dst 28m
   Indium........................ buy   5,805cr sell   6,762cr gain     957cr src 13m dst 28m
   Lithium....................... buy   1,129cr sell   1,893cr gain     764cr src 13m dst 28m
   Uranium....................... buy   2,382cr sell   3,135cr gain     753cr src 13m dst 28m
   Gallium....................... buy   4,984cr sell   5,476cr gain     492cr src 13m dst 28m
   Titanium...................... buy     818cr sell   1,227cr gain     409cr src 13m dst 28m
   Copper........................ buy     239cr sell     488cr gain     249cr src 13m dst 28m
   Aluminium..................... buy     139cr sell     324cr gain     185cr src 13m dst 28m
   Hydrogen Fuels................ buy      48cr sell      62cr gain      14cr src 13m dst 28m

  + TRADE ROUTE: ACIHAUT/Cuffey Plant->LP 98-132/Freeport
   Explosives.................... buy     163cr sell     337cr gain     174cr src 13m dst 11m

  + TRADE ROUTE: ACIHAUT/Cuffey Plant->ROSS 1015/Bowersox Mines
   Explosives.................... buy     163cr sell     343cr gain     180cr src 13m dst  2h

  + TRADE ROUTE: ACIHAUT/Cuffey Plant->I BOOTIS/Chango Dock
   Beryllium..................... buy   8,209cr sell   9,299cr gain   1,090cr src 13m dst  5m
   Gallium....................... buy   4,984cr sell   5,916cr gain     932cr src 13m dst  5m
   Indium........................ buy   5,805cr sell   6,604cr gain     799cr src 13m dst  5m
   Lithium....................... buy   1,129cr sell   1,884cr gain     755cr src 13m dst  5m
...

kfsone · Sep 24, 2014

IxForres said:
No, other than using the snapshot. With 50 stations that approach might work, for more (and there'll be more) the periodic snapshots are a more scalable approach.

http://forums.frontier.co.uk/showthread.php?t=39671 might be a better place to discuss Elite Metrics, though, rather than derailing this fine thread

TD is modular for the exact purpose of making it easy to replace things like the backend data source.

I'll take a look at EM later and formulate a couple of questions for you, Ix; (this isn't my first rodeo)

-Oliver

wolverine2710 · Sep 24, 2014

Wanted to have a look at the database using DbVisualizer Free but I'm getting an error.

An internal error occurred in:

com.onseven.dbvis.sql.M: java.sql.SQLException: [SQLITE_CORRUPT] The database disk image is malformed (malformed database schema (UpgradeVendor) - near "WITHOUT": syntax error)

The error may have affected the application state. Please to vendor.

Long Message:
[SQLITE_CORRUPT] The database disk image is malformed (malformed database schema (UpgradeVendor) - near "WITHOUT": syntax error)

Details:
***Type: com.onseven.dbvis.sql.M

Stack Trace:
com.onseven.dbvis.sql.M: java.sql.SQLException: [SQLITE_CORRUPT] The database disk image is malformed (malformed database schema (UpgradeVendor) - near "WITHOUT": syntax error)
***at com.onseven.dbvis.db.AbstractFacade.getTables(Z:456)
***at com.onseven.dbvis.J.B.N.ā(Z:792)
***at com.onseven.dbvis.J.B.N.execute(Z:315)
***at com.onseven.dbvis.J.B.Y.ā(Z:1386)
***at com.onseven.dbvis.J.B.Y.run(Z:337)
***at com.onseven.dbvis.J.B.B.এ(Z:3320)
***at com.onseven.dbvis.J.B.B.execute(Z:1626)
***at com.onseven.dbvis.J.B.Y.ā(Z:1386)
***at com.onseven.dbvis.J.B.K.Ă(Z:1374)
***at com.onseven.dbvis.J.B.K.doInBackground(Z:1521)
***at javax.swing.SwingWorker$1.call(SwingWorker.java:296)
***at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
***at java.util.concurrent.FutureTask.run(FutureTask.java:166)
***at javax.swing.SwingWorker.run(SwingWorker.java:335)
***at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
***at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
***at java.lang.Thread.run(Thread.java:724)
Caused by: java.sql.SQLException: [SQLITE_CORRUPT] The database disk image is malformed (malformed database schema (UpgradeVendor) - near "WITHOUT": syntax error)
***at org.sqlite.DB.newSQLException(DB.java:383)
***at org.sqlite.DB.newSQLException(DB.java:387)
***at org.sqlite.DB.throwex(DB.java:374)
***at org.sqlite.NativeDB.prepare(Native Method)
***at org.sqlite.DB.prepare(DB.java:123)
***at org.sqlite.Stmt.executeQuery(Stmt.java:121)
***at org.sqlite.MetaData.getTables(MetaData.java:1117)
***at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
***at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
***at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
***at java.lang.reflect.Method.invoke(Method.java:606)
***at com.onseven.dbvis.g.B.E.ā(Z:3526)
***at com.onseven.dbvis.g.B.F$A.call(Z:1474)
***at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
***at java.util.concurrent.FutureTask.run(FutureTask.java:166)
***... 3 more

System Information:
Product: DbVisualizer Free 9.1.10
Build: #2295 (2014-08-08 18:38:31)
Java VM: Java HotSpot(TM) 64-Bit Server VM
Java Version: 1.7.0_25
Java Vendor: Oracle Corporation
OS Name: Windows 7
OS Arch: amd64
OS Version: 6.1

I've used the program before to check the BPC sqlite database and that went fine. Would it be possible for you to see if you can open the database with the mentioned program. That way I can rule out if it something on my site or perhaps something different. Note: it open fine with sqlite3.exe.

Edit: Extremely happy that you are considering using lx's web-api as an additional datasource.

kfsone · Sep 25, 2014

wolverine2710 said:
Wanted to have a look at the database using DbVisualizer Free but I'm getting an error.

An internal error occurred in:

com.onseven.dbvis.sql.M: java.sql.SQLException: [SQLITE_CORRUPT] The database disk image is malformed (malformed database schema (UpgradeVendor) - near "WITHOUT": syntax error)

The error may have affected the application state. Please to vendor.

Long Message:
[SQLITE_CORRUPT] The database disk image is malformed (malformed database schema (UpgradeVendor) - near "WITHOUT": syntax error)

Details:
***Type: com.onseven.dbvis.sql.M

Stack Trace:
com.onseven.dbvis.sql.M: java.sql.SQLException: [SQLITE_CORRUPT] The database disk image is malformed (malformed database schema (UpgradeVendor) - near "WITHOUT": syntax error)
***at com.onseven.dbvis.db.AbstractFacade.getTables(Z:456)
***at com.onseven.dbvis.J.B.N.ā(Z:792)
***at com.onseven.dbvis.J.B.N.execute(Z:315)
***at com.onseven.dbvis.J.B.Y.ā(Z:1386)
***at com.onseven.dbvis.J.B.Y.run(Z:337)
***at com.onseven.dbvis.J.B.B.এ(Z:3320)
***at com.onseven.dbvis.J.B.B.execute(Z:1626)
***at com.onseven.dbvis.J.B.Y.ā(Z:1386)
***at com.onseven.dbvis.J.B.K.Ă(Z:1374)
***at com.onseven.dbvis.J.B.K.doInBackground(Z:1521)
***at javax.swing.SwingWorker$1.call(SwingWorker.java:296)
***at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
***at java.util.concurrent.FutureTask.run(FutureTask.java:166)
***at javax.swing.SwingWorker.run(SwingWorker.java:335)
***at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
***at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
***at java.lang.Thread.run(Thread.java:724)
Caused by: java.sql.SQLException: [SQLITE_CORRUPT] The database disk image is malformed (malformed database schema (UpgradeVendor) - near "WITHOUT": syntax error)
***at org.sqlite.DB.newSQLException(DB.java:383)
***at org.sqlite.DB.newSQLException(DB.java:387)
***at org.sqlite.DB.throwex(DB.java:374)
***at org.sqlite.NativeDB.prepare(Native Method)
***at org.sqlite.DB.prepare(DB.java:123)
***at org.sqlite.Stmt.executeQuery(Stmt.java:121)
***at org.sqlite.MetaData.getTables(MetaData.java:1117)
***at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
***at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
***at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
***at java.lang.reflect.Method.invoke(Method.java:606)
***at com.onseven.dbvis.g.B.E.ā(Z:3526)
***at com.onseven.dbvis.g.B.F$A.call(Z:1474)
***at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
***at java.util.concurrent.FutureTask.run(FutureTask.java:166)
***... 3 more

System Information:
Product: DbVisualizer Free 9.1.10
Build: #2295 (2014-08-08 18:38:31)
Java VM: Java HotSpot(TM) 64-Bit Server VM
Java Version: 1.7.0_25
Java Vendor: Oracle Corporation
OS Name: Windows 7
OS Arch: amd64
OS Version: 6.1

I've used the program before to check the BPC sqlite database and that went fine. Would it be possible for you to see if you can open the database with the mentioned program. That way I can rule out if it something on my site or perhaps something different. Note: it open fine with sqlite3.exe.

Edit: Extremely happy that you are considering using lx's web-api as an additional datasource.

Well, I'm not specifically considering doing anything with his api, just saying if someone is interested in pulling the data into TradeDangerous.db or TradeDangerous.prices, TD won't care where the data came from.

As to your sql problem, I don't do Java. It sounds like either you opened the .sql file instead of the .db file, or your tool uses an older version of the SQLite database driver:

http://www.sqlite.org/withoutrowid.html

1.2 Compatibility

SQLite version 3.8.2 or later is necessary in order to use a WITHOUT ROWID table. An attempt to open a database that contains one or more WITHOUT ROWID tables using an earlier version of SQLite will result in a "malformed database schema" error.

Alienfish360 · Sep 26, 2014

I started developing a UDP server in this to communicate with the rPi and small LCD display to display the data shown when you run the tool, without having to alt+tab between screens.

I notice now though that with EMDN now being taken down, does this mean that Trade Dangerous will only be functional with manual entry of market data?

gazelle · Sep 26, 2014

no more routes after manual update?

@kfsone:

I read about your tool and installed it today to try it out (via git poll, v3.9?).

After I used the "update" to put in the new prices via an text editor all I get is "No routes match your selected criteria."

After some research my guess would be: The manual update does set the demand/stock to -1 and your tool will ignore it afterwards.

The exported prices.tmp file does not have a column for the demand/stock?

Code:

# Source for TradeDangerous' price database.

@ DAHAN/Dahan Gateway
   + Chemicals
      Mineral Oil               199      0
      Hydrogen Fuels             80     81
      Explosives                145    162
...

kfsone · Sep 27, 2014

Alienfish360 said:
I started developing a UDP server in this to communicate with the rPi and small LCD display to display the data shown when you run the tool, without having to alt+tab between screens.

I notice now though that with EMDN now being taken down, does this mean that Trade Dangerous will only be functional with manual entry of market data?

That's certainly the clearest fallback, but people could also share their data and it wouldn't be difficult to come up with a way for people to collaborate on gathering data. TD is set up pretty nicely for coping with that via the ".prices" mechanism of editing data, and I should probably pull my finger out and add the manual, guided way of adding prices for people.

But anyone can feed the backend SQLite db or the .prices file, the bulk of TradeDangerous is actually concerned with the processing of the data.

-Oliver

kfsone · Sep 27, 2014

gazelle said:
@kfsone:

I read about your tool and installed it today to try it out (via git poll, v3.9?).

After I used the "update" to put in the new prices via an text editor all I get is "No routes match your selected criteria."

After some research my guess would be: The manual update does set the demand/stock to -1 and your tool will ignore it afterwards.

The exported prices.tmp file does not have a column for the demand/stock?

Code:

# Source for TradeDangerous' price database. @ DAHAN/Dahan Gateway + Chemicals Mineral Oil 199 0 Hydrogen Fuels 80 81 Explosives 145 162 ...

It's because I was using "stock_level > 0" as a constraint on loading the data, ugh!

Fixed, committed & pushed. Grab the latest version and you won't have the problem.

-Oliver

kfsone · Sep 27, 2014

I've removed the EMDN code in v3.9.1 to comply with Michael Brookes' post.

In-Development TradeDangerous: power-user trade optimizer

kfsone

Codec

wolverine2710

Tutorial & Guide Writer

kfsone

kfsone

wolverine2710

Tutorial & Guide Writer

wolverine2710

Tutorial & Guide Writer

IxForres

Codec

IxForres

kfsone

kfsone

kfsone

wolverine2710

Tutorial & Guide Writer

kfsone

Alienfish360

gazelle

kfsone

kfsone

kfsone