[RELEASE] Slopey's BPC Market Tool! - New version!

Slopey

Volunteer Moderator
Slopey
You should really update to 4.2.0.2, 4139 is bugged and has issues with updates.

If in doubt, get the download from the OP, I've updated that to 4202, and delete (if you don't want to keep it) your ED4.db in the BPC directory (or back it up if you do), between an uninstall/re-install.
 

Mereidos

M
Slopey said:
You should really update to 4.2.0.2, 4139 is bugged and has issues with updates.

If in doubt, get the download from the OP, I've updated that to 4202, and delete (if you don't want to keep it) your ED4.db in the BPC directory (or back it up if you do), between an uninstall/re-install.
Click to expand...

^ thats what i did ... (delete folder and download from OP)
 

Yokai

Yokai
Also, I've found over the past two weeks that whenever your install of Slopey's starts acting funny/sluggish, the following basic process can do wonders:

1. Kill and restart Slopey's (use Task Manager if you have to).
2. Wait for the restart to finish its download > parse routine, without touching ANYTHING.
3. Go to the Options tab.
4. Delete the database; wait for a success message.
5. Delete the commodity list.
6. Restart Slopey's and wait for the restart to finish its download > parse routine, without touching ANYTHING.

Now you're golden and you'll see it's behaving much better overall.


Two more tips:

A. It's a good idea to blow away the database and commodity list (per the above procedure) every few days at most. Why? Because lots of commanders are constantly populating the DB with bad data (unintentionally or not), and the DB server on the back end is constantly culling older (potentially bad) data with a 48-hour rolling window. However, your local client's DB is NOT being culled, so you're just accumulating bad data constantly. Clearing your local DB gives you the best chance to work with the latest, greatest data that is as clean as possible.

B. If you are not currently using EliteOCR to add/update station commodity data in Slopey's, you're doing it the hard way, and you're potentially messing up the crowdsourced data for everyone else. I cannot tell you now many instances of bad data I see that is obviously either transposed digits or missing digits from what was obviously manual data entry. Do yourself a favor and do the community a favor: get EliteOCR and get comfy with it. It's a snap to ACCURATELY add/update commodity data. I can do a station add/update in less than 3 minutes and be 100% confident of my input.
 

Slopey

Volunteer Moderator
Slopey
Yokai said:
B. If you are not currently using EliteOCR to add/update station commodity data in Slopey's, you're doing it the hard way, and you're potentially messing up the crowdsourced data for everyone else. I cannot tell you now many instances of bad data I see that is obviously either transposed digits or missing digits from what was obviously manual data entry. Do yourself a favor and do the community a favor: get EliteOCR and get comfy with it. It's a snap to ACCURATELY add/update commodity data. I can do a station add/update in less than 3 minutes and be 100% confident of my input.
Click to expand...

Actually, handballed data is far more accurate than EliteOCR. Especially with Ds and Os, and 6s and 9s. Look at Edwaros station rather than Edwards in Sol for example, and various other digit errors.

EliteOCR is NOT more accurate, but it is faster - BUT YOU MUST MANUALLY CHECK the data it creates otherwise you screw up the database faster than people doing it by hand.

So which ever method you use - please ENSURE it is accurate! Don't just press OCR All in EliteOCR and assume it's ok!!!!
 

Corwyn

C
Slopey said:
Actually, handballed data is far more accurate than EliteOCR. Especially with Ds and Os, and 6s and 9s. Look at Edwaros station rather than Edwards in Sol for example, and various other digit errors.

EliteOCR is NOT more accurate, but it is faster - BUT YOU MUST MANUALLY CHECK the data it creates otherwise you screw up the database faster than people doing it by hand.

So which ever method you use - please ENSURE it is accurate! Don't just press OCR All in EliteOCR and assume it's ok!!!!
Click to expand...

Hope you're feeling better, Slopey, I hope 2015 finds you happy, healthy, and prosperous.

As for EliteOCR being inaccurate, you're correct, I'm always correcting O's and D's, 5 and 0 seem a bit odd, 2 and 8, 6 and 8.

I did some experimenting last night, playing with Nvidia's DSR, after seeing a post somewhere about it. I think I finally got it working, but the lowest resolution above 1080 I get with it is 1402X1122. I set my screen resolution to this, and the game, and took the screenshots. I ran it through EliteOCR, and it was a 100% success rate.

I did the OCR twice, once to get a clean sampling, visually corrected as need (one entry at a time), and although I reviewed each entry, none were incorrect.

I then did new screenshots and ran it again, this time doing the automatics in OCR.

I ran both resulting cvs', less timestamps, through a diff, and they came up 100% the same.

The only problem I ran into with the DSR resolution was, after ALT Tab out of ED, when I tried to go back in, the game client locked up. I'll have to do some more experimenting.

I still prefer going through every entry visually, to be sure they're accurate, I know too much about computers to trust them, since OCR is still not 100% accurate, even though these particular test were. Call me a cynical if you will, but I still don't trust computers 100%.

Cheers
 
Last edited:

Yokai

Yokai
Slopey said:
Actually, handballed data is far more accurate than EliteOCR. Especially with Ds and Os, and 6s and 9s. Look at Edwaros station rather than Edwards in Sol for example, and various other digit errors.

EliteOCR is NOT more accurate, but it is faster - BUT YOU MUST MANUALLY CHECK the data it creates otherwise you screw up the database faster than people doing it by hand.

So which ever method you use - please ENSURE it is accurate! Don't just press OCR All in EliteOCR and assume it's ok!!!!
Click to expand...

I understand what you're saying. For me, I'm far less error prone to just visually compare the two lines (screenshot and OCR) one by one, then click one button, than it is to manually enter numbers with lots of tabs and returns/spaces. Too many points of failure the manual way, and that's what jumps out at me when I see obviously bad data in the database. Most common? a price that should be 4 digits that has only one digit. Next most common? transposed digits. If EliteOCR mistaken replaces a 6 with 9, that's a far smaller error than leaving out a digit entirely or transposing a 1 and a 9, especially in the 100s or 1000s column.

Maybe I have better luck with EliteOCR because I run at 1920 x 1080 on an Nvidia card? Also, I have good hygiene taking screenshots (such as moving my cursor completely to the right for each shot, lining up rows by manually dragging the scroll bar, and also enabling "eliminate duplicates" in the tool).

The only problem I usually encounter with EliteOCR is a bad interpretation of the station name. You just have to double-check the station name drop down for every single screenshot to ensure you're picking the correct one, when the tool is confused.
 

Corwyn

C
Yokai said:
I understand what you're saying. For me, I'm far less error prone to just visually compare the two lines (screenshot and OCR) one by one, then click one button, than it is to manually enter numbers with lots of tabs and returns/spaces. Too many points of failure the manual way, and that's what jumps out at me when I see obviously bad data in the database. Most common? a price that should be 4 digits that has only one digit. Next most common? transposed digits. If EliteOCR mistaken replaces a 6 with 9, that's a far smaller error than leaving out a digit entirely or transposing a 1 and a 9, especially in the 100s or 1000s column.

Maybe I have better luck with EliteOCR because I run at 1920 x 1080 on an Nvidia card? Also, I have good hygiene taking screenshots (such as moving my cursor completely to the right for each shot, lining up rows by manually dragging the scroll bar, and also enabling "eliminate duplicates" in the tool).

The only problem I usually encounter with EliteOCR is a bad interpretation of the station name. You just have to double-check the station name drop down for every single screenshot to ensure you're picking the correct one, when the tool is confused.
Click to expand...


Tell ya what, do a test:

1: take a set of screenshots of the commodities.
2: run it through EliteOCR, verifying each and every entry (I do this routinely and it takes just a matter of minutes for the whole set)
3: save it as visualtest.CVS
4: run steps 2 and 3 again, but run the automatics on EliteOCR instead of visually checking each and every entry and save the file as OCRtest.cvs
5: Run both files through some editor that uses RegEx, and clear the time/date stamps, you can use this regular expression for the search:

Code: 
2015(\-[\d]{2}){2}T([\d]{2}\:){2}[\d]{2}\+[\d]{2}\:[\d]{2}\;

6: Run a diff on the two resulting files. See if there are differences, and make note of which file contains the differences.

I've done similar tests, and yes, I've also found where digits are excluded, and don't even show in the OCR scan, but, it's like a glaring error to me, the numbers just seem so far off from what you'd expect, that I go back to ED and confirm it, and manaually correct it before I commit it to the CVS. Due to my visual checks on each and every entry, I surmise I have a .0001% error rate over the course of a full game session, if that. I'm very detail oriented, and a bit on the OCD side where it comes to data collection and validation.

The whole process I do takes no more than 3 to 5 minutes per station data collection, I can spare that amount of time to be sure I'm as close to 100% accurate as I can be.
 
Last edited:

Shismar

S
Corwyn said:
Tell ya what, do a test:
I've done similar tests, and yes, I've also found where digits are excluded, and don't even show in the OCR scan, but, it's like a glaring error to me, the numbers just seem so far off from what you'd expect, that I go back to ED and confirm it, and manaually correct it before I commit it to the CVS. Due to my visual checks on each and every entry, I surmise I have a .0001% error rate over the course of a full game session, if that. I'm very detail oriented, and a bit on the OCD side where it comes to data collection and validation.

The whole process I do takes no more than 3 to 5 minutes per station data collection, I can spare that amount of time to be sure I'm as close to 100% accurate as I can be.
Click to expand...
Comparing line by line is safer and faster than looking at the data and going back to ED (or the screenshots).

I don't know EliteOCRs check dialog, I am using a different OCR solution that does need the checks, but if it is anything like that I am done in 2 minutes max and sure prices are correct. Station names are a different topic, they come out badly so I need to enter them any way if not already present.

With visual control the OCR solutions are certainly more reliable than pure manual entry and produce data much faster. I'd still like an API from Frontier that gives me the data of systems I visit.

P.S. Thanks for you tool Slopey. Much appreciated.
 

Inspector Laz0r

I
Please offer an option to import .csv files in local mode. I can't use EliteOCR anymore, since it doesn't allow me to export .bpc files with screenshots lower than 1080p resolution (I'm using 1680x1050). VSR isn't an option, since it will never support my HD 7950.
 

Slopey

Volunteer Moderator
Slopey
Inspector Laz0r said:
Please offer an option to import .csv files in local mode. I can't use EliteOCR anymore, since it doesn't allow me to export .bpc files with screenshots lower than 1080p resolution (I'm using 1680x1050). VSR isn't an option, since it will never support my HD 7950.
Click to expand...

That's by design - you need to go and speak to Seeebek about limits in EliteOCR. He has said he'll relax it, but go check with him. And local mode still uploads to the cloud, so you'd still be overwriting current data with historic - "local mode" (which is no-updates) just stops your data being updated at your end. That's all.

- - - - - Additional Content Posted / Auto Merge - - - - -

The Spook said:
Clean install of 4.2.0.2

Parsing extremely slow and 'Database table locked' error popping up.

Any ideas?
Click to expand...

Restart it.
 
Last edited:

Dielos

Dielos
Hello Slopey,

I know this has been explained to death, but I'm still having trouble with it. I can't get the application to detect my location. Your last version detected my logs location correctly "D:\Games\Frontier\EDLaunch\Products\FORC-FDEV-D-1010\Logs", but location is still unknown.

The appconfig.xml file has the VerboseLogging="1" in it, several times even:
Code: 
<AppConfig>
    <Display>
        <ScreenWidth>1280</ScreenWidth> 
        <ScreenHeight>720</ScreenHeight> 
        <FullScreen>false</FullScreen> 
        <VSync>true</VSync>
        <PresentInterval>1</PresentInterval>
        <Adapter>0</Adapter>
        <Monitor>0</Monitor> 
        <DX11_RefreshRateNumerator>60</DX11_RefreshRateNumerator>
        <DX11_RefreshRateDenominator>1</DX11_RefreshRateDenominator>
        <LimitFrameRate>false</LimitFrameRate>
        <MaxFramesPerSecond>30</MaxFramesPerSecond>
        <!-- If you are looking for the SSAA setting this has moved in to Settings.xml in the save directory
          ** typically: C:\Users\{username}\AppData\Roaming\Frontier Developments\Elite Dangerous\Options\Graphics.
          ** If this file does not exist you can generate it by entering the graphics options screen in game
          ** and hitting apply. You can then add <SSAA>X</SSAA> when X is the SSAA amount (1 by default) -->
    </Display>
    
    <Threads
        RenderThreadStackSize="1048576"
        WorkerThreadStackSize="1048576"
        NumWorkerThreads="6"
        RenderJobQueueSize="10240"
        KernelJobQueueSize="10240"
        MinSpareCores="0"
        OptimiseForPerformance="1"
        UseThreadPriorities="0"
        PerformanceScaling="1"
    />
    
    <Files
        MaxLoadRetries="10"
    />

    <Network
VerboseLogging="1"
      Port="0"
      upnpenabled="1"
      LogFile="netLog"
      DatestampLog="1"
      >

<Network Port="0"
upnpenabled="1"
LogFile="netLog"
DatestampLog="1"
VerboseLogging="1"
ReportSentLetters="1"
ReportReceivedLetters="1"
>

    </Network>
VerboseLogging="1"
  <GameObjects />
</AppConfig>

As I understand, until the application doesn't detect my location, I can't help with uploading data. Is this correct? My guess is that the problem lies with the config file? I'm running the program as admin.

For the rest it works like a charm. Thanks!
 

SilverlightPony

SilverlightPony
Inspector Laz0r said:
Please offer an option to import .csv files in local mode. I can't use EliteOCR anymore, since it doesn't allow me to export .bpc files with screenshots lower than 1080p resolution (I'm using 1680x1050). VSR isn't an option, since it will never support my HD 7950.
Click to expand...
Seeebek locked out BPC export from sub-1080p screenshots at Slopey's request, IIRC, because the error rate increases drastically below 1080p, and too many people don't verify their f'n numbers before uploading them. You can manually resize the screenshots to 1080p or higher before OCR'ing them, but you MUST verify the OCR'd data before exporting it.

If you're using an nVidia graphics card from 400-series or newer, you can try running the game in Dynamic Super Resolution mode; this basically lets you run the game at a higher resolution than your monitor actually supports, and the vid card will downscale it to fit the monitor. Since the game is running at higher res, any screenshots you take should also be higher res.
 

ixobelle

ixobelle
I'm missing something fundamental, and have spent an hour or so trying to figure it out... I'm running a high enough resolution, and don;t get any sort of resolution errors, but importing BPC files into slopey using Elite OCR, just complains that it's:

Unable to parse the BPC file - is it in the correct format?

I can open the file in Notepoad++, and verify the data is there, and see no option to import CSV or any other format... I also can't find an old version of Slopey's anywhere that may work instead.

I'm trying to use it offline, and just do this for myself, but hitting a wall.
 
Last edited:

Rhysling

R
Love the tool, but is there any preferred method for reporting errors in the database? e.g. misplaced spaceports or commodity prices. Increasingly there are also multiple instances of the same commodity at a given station (although that could be my local db, I suppose... not sure how that hangs together, or how to fix it if it's a local problem, short of deleting the db and starting again).
 

SilverlightPony

SilverlightPony
ixobelle said:
I'm missing something fundamental, and have spent an hour or so trying to figure it out... I'm running a high enough resolution, and don;t get any sort of resolution errors, but importing BPC files into slopey using Elite OCR, just complains that it's:

Unable to parse the BPC file - is it in the correct format?

I can open the file in Notepoad++, and verify the data is there, and see no option to import CSV or any other format... I also can't find an old version of Slopey's anywhere that may work instead.

I'm trying to use it offline, and just do this for myself, but hitting a wall.
Click to expand...
Even offline mode still uploads to the server. BPC is rejecting the file because it thinks something is wrong with it that could lead to bad data in the server-side database...but exactly what's causing this, I'm not sure. Are the system and station names in the filename correct?

- - - - - Additional Content Posted / Auto Merge - - - - -

Rhysling said:
Love the tool, but is there any preferred method for reporting errors in the database? e.g. misplaced spaceports or commodity prices. Increasingly there are also multiple instances of the same commodity at a given station ...
Click to expand...
Report it here, pretty much. Slopey goes through every now and then and sanitizes it manually.

Rhysling said:
... (although that could be my local db, I suppose... not sure how that hangs together, or how to fix it if it's a local problem, short of deleting the db and starting again).
Click to expand...
No, that's pretty much it for trying to clear up client-side errors. Wipe the database, wipe the commodity list, restart BPC, do a fresh download-and-parse.
 

Code

C
Clean install of 4.2.0.2 and noticed a missing system which was previously there. (Granthaimi)

Should I just add it as new data?

Keep up the good work!
 
You must log in or register to reply here.
Back
Top Bottom