RegulatedNoise - trade tool with zxctypo's best-in-class OCR (EliteOCRReader as-was)

sutex · Dec 30, 2014

View attachment 6367 Followed the instructions to calibrate three times still getting this error , Using EliteOCR no problems with BPC ,

jippyuk · Dec 30, 2014

Probably got something to do with your screen res or colour depth.

I had this issue when I was connecting to my pc via teamviewer in which the res downscales and got that error message. When running in native 1920 it was fine. You got a different res running?

Treb42 · Dec 30, 2014

After I put this in the EDDN thread by accident, let's try this again:

I had a go with RN yesterday and found the OCR kinda lacking...
For one, I had no option in the GUI to correct certain errors, the CSV textbox stayed empty and whatever was recognized landed directly in the db.
I have no clue of how the OCR for commodities works but since it is a very limited dataset (just looking at the english commodities) wouldn't it be possible to automaticly check the OCR results against a defined list of known commodities!? Esp. since I can't simply edit the result in the app like i can for station and system names.

I'll post a few sample screenshots when im back home, but comparing the hit rates (running 2650x1440) eliteOCR performed alot better :/ but I realy like the workflow with RN

maxh2003 · Dec 30, 2014

sutex said:
Followed the instructions to calibrate three times still getting this error

This is an interesting one, I can't immediately see why it's crashing here. I'll put a bit of extra debug information into version 1.1 so when it crashes it will give more info I can use to debug it. It would be helpful if you could post a link to a screenshot that shows the problem, plus your calibration.txt values.

Torqued said:
looks like some nasty critters got into your distribution... VirusTotal

Well first up I can tell you there is nothing nasty in it, these are false positives. I've run various components of RegulatedNoise through VirusTotal to investigate. The result is as you'd suspect given that EliteOCRReader never triggered this - the addition of EliteBrainerous - a compiled, scripted Python app - is giving false positives in virustotal. Symantec, for instance, is giving "WS.Reputation.1" - which means they've never heard of this, and they consider (say) scripted Python to be a high-risk attack vector. For anyone who wants to replicate these results, just find "nn_training.exe" in the install, zip it and send it to virustotal. Nn_training.exe is the front-end to Brainerous and I have tweaked it ever-so-slightly; other than that it is just a compiled Python exe.

Bottom-line: there's nothing at all I can do about this without removing EliteBrainerous, which I'm not going to do. About fifty of VirusTotal's engines say it's clean, the user-base hasn't found anything to complain about and it doesn't run with Admin rights (any more

so it couldn't do much if it wanted to. Maybe someone can tell me something I don't know about compiling Python here, but otherwise, I can't really help you.

SteelyEyed said:
(Some) station(s) will still buy when there is ZERO demand for a product

This is a bit of a nightmare though, isn't it? There is no way to detect this so, unless you were willing to actively tick a box that means "this station buys even with zero demand", there's no other way to handle it. I can't remove the "only include stations which actually have demand" bit of the stat calculations, because most stations *don't* buy unless there is demand, so the stats become useless...?

SteelyEyed said:
The "Start Monitoring..." button is missing from the OCR tab.

This relates to the help information, I presume. The button is now labelled "Monitor Directory for Commodity Screenshots"; I will fix up the help in v1.1.

jippyuk said:
Any chance of a button to delete screenshots used during OCR

A delete-the-screenshot-immediately-after-import checkbox, I guess? Or specifically a button to do it on demand?

Treb42 said:
For one, I had no option in the GUI to correct certain errors, the CSV textbox stayed empty and whatever was recognized landed directly in the db.

Yep, that's how it works at the moment, because most of the time there's nothing to correct at all. But if people want it I can add a checkbox to force it to pause for inspection/amendment before it imports the data?

Treb42 said:
wouldn't it be possible to automaticly check the OCR results against a defined list of known commodities!?

It does, but it's not a big list. It does add any names you OCR, to that list, so once it's seen a name once it'll recognise it again. But only if there are only two or fewer character differences between what it's seen before, and what it's just scanned. I can back that off to three characters, but maybe only for longer-than-six-character names, because we run the risk of false matches otherwise...

various said:
UI colour other than orange

I'll se if I can make this configurable. It's not *completely* straightforward but should be possible. Probably in v1.2, in the meantime I've added better handling of when it falls over for this reason.

Snake Man said:
Can you please remove the 'categoryName': '$1$' from EDDN export as it kind of messes the normal format there.

I think it has to be present, but I can set it to null or Unknown...?

Treb42 · Dec 30, 2014

maxh2003 said:
Yep, that's how it works at the moment, because most of the time there's nothing to correct at all. But if people want it I can add a checkbox to force it to pause for inspection/amendment before it imports the data?

Yes please, a toggle would be perfect I think, so I can do 2-3 stations and check everything is recognised correctly and then go full auto. Yesterday i kept getting "Advance 0 Catalysers" and "HiE Suits" all the time, nothing I would want to push out to the EDDN

And btw, what's with the color highlighting on the "by station" tab? couldn't figure it out...

maxh2003 · Dec 30, 2014

Treb42 said:
Yes please, a toggle would be perfect I think, so I can do 2-3 stations and check everything is recognised correctly and then go full auto. Yesterday i kept getting "Advance 0 Catalysers" and "HiE Suits" all the time, nothing I would want to push out to the EDDN

OK, I'll do that. It sounds a lot like a straightforward editable dictionary of auto-corrects would help people too, because it sounds like people are consistently getting the same errors.

And btw, what's with the color highlighting on the "by station" tab? couldn't figure it out...

It highlights where the best buy or sell is available at the currently-selected station...

Snake Man · Dec 30, 2014

maxh2003 said:
I think it has to be present, but I can set it to null or Unknown...?

There is no categoryname, here are the EDDN format specs.

maxh2003 · Dec 30, 2014

Snake Man said:
There is no categoryname, here are the EDDN format specs.

Ah, it has been removed. Good! I will update accordingly

maxh2003 · Dec 30, 2014

Updated with a few features and bugfixes, see first post

Snake Man · Dec 30, 2014

Great stuff, RegulatedNoise just got that much better. Good work

Kepler · Dec 30, 2014

Loving RegulatedNoise using it all the time and it is working very well for me.

Any chance of a skin for it ??

I use it on a second monitor/extended desktop and the default colours are quite bright/distracting when playing in a dark room. Ideally a dark skin with blacks and oranges would match really well.

Oddity · Dec 30, 2014

Unfortunately it starts with:

---------------------------

---------------------------
Application has encountered an error and will close.

Exception details:

Index was out of range. Must be non-negative and less than the size of the collection.

Parameter name: index

Inner exception:

Stack:

at System.ThrowHelper.ThrowArgumentOutOfRangeException()

at System.Collections.Generic.List`1.get_Item(Int32 index)

at RegulatedNoise.Form1.CheckAndRequestVerboseLogging() in c:\Users\Max\Documents\Visual Studio 2013\Projects\RegulatedNoise\RegulatedNoise\Form1.cs:line 126

at RegulatedNoise.Form1..ctor() in c:\Users\Max\Documents\Visual Studio 2013\Projects\RegulatedNoise\RegulatedNoise\Form1.cs:line 84

at RegulatedNoise.Program.Main() in c:\Users\Max\Documents\Visual Studio 2013\Projects\RegulatedNoise\RegulatedNoise\Program.cs:line 24
---------------------------
OK
---------------------------

(Win8.1x64)
Can I get you something else to help narrow down on this?

maxh2003 · Dec 30, 2014

Oddity said:
Unfortunately it starts with ... CheckAndRequestVerboseLogging() in c:\Users\Max\Documents\Visual Studio 2013\Projects\RegulatedNoise\RegulatedNoise\Form1.cs:line 126

In this case it's b0rking because it's looking for your Frontier Products directory (which you browse to when you first run RN), and it can't find any FORC-FDEV-.... directories in there that hold your game and its log files.

Can you either:
a) delete your RegulatedNoiseSettings.xml file, run the app and re-try setting your products directory?
or
b) open RegulatedNoiseSettings.xml in Notepad or similar, and post the <ProductsPath> line?

Hastarin · Dec 31, 2014

In my case it's looking in the wrong folder. For some reason it's looking in Win32/Logs rather than Logs.

maxh2003 · Dec 31, 2014

Hastarin said:
In my case it's looking in the wrong folder. For some reason it's looking in Win32/Logs rather than Logs.

It shouldn't be looking in "Logs" for Frontier products (FORC-FDEV-...) at all so that seems very odd. Are you getting the same error as Oddity? If so, again try either:
a) delete your RegulatedNoiseSettings.xml file, run the app and re-try setting your products directory?
or
b) open RegulatedNoiseSettings.xml in Notepad or similar, and post the <ProductsPath> line?

I will add extra code around this area to help out in the next release, seems it's catching out a few people.

Oddity · Dec 31, 2014

Ah yes, thanks, settings pointed directly to the log directory. Up & running now.

Unfortunately.... (sry ^.^)

Application has encountered an error in Crop() and will close.

Width = -74; Height = 23; checkpoint = 1

Exception details:

Parameter is not valid.

Inner exception:

Stack:

at System.Drawing.Bitmap..ctor(Int32 width, Int32 height, PixelFormat format)

at System.Drawing.Bitmap..ctor(Int32 width, Int32 height)

at RegulatedNoise.Ocr.Crop(Bitmap b, Rectangle r) in c:\Users\Max\Documents\Visual Studio 2013\Projects\RegulatedNoise\RegulatedNoise\Ocr.cs:line 428

Application has encountered an error and will close.

Exception details:

Object reference not set to an instance of an object.

Inner exception:

Stack:

at RegulatedNoise.Ocr.PerformOcr(List`1 textRowLocations) in c:\Users\Max\Documents\Visual Studio 2013\Projects\RegulatedNoise\RegulatedNoise\Ocr.cs:line 271

at RegulatedNoise.Ocr.ProcessNewScreenshot(String screenshot) in c:\Users\Max\Documents\Visual Studio 2013\Projects\RegulatedNoise\RegulatedNoise\Ocr.cs:line 179

at RegulatedNoise.Ocr.ScreenshotCreated(String filePath) in c:\Users\Max\Documents\Visual Studio 2013\Projects\RegulatedNoise\RegulatedNoise\Ocr.cs:line 47

kurush · Dec 31, 2014

maxh2003 said:
Hm. It hasn't managed to find any rows to load. Can you post or link to the screenshot you were using, please? Also your calibration.txt values.

I actually get this View attachment 6444
right after selecting the game folder and any time I try to start it. Is it trying to go for my screenshots right away? I have a lot of garbage in that folder. I have verbose logging already enabled and I tried both the application folder and its logs subfolder with the same results (not actually the same, Win32/Logs if I use the app folder, this exception if I select "Logs" subfolder). From the stack it looks like it might be trying to mess with the network logging XML there. Here is what I have at the end of it:
<Network
VerboseLogging="1"
Port="0"
upnpenabled="1"
LogFile="netLog"
DatestampLog="1"
>

</Network>
VerboseLogging="1"
<GameObjects />
</AppConfig>

Treb42 · Dec 31, 2014

here is an example of the OCR messing up in v1.1 and the screenshot it got it from:
http://imgur.com/a/9zLvV

maxh2003 · Dec 31, 2014

kurush said:
From the stack it looks like it might be trying to mess with the network logging XML there.

Spot-on, though I don't get where the second "VerboseLogging" entry came from, and what you posted looks normal in every other respect.

Can you try removing *both* "VerboseLogging" lines from your config xml, saving it, and then starting RN? It should give you dialog boxes to talk you through what it does to appConfig.xml - and once it's finished and the app loads, your appConfig.xml should contain one reference to VerboseLogging, immediately underneath the "<Network" line...

- - - - - Additional Content Posted / Auto Merge - - - - -

Oddity said:
Application has encountered an error in Crop() and will close.

Width = -74; Height = 23; checkpoint = 1

"Width = -74": RN has somehow decided, based on your calibration values and the screenshot details, that it needs to crop out a bitmap of minus 74 pixels width. Amusingly, "23" is about the right height... but -74 isn't a good width.

Can you post the contents of your Calibration.txt so I can replicate this exactly, please? Without the calibration values it's just guesswork

maxh2003 · Dec 31, 2014

Treb42 said:
here is an example of the OCR messing up in v1.1 and the screenshot it got it from:[/URL]

"Advance 0 Catalysers" is a good one, I will look at that one specifically. "He Suits" is another matter... strictly everything should be in capitals, any time you see an app using CamelCase it's faking it like I am here, and "He" is the result... plus dropping punctuation from commodity names improves the OCR consistently... hence "He Suits". Again, maybe we need an autocorrect dictionary here.