RegulatedNoise - trade tool with zxctypo's best-in-class OCR (EliteOCRReader as-was)

sutex · Jan 3, 2015

Using RegulatedNoise v1.32 , working very well YOUR program is a lot faster than using BPC, given I have to use EliteOCR , then delete the images. Time is money

http://i.imgur.com/3yAz29i.png This image shows , something that needs correcting, at the moment tabbing to commodity list to get the correct spelling as it appears

SUGGESTION that you have a drop down in correct value that has a list of commodity's to choice from. Same for stations & systems , if possible, would save the spelling mistakes

Is there a way that the system logs ( the places I'm always travelling too & from , be apart of a more detail trading run IF my screenshots of those stations are up to date

Also would like a YouTube video by those who DO know how it all works , to explain the inner workings of this fine program, sure I'm missing those finer points

UPDATE IMPORTANT .. I think most of my data is wrong seems the last NUMBER has been left of the SELL BUY or wherever else there's something to be read , not all commodity are missing the LAST NUMBER , so it should be 123 but it stored as 12 , CLOTHING was missed completely

Means If this is happened to me then other data cant be trusted either, its a big problem, this calibration is just not happening as it should
He's me thinking it was all going sweet, Just only noticed it when I started looking low end commodity to move around while a travel around to scan ports

cheers

Critical · Jan 3, 2015

Is there any way to edit the info imported from the screenshots? It just keeps on messing with station names and MED/HIGH demand words all the time.

iAmLegion · Jan 3, 2015

Critical said:
Is there any way to edit the info imported from the screenshots? It just keeps on messing with station names and MED/HIGH demand words all the time.

Not to my knowledge, but it would be good to be able to edit info.

Thuffyr · Jan 3, 2015

Critical said:
Is there any way to edit the info imported from the screenshots? It just keeps on messing with station names and MED/HIGH demand words all the time.

What I do is controlling after import.
I go to "Price Analysis" tab, then select "By Station" tab, select my last imported station, and check the prices.
When a price or something else is not correct, I select the row ans click on "Edit Row" button on botton right of the panel.
A small window open and I change what I want.
T.

Unoid · Jan 3, 2015

I've noticed that If I have EDDN listening on and importing, that the client becoems very laggy and I can hardly use it for station to station or OCR.

Is this common?

ImaginaryEd · Jan 3, 2015

Code for automatic Low/Med/High corrections:

Code:

else if (_correctionColumn == 5 || _correctionColumn == 7) // hacks for LOW/MED/HIGH
{
	var commodityLevelUpperCase =
		textInfo.ToUpper(_commodityTexts[_correctionRow, _correctionColumn]).Replace(" ", "").Replace("-", "").Replace(".", "").Replace(",", "");

	var levenshteinLow = l.LD("LOW", commodityLevelUpperCase);
	var levenshteinMed = l.LD("MED", commodityLevelUpperCase);
	var levenshteinHigh = l.LD("HIGH", commodityLevelUpperCase);
	var levenshteinBlank = l.LD("", commodityLevelUpperCase);

	//Pick the lowest levenshtein number
	var lowestLevenshtein = Math.Min(Math.Min(levenshteinLow, levenshteinMed), Math.Min(levenshteinHigh, levenshteinBlank));

	if (lowestLevenshtein == levenshteinBlank)
	{
		_commodityTexts[_correctionRow, _correctionColumn] = "";
		_originalBitmapConfidences[_correctionRow, _correctionColumn] = 1;
	}
	else if (lowestLevenshtein == levenshteinLow)
	{
		_commodityTexts[_correctionRow, _correctionColumn] = "LOW";
		_originalBitmapConfidences[_correctionRow, _correctionColumn] = 1;
	}
	else if (lowestLevenshtein == levenshteinMed)
	{
		_commodityTexts[_correctionRow, _correctionColumn] = "MED";
		_originalBitmapConfidences[_correctionRow, _correctionColumn] = 1;
	}
	else if (lowestLevenshtein == levenshteinHigh)
	{
		_commodityTexts[_correctionRow, _correctionColumn] = "HIGH";
		_originalBitmapConfidences[_correctionRow, _correctionColumn] = 1;
	}
}

ImaginaryEd · Jan 3, 2015

Treb42 said:
H.E. Suits still cause some issues in 1.31, on the first screenshots it's ok and the 2nd one drops the dots:
http://imgur.com/a/wdUmd#2

Hi Treb, this is because RegulatedNoise doesn't have the dots in the hard-coded name for this commodity, so if you have a variant name for this commodity already in your data, it might match on either (depending on which it thinks is the closer match).

I noticed this when I was writing the new matching code, but I worked around the issue by opening AutoSave.csv and replacing all occurrences of H.E. Suits with He Suits. I don't know what other systems on EDDN use, but using the full name (H.E. Suits) is probably correct.

I mentioned previously that the hard-coded commodity list should be pulled out into a separate file, and I think this reinforces it.

- - - - - Additional Content Posted / Auto Merge - - - - -

sutex said:
UPDATE IMPORTANT .. I think most of my data is wrong seems the last NUMBER has been left of the SELL BUY or wherever else there's something to be read , not all commodity are missing the LAST NUMBER , so it should be 123 but it stored as 12 , CLOTHING was missed completely

Means If this is happened to me then other data cant be trusted either, its a big problem, this calibration is just not happening as it should
He's me thinking it was all going sweet, Just only noticed it when I started looking low end commodity to move around while a travel around to scan ports

cheers

That sounds like it could be a calibration issue - can you post a screenshot and the contents of calibration.txt please?

AngelVI · Jan 3, 2015

Seems like a very useful tool! Thanks for that! I have some isues with the buy/sell commodity prices in version 1.32 - they do not display the last number of the buy or sell price. For example the best buy price for paladium shows 1424 while the real buy price in game is 14246. This goes for all commodities. Is there any way to fix that?

maxh2003 · Jan 3, 2015

ImaginaryEd said:
RegulatedNoise doesn't have the dots in the hard-coded name for this commodity... it might match on either ... using the full name (H.E. Suits) is probably correct ... should be pulled out into a separate file

That sounds like it could be a calibration issue - can you post a screenshot and the contents of calibration.txt please?

+1, this is all accurate. I don't have much time this weekend but what I will try and do is get the source code into GitHub. The more people who care to poke around and help fix stuff, the better as far as I'm concerned

Kepler · Jan 3, 2015

Only problem I am currently having, are errors being made while OCRing the station names. I know this was mentioned earlier in the thread, and is probably related to changing light intensity around that areas of the photo. It then auto inserts/saves this info as a new station. I have been manually correcting the spelling in the stations tab.

Also noticed that when correcting the station name I need to leave a space after it before saving, or the formatting is wrong (i.e. no gap is left between station and system which is in brackets). If I don't do this, it is also saved as a new station.

ImaginaryEd · Jan 3, 2015

maxh2003 said:
get the source code into GitHub

Yes please!

- - - - - Additional Content Posted / Auto Merge - - - - -

AngelVI said:
Seems like a very useful tool! Thanks for that! I have some isues with the buy/sell commodity prices in version 1.32 - they do not display the last number of the buy or sell price. For example the best buy price for paladium shows 1424 while the real buy price in game is 14246. This goes for all commodities. Is there any way to fix that?

This is another calibration issue - where you place the point between sell and buy, try to get it as close to the line as possible. If you could, can you also post a screenshot and the contents of calibration.txt here please?

maxh2003, do you think we could work out the best calibration values for each screen resolution and use those if the screenshot matches a known good resolution? It would remove the possibility of calibration errors occurring.

maxh2003 · Jan 3, 2015

ImaginaryEd said:
maxh2003, do you think we could work out the best calibration values for each screen resolution and use those if the screenshot matches a known good resolution? It would remove the possibility of calibration errors occurring.

I was pondering that earlier in lieu of proper auto-calibration, I think it's worth a shot. I guess we can pick up the majority of users who haven't done some strange FOV customization and are running standard resolutions. In my mind's eye at least, it needn't be any more complex than a drop-down on the OCR Calibration tab, with a list of resolutions, which just auto-populates the calibration values listbox and writes out the calibration.txt file. What do you think?

The Github upload is done btw, and as a test push I incorporated your HIGH/MED/LOW tweaks. I have no idea how pull requests work, but I look forward to receiving them...!

ImaginaryEd · Jan 3, 2015

I saw the github upload - I'm watching it now (as "EdTorbett") and pulling it so I can compile the latest version (as I don't think the latest compiled release contains the high/med/low fixes).

I didn't think we'd need the dropdown selection if we can read the size of the bitmap from the bitmap itself, but if that's offloaded to the python code and pretty much invisible then a dropdown should be sufficient (short of FOV adjustment, which may include rift DK1/2 craziness). Manual calibration should hopefully end up as a fallback method at that point.

maxh2003 · Jan 3, 2015

ImaginaryEd said:
I don't think the latest compiled release contains the high/med/low fixes).

Correct. I probably need to refine the release "process" so it's a bit more automatic than it currently is, producing a release is mildly annoying at the moment. That said, if I never build another installer it'll be ten minutes too soon; I *do* like the big-old-zip-file-of-binaries approach

ImaginaryEd said:
read the size of the bitmap from the bitmap itself, but if that's offloaded to the python code

ImaginaryEd said:
read the size of the bitmap from the bitmap

ImaginaryEd said:
read the bitmap

...oh yeah. Didn't even think of that but it's a no-brainer. I suggest we use the existing Calibrate button, load up the sample screenshot as we currently do, but then check its resolution. If it's a known resolution, offer to use reference values (but allow the user to calibrate manually if they want to). The offer probably needs to point out that if someone's tinkered with (say) FOV then reference values may not work well. How does that sound for a workflow? We avoid any more "Click here to do this" buttons cluttering the UI, and as you rightly say it will mop up 95% (?) of calibration issues...

In passing, the only python involved in the whole app is EliteBrainerous, and (because I don't "do" python) it's a super-simple interface. An ever-so-slightly-tweaked EB receives up to 100 black-and-white cropped numbers (e.g. 1234) and spits out the results in one go. When we need to OCR, we write out those images to a directory and call EB (as a compiled executable). The net effect is reasonable - we only incur the 2-second overhead of invoking EB once per screenshot, and I didn't have to spend long stringing it together either.

So python only ever sees small crops of the image, and all the cropping/image processing takes place in C#. Check out the little raft of image-processing methods at the end of Ocr.cs.

iAmLegion · Jan 4, 2015

As for the smartscreen issue, only option is this http://www.eightforums.com/tutorials/2631-windows-smartscreen-turn-off-windows-8-a.html
Unless you find a way to get it recognized by windows and authorized.

sutex · Jan 4, 2015

Thuffyr said:
What I do is controlling after import.
I go to "Price Analysis" tab, then select "By Station" tab, select my last imported station, and check the prices.
When a price or something else is not correct, I select the row ans click on "Edit Row" button on botton right of the panel.
A small window open and I change what I want.
T.

Assuming other are doing the same. which they are not, , this is a problem , data cant be trusted , If you have to edit the capture price data , there is no point

sutex · Jan 4, 2015

That sounds like it could be a calibration issue - can you post a screenshot and the contents of calibration.txt please?

Yes it is which is the fundamental problem , it's up to the USER to get it right , if they don't then everyone gets the wrong data. Given I have been really careful with the calibration, twice after noticing the missing numbers ,and it still happening. the whole OCR needs to be looked at , Elite OCR gets it right

http://i.imgur.com/1saiXkJ.png calibration.txt

maxh2003 · Jan 4, 2015

sutex said:
Yes it is which is the fundamental problem , it's up to the USER to get it right , if they don't then everyone gets the wrong data. Given I have been really careful with the calibration, twice after noticing the missing numbers ,and it still happening. the whole OCR needs to be looked at , Elite OCR gets it right

http://i.imgur.com/1saiXkJ.png calibration.txt

I do understand your frustration, but until someone with more time on their hands than I currently have, comes along and writes an auto-calibrate, we are where we are. It's also worth remembering that EliteOCR is a different program, written in a different language, by a different person, with different goals in mind, over a different timeframe. There's not much point in comparing them, except when deciding which one to use.

I'd also like to point out that ImaginaryEd's suggestion about presets for common resolutions would probably give us the best reward-to-effort ratio, so I wanted to check your resolution. But I looked through your posts on this thread and, unless I'm much mistaken, you haven't posted or linked to an actual commodities screenshot. What we have is a screenshot of your calibration values, but what ImaginaryEd wanted was two things: 1) an example of your commodities screenshot taken in ED - ideally one that hasn't worked well for you, and separately 2) the contents of the calibration.txt file that's in your RegulatedNoise directory, which you should be able to copy-and-paste into the forum (it only contains one CSV line if I remember correctly). If you have already posted such a commodities screenshot please forgive my error and re-link to it in a new post.

With those two things we should be able to identically replicate your problem and thus (hopefully) fix it. But if we're missing either one of those, it's all guesswork.

Andrew Sayers · Jan 4, 2015

Moved to the "dangerous discussion" forum.

sutex · Jan 4, 2015

maginaryEd wanted was two things: 1) an example of your commodities screenshot taken in ED - ideally one that hasn't worked well for you, and separately 2) the contents of the calibration.txt file that's in your RegulatedNoise directory, which you should be able to copy-and-paste into the forum (it only contains one CSV line if I remember correctly).

Ok , will do,

http://i.imgur.com/YPsGNYQ.png this is one of the images that didn't work My Res is 1920x1080 32 true @144 hz

https://www.dropbox.com/s/6aijpm1ol9qajos/Calibration.txt?dl=0 calibration.txt file from RegulatedNoise directory,