In-Development Remlok Intelligent HUD for VR

Lordxenu · Apr 23, 2018

I have some very early prototype screenshots for an app I'm developing to give commanders a VR overlay.

What will it do?

I play Elite primarily in VR and really enjoy mission running but wish I had some more information more easily available. Initially I set out to deliver the following:

- Display remaining targets on screen at all time
- Order my missions by value, Time remaining and distance
- Display commander info (name, ranks, credit balance etc)
- Show co-ordinates and headings on planets

After exploring the Journal file and some early prototypes I realised commander details are pointless! you know your name and you only need credits at times when you're not under much pressure. So I removed that aspect and focused on missions.

However, the journal has some useful information which made me wonder could I build more personal narrative using historic mission details. I have been looking at highlighting to the player which factions we have been supporting and which working against perhaps adding more context (e.g. warning!: we have been undermining the xyz faction they may not be pleased to see us) type thing when you jump to a new system where that faction is present.

I have also been looking at how best to handle these interactions (switching from joystick to VR controllers is clunky and I would like to avoid using voice attack (just because it would require coding different things like a .vap to run and I would also like to make this completely free if I ever release).

So I am now looking at integrating IBM Watson to provide text to speech and speech to text as well as potentially using these methods to handle a chatbot type interaction. There is an amount of free stuff you can do with Watson so I will see how far I get. The chatbot would be a verbal gateway to the real world perhaps running searches or even using a commanders details to tell them things (e.g. tell me where the nearest friendly faction is)

What can it do today

As of right now it can do the following:

- Inject an overlay on top of the game into steamVR
- retrieve system information from EDDB
- retrieve and calculate active missions from the journal
- calculate the distance from sol to the missions (I will be getting the players location)
- order missions by reward value, time left and distance (currently from sol but you get the idea)

I have a couple of screenshots (I'm alright at code but art is not my strong point!)

Note I'm using a custom journal file (i've been creating missions to test things) hence the factions / payouts etc looking strange but they are all running from my journal file

What am I doing next?

The mission system needs more work (at the moment if you're not killing things it doesn't care!

Then I want to add more of the context around players relationships with factions

Then I want to try implement the planet heading system

Then I want to add Watson

Then I will probably have more ideas!

Why did I do it?
2 reasons really

1. I was inspired by this post on reddit https://www.reddit.com/r/EliteDangerous/comments/83re2i/rainmeter_hud_overlay/ and really wanted Na'Qan's overlay in VR.
2. I could do some web page stuff but had never coded in unity and definately not C# before and so this seemed like a great project to learn!
Why am i sharing this?

Again 2 reasons I suppose

1. As I said I've never coded Unity or C# before and have managed to pull this together in a few weeks - I'm quite proud of that and wanted to share it with a community where my App might be relevant.
2. If anyone is interested in this and has any suggestions I can take them on board and see what I can do!

I have no release plans etc I may never finish this but if I do I will certainly be releasing it for free.

Thanks for getting this far! All feedback is welcome!

ArachAu · Apr 23, 2018

This seems super neat. I look forward to seeing it develop along.

Dr Nibelungen · Apr 24, 2018

Do you have a github link? Im interested in following the development of this.

Lordxenu · Apr 24, 2018

Dr Nibelungen said:
Do you have a github link? Im interested in following the development of this.

Not on github yet only because I'm still new at this will try get one setup over the next few days

Lordxenu · Apr 24, 2018

I've created the GitHub for this so you can see my progress https://github.com/AArchAngel/Remlok-HUD

besieger · May 2, 2018

Does this not put a fair amount of strain on the system given you are running the unity engine alongside elite?

Marcus Seldon · May 7, 2018

That's very nice, as a developer myself, I would suggest making this overlay ubiquitous, so you can concentrate on the code to make the overlay stable (I suck at graphics too

) and let you and other people inject messages with a message queue or something like that. So you can sperate logic and content.

besieger · May 7, 2018

Marcus Seldon said:
That's very nice, as a developer myself, I would suggest making this overlay ubiquitous, so you can concentrate on the code to make the overlay stable (I suck at graphics too ) and let you and other people inject messages with a message queue or something like that. So you can sperate logic and content.

This project seems dead though I would like to pick it up but before I do, would people even use it?

Marcus Seldon · May 7, 2018

besieger said:
This project seems dead though I would like to pick it up but before I do, would people even use it?

I don't think this project is dead at all, the author committed on GitHub just 6 hours ago.

I would really like having something like this for VR, the ED interface is not the best on VR, having a well-planned HMD UI would be very helpful. I already use EDMC plugins with an Overlay (EDMCOverlay by Ian Norton) like EDR (from lekeno) and OmniScanner (I'm the author for this one), having a framework or a library for VR overlaying would open a lot of possibilities for extensions.

besieger · May 7, 2018

Marcus Seldon said:
I don't think this project is dead at all, the author committed on GitHub just 6 hours ago.

I would really like having something like this for VR, the ED interface is not the best on VR, having a well-planned HMD UI would be very helpful. I already use EDMC plugins with an Overlay (EDMCOverlay by Ian Norton) like EDR (from lekeno) and OmniScanner (I'm the author for this one), having a framework or a library for VR overlaying would open a lot of possibilities for extensions.

Well while it can be done unfortunately the way you have to hook into the compositor would mean a developer would have to extend the overlay to support the apps you linked, also it worth pointing out this overlay is strictly 2D and with a smaller FOV than what you get with a traditional 2d screen you might find it hard to allocate room for all your apps.

Lordxenu · May 23, 2018

Hey there not dead just difficult! I've just about got a first release working which just does missions. In terms of system strain I don't see a framrate drop whilst using the injection to steam vr is simple and there isn't a lot of moving parts. Besieger if you want to get involved I would be more than happy for support. I also messed up my github hence the commits being a bit weird but will hope to update tonight.

besieger said:
Well while it can be done unfortunately the way you have to hook into the compositor would mean a developer would have to extend the overlay to support the apps you linked, also it worth pointing out this overlay is strictly 2D and with a smaller FOV than what you get with a traditional 2d screen you might find it hard to allocate room for all your apps.

Also very much this I've found more than anything space is a constraint and the overlay is very much my app taking feeds from eddb but I have to place the info

Lordxenu · May 23, 2018

besieger said:
Does this not put a fair amount of strain on the system given you are running the unity engine alongside elite?

And finally for this you can test on your system using something like ovrlay which uses a similar injection method but has the extra strain of live recording another window which again runs fine on my system but your mileage may vary I'm hoping to get a video of it in action out as soon as possible

Lordxenu · Jun 6, 2018

Alright another quick update so I've managed to get the app sending commands to the game but more excitingly I've got watson speech to text integrated which means in game text chat through voice dictation I would say with about 90% accuracy.

I've reworked the ui and am now in the process of setting variables up to create you own voice commands similar to voice attack but it looks for keywords not statements (e.g. if sentence contains landing and gear press the landing gear button so you won't need to remember exact commands).

The final thing I'm working on is story context for missions so adding more info about who you're working for and what they're trying to achieve (e.g. "we've been asked to deliver x tons of widgets for xyz they are currently trying to expand and this will help as their people need widgets)

The story system is something I'd like to expand to add triggers for events for example if you do x amount of work for the children of raxxla it will lead you to the formadine rift and so on. This deeper story system will be a second release as I'm really keen to get this out and let people see what they think.

As always this is a slow process ad I'm one guy with very little skill or ability!

amcb10 · Jun 6, 2018

Keep it up, I have VR and it is a Royal pain to type stuff as I am unable to touch type so this will be brilliant.......

amcb10 · Jun 7, 2018

If you require anything testing etc. just send a PM with what is required.

Lordxenu · Jun 7, 2018

amcb10 said:
If you require anything testing etc. just send a PM with what is required.

Hey thanks I will just got the first round of menu animations working I now have every feature working but within certain parameters (e.g. passenger missions not included voice commands are limited to chat and opening / closing the menu) so I have a feeling I may end up taking some time getting it running with all the variables I'll try get an early early release out with many caveats for testing as feedback will be really helpful

Lordxenu · Jun 14, 2018

For anyone still paying any attention to this, another update!

Firstly I've totally screwed my github somehow which is why there haven't been any commits. I'll get round to it but as I'm the only one working on this it's really low priority for me.

So my "little app" has turned into thousands of lines of code and half a dozen plugins.

I'm by no means giving up but it's feeling a bit daunting right now.

So where I've got to.

I'm now able to generate stories on the fly including context around the mission impacts (at the moment just is it positive or negative to the target faction) whilst doing this I'm also creating characters and storing them against factions so for each faction you'll have a contact for security, military, logistics etc and every time you get a mission from a faction you'll get the same person giving you the mission but the flavour text and faction impacts will change.

What I think is pretty cool is that I also generate a voice profile for each one using text to speech through watson so each character has their own voice and the voice is persistent so whenever you take a delivery mission for faction x you'll get the same guy or girl.

I did make a video yesterday but as is always the way a bug appeared and it all fell apart (fixed now).

It feels like I'm really close but I've been on this until 1-2 am for the last few nights and when you have 2 kids (who think 6.30am is a lie in!) It gets a bit much.

I really am going to try get a video up (I say this every time!).

amcb10 · Jun 14, 2018

I'm still here, keep plugging away CMDR......

Marcus Seldon · Jun 17, 2018

Hey Lordxenu,

I played a little with OVRLay which is the base for your Remlok-HUD, what I discovered is that I can't use OVRLay with my Rift DK2 in ED, it works outside ED for example in the SteamVR lounge but once ED takes over and I think Oculus API takes control over SteamVR the Overlay doesn't work.

So my question is, are you using an HTC Vive for testing? Or an Oculus CV1 ?
Is there a way of how can I fix this problem with the DK2 ?

Thanks

Lordxenu · Jun 17, 2018

Marcus Seldon said:
Hey Lordxenu,

I played a little with OVRLay which is the base for your Remlok-HUD, what I discovered is that I can't use OVRLay with my Rift DK2 in ED, it works outside ED for example in the SteamVR lounge but once ED takes over and I think Oculus API takes control over SteamVR the Overlay doesn't work.

So my question is, are you using an HTC Vive for testing? Or an Oculus CV1 ?
Is there a way of how can I fix this problem with the DK2 ?

Thanks

Hey I'm using pimax 4k through steamvr so not sure re oculus the only thing I can think is that ED is launching through oculus not steamvr can you access the steam menu when playing? Maybe check if ed is installed through steam or oculus if it's oculus this will probably be the problem