Use Kinect for IR Head Tracking

Roelon · Feb 16, 2015

Hey all. I'm a newcomer to Elite Dangerous, and I'm loving it so far.

Within the first 5 minutes of play, I exited, and went out and bought a joystick. Made all the difference!

Now, I'm looking into other ways to immerse myself into the game - but on a budget.

I've got an old XBox 360 Kinect. I was wondering if anyone has used it for head tracking?

After a Bing search (lol, yea right) I found Kinect2IR. It's a software solution that turns a Kinect into a (presumably less-reliable) TrackIR. Any game that supports TrackIR can supposedly use this software with a Kinect.

Has anyone used this? Any other thoughts? Of course, I've heard of FaceTrackNoIR, but I imagine that a IR solution would be better if given the choice.

Masta Squidge · Feb 16, 2015

try it and let us know. you do need to be sure your kinect uses a USB connection though. I wasn't aware the one for the 360 was usb, the One certainly isnt

Roelon · Feb 17, 2015

Masta Squidge said:
try it and let us know. you do need to be sure your kinect uses a USB connection though. I wasn't aware the one for the 360 was usb, the One certainly isnt

Good point. Just got an adapter from Amazon. With a prime account, it's less than $10 US. I'll let you know Wed. evening!

NewYears1978 · Feb 17, 2015

Good question. I know the PS3 Camera will work, you can get it for 10 bucks on Ebay/Amazon.

Kinect was a lot more ...clunky than the PS3 Camera but it might still work. I may even grab myself a PS3 Cam soon.

seansamurai1 · Feb 20, 2015

Any further news on this?
cheers

Roelon · Feb 26, 2015

Well, after much testing, I was indeed able to utilize the kinect for head tracking.

Setup:
- 1x XBox 360 Kinect
- 1x Kinect Adapter for PC (Amazon ~$8 US)
- Kinect Developer software - Bing it. (Haha.)
- Kinect2IR

Installation:
- Not going to go through details here. It's pretty straight forward. Install software, Connect the Kinect. ....? Profit.

Observations:
- It performs "ok" tracking. The IR does indeed track well in low-light, so that's a plus over the NoIR solutions.
- It has to be positioned quite far away. It seems there's an optimal range, and that range is a good foot-and-a-half further than normal viewing distances. So, it cannot be mounted on top of my 24" monitor. I had to pull my desk out away from the wall, and rig a makeshift stand for the device.
- Then, the angle was way off. Because of the viewing distance, it had to look down on me from a steep angle. Because it was about 18" behind my monitor the monitor itself blocked the IR.
- That made for an interesting render of my head. Now, the software allows for one to "center" one's head. The software identifies a point on the IR map of your head, and puts a little "thumbtack" there. A pick-point, I 've heard them called. It's a reference point on the render. When the thumbtack moves, your "virtual head" moves. So, I looked straight ahead at the monitor, and centered my head. All was well until I looked down a bit. Due to the already steep angle, and recentering of virtual head, It wouldn't further render me looking down. When I did, the pick point jumped from the middle of my face to my forehead.

I have a pretty decent rig, but had no webcam. I ended up ditching the IR tracking functions, turned my Kinect into a webcam, and am now using FaceTrackNoIR with my Kinect as the webcam.

In summary:
- IR tracks well in low-light, but the narrow "sweet spot" and resulting position of the sensor in relation to me just made it not work well at all.
- FaceTrackNoIR works very (surprisingly) well, but you need to have relatively-even lighting to make it work properly. I only use it when I play during the day. Most often, I don't go for head tracking at all.

IAM GROOT · Feb 16, 2016

Ok I had no idea what I was doing but I finally got kinect2tir tracking my head through the kinect but it seems the dll file is only written for 32bit OS and doesn't work on Horizons which is 64 bit.
The facetracknoir is working though however I'm not finding it easy as it moves way too much at the slightest head movement. I haven't figured out how to tweak it yet.

MC Stryker · Feb 19, 2016

Roelon said:
Hey all. I'm a newcomer to Elite Dangerous, and I'm loving it so far.

Within the first 5 minutes of play, I exited, and went out and bought a joystick. Made all the difference!

Now, I'm looking into other ways to immerse myself into the game - but on a budget.

I've got an old XBox 360 Kinect. I was wondering if anyone has used it for head tracking?

After a Bing search (lol, yea right) I found Kinect2IR. It's a software solution that turns a Kinect into a (presumably less-reliable) TrackIR. Any game that supports TrackIR can supposedly use this software with a Kinect.

Has anyone used this? Any other thoughts? Of course, I've heard of FaceTrackNoIR, but I imagine that a IR solution would be better if given the choice.

I personally haven't. However, I did development with the Kinect v1 and v2 for quite a while and I'm not to sure that would be the best device to achieve an alternative and potentially low-cost solution for head tracking, at least v1. The Kinect typically does a pretty good job but the tech is also finicky at times, especially the v1 from the original Xbox 360. v2 is far better but it can still have some issues, especially with the person in a seated position. With the v1, the thing you will notice most is lighting will greatly affect the results of how well it works.

One solution was Johnny Lee's implementation using the Nintendo Wii-motes (low-cost track ir solution). You definitely are going to require a little bit of programming skills of course to get this working with Elite for head tracking but it would work pretty good. Its possible though with a little bit of searching, you may come across someone whos already used this technique for a general headtracking app similar to TrackIR or one that would be compatible with it. Its a cheap solution, but then again, you already own the older Kinect. I've worked with his code on this before and was able to get things up and running a couple years back. The concept is using the IR camera in a Wii-mote to handle the tracking and you can create a makeshift solution as Johnny does in the video below using Safety Glasses from home depot and some IR LEDs... its fairly easy, its just the programming and/or someone having used this and getting it to work with TrackIR. I think this is actually where TrackIR originally came from and how the people behind it developed it but not 100% on that. But when Johnny did this many years back, he ended up doing a TED event talking about this and it was all over the place back then so its pretty likely to assume that considering TrackIR came some time after, that they created the product from this original video and Johnny's code.

This may not be relevant to you but if so, hope it helps and maybe points you in a potential direction. Here is the link to the video: https://www.youtube.com/watch?v=Jd3-eiid-Uw

Pickle · Nov 23, 2016

Roelon said:
Well, after much testing, I was indeed able to utilize the kinect for head tracking.

Setup:
- 1x XBox 360 Kinect
- 1x Kinect Adapter for PC (Amazon ~$8 US)
- Kinect Developer software - Bing it. (Haha.)
- Kinect2IR

Installation:
- Not going to go through details here. It's pretty straight forward. Install software, Connect the Kinect. ....? Profit.

Observations:
- It performs "ok" tracking. The IR does indeed track well in low-light, so that's a plus over the NoIR solutions.
- It has to be positioned quite far away. It seems there's an optimal range, and that range is a good foot-and-a-half further than normal viewing distances. So, it cannot be mounted on top of my 24" monitor. I had to pull my desk out away from the wall, and rig a makeshift stand for the device.
- Then, the angle was way off. Because of the viewing distance, it had to look down on me from a steep angle. Because it was about 18" behind my monitor the monitor itself blocked the IR.
- That made for an interesting render of my head. Now, the software allows for one to "center" one's head. The software identifies a point on the IR map of your head, and puts a little "thumbtack" there. A pick-point, I 've heard them called. It's a reference point on the render. When the thumbtack moves, your "virtual head" moves. So, I looked straight ahead at the monitor, and centered my head. All was well until I looked down a bit. Due to the already steep angle, and recentering of virtual head, It wouldn't further render me looking down. When I did, the pick point jumped from the middle of my face to my forehead.

I have a pretty decent rig, but had no webcam. I ended up ditching the IR tracking functions, turned my Kinect into a webcam, and am now using FaceTrackNoIR with my Kinect as the webcam.

In summary:
- IR tracks well in low-light, but the narrow "sweet spot" and resulting position of the sensor in relation to me just made it not work well at all.
- FaceTrackNoIR works very (surprisingly) well, but you need to have relatively-even lighting to make it work properly. I only use it when I play during the day. Most often, I don't go for head tracking at all.

I would really really appreciate it if you could tell me how you got Kinect2TIR to control the camera movement on Elite Dangerous I have spent all week looking for a way to do this, I have everything working on Kinect2TIR and the 'INGAME HEAD' skull is moving the way I want it to, but nothing happens when I play Elite Dangerous, I have looking in the controls in opetions on ED but I can't find a way to switch the camera movement from 'mouse' to 'Kinect2TIR'.

could you please help?

thanks, nicholas.

Pickle · Nov 23, 2016

Roelon said:
Well, after much testing, I was indeed able to utilize the kinect for head tracking.

Setup:
- 1x XBox 360 Kinect
- 1x Kinect Adapter for PC (Amazon ~$8 US)
- Kinect Developer software - Bing it. (Haha.)
- Kinect2IR

Installation:
- Not going to go through details here. It's pretty straight forward. Install software, Connect the Kinect. ....? Profit.

Observations:
- It performs "ok" tracking. The IR does indeed track well in low-light, so that's a plus over the NoIR solutions.
- It has to be positioned quite far away. It seems there's an optimal range, and that range is a good foot-and-a-half further than normal viewing distances. So, it cannot be mounted on top of my 24" monitor. I had to pull my desk out away from the wall, and rig a makeshift stand for the device.
- Then, the angle was way off. Because of the viewing distance, it had to look down on me from a steep angle. Because it was about 18" behind my monitor the monitor itself blocked the IR.
- That made for an interesting render of my head. Now, the software allows for one to "center" one's head. The software identifies a point on the IR map of your head, and puts a little "thumbtack" there. A pick-point, I 've heard them called. It's a reference point on the render. When the thumbtack moves, your "virtual head" moves. So, I looked straight ahead at the monitor, and centered my head. All was well until I looked down a bit. Due to the already steep angle, and recentering of virtual head, It wouldn't further render me looking down. When I did, the pick point jumped from the middle of my face to my forehead.

I have a pretty decent rig, but had no webcam. I ended up ditching the IR tracking functions, turned my Kinect into a webcam, and am now using FaceTrackNoIR with my Kinect as the webcam.

In summary:
- IR tracks well in low-light, but the narrow "sweet spot" and resulting position of the sensor in relation to me just made it not work well at all.
- FaceTrackNoIR works very (surprisingly) well, but you need to have relatively-even lighting to make it work properly. I only use it when I play during the day. Most often, I don't go for head tracking at all.

*appologies if the first post worked and you get this replay twice*

I would really appreciate if you could tell me how you got Elite Dangerous to recognize Kinect2TIR as the controller for camera movement, I have the Kinect set up and Kinect2TIR seems to be working well, the 'INGAME HEAD' skull is moving just the way I want it to, however when I start up Elite nothing is happening with the camera movement, i've searched the control settings in ED but can't find a way to change camera movement from 'mouse' to 'Kinect2TIR' please help me, I have no clue what to do now and it has literally taken me all week to get this far, I feel like i'm so close to getting it to work now but there's that one step left lol.

again I'd really appreciate your help in getting this to work, thanks!

Slion · May 21, 2019

Open Track now supports Kinect V2 head tracking in multiple ways.