I tend to minimize installation and removal of the Rift due to potential issues with scratching the lens ( Oculus had some issues with dk2 in this regard), even if you aren't worried about actual damage to the rift, you have the problems of steaming up lens, getting sweat on them or just flakes of dust each time. So generally I just sit there and wait, sometimes i'll check out galaxy/system map, but as you point out eventually you get bored anyway.
My best advice is to queue up podcasts or audio books to listen while you play. I believe its possible to set up some control of audio via Voice Attack, but haven't looked into it. Of course in an ideal world Frontier would have implemented a audio player or perhaps even a video player ( stick it on that panel between your legs) to hel- pass the time on some of those very long SC ( 15-40 mins )