I do know about port forwarding, I just don't like it because it leaves my PC wide open on that port.
This is in addition to Northpin's comment (hopefully some clarification).If you dont have ED running, there is nothing to listen on that port. So there is nothing to be wide open on your PC
Yes, port forwarding will send the packets to your PC, but if you set it up correctly (usually pretty easy to get right, actually), it will be ONLY udp packets, and ONLY those with the correct destination port number (the one you specify). However, even though those packets now get to your PC, there has to be a program listening to that particular UDP port for the OS to pass the packet on (otherwise, the packet is simply dropped). This means that for your PC to be vulnerable, a nefarious program needs to be listening to that port. And part of the way the internet protocol stack (the relevant code in your OS) works is only one program can listen to a port, thus if such a nefarious program was running, Elite would not be able to use the port and you would not be able to connect (meaning Elite's failure to operate would inform you that something fishy is going on). It also means that when Elite is running (in port-forwarding mode), no other program (nefarious or otherwise) can listen to that port.
Also, while not particularly convenient, you can always disable the port-forwarding when not playing Elite, and re-enable it when you are (assuming you have ready access to your router's settings, of course).