OK, just been having a quick play and I think I've figured it out.
If you open the comms panel by either pressing '2' or by pressing UI-focus/Up, then, if you have the "auto focus on text input field" Control option set, the focus will go into the text typing area (which is highlighted in blue) and will stay there until you press escape (i.e. you can type a message, press Enter, type another message, press Enter, etc). You need to press escape to de-focus that text box.
If you open the comms panel with the "auto focus on text input field" Control option turned off then the focus won't go into the text area until you press Enter. Once you do press Enter to start typing a message however the focus will again stay in there until your press escape.
If you don't open the comms panel but simply press the key bound to "quick comms" (typically Enter) then, having typed your message, when you press Enter again to send the message the comms panel will automatically close again.
Personally I'd say turn off the "auto focus on text input field" Control option and use quick comms to send messages. That way you won't be stuck in a situation where you have to press escape to regain control of your ship.
I think perhaps the design is wrong in leaving the input focus in the comms text box after you've pressed Enter to send your message on those first two cases but only time will tell.