I don't think you broadband speed has much to do with it at all. I have a free network monitor program in my system tray that I can look at if I want when playing ED. It shows live data received and sent data, it rarely goes above 25 kB/s for upload and download when playing. In fact it usually sit below 1 kB/s for both the majority of the time. Just under 10 kB/s for most instance transitions.
Technically you should be able to play most of ED on a dial-up modem (transiting some instances would be a bit longer than usual). The vast majority of data needed to play is already on your computer. This is the case for pretty much all MMOs I've played.
I'm not a computer guy, but logically that means the problem is not about the speed at which data can be sent and received, but the responsiveness between your computer and the ED server once it has received data. Essentially how long it takes to make and fulfil requests.
I usually have a very smooth gameplay experience, but I find this is mixed when it comes to opening up the galaxy map, mission board etc. I would say usually 10-15 secs to open, but sometimes for no discernible reason it can take 30sec - 1 min. Both can happen in the same play session. Since my computer hasn't changed I assume that it has to do with the responsiveness of the server.
I don't know if it is the p2p network, server, the coding, or all three. But I agree with the OP these features take way too long to open and is quite irritating after a while. Not really sure why it isn't pretty much instant. May be because the mission board is rng and the galaxy map needs to recheck system states etc? Whatever the reason we spend enough time jumping between systems and SC, without having to also wait for a window to pop up.