I've always been a proponent of more plausible economic, insurance, liability, and law enforcement systems...it's all part of that context and consequence ED sorely lacks.
I'm sure this would change the form ganking takes, but I doubt it would make whatever subset of who are avoiding Open to avoid the possibility of being ganked happy, unless the new system was so arbitrarily heavy handed as to be the opposite of plausible. Even then, it would still be part of a formulaic system with judgements rendered based on technicalities, which would invariably be harsher on the inexperienced or reckless than the experienced and malicious...much like real life.
Organized PvP would need some sort of in-game 'simulator', if it were to be viable without carefully avoiding ship destruction.
The court judgement can force the hooligan to pay compensation for all the property damage he did, but the insurance of the hooligan's own car has literally nothing to do with that.
This is what liability insurance is.
If I have liability insurance and I run someone over, their medical bills are covered by my insurer. In my area, it's illegal to operate a motor vehicle on public roads without liability insurance. You can't get registration or license plates without proving you're insured and if you are caught driving without insurance, you lose your license. Of course, none of this is an absolute prohibition and there are huge number of unlicensed drivers, but it does deter enough people to be significant.
In the
Elite: Dangerous context ship insurance could work in a similar matter. If one is found liable for the destruction of another ship, one's own insurance would cover that, and one's premium would go up. If one's insurance couldn't cover it, then penalties could involve seizure of assets, garnishment of income, or a stint in a debtor's prison until one's cohorts could effectively ransom for one's freedom. Even if the system allowed for the possibility of not getting caught or successfully persecuted (and it absolutely should, if it's to make any sense) prolific criminals would need to exist as outlaws, with access to few of the services others take for granted.