The exact origin of the phrase is not completely known, but general consensus traces it back to the 1950s in the United States and literal cans of worms. Fisherman used to buy sealed metal cans of earthworms, as opposed to the plastic containers or Styrofoam cups of today. After arriving at their fishing spot, they would set the metal can down and open it. What was inside was alive and if the top was left open for too long or the can was tipped over, well, your biggest problem would no longer be catching fish. It would be catching your bait that’s wriggling away.
The earliest documented example of this phrase is found in 1951 edition of the Edwardsville Intelligencer in Illinois,
The question of command for Middle East defense against Soviet aggression is still regarded as ‘a can of worms’ at General Eisenhower’s SHAPE headquarters here.
Linguists believe that the expression is actually a more modern, Americanized version of the expression “opening Pandora’s Box,” which originally was actually a jar, but mistranslations gave us “Pandora’s Box.”
...
I am now going to name one of my ships Pandora's Jar just to see what reaction people have to it...