... Finding the sweet spot will be complicated, since I still also need to figure out exactly what weapons the opponent is using,...
Short Version: Aim for having even resists for all damage types for hull and shield.
Detailed:
For shields (PvE):
Bi-weave shields (in most cases the best solution for PvE anyway) modded for thermal resistance, shield boosters with resistance/super caps. The trick with bi weave is that you don't want high base shield numbers, much more important is the ability to recharge them fast, giving you more or less unlimited shields in PvE.
For shields (PvP): I'm no expert here, from what i heard:
Mostly base value matters, so reinforced shield (standard shield or prismatic), heavy duty boosters, maybe one thermal modded booster. Most in PvP seem to use plasma weapons, these ignore resists.
For hull (PvE against standard NPC):
Go with reacive armor heavy duty+deep plating modded, one hull booster (the one with the lowest class you have) thermal/deep plating modded, any further hull boosters with heavy duty/deep plating.
For hull (PvE against Thargoids - again, i'm no expert here):
Some caustic resistance, other than that just raw hull hitpoints since they ignore resists.
For hull PvP (no expert...you know...):
Basically the same as against Thargoids minus the caustic (again, plasma weapons do absolute damage). You will want some module protection boosters.
... and what Verity means with "taking heat damage" when my thermometer seems happy etc. ...
If your ship gets too hot (over 100% heat) you take heat damage. Nothing you can do about that, except getting too hot... A class A powerplant helps here, as does low emissions modding / thermal spread special effect. For emergencies you might equip a heat sink launcher, but i have never needed that for PvE.