Ran some pcaps and found the frontier servers getting denied on my outside interface. These where UDP packets using UDP port that showed as correctly working for awhile. Checked my router and found UDP timeout set to 30 seconds (default) which was a few seconds more than the last good response...
