If you're going to change wire, I'd go with #3 and a 100A breaker... and call it a primary only protection scheme for the transformer....
It's only like 7' x 3 of wire to change it to #4 from #6 and go with the 80 amp, so that's the plan now. Agree?