Thermal overloads should be set to 100% 3.5A in this case, the fuse is to protect the cable not the motor and should be rated to match your codes and current ratings.
Have you tested the motor and got continuity and insulation resistance values? If so what are they?
There a good chance the motor has survived and the fuses failed simply through fatigue.
I agree, the OL was set wrong, should have been set at 3.5A. Many many people do this wrong, because they read in the NEC that the OL should not allow more than 125% current to flow to the motor (with a 1.15SF). But what they fail to do is read the INSTRUCTIONS for the OL relay, which tell you that the relay is ALREADY calibrated for 125% as the pick-up point of the trip curve, so if you SET it for 125% AGAIN, you are at 125% of 125%... too high to protect the motor.
As for single phasing, it all depends on the LOAD on the motor. If a motor is FULLY LOADED, and a phase is lost, the current on the other two will increase by about 50%, which because it is 25% higher than the pick-up point, trips the OL eventually. But if the motor is only 75% loaded at the time the phase is lost, then the current only increases to 112.5% and the OL will NOT trip, because it is below the 125% pick-up point.
In your case you had the double whammy: the pick-up point was ALREADY at 125% of 5A, so 6.25A. which compared to FLC is 179% of the motor nameplate. So even IF the motor was fully loaded when the phase was lost, the current on the other two phases would have increased to around 5.5A, but that was still WAY below the pick-up point of the OL relay. So it held in until the fuse finally cleared.
Your motor winding insulation is likely compromised too by the way. It may still work for a while, but it's life has been cut short now. The heating effect on the winding insulation is INCREASED when the motor is single phased.