The voltage unbalance in percent is defined by the National Electrical Manufacturers Association (NEMA) in Standards Publication no. MG 1-1993 as:
% Unbalance = (Maximum Deviation from Average Phase to Phase Voltage) / (Average of Three Phase-to-Phase Voltages)
Note that the line voltages are used in this NEMA standard as opposed to the phase voltages. When phase voltages are used, the phase angle unbalance is not reflected in the % Unbalance and therefore phase voltages are seldom used to calculate voltage unbalance.
There is also the IEC Standards to indicate the degree of unbalance is the voltage unbalance factor (VUF) which is the ratio of the negative sequence voltage to the positive sequence voltage represented as:
%Unbalance = [V2 (the Negative Sequence Voltage) / V1 (the Positive Sequence Voltage)] x 100
and are the positive and negative sequence voltages, respectively, and can be obtained using symmetrical components. If you are not familiar with Sequence Components calculations the following will allow you to calculate the ratio of V2/V1
The effect unbalance has and to what limit you should use is somewhat subjective. A NEMA curve for the derating of a squirrel cage induction motor is give in the graph below:
Derating factor for squirrel cage induction motors due to unbalanced voltage. Standard ANSI/NEMA MG1
Hope this helps.