To answer your question, it depends on the type of OL but most likely the motor would fry before anything would act. If the 3rd leg was a neutral you would not have a rotating magnetic field; it would be oscillating back and forth on the other two legs but not spinning. When not spinning, a motor winding is essentially a high resistance short circuit. So technically your Short Circuit Protection Device should have operated, but there may have been enough resistance in the windings to prevent that, it depends on a lot of other factors.
As to the OLR, a solid state OLR that has Ground fault and/or Current Imbalance protection would have likely tripped immediately but a standard bi-metal or melting alloy OLR probably would not have done so, or at least not quick enough to prevent motor damage. There would have been current seen on all 3 circuits. It would have been more than what the 3 phase current should have been, but the effect of such a severe current imbalance would have caused motor heating in extreme disproportion to the same of amount of balanced current, so winding damage would be likely before the amount of current seen by the heater elements caused a trip.
So essentially you would have a race between the fuse / CB, the OLR and the motor winding insulation, with the insulation the likely loser no matter what.