That breaker is an MCP, "Motor Circuit Protector" or "Mag-Only" breaker, also known as an Instantaneous Trip (IT) breaker, which is how it is referenced in the NEC. The rules for sizing it are both in table 430.52, and also in paragraph 430.52.C.3 (on the next page in most editions) where is references exceptions. Essentially you can set the IT breaker to 800% of the motor FLA, but if you show that this doesn't work, you can increase that to 1100% for a "standard" motor, or even 1700% if it is an energy efficient motor.
You have a 150A continuous rated breaker, that is not the correct size MCP for a 100HP motor, you need a 250A frame. The 150A frame is good only up to 60HP, as it was apparently originally used. Fix that issue first.
Your FLA is 127A, so you can set the IT breaker trip point at 1000A without question, 1397A if you demonstrate that doesn't work, and 2159A if it is a newer energy efficient motor.
If you have an older motor, NOT energy efficient, and the breaker is tripping at 1750A, then you have a problem in the motor that must be addressed; you cannot have it set that high legally. If it is a newer EE motor, then you have a little more wiggle room on the allowable settings. That does not solve the breaker size issue, but until you can get the proper sized breaker it can allow you to run temporarily.
Another issue is that of it being an MCP in the first place. You CANNOT use an MCP in a "home brew" starter arrangement, it is only allowed to be used in "listed combination motor controller" (430.52.C.3), which effectively means built by the starter manufacturer. If you have built your own controller, you must use a Thermal-Magnetic (Inverse time) circuit breaker, which changes the sizing rules.
PS: I'm pretty much redundant to Cold Fusion's post, although I guess I need to check my math...
But the last part is important.