You have multiple issues here, I'm not clear on which one you want solved. Also, you say Wye-Delta, then Solid State, then across the line Delta only, I don't understand what you mean by all those conflicting statements. Just because the motor is capable of Wye-Delta starting doesn't mean you have to use it. I'm going to assume you have a soft starter connected in-line to the motor with the motor connections in Delta, as it should be.
1) Conductor size. Nothing in your description (and this assumption) warrants anything other than standard NEC conductor sizing practices right out of article 430, except the fact that 766HP is not going to be on the charts. So in that case you would size the conductors at 125% of the motor nameplate FLA, rounded up to the next standard size. All the other rules about voltage drop over distance, number of conductors in the raceway, insulation rating etc. still apply, I cannot see your installation to recommend anything specific.
2) Assuming again that the soft starter is connected in-line to a Delta connection on the motor, I would imagine that if "the soft start only lets the motor run at 800amps before it overloads" means that you have programmed or set the soft starter's overload setting at 800A. If it cannot be adjusted to 853A, that means you have a soft starter that is too small.
3) If you meant that the soft starter itself is overheating and tripping on OT, then that's a different issue. If you do not have a bypass contactor and the soft starter is in a vented enclosure, you have to move a LOT of air through a starter that big. If it is 100 deg. F air, you are already close to the 104 deg. maximum operating temperature of that starter. Benshaw soft starters can be used up to 50deg C (122F) but require derating, I think it is something like 1% for every 1 degree C over maximum.
4) "I've seen the Benshaw incloser reach around 115 degrees F." No idea what an "incloser" is. Do you mean the Bypass Contactor? If your soft starter is only capable of 800A and you have a bypass contactor, it is likely only rated for 800A as well. That is a standard contactor size for Benshaw, the next size up is I think 1000A or so and is a lot larger and more expensive. Same for the soft starter itself. So if someone tried to get cheap by buying an 800A rated starter for a motor capable of 853A and you are operating it above it's maximum ambient, that is the root cause of all of your problems.
It sounds to me like you might be in over your head on this. I would recommend that you enlist the services of an electrician with experience in large motor systems.
