So to answer the op, I would say yes that is acceptable but not for the reasons you state. It seems the service disconnect and ocpd still need to be adjacent, but you can choose to either Mount them at the distribution point or at each building. If you have a disconnect only at the distribution point, you can consider it the site isolation device, feeding service conductors, and the service disconnect is at each building (with the OCPD).
All the rural POCO's here provide that disconnect at the pole, often is a meter and disconnect combination, some have OCPD's some don't, as mentioned some even have a manual transfer switch incorporated into them - usually with no OCPD's though.
The State AHJ just sees it as a convenience switch provided by the POCO, and also considers that the POCO could possibly change that switch to another type with/without OCPD or with/without a manual transfer switch so they don't consider it to be the service disconnecting means regardless of what options it may have. Been times I have set up a temporary service off one of those maybe even mounted on same pole, I still needed to provide my own service disconnect even though there was a breaker on meter/main the POCO installed. I am fine with it as long as it is consistently enforced that way.
These rural POCO's do that for up to 200 amps pretty much any voltage, some do up to 400 amps 240 volts single phase and some even 240 400 amp three phase. higher capacities are usually done on case by case basis with certain amount of standard approaches, but often the "service point" is the transformer secondary or maybe in a CT cabinet.