I see it slightly differently...
RSD requirements only apply to conductors installed on or in buildings. That's in the first sentence of 690.12. From there, controlled conductors are subdivided into two categories: within and without the array boundary. For a groundmount, you have no conductors that are both within the array boundary and on or in a buidling, so we can dispense with worrying about those requirements. However, you've described bringing a PV DC circuit on or in a building to an inverter. So now that portion of the circuit that is on or in the building is controlled. But in my opinion, that only applies to that portion that is on or in the building. By putting a DC disconnect at the point where they come onto the building, or at a readily accessible point before that, you provide an initiation device that allows the contrlled conductors on/in the building to meet the requirements when the disconnect is opened.
One loophole that I don't think was intended is that they don't specify how far away from the building your rapid shutdown switch can be. If it's 1000ft away I think that violates the intent of the requirement, although not the letter. They should have required the initiation device to be at a "readily accessible location outside *and within sight of* the building". i.e. no more than 50ft away. If it were me, I wouldn't push your interpretation too hard with an AHJ if the groundmount were farther away than that.