Some ATS's use relays (called contactors in bigger sizes), but these must still incorporate a mechanism that physically prevents simultaneous energization, not merely by the electrical circuit arrangement.
A service rated ATS will still require a main breaker. It becomes the service disconnect when placed ahead of the panel.
While small manual transfer switches use double-throw switches and a separate set of breakers to switch between utility and generator feed, an ATS will always use a contactor to perform this function. (At least, I have not seen any that don't.)
Thanks everyone for your input. Issue that i am having is i am trying to coordinate the breaker in the ATS with utility fuse and its really hard to do so. Thats why i was thinking why cant we use relays just like we do in medium voltage switch gear and make the coordination lot better?