Well remember, he's an important government official & you're only a lowly peasant who does the work, pays for the permits & pays the taxes for his salary & benefits. How dare you ask him a question? You should be shining his shoes, waxing his car, etc.
I think you should carefully document every conversation, starting now, while some/all of it is still fresh. Record phone calls if practical & legal in your state. If his local boss is uncooperative, go to the state level, as someone else suggested. I don't favor starting wars, but it sounds like he's already started one with you. Check around and see if anyone else has had problems with him. If so, some of you could complain as a group. You paid for permits, you worked the job, or paid employees to do it. He has no right to disallow your presence. If he wanted to pick you about code, do you think he would hesitate? Not for a moment. You have every right to know what he bases a decision on, especially if it's some grey area or peculiar interpretation. While some here are tough, very few are such jerks. We did have 1 guy that loved to pick and sound sympathetic on some trivial issue, cry crocodile tears, then grin & make you redo something he knew was hairsplitting. He's now a contractor again. His guys say he's good to work for. I think authority just went to his head too much back then.
Most inspectors here want someone on site if it's a bigger job, can go either way on smaller ones. I like to be there, as I can answer any questions he has & if he finds a small problem, I can fix it while he's there. Most inspectors here are good with that. They've worked in the field too & haven't forgotten the headaches. On some jobs, that's also when I get final payment, upon passing inspection.