You are right and it can be hard to nail down just who or what the AHJ is.
The real authority rest with the local governent itself. The position of building official is created an filled by someone. I had to look up the building official for the City of Atlanta and even after I found it I can't pronounce it ( Ibrahim Maslamani ). Never met the man and hope I don't have to.
The reason that I say that an inspector is not the AHJ is this. I have a small problem at present. A certain inspector decided to overlook a small fault on an inspection two years ago, even though it was pointed out to him ( didn't want to make the homeowner mad ). Now I have a new inspection at the same property that failed, not because of anything done on this permit but because of something that was OK'd two years ago. The new inspector is a bit more knowledgeable and understands that the first inspector didn't have the authority to make said decision.
Monday I will call Lee Mclead, CBO for Cobb county and see if he will Ok a variance for this little problem, if not I'll turn the carpenters loose on it. It's a clearance issue and nothing I can do anyway.
It's really my fault, I should have insisted that the problem be corrected two years ago. I thought the inspector would insist once I pointed the fault out to him but he didn't and now it's come back to bite me.