I certainly would hope that leo has the right to ask you if you have a carry permit IF he has reason to believe you are carrying. If he does not, it is like a leo not being able to stop or question someone he has reason to suspect they just robbed a bank. If I were to be asked by leo if I am carrying I would tell them yes and I have a carry permit. Would not bother me at all.
Robbing banks is a crime, carrying a firearm with a licence is not. It is like having to prove I didn't steal the money in my wallet. Or it would be the same as leo pulling you over just to see if you have a driver license. It's called being free of illegal search or seizure. We do have a constitution you know.