I can't agree with you more. I have mine processed and they hang it for at least 7 days. My 2.5yr old buck and my 7.5yr old buck tasted the same. Killed them both last year 2 weeks apart in mid Novemberl. Both showed signs of rutting activity.Some people comment they don't like to eat bucks, or maybe just not during the rut, because they taste to 'gamey' or even 'musky' etc...
So the question is, do you notice any difference in taste in deer and do you not shoot them because of that.
I've never noticed any difference and my personal opinion is the timeliness and method of processing has infinitely more to do with the taste than the sex or time of year. But interested in yours.