I kept one for about 15 years (full, no neck) and shaved it off about 6 years ago. I look 10 years older than I am (at least) with it and 10 years younger (or so I've been told) than I am without it.
I feel ya. I had a raggedy little Joe Dirt goatee for several years. Screwed up trimming it about 6 months ago and had to kill it. The first day 3 people told me how much younger I looked. Been clean shaven since.