That's the first thing I heard when I got my bike. I even had one guy say," Get you a hack saw and just cut the mufflers in half." Not the baffles mind you, the whole muffler. Yeah, ok whatever, there's some advice that will make you cringe.
Bottom line, like everyone has said, if YOU like them, and since it's on YOUR bike, then keep it like it is.
There's a lot of guys, some old line Harley guys, some noobies, who thing it's desireable for a Harley to sound as harsh as 50 caliber machine gun and have about the same db. Not me.
My V-Rod is very quiet, by Harley standards, my LowRider is not, but it's not harsh either, and I can tell you, when you feel like running it hard, the quieter bike is a lot easier to run hard, a lot less conspicuous.