If it's just a riding club, you shouldn't be hassled as long as their patch isn't a 3 piece and doesn't look like a MC patch.
That said, I never heard of a RC with "colors" or a requirement to wear them. I have a big back patch on my vest of a unicorn but no rocker patches or name, not part of any group and I've never been hassled by anyone over it, cop or MC members. Well, I was hassled once by some random lady in a Harbor Freight check out that thought I was lying to her when I said I wasn't in a club. "Oh, you can't talk about it huh?". No lady, this is blatantly not a club jacket, stop watching so much TV.