Less so than the size.

I hear that they'll laugh at you on beer nights at the local Sith lodge if you show up with a pink one....

Aside from rumors that certain corruptions in red crystals can cause another lightsaber to snuff at the wrong time, no. I can serenely whip your ass with a red lightsaber and savagely overdo it with a blue one...

The red lightsaber uses a different crystal that is supposedly one that can only be used by someone who pours their anger and hatred into it - so in the case of a red lightsaber it is very symbollic of what person they are.

In the case of blue, green and yellow (let's ignore Windu's purple one for now) according to some source material such as the Jedi Academy game they symbolise the type of Jedi - guardian, consular and sentinel. However the majority of subject matter states that it does not matter and is the personal choice of the Jedi Padawan (or Knight/Master if it is a 2nd,3rd,etc. lightsaber).