Not so fast..a bad connection or a bad contact in the switch can cause this problem too.
I would replace/bypass the switch first as a test, if that doesn't work I'd check all the connections, HOTS and neutrals, and finally I'd check/replace the breaker if all else fails.
Hopefully the OP will return with the end findings....