If a judge's spouse is politically active, does it "test" his "impartiality"?

Surely, the answer can't be, only if the judge is conservative.