You have a nick in the wire that's in the conduit! Probably the undgrounded wire!
Or
You have a nick or pinch in the can light branch! I"m thinking the latter, due to you OP description.
Rememeber that you should not read any voltage on a single circuit of neutral to ground! Thus your 50 volts!
Maybe I'm totally wrong to say "remember" that but I've seen a pinched wire on a circuit were I could read 50 volts on neutral to ground, shorted the wire (not by me) that then blown out the pinch point on a ungrounded circuit.
PS
The stray voltage on the neutral also smoked a majastic motion sensor that was put in without checking for this shorted or grounded circuit voltage, by me! I don't think it's phantom in this/(your) case!