If, as Iwire notes, there is current flowing on the neutral conductor, then the voltage difference between the neutral and the equipment grounding conductor can simply be caused by normal voltage drop (say if you have long runs or a small neutral), or it could be caused by poor connections in the neutral return path.
Another possibility is that you have a bad equipment grounding/bonding conductor, with local grounding via grounding electrodes but no true connection between your neutral/ground bond at the source, and the grounding bus at the panel. In this case, you _may_ be measuring the difference between local ground at the source and at the panel.
Finally, I think that you have a small error in your assumptions. In a perfectly balanced situation, the vector sum of the three phase currents should be zero. However the_purpose_ of that neutral conductor is to permit the flow of unbalanced currents. If you have any unbalance at all, you will have current flow on that neutral, and if you have current flow on the neutral, then you can have voltage drop on the neutral.
-Jon