I was following you up to here. You seem to be stating something you're not showing. Translation: Huh?
Remember that the capital letters A, B, C, and N all indicate vectors, rather than simple numbers (scalars). This is a 2-D problem, so that means they each have two components. One in the x-direction, and one in the y-direction.
The following vectors that define direction of the larger vectors, are called unit vectors. Because their length is one unit. I used these, as starting points to define the directions of A, B, and C.
a = <1, 0>
b = <-1/2, -sqrt(3)/2>
c = <-1/2, +sqrt(3)/2>
Distribute the magnitudes (Ia, Ib, and Ic) throughout the terms of the unit vector, for vectors A, B, and C:
A = <Ia, 0>
B = <-1/2*Ib, -sqrt(3)/2 * Ib>
C = <-1/2*Ic, -sqrt(3)/2 * Ic>
The first component of each vector is the part of the vector projected on the x-axis. That is, the terms between "<" and ",".
Ax = Ia
Bx = -1/2*Ib
Cx = -1/2*Ic
Likewise, the second component of each vector, is the part projected on the y-axis.
Ay = Ia
By = -sqrt(3)/2*Ib
Cy = +sqrt(3)/2*Ic
Add up the x-terms to get zero, leaving Nx as an unknown. And repeat with the y-terms, with Ny as an unknown.
Ax + Bx + Cx + Nx = 0
Ay + By + Cy + Ny = 0
Solve for Nx and Ny, which are the components of the neutral current vector:
Nx = -(Ax + Bx + Cx)
Ny = -(Ay + By + Cy)
When we get our results, this is the neutral current vector:
N = <Nx, Ny>
To get the magnitude of a vector, you combine the components in the Pythagorean theorem. Because placing the individual component vectors head-to-tail as we do when we add vectors, forms the two legs of a right triangle. The resultant of adding them, follows the diagonal.