I was on top of an "A" frame ladder about 16 feet above the concrete floor when I fell.
Now that the actual force and my choice of direction was arbitrary I sure wish I had fell upwards as I was only a couple of feet from the ceiling.
Damn Newton and his apple
I don't believe my choice has much to do with what is happening
But your loss was another's gain. The potential energy you lost was gained by the Earth when you interacted with it.
You would testify in court that you fell to Earth. Had you weighed more, a lot more, you would say the Earth ran into you. That is a direction choice.
The fact is that the distance between you and the Earth was reduced, probably rather quickly. Now, we have become accustomed to the direction convention that says you climbed up the ladder and then you fell down from the ladder. But if we go back to the fundamentals, we know that the gravitational force between two objects is G*m1*m2 / r^2.
There is no universally defined direction for r. But because the mass of the Earth is about 6x10^24 kg, we usually reference gravitational forces relative to the Earth's position. Gravitational fundamentals will tell you that both directions are valid and the choice of direction is arbitrary, even though your sore behind would argue that only one direction is valid.
It may be of little comfort, but had you fallen when the Moon was directly overhead, it would have lessened your impact ever so slightly.