When I worked for Siemens and went through what was essentially the original ITE training program (meaning it was very old by then), the 42 pole limit (not CIRCUITS by the way, POLES) was based upon the following:
1. It only applied to "lighting and appliance panelboards", defined as having 10% or more pole spaces that were 30A or less, so basically residential and commercial panels. This rule didn't apply to "industrial panelboards".
2. Add that with the fact that the boxes were designed to fit in between standard wall studs.
3. Now add that the topmost breaker handle can't be higher than 6'7" off the floor, using more than 42 poles in a panel resulted in the bottom most breaker being down closer to "toddler height" off of the floor. There is no actual lower height restriction in the NEC, but the panel mfrs were concerned about Resi panels allowing little ones to get too close, so they wanted to stick to a self-imposed limit of 24" up off of the floor (which interestingly made it into the Mobile Home paragraph).
So within those conditions and limitations, the shear number of possible conductors was not going to fit from a box fill perspective, because 42 poles could POTENTIALLY be 42 single pole breakers that would need neutrals, resulting in 84 branch conductors crammed into that annular space around the bus and breaker frame.
So why has it changed? No idea.