The following is purely hypothetical...I'm an engineer...so I have fun doing this stuff. Until we have more test data, we don't have much to go on...but I'll just throw out an example of what "could" be the problem.
Let's say we have a 100 ohm resistance somewhere in the neutral from the outlet back to the panel. Without a load (i.e., an open circuit at the outlet), the voltage across the outlet (hot to ground) will read 120V.
For sake of argument, let's say the refrigerator runs off 720W (ignoring PF for the sake of simplicity) and pulls 6A, resulting in an equivalent resistance of 20 ohms. That is, Power is 120V X 6A = 720W. The main thing here is the 20 ohms.
When the load is connected, the open circuit that once existed at the outlet (showing 120V) now has the 20 ohm at the outlet. And if the resistance to/from the outlet back to the source is nigh zero, then we have 120V @ 6A, and the fridge happily churning at 720W within spec.
But if we have 100 ohm of resistance in the neutral...in series with the outlet, then the 120V is now impressed across 120 ohms (100 from the neutral resistance and 20 from the Fridge). 120V and 120 ohms means 1A will flow. The 1A across the fridge 20 ohms will result in a 20V reading across the outlet...and the 1A across the neutral resistance will show the remaining 100V.
As the EGC is connected to the neutral at the SE, If there is a 100V drop on the neutral on the way back to the SE panel, then 100V will show between the neutral and the ground at the outlet when the 20 ohm load (Fridge) is connected.
The same problem can occur if the 100 ohm resistance is on the hot side, but checking for that occurrence is slightly more complicated.
Basically, if you see 120V in open condition (no load) and 20V in closed condition, then the other 100V is somewhere in series. Be it along the neutral as described above...or along the hot. It could manifest in the wire itself...or more likely...in a bad connection (including bad connection of hot or neutral in the panel or a receptacle in the circuit)....not bad enough to be open...but bad enough to induce approximately 100 ohm of resistance or there about.