As ulanini posted, you save by reducing line loss. If you put the capacitor right next to your meter, you really save nothing (the power company will see the savings, but don't expect a thank-you note). You want the capacitor close to the inductor.
Think about a simplified view of resistor, inductor (coils or motors) and capacitor loads. Resistive loads use extra energy because they pull current down a wire. When current goes through the wire, you must overcome the wire resistance, and that extra effort causes heat loss. Heat loss is real energy that you pay for at the meter.
In the same manner, inductors and capacitors also pull (actually "push" and "pull") current down the wire. Inductors and capacitors are stored energy systems.
A motor pulls some current down a wire for productive output. It also has to have some extra power to create magnetic fields so it can function. This extra power produces no real work but is necessary. The extra current needed to feed this stored energy system is pushed up and down the wire because the AC wave cycles from positive to negative. This causes heat loss that you pay for at the meter.
If we could figure out a way to keep this energy stored locally without having to move it up and down the line, we could stop pushing this extra current around and save heat losses.
The capacitor is a stored energy system that is the opposite of the inductor system. When the AC wave cycles from positive to negative, these two systems can swap the stored energy back and forth.
The solution is to put a capacitor close to the inductor so the stored energy current will only flow between the capacitor and inductor. This way, you will not have the extra heat loss from the meter to the motor, but only from the capacitor to the motor.
At start-up, the power needed to charge the stored energy system still has to travel from the meter down the wire to the capacitor/inductor pair but that is a small amount of time compared to how long the motor will be running.
[edit: If you do not have a concentrated inductor load, you must make a compromise and figure out where to put a capacitor to optimize savings. A capacitor by itsself is just as bad as a motor by itsself. FWIW, pf correction is usually only worried about in commercial and industrial settings. If I had a big motor in my house, I might consider putting in a capacitor but it would have to be a really big motor with a really long feed. Right now, I can't think of a residential application that would make it worth it].
[one more edit: In an industrial setting, I would think several hundred horsepower, located several hundred feet from the meter, that had a significant amount of run time, would be worth investigating].
