One way to look at this would be if you see a lot of Sparks, then the Fault is not "Bolted".
Not that a given L-L-L Bolted Fault doesn't produce Sparks, it has a lot less Sparks per Arcs / Flames / Smoke / Sound.
Sparks are normally produced from blasting the Metallic items (box + conductor) into smoldering little pieces of flying debris.
So, when this occurs, there is typically a corresponding drammatic reduction in fault current.
In the case of your Lighting Circuit fault, the level of fault current is very easilly Impeded by the poor conduction of the Fault Current's Path, due to the conditions mentioned (water filled contactor enclosure) + possibly loose fittings and locknuts.
The Sparks 'a flying + Impeded Fault Current Path = Low Fault Current Levels - like peaking at maybe 50 - 100 Amps for 1/2 Second, averaging 40 Amps over a 10 Second time frame.
The Branch Circuit OCPD will see this as a normal thing, and will only trip when the Current Levels for it's Inverse-Time Design points are exceeded (more on this later ..)
So to make it short and simple, the Circuit Breaker would not trip until the Fault became more solid, or something allowed the Fault to draw an extremely higher and steadier Current level - when at such time _SHOULD_ open the OCPD (or something upstream if the last one is unable to respond).
The Main Service Disconnect (Main Breaker) did not trip from this type of Scenario - rather it "Finally" tripped from reading a "Leakage To Ground" or simply, thought it saw an Arc Fault between an Ungrounded Conductor and some metallic enclosure.
That's what the GFPE Main is for.
I bet the Service Capacity is 1000 Amps or higher, and as you mentioned, the System is 480Y/277V 3 Phase 4 Wire.
The fact that 6 to 7 Seconds went by before the GFPE tripped, sounds like someone has tweeked the settings to their highest time + current settings, or lowest sensitivity points (trimpots spun by jokers is not a good idea!).
This is sometimes done by Electrical Personnel, in an attempt to keep from tripping the GFPE Main while they are working on stuff - in case they cause an accidental Ground fault (so the Building's Power is not suddenly lost, and every Tenant is mad at them!).
Anyhow, not that this is 100% exactly what happened, but it is the most common result for a Scenario such as yours.
Best way to eliminate the possibility of bringing down the entire Building (with the corresponding sound of the Back-Up Genny's Prime Mover starting up... + complaining Tenants looking for you.... heee heee heee), is not to work on things Live!
If Live work must be done - and the risks are assessed + addressed properly (PPE, tell everyone that "Poo-Poo Occurs" regardless of how much care is taken, procedures created, etc.), do it the right way and only because there is no option to de-energize.
"You Just Cannot Turn That Off Because I Need To Use My Calculator" is not a qualifying reason to do live work.
"If The Power Is Lossed, The Plant Will Explode - Because It Is A Continuous Production Type Manufacturing Process" is a much better excuse, which may warrant live work.
(but if coupled with the old "My Calculator Must Be On 24/7" excuse - and you will only be working on the Branch Circuit for the Calculator - tell them to "Take A One-Way Trip To He**" , and maybe help them find the Depot / Terminal, for that particular Trip).
Scott35
