NYC power outage

[mbrooke]

West 42nd:

 

[mbrooke]

West 49th and West 50th st substations.

Yellow arrows show 420MVA 345kv to 138kv auto transformers also based on a second level factor. Red arrows show 65MVA 138kv to 13.8kv units.

 

[mbrooke]

Stations supplied by West 49th st 345kv-138kv autotransformers:

 

[mbrooke]

......

 

[mbrooke]

Confirming the 345kv-138kv auto transformer ratings:

 

[mbrooke]

Confirmation of the 138kv-13.8kv transformer ratings:

 

[mbrooke]

Now, with all that said-


If any 138-13.8kv transformer needs to be de-energized for an uncleared secondary fault only one 138kv cable is tripped. In the case of Astor it will result in a circuit switcher trip, as for the other 3 stations I am not sure but I'd say a cable trip due to their age (hence no circuit switcher).



345kv Breaker Diagram of West 49th st:

 

[mbrooke]

From the diagram above, a 138kv cable clearing will simply result in one auto transformer and 6 out of 30 138kv-13.8kv transformers being cleared. The remaining 4 autos, 138kv cables and 24 area substation transformers would remain energized and due to their 13.8kv secondaries being in parallel absolutely no network load would be lost.

 

[mbrooke]

The assumption above is made that the 345kv auto transformers feed the 138kv cables directly with at most a disconnect between them.

Even if the 5 auto transformers feed an elaborate 138kv ring bus at 49th st- the fault clearing topology would still remain the same and thus the outcome. Only the breakers around (for) the 138kv cable requiring de-energizing would open.

I can only conclude that:

1) In addition to the relaying failures at West 65th relay misoperations also took place at 49th st.

-OR-

2) West 65 lacks directional over current or any over current for that matter on its 13.8kv transformer breakers. The uncleared 13.8kv feeder fault presented itself as an overload condition. Eventually some device caught the 5 paralleled transformers over heating and issued a trip command for all 5 clearing all the 138kv cables and the area substations they supply. But I can't see not having any 50/51 protection with secondary trip on the 65MVA units.

I'm left with more questions then answers. :lol::blink:

 

[mbrooke]

Lastly- This I team investigation makes a very good case that old outdated electromechanical relays may have been the culprit:


https://www.nbcnewyork.com/news/loc...y-Systems-Blamed-for-Blackout--512922931.html


-However-


Here they talk about "Software" leading me to think they are talking about new microprocessor devices having been involved in the incident:


https://www.nbcnewyork.com/on-air/a...m-Caused-NYC-Blackout_New-York-512760902.html


There is a need for more information. Con Ed is far from forth coming and the media is left spinning it wheels despite the news coverage of this incident being good and mostly accurate.


I know public documents have been far more transparent in the past up to including the make and settings of the equipment ie, for example the Astoria Charles Poletti generation project relaying. (This was not the relaying involved in the December blue light incident, but simply that which went into a new generation project)


https://files.engineering.com/getfi...sc=212727627.4.1564948456035&__hsfp=165344926


https://files.engineering.com/getfi...sc=212727627.4.1564948456035&__hsfp=165344926


I'm not sure why the blackout on information and vague responses from Con Ed.


As of now I'm sticking to me original theory that there is a manufacturer that sold some some defective relaying to Con Edison possibly involving IEC-61850 communications.

 

[paulengr]

It is really easy to mess up 61850 programming.

Tertiary distance relays are basically set to monitor a large area such as an entire generating plant as a kind of fail safe in case everything else fails to trip. It looks at impedance, basically Volts divided by Amps. If this gets too low which would be a short somewhere the distance relay trips. Because they usually cover a large area the result is usually catastrophic. So it sounds either like the feeder protection on the cable missed it or the breaker(s) failed to open, or else the distance relay was set too tight causing miscoordination.


Sent from my iPhone using Tapatalk

 

[romex jockey]

It is really easy to mess up 61850 programming.

Tertiary distance relays are basically set to monitor a large area such as an entire generating plant as a kind of fail safe in case everything else fails to trip. It looks at impedance, basically Volts divided by Amps. If this gets too low which would be a short somewhere the distance relay trips. Because they usually cover a large area the result is usually catastrophic. So it sounds either like the feeder protection on the cable missed it or the breaker(s) failed to open, or else the distance relay was set too tight causing miscoordination.


Sent from my iPhone using Tapatalk

I thought scada was involved Paul? :huh: ~RJ~

 

[mbrooke]

It is really easy to mess up 61850 programming.

Tertiary distance relays are basically set to monitor a large area such as an entire generating plant as a kind of fail safe in case everything else fails to trip. It looks at impedance, basically Volts divided by Amps. If this gets too low which would be a short somewhere the distance relay trips. Because they usually cover a large area the result is usually catastrophic. So it sounds either like the feeder protection on the cable missed it or the breaker(s) failed to open, or else the distance relay was set too tight causing miscoordination.


Sent from my iPhone using Tapatalk

Ditto- even for the manufacturer.

From reports is sounds like the primary and secondary 13.8kv feeder protection relaying failed from the "censors" not communicating.

Since we issued our preliminary findings into the cause of the July 13th power outage on the West Side of Manhattan, we have conducted extensive testing of equipment and a thorough review into why our relay protection system did not perform as designed. Our engineers have determined the root cause and taken steps to prevent a recurrence.


As part of our investigation, we reviewed 15 years of operating data and took equipment out of service at our 65th Street substation to conduct diagnostic testing, which allowed us to simulate the event. We identified a flawed connection between some of the sensors and protective relays at the substation. We have corrected that condition.


Out of an abundance of caution, we have taken preventive measures by isolating similar relay equipment at other substations. We will analyze and test the equipment before we put it back in service. Our electrical delivery system continues to operate with multiple layers of relay protection.

The New York City grid is one of the most complex and technologically advanced in the world, with multiple layers of redundancy. In electrical systems, a relay detects abnormal conditions on the electrical delivery system and instantly sends signals to circuit breakers to open and isolate the problem. We are confident that we have identified the cause of the July 13th outage, and taken the corrective actions necessary to provide safe, reliable service for our customers.

https://www.coned.com/en/about-us/media-center/news/20190729/con-edison-statement


Step distance could have been reaching through those transformers and well beyond.

But IMHO I still think the 13.8kv breaker immediately after the transformer should have opened and not the transmission system. Even if it meant that all 5 secondary trafo breakers would open.

 

[mbrooke]

And of note they say they could re-simulate the event. I want to think that means its a programming (software) issue... with all else done right on the outside.

 

[mbrooke]

I thought scada was involved Paul? :huh: ~RJ~

You mean like this?

 

[romex jockey]

I don't understand why, given poco configured ring circuits, how one 'leg' can black out, and not be fed from the other here?
:?
~RJ~

 

[mbrooke]

I don't understand why, given poco configured ring circuits, how one 'leg' can black out, and not be fed from the other here?
:?
~RJ~

Well, your train of thought is on the right track- thats the same question I'm asking >>

All the breakers on a ring bus are normally closed. When a fault happens on a cable, line, or transformer, only the two breakers directly feeding the failed element trip. Power continues to flow around the now open ring.


Often times to prevent a second fault from breaking the the ring in two or more separate halves (forming undesirable circuit configurations), the outgoing terminal disconnect is equipped with a motor operator which opens and the two tripped breakers re-close closing the ring back up but without the failed equipment (due to the disconnect now being open)

Motor disconnect opening:

https://www.youtube.com/watch?v=7jzTeKVLLCE


The GIS version as would be seen at West 49th st:


https://www.youtube.com/watch?v=q5aCilA_47c


And no that arc is not load, but busbar capacitance which gets very extreme at those voltages.


Even if one breaker fails to open (sticks) for a fault, the scheme is still equipped with breaker failure protection.

That means when the two breakers are called to trip, relaying monitors the current and the breakers' auxiliary contacts. If current is still detected passing through either CTs after 6 cycles of tripping being initiated- it means the breaker has jammed. Relaying then initiates the breaker failure sequence where the next breaker over is then tripped. A DDT signal is also sent to trip any remote breakers supplying the line over as its still feeding though the stuck breaker. This ensure the failed element is truly de-energized and the system is not perpetually feeding a fault.


All this takes place under 30 cycles from the start of the first trip initiation.


A stuck breaker results in two elements out of service- however- this is no problem as only 1 transformer and 1 line is removed at West 49th. This is still within the second contingency design. In fact in theory you could loose two transformers and two 345kv cables in secession and not have any over loaded equipment.

Reason for the stuck breaker scheme is that if the jammed breaker is not dealt with the fault will persist possibly clearing the entire station remotely through Zone 2 / Zone 3 MHO or over current. A cleared station is not only lost load, but also severing a power flow path.

 

[mbrooke]

From public docs here is the cross Hudson interconnection section of the 49th st ring bus.


The switch to the right of the pothead is the line disconnect discussed above. The squares (8,7) are the breakers and the switches next to them are their respected disconnects.


50F stands for over current failed breaker- meaning if current is still sensed 6 cycles after a breaker has been called to trip BF trip is initiated by that relay.

87 is differential protection

86 is a lockout relay.

F.O. is fiber optic cable that runs to the PSEG substations to exchange data for 87 differential protection among other things.

R= normal primary relaying

BU= backup relaying


CCPD= Capacitive coupled potential divider. Con Eds name for a capacitive coupled voltage transformer. Takes the line to ground voltage of 345kv and steps it down to 110 volts for the relaying.

 

[mbrooke]

REL 350 relay if curious for short circuit and ground fault protection of the Hudson cable:


https://library.e.abb.com/public/08e5986899d019e685256f320072479e/DB41-461m REL350.pdf

 

[mbrooke]

Differential protection as common for Con Edison.


(I'd imagine it was also one of these devices that failed during the Astoria incident in December.)