Entergy Noncompliant with PRC-005-6 R3 Resulting in Penalty of $141,000

Summary:

Entergy notified SERC that it was noncompliant with PRC-005-6 R3. Entergy failed to maintain its Protection System Components that are included within the time-based maintenance program in accordance with the minimum maintenance activities and maximum maintenance intervals prescribed within Table 1-1, Table 3, and Table 4-1 for 18 relays at a single substation (Instance 1). Entergy performed an extent of condition (EOC) for Instance 1 and discovered an additional instance (Instance 2) involving eight additional relays. Entergy identified two additional instances that involved the battery at one substation (Instance 3) and an additional 10 relays (Instance 4).

Additional Discussion:

On March 14, 2022, during scheduled relay maintenance at the Hot Springs Extreme High Voltage (EHV) substation, relay personnel inquired about the status of maintenance on adjacent panels 1R, 2R, 2A, and 2B. In researching the status of work orders associated with the adjacent panels, the Planner/Scheduler discovered that the four panels were suspended (auto generated maintenance on these relays was turned off).

For Instance 1, Entergy determined that it did not perform maintenance and testing activities on four relay panels at Hot Springs EHV including (AT1) 2A, and 2R and (AT2) 1R, and 2B. The relay panels were inadvertently suspended in the Substation Work Management System (SWMS) during a Single Point of Failure (SPOF) project to improve reliability of the Bulk Electric System. During the project, Entergy installed a new high voltage (HV) line relay panel, and it removed HV line relay components from an existing autotransformer panel. The existing autotransformer panel remained in place with the HV line protection removed. However, bus differential and breaker control relaying components in the autotransformer panel remained in service. Two of the four panels currently have electromechanical differential relays (three in each panel), as well as breaker fail fault detection and timer relays, one of each per panel, for a total of 10 relays in the two panels. The other two panels contain breaker failure relays (one SEL relay per each) along with the differential relays (three in each panel), for a total of 8 relays in the two panels. There was a total of 18 relays involved in Instance 1.

Entergy last performed the scheduled maintenance and testing for these relays on October 15, 2013. The next scheduled maintenance and testing should have occurred on or before December 31, 2019. Instance 1 ended on April 9, 2022, when Entergy completed required maintenance and testing for the panels in question.

On February 12, 2023, Entergy completed its EOC review. The review consisted of a total of 1,119 suspended PRC relay panels from 2016 to 2021 across the system resulting from a variety of relay upgrades, modifications, and failure replacements. The majority of these relay panel replacements were one for one. However, the stations involving SPOF projects during this time frame had a higher probability of relay panel modifications to address the SPOF condition. Of these 1,119 suspensions, Entergy identified 208 in stations where it completed an SPOF project. As a result, it reviewed all 208 suspended panels and found two deviations at Mabelvale substation (Panel 10R and 11R), explained in more detail in the discussion of Instance 2 below.

In Instance 2, on February 12, 2023, while conducting an EOC evaluation for the 208 SPOF projects, Entergy found components of the protective relay panels 10R and 11R at Mabelvale substation deactivated/suspended in the SWMS application; however, they were physically still in service. During the EOC evaluation of the 208 identified panels, Entergy found components of Mabelvale EHV panel 10R pertaining to breaker failure protection (five relays) on Breaker B6400 to be still in service with the panel suspended. Additionally, it found components on panel 11R pertaining to Mabelvale station service and reactor protection (three relays) still in service with the panel suspended. There was a total of eight relays involved for Instance 2. According to PRC-005-6, Table 1-1 protective relay maintenance activities have a maximum maintenance interval of six calendar years. The last relay maintenance and testing for relays associated with panel 10R occurred on April 24, 2013. The next maintenance and testing cycle should have occurred on or before December 31, 2019. For panel 11R, Entergy performed the last relay maintenance and testing on April 24, 2012. The next maintenance and testing cycle should have occurred on or before December 31, 2018. On April 13, 2023, Entergy’s relay maintenance group performed the required maintenance and testing for all relays associated with panels 10R and 11R, which brought Entergy back into compliance for Instance 2.

For Instance 3, on September 12, 2022, while reviewing a listing of potential overdue work, Entergy discovered one instance of potential noncompliance related to the Pintail Substation. Entergy discovered that it failed to complete an 18-calendar month impedance test on the Pintail 138 kV substation battery in accordance with Table 1-4(a). The enterprise asset management system (Maximo) application reflects that Entergy last completed the task on February 18, 2021. The testing should have occurred on or before August 31, 2022. Entergy has a built-in compliance margin of six months in Maximo. The internal due date was set for February 28, 2022, based on the last time Entergy completed the task, which was February 18, 2021. Entergy then scheduled the work order to conduct the impedance test on March 22, 2022, to be completed on April 4, 2022; however, Entergy rescheduled it several times. It eventually completed the work by September 13, 2022, which exceeded the NERC maximum maintenance interval of 18 months. Entergy performed an EOC review. The review consisted of evaluating all battery maintenance for the Transmission business unit required under PRC-005-6 (i.e., impedance testing, required battery inspection, and battery capacity testing). The scope also included evaluating Power Generation and Nuclear PRC-005 battery components. The evaluation included the time frame of the SWMS to Maximo application transition in December 2021 through current planning period to the end of December 2022. Entergy did not discover any additional instances of noncompliance.

In Instance 4, on August 1, 2023, while enabling newly installed components in the Maximo application, the area planner noticed transformer protection panels in the Arkansas area that were unfamiliar and warranted further investigation. Upon review, the planner determined that the required maintenance at two panels was past due. A SERC applicability review of a non-PRC-005 designated asset (which was determined to be correctly designated as non-PRC-005) prompted the planner to review other similar transformer protection panels in the state, leading to the discovery of a distribution transformer protection scheme at Independence 500/161/13.8kV substation that was of similar configuration to the original asset investigated. The review of Independence 161/13.8 XFMR #2/Breaker 2115 transformer protection panel AC schematic (consisting of two relays) showed protection wrapping both 161kV breaker feeds as well as transformer high side indicating that it could potentially be a PRC-005 designated asset. Entergy placed this panel (T2) in service on December 19, 2016, and enabled it in the previous generation asset management system (SWMS) on February 10, 2017, with a “N/A” PRC-005 designation. Due to the “N/A” PRC-005 designation, PRC-005 appropriate maintenance tasks/intervals were not generated for this asset, and Entergy did not perform PRC-005 designated maintenance at time of discovery. It performed the last relay maintenance and testing during installation on June 29, 2016. The next maintenance and testing cycle should have occurred on or before December 31, 2022. Further review of similarly configured PRC-005 “N/A” designated protection panels uncovered one additional transformer panel 5R T1 (consisting of eight relays) with incorrect designation at Little Rock Gaines 115kV substation. Entergy performed the last relay maintenance and testing on panel 5R T1 during installation on March 26, 2015. The next maintenance and testing cycle should have occurred on or before December 31, 2021. On December 11, 2023, Entergy’s relay maintenance group performed the required maintenance and testing for all relays associated with panel T2 at Independence substation. On December 14, 2023, Entergy maintained and tested all relays associated with panel 5R T1 at Little Rock Gaines, which brought Entergy back into compliance for Instance 4.

The initial EOC review consisted of 386 transformer protection panels with non-PRC-005 designation in the Arkansas jurisdiction. The results of the review yielded one transformer panel (5R T1) at Little Rock Gaines 115 kV substation that had the wrong PRC-005 designation and past due maintenance. Entergy reviewed all remaining transformer protection panels with non-PRC-005 designation system-wide and did not identify additional instances.

This violation started on January 1, 2019, (Instance 2) when Entergy failed to maintain its Protection System Components that are included within the time-based maintenance program in accordance with the minimum maintenance activities and maximum maintenance intervals; and ended on December 14, 2023, (Instance 4) when Entergy completed its maintenance and testing in accordance with the minimum maintenance activities and maximum maintenance intervals. The duration was 1,808 days.

Cause

The causes for this violation were ineffective communication, ineffective internal controls, deficient process/procedure, and ineffective training program. Specifically, there was inadequate communication of the energization notice (EN) submittal for the initial suspension of panels. The EN program at the time of events did not include the same verification parameters (peer checks, manager verifications, SWMS Asset Data Form (ADF)) as the current process (Instances 1 and 2). Second, there were ineffective internal controls to properly verify the component status in the field (Instance 1 and 2), and there was an overreliance on rescheduling of non-outage tasks without proper controls to ensure no resulting compliance implications. For example, Preventative Maintenance tasks had been pushed out due to competing work considered more critical, but without adjustments to the Preventative Maintenance schedule to address potential compliance risks (Instance 3). Third, the Substation Asset Configuration Manual procedure included vague or interpretive guidance for unique distribution transformer protection configurations and did not provide enough specificity to the Planners/Scheduler, which would allow them to properly identify the relays as PRC-005-6 assets requiring maintenance and testing as described within the PRC-005-6 Tables (Instance 4).

Disposition

SERC considered Entergy’s PRC-005 R3 compliance history in determining the penalty of $141,000, SERC considered Entergy’s PRC-005 R3 compliance history to be an aggravating factor in the penalty determination. Entergy had five prior relevant noncompliances and SERC considered four as aggravating for penalty purposes. The other noncompliance involved different causes, and mitigation would not have prevented the current instances.

SERC applied a mitigating credit for the following reasons:

• Entergy staff were professional and sought to work cooperatively with SERC throughout the investigation of the violation.
• Entergy self-reported the violation in a timely manner from the date of discovery.
• Entergy accepted responsibility and admitted to the violation.
• Entergy agreed to settle the violation.