“The fuel is free. So, let’s drive down the cost of O&M”
With more than 30 years of proven experience, SunPower is a global leader in solar innovation, and has consistently led the market with the highest performing solar power systems, holding the record for the most efficient solar panel on the market. SunPower provides outstanding service and impressive electricity cost savings globally, for residential, commercial, and utility power plant customers. To date, SunPower solar systems have delivered over 18,000 GWh of electricity, enough to power over 1.6 million US homes for a year.
To manage this large fleet of power plants, SunPower has two remote operations centers – the Remote Operations and Control Center (ROCC) in Austin, TX, and the Command Center in the Philippines. These centers operate 24/7, 365 days a year, and together monitor and control over 2.5 GW of commercial, industrial, and utility scale solar plants distributed over 700 sites around the globe. Their mandate is to optimize the performance and reliability of these plants and to maintain high customer satisfaction.
Monitoring over 2.5GW of SunPower’s solar assets across over 700 power plants, processing billions of data points per day.
Integration of data from disparate systems, including Data Historian, Enterprise Asset Management, and SCADA, Commercial and Utility Power Plant fleets into a unified database and user interface
New tools for operators to observe, troubleshoot, and diagnose performance and reliability issues in real-time
Deploying Krypton resulted in a 50% reduction in nuisance alerts, and an order-of-magnitude improvement in alert Response Time in the first quarter of operation
As SunPower’s business has grown, their remote operations teams have faced a series of challenges. The number of power plants that SunPower monitors and controls have increased rapidly. And, the volume of data generated from these solar assets has increased in kind, as new devices such as microinverters, each with independent sensor reporting, have been rolled out within the fleet. Sensors in every key subsystem (e.g. inverters, trackers, combiner boxes, transformers, weather stations) provide a continuous high volume stream of data, often delivering that data into different, siloed data stores and operator tools. Before Krypton, SunPower’s operators engaged multiple software tools to gain clarity into underlying data and surrounding equipment issues. These tools run on databases with a single-server architecture, and require tremendous hardware expense to scale with the growing volumes of data. This design also introduces a single point of failure.
All of this combined to make it difficult for operators to gain full situational awareness. Operators were flooded with alerts, suffering from alert fatigue as they navigated between multiple systems to address each issue. “Our operators were getting too many alerts, and had to make too many clicks to get anything done,” said Sarah Herman, Senior Manager of Monitoring Operations at SunPower. “It was challenging to standardize our operator’s behavior in this kind of environment.”
SunPower had internally built a number of software tools to help their operators manage this data flow, and several of those tools were straining under the increased load. SunPower’s alarm management system, for example, was producing a high level of false, or nuisance alarms, and did not have the flexibility needed to modify and create new alarms to improve the signal to noise ratio. SunPower had also built a custom system to calculate key performance metrics in real-time, such as PV inverter availability and a performance index. This server was unable to keep up with the volume of calculations required by the increased data flow, which limited SunPower’s ability to implement new performance metrics and alerts that could help the team better manage the fleet.
SunPower not only wanted to address the limitations of their current solutions – they wanted to dramatically improve capability and efficiency. “We want to bring more capability into our remote operations centers, and not only be control room operators, but have the capability to do root cause analysis and performance evaluations,” said Dierauf. “To lead this industry, you have to innovate – machine learning is the direction we want to go.”
Krypton engaged with SunPower in 2015 to help them address many of these challenges. SunPower was intrigued by Krypton’s value proposition to “learn from every O&M data point,” as they recognized they were not extracting maximum value out of the growing amount of data they had. The companies worked together over a series of months to identify the critical issues, and how Krypton’s “Asset Intelligence as a Service” offering could address them.
SunPower deployed Krypton Collect, which uses Krypton’s data agents to bring data together from multiple siloed systems, including their Data Historian (OSIsoft PI) their Enterprise Asset Management system (IBM Maximo), and SCADA systems. Krypton Collect’s modern distributed architecture allowed all of this data to be instantly accessible and searchable, in real-time. And for the first time brought both human and machine-generated data into the same place, enabling correlation and learning previously not possible.
SunPower also deployed Krypton Decision Engine, a real-time stream processing engine to allow SunPower to process literally billions of data points per day, enabling them to process a significantly greater volume and complexity of alert rules and performance calculations to better detect issues real-time.
And finally, SunPower implemented several Krypton Applications, including Unified Monitoring and Alert Engine, replacing SunPower’s custom-built solutions with a modern system that resides in a single tool for SunPower’s operations staff.
In the first quarter of deployment, SunPower’s Command Center reported 50% fewer false alarms and nuisance alerts, which had a significant improvement on operator performance. This allows the operations teams to manage more assets with less resources. Performance issues and alerts can now be processed and presented to operators much more quickly, resulting in an order of magnitude reduction in Response Time, a critical metric in assessing the effectiveness of SunPower’s operations. Krypton is now the primary interface for operators to do their job — allowing them to see all of their data and manage resolutions. “Our data is now all in one place — we don’t have to go look in separate servers for different kinds of information and alerts,” Herman said. Dierauf calls Krypton “our operator’s eyes and ears to the world.”
When an alarm comes in, operators can quickly see relevant time-series data, search for similar prior events to help diagnose and troubleshoot, and then take action – all without having to leave the Krypton interface. The Krypton solution has also increased collaboration between SunPower’s two remote operations centers, as they are both using the same tools and seeing the same data. The allows for resources sharing across both locations.
Krypton and SunPower have continued to work together, post-deployment, to improve the usefulness of the solution – adding new features, such as reporting metrics on operator performance, which allows SunPower to automatically learn how operators are responding to alarms.
“It’s been great how responsive Krypton has been to SunPower. Krypton listens to our operators and realizes that their ideas can be valuable to the final product. They have taken a very user-centric approach to deployment in SunPower”
SunPower and Krypton are working to shift SunPower’s O&M organization from reactive to predictive, by fully leveraging the power of the data they already have. The teams are working together to learn from SunPower’s vast historical dataset of PV system performance, to introduce machine learning algorithms that will help SunPower better predict and diagnose failures, and allow them to operate increasingly larger and more diverse fleets cost effectively.