CONTACT US
1-844-4ZPE-SYS
ZPE CLOUD

Providing Out-of-Band Connectivity to Mission-Critical IT Resources

Home » Improve Network Security » User Management

Edge Computing Use Cases in Banking

by | Aug 13, 2024 | Actionable Data, Application Hosting, Consolidation, Edge Computing, EdgeOps, Improve Network Security, Increase Productivity, Industry Use Cases, Monitoring & Reporting, Network Automation, Network Edge Orchestration, Out of Band Management, Remote Network Management, Simplify Branch Infrastructure, Streamline Deployments, User Management, Vendor Neutral Platform, Virtualization, Zero Touch Provisioning (ZTP), Zero Trust Security

financial services

The banking and financial services industry deals with enormous, highly sensitive datasets collected from remote sites like branches, ATMs, and mobile applications. Efficiently leveraging this data while avoiding regulatory, security, and reliability issues is extremely challenging when the hardware and software resources used to analyze that data reside in the cloud or a centralized data center.

Edge computing decentralizes computing resources and distributes them at the network’s “edges,” where most banking operations take place. Running applications and leveraging data at the edge enables real-time analysis and insights, mitigates many security and compliance concerns, and ensures that systems remain operational even if Internet access is disrupted. This blog describes four edge computing use cases in banking, lists the benefits of edge computing for the financial services industry, and provides advice for ensuring the resilience, scalability, and efficiency of edge computing deployments.

4 Edge computing use cases in banking

1. AI-powered video surveillance

PCI DSS requires banks to monitor key locations with video surveillance, review and correlate surveillance data on a regular basis, and retain videos for at least 90 days. Constantly monitoring video surveillance feeds from bank branches and ATMs with maximum vigilance is nearly impossible for humans, but machines excel at it. Financial institutions are beginning to adopt artificial intelligence solutions that can analyze video feeds and detect suspicious activity with far greater vigilance and accuracy than human security personnel.

When these AI-powered surveillance solutions are deployed at the edge, they can analyze video feeds in real time, potentially catching a crime as it occurs. Edge computing also keeps surveillance data on-site, reducing bandwidth costs and network latency while mitigating the security and compliance risks involved with storing videos in the cloud.

2. Branch customer insights

Banks collect a lot of customer data from branches, web and mobile apps, and self-service ATMs. Feeding this data into AI/ML-powered data analytics software can provide insights into how to improve the customer experience and generate more revenue. By running analytics at the edge rather than from the cloud or centralized data center, banks can get these insights in real-time, allowing them to improve customer interactions while they’re happening.

For example, edge-AI/ML software can help banks provide fast, personalized investment advice on the spot by analyzing a customer’s financial history, risk preferences, and retirement goals and recommending the best options. It can also use video surveillance data to analyze traffic patterns in real-time and ensure tellers are in the right places during peak hours to reduce wait times.

3. On-site data processing

Because the financial services industry is so highly regulated, banks must follow strict security and privacy protocols to protect consumer data from malicious third parties. Transmitting sensitive financial data to the cloud or data center for processing increases the risk of interception and makes it more challenging to meet compliance requirements for data access logging and security controls.

Edge computing allows financial institutions to leverage more data on-site, within the network security perimeter. For example, loan applications contain a lot of sensitive and personally identifiable information (PII). Processing these applications on-site significantly reduces the risk of third-party interception and allows banks to maintain strict control over who accesses data and why, which is more difficult in cloud and colocation data center environments.

4. Enhanced AIOps capabilities

Financial institutions use AIOps (artificial intelligence for IT operations) to analyze monitoring data from IT devices, network infrastructure, and security solutions and get automated incident management, root-cause analysis (RCA), and simple issue remediation. Deploying AIOps at the edge provides real-time issue detection and response, significantly shortening the duration of outages and other technology disruptions. It also ensures continuous operation even if an ISP outage or network failure cuts a branch off from the cloud or data center, further helping to reduce disruptions and remote sites.

Additionally, AIOps and other artificial intelligence technology tend to use GPUs (graphics processing units), which are more expensive than CPUs (central processing units), especially in the cloud. Deploying AIOps on small, decentralized, multi-functional edge computing devices can help reduce costs without sacrificing functionality. For example, deploying an array of Nvidia A100 GPUs to handle AIOps workloads costs at least $10k per unit; comparable AWS GPU instances can cost between $2 and $3 per unit per hour. By comparison, a Nodegrid Gate SR costs under $5k and also includes remote serial console management, OOB, cellular failover, gateway routing, and much more.

The benefits of edge computing for banking

Edge computing can help the financial services industry:

  • Reduce losses, theft, and crime by leveraging artificial intelligence to analyze real-time video surveillance data.
  • Increase branch productivity and revenue with real-time insights from security systems, customer experience data, and network infrastructure.
  • Simplify regulatory compliance by keeping sensitive customer and financial data on-site within company-owned infrastructure.
  • Improve resilience with real-time AIOps capabilities like automated incident remediation that continues operating even if the site is cut off from the WAN or Internet
  • Reduce the operating costs of AI and machine learning applications by deploying them on small, multi-function edge computing devices. 
  • Mitigate the risk of interception by leveraging financial and IT data on the local network and distributing the attack surface.

Edge computing best practices

Isolating the management interfaces used to control network infrastructure is the best practice for ensuring the security, resilience, and efficiency of edge computing deployments. CISA and PCI DSS 4.0 recommend implementing isolated management infrastructure (IMI) because it prevents compromised accounts, ransomware, and other threats from laterally moving from production resources to the control plane.

IMI with Nodegrid(2)

Using vendor-neutral platforms to host, connect, and secure edge applications and workloads is the best practice for ensuring the scalability and flexibility of financial edge architectures. Moving away from dedicated device stacks and taking a “platformization” approach allows financial institutions to easily deploy, update, and swap out applications and capabilities on demand. Vendor-neutral platforms help reduce hardware overhead costs to deploy new branches and allow banks to explore different edge software capabilities without costly hardware upgrades.

Edge-Management-980×653

Additionally, using a centralized, cloud-based edge management and orchestration (EMO) platform is the best practice for ensuring remote teams have holistic oversight of the distributed edge computing architecture. This platform should be vendor-agnostic to ensure complete coverage over mixed and legacy architectures, and it should use out-of-band (OOB) management to provide continuous remote access to edge infrastructure even during a major service outage.

How Nodegrid streamlines edge computing for the banking industry

Nodegrid is a vendor-neutral edge networking platform that consolidates an entire edge tech stack into a single, cost-effective device. Nodegrid has a Linux-based OS that supports third-party VMs and Docker containers, allowing banks to run edge computing workloads, data analytics software, automation, security, and more. 

The Nodegrid Gate SR is available with an Nvidia Jetson Nano card that’s optimized for artificial intelligence workloads. This allows banks to run AI surveillance software, ML-powered recommendation engines, and AIOps at the edge alongside networking and infrastructure workloads rather than purchasing expensive, dedicated GPU resources. Plus, Nodegrid’s Gen 3 OOB management ensures continuous remote access and IMI for improved branch resilience.

Get Nodegrid for your edge computing use cases in banking

Nodegrid’s flexible, vendor-neutral platform adapts to any use case and deployment environment. Watch a demo to see Nodegrid’s financial network solutions in action.

Watch a demo

AI Orchestration: Solving Challenges to Improve AI Value

by | Aug 2, 2024 | Actionable Data, Application Hosting, Consolidation, Edge Computing, EdgeOps, Improve Network Security, Increase Productivity, Industry Use Cases, Monitoring & Reporting, Network Automation, Network Edge Orchestration, Out of Band Management, Remote Network Management, Simplify Branch Infrastructure, Streamline Deployments, User Management, Vendor Neutral Platform, Virtualization, Zero Touch Provisioning (ZTP), Zero Trust Security

AI Orchestration(1)
Generative AI and other artificial intelligence technologies are still surging in popularity across every industry, with the recent McKinsey global survey finding that 72% of organizations had adopted AI in at least one business function. In the rush to capitalize on the potential productivity and financial gains promised by AI solution providers, technology leaders are facing new challenges relating to deploying, supporting, securing, and scaling AI workloads and infrastructure. These challenges are exacerbated by the fragmented nature of many enterprise IT environments, with administrators overseeing many disparate, vendor-specific solutions that interoperate poorly if at all.

The goal of AI orchestration is to provide a single, unified platform for teams to oversee and manage AI-related workflows across the entire organization. This post describes the ideal AI orchestration solution and the technologies that make it work, helping companies use artificial intelligence more efficiently.

AI challenges to overcome

The challenges an organization must overcome to use AI more cost-effectively and see faster returns can be broken down into three categories:

  1. Overseeing AI-led workflows to ensure models are behaving as expected and providing accurate results, when these workflows are spread across the enterprise in different geographic locations and vendor-specific applications.
    .
  2. Efficiently provisioning, maintaining, and scaling the vast infrastructure and computational resources required to run intensive AI workflows at remote data centers and edge computing sites.
    .
  3. Maintaining 24/7 availability and performance of remote AI workflows and infrastructure during security breaches, equipment failures, network outages, and natural disasters.

These challenges have a few common causes. One is that artificial intelligence and the underlying infrastructure that supports it are highly complex, making it difficult for human engineers to keep up. Two is that many IT environments are highly fragmented due to closed vendor solutions that integrate poorly and require administrators to manage too many disparate systems, allowing coverage gaps to form. Three is that many AI-related workloads occur off-site at data centers and edge computing sites, so it’s harder for IT teams to repair and recover AI systems that go down due to a networking outage, equipment failure, or other disruptive event.

How AI orchestration streamlines AI/ML in an enterprise environment

The ideal AI orchestration platform solves these problems by automating repetitive and data-heavy tasks, unifying workflows with a vendor-neutral platform, and using out-of-band (OOB) serial console management to provide continuous remote access even during major outages.

Automation

Automation is crucial for teams to keep up with the pace and scale of artificial intelligence. Organizations use automation to provision and install AI data center infrastructure, manage storage for AI training and inference data, monitor inputs and outputs for toxicity, perform root-cause analyses when systems fail, and much more. However, tracking and troubleshooting so many automated workflows can get very complicated, creating more work for administrators rather than making them more productive. An AI orchestration platform should provide a centralized interface for teams to deploy and oversee automated workflows across applications, infrastructure, and business sites.

Unification

The best way to improve AI operational efficiency is to integrate all of the complicated monitoring, management, automation, security, and remediation workflows. This can be accomplished by choosing solutions and vendors that interoperate or, even better, are completely vendor-agnostic (a.k.a., vendor-neutral). For example, using open, common platforms to run AI workloads, manage AI infrastructure, and host AI-related security software can help bring everything together where administrators have easy access. An AI orchestration platform should be vendor-neutral to facilitate workload unification and streamline integrations.

Resilience

AI models, workloads, and infrastructure are highly complex and interconnected, so an issue with one component could compromise interdependencies in ways that are difficult to predict and troubleshoot. AI systems are also attractive targets for cybercriminals due to their vast, valuable data sets and because of how difficult they are to secure, with HiddenLayer’s 2024 AI Threat Landscape Report finding that 77% of businesses have experienced AI-related breaches in the last year. An AI orchestration platform should help improve resilience, or the ability to continue operating during adverse events like tech failures, breaches, and natural disasters.

Gen 3 out-of-band management technology is a crucial component of AI and network resilience. A vendor-neutral OOB solution like the Nodegrid Serial Console Plus (NSCP) uses alternative network connections to provide continuous management access to remote data center, branch, and edge infrastructure even when the ISP, WAN, or LAN connection goes down. This gives administrators a lifeline to troubleshoot and recover AI infrastructure without costly and time-consuming site visits. The NSCP allows teams to remotely monitor power consumption and cooling for AI infrastructure. It also provides 5G/4G LTE cellular failover so organizations can continue delivering critical services while the production network is repaired.

A diagram showing isolated management infrastructure with the Nodegrid Serial Console Plus.

Gen 3 OOB also helps organizations implement isolated management infrastructure (IMI), a.k.a, control plane/data plane separation. This is a cybersecurity best practice recommended by the CISA as well as regulations like PCI DSS 4.0, DORA, NIS2, and the CER Directive. IMI prevents malicious actors from being able to laterally move from a compromised production system to the management interfaces used to control AI systems and other infrastructure. It also provides a safe recovery environment where teams can rebuild and restore systems during a ransomware attack or other breach without risking reinfection.

Getting the most out of your AI investment

An AI orchestration platform should streamline workflows with automation, provide a unified platform to oversee and control AI-related applications and systems for maximum efficiency and coverage, and use Gen 3 OOB to improve resilience and minimize disruptions. Reducing management complexity, risk, and repair costs can help companies see greater productivity and financial returns from their AI investments.

The vendor-neutral Nodegrid platform from ZPE Systems provides highly scalable Gen 3 OOB management for up to 96 devices with a single, 1RU serial console. The open Nodegrid OS also supports VMs and Docker containers for third-party applications, so you can run AI, automation, security, and management workflows all from the same device for ultimate operational efficiency.

Streamline AI orchestration with Nodegrid

Contact ZPE Systems today to learn more about using a Nodegrid serial console as the foundation for your AI orchestration platform. Contact Us

Edge Computing Use Cases in Telecom

by | Jul 31, 2024 | Actionable Data, Application Hosting, Consolidation, Edge Computing, EdgeOps, Improve Network Security, Increase Productivity, Industry Use Cases, Monitoring & Reporting, Network Automation, Network Edge Orchestration, Out of Band Management, Remote Network Management, Simplify Branch Infrastructure, Streamline Deployments, User Management, Vendor Neutral Platform, Virtualization, Zero Touch Provisioning (ZTP), Zero Trust Security

This blog describes four edge computing use cases in telecom before describing the benefits and best practices for the telecommunications industry.
Telecommunications networks are vast and extremely distributed, with critical network infrastructure deployed at core sites like Internet exchanges and data centers, business and residential customer premises, and access sites like towers, street cabinets, and cell site shelters. This distributed nature lends itself well to edge computing, which involves deploying computing resources like CPUs and storage to the edges of the network where the most valuable telecom data is generated. Edge computing allows telecom companies to leverage data from CPE, networking devices, and users themselves in real-time, creating many opportunities to improve service delivery, operational efficiency, and resilience.

This blog describes four edge computing use cases in telecom before describing the benefits and best practices for edge computing in the telecommunications industry.

4 Edge computing use cases in telecom

1. Enhancing the customer experience with real-time analytics

Each customer interaction, from sales calls to repair requests and service complaints, is a chance to collect and leverage data to improve the experience in the future. Transferring that data from customer sites, regional branches, and customer service centers to a centralized data analysis application takes time, creates network latency, and can make it more difficult to get localized and context-specific insights. Edge computing allows telecom companies to analyze valuable customer experience data, such as network speed, uptime (or downtime) count, and number of support contacts in real-time, providing better opportunities to identify and correct issues before they go on to affect future interactions.

2. Streamlining remote infrastructure management and recovery with AIOps

AIOps helps telecom companies manage complex, distributed network infrastructure more efficiently. AIOps (artificial intelligence for IT operations) uses advanced machine learning algorithms to analyze infrastructure monitoring data and provide maintenance recommendations, automated incident management, and simple issue remediation. Deploying AIOps on edge computing devices at each telecom site enables real-time analysis, detection, and response, helping to reduce the duration of service disruptions. For example, AIOps can perform automated root-cause analysis (RCA) to help identify the source of a regional outage before technicians arrive on-site, allowing them to dive right into the repair. Edge AIOps solutions can also continue functioning even if the site is cut off from the WAN or Internet, potentially self-healing downed networks without the need to deploy repair techs on-site.

3. Preventing environmental conditions from damaging remote equipment

Telecommunications equipment is often deployed in less-than-ideal operating conditions, such as unventilated closets and remote cell site shelters. Heat, humidity, and air particulates can shorten the lifespan of critical equipment or cause expensive service failures, which is why it’s recommended to use environmental monitoring sensors to detect and alert remote technicians to problems. Edge computing applications can analyze environmental monitoring data in real-time and send alerts to nearby personnel much faster than cloud- or data center-based solutions, ensuring major fluctuations are corrected before they damage critical equipment.

4. Improving operational efficiency with network virtualization and consolidation

Another way to reduce management complexity – as well as overhead and operating expenses – is through virtualization and consolidation. Network functions virtualization (NFV) virtualizes networking equipment like load balancers, firewalls, routers, and WAN gateways, turning them into software that can be deployed anywhere – including edge computing devices. This significantly reduces the physical tech stack at each site, consolidating once-complicated network infrastructure into, in some cases, a single device. For example, the Nodegrid Gate SR provides a vendor-neutral edge computing platform that supports third-party NFVs while also including critical edge networking functionality like out-of-band (OOB) serial console management and 5G/4G cellular failover.

Edge computing in telecom: Benefits and best practices

Edge computing can help telecommunications companies:

  • Get actionable insights that can be leveraged in real-time to improve network performance, service reliability, and the support experience.
  • Reduce network latency by processing more data at each site instead of transmitting it to the cloud or data center for analysis.
  • Lower CAPEX and OPEX at each site by consolidating the tech stack and automating management workflows with AIOps.
  • Prevent downtime with real-time analysis of environmental and equipment monitoring data to catch problems before they escalate.
  • Accelerate recovery with real-time, AIOps root-cause analysis and simple incident remediation that continues functioning even if the site is cut off from the WAN or Internet.

Management infrastructure isolation, which is recommended by CISA and required by regulations like DORA, is the best practice for improving edge resilience and ensuring a speedy recovery from failures and breaches. Isolated management infrastructure (IMI) prevents compromised accounts, ransomware, and other threats from moving laterally from production resources to the interfaces used to control critical network infrastructure.

IMI with Nodegrid(2)
To ensure the scalability and flexibility of edge architectures, the best practice is to use vendor-neutral platforms to host, connect, and secure edge applications and workloads. Moving away from dedicated device stacks and taking a “platformization” approach allows organizations to easily deploy, update, and swap out functions and services on demand. For example, Nodegrid edge networking solutions have a Linux-based OS that supports third-party VMs, Docker containers, and NFVs. Telecom companies can use Nodegrid to run edge computing workloads as well as asset management software, customer experience analytics, AIOps, and edge security solutions like SASE.

Vendor-neutral platforms help reduce hardware overhead costs to deploy new edge sites, make it easy to spin-up new NFVs to meet increased demand, and allow telecom organizations to explore different edge software capabilities without costly hardware upgrades. For example, the Nodegrid Gate SR is available with an Nvidia Jetson Nano card that’s optimized for AI workloads, so companies can run innovative artificial intelligence at the edge alongside networking and infrastructure management workloads rather than purchasing expensive, dedicated GPU resources.

Edge-Management-980×653
Finally, to ensure teams have holistic oversight of the distributed edge computing architecture, the best practice is to use a centralized, cloud-based edge management and orchestration (EMO) platform. This platform should also be vendor-neutral to ensure complete coverage and should use out-of-band management to provide continuous management access to edge infrastructure even during a major service outage.

Streamlined, cost-effective edge computing with Nodegrid

Nodegrid’s flexible, vendor-neutral platform adapts to all edge computing use cases in telecom. Watch a demo to see Nodegrid’s telecom solutions in action.

Watch a demo

Edge Computing Use Cases in Retail

by | Jul 25, 2024 | Actionable Data, Application Hosting, Consolidation, Edge Computing, EdgeOps, Improve Network Security, Increase Productivity, Industry Use Cases, Monitoring & Reporting, Network Automation, Network Edge Orchestration, Out of Band Management, Remote Network Management, Simplify Branch Infrastructure, Streamline Deployments, User Management, Vendor Neutral Platform, Virtualization, Zero Touch Provisioning (ZTP), Zero Trust Security

Automated transportation robots move boxes in a warehouse, one of many edge computing use cases in retail
Retail organizations must constantly adapt to meet changing customer expectations, mitigate external economic forces, and stay ahead of the competition. Technologies like the Internet of Things (IoT), artificial intelligence (AI), and other forms of automation help companies improve the customer experience and deliver products at the pace demanded in the age of one-click shopping and two-day shipping. However, connecting individual retail locations to applications in the cloud or centralized data center increases network latency, security risks, and bandwidth utilization costs.

Edge computing mitigates many of these challenges by decentralizing cloud and data center resources and distributing them at the network’s “edges,” where most retail operations take place. Running applications and processing data at the edge enables real-time analysis and insights and ensures that systems remain operational even if Internet access is disrupted by an ISP outage or natural disaster. This blog describes five potential edge computing use cases in retail and provides more information about the benefits of edge computing for the retail industry.

5 Edge computing use cases in retail

.

1. Security video analysis

Security cameras are crucial to loss prevention, but constantly monitoring video surveillance feeds is tedious and difficult for even the most experienced personnel. AI-powered video surveillance systems use machine learning to analyze video feeds and detect suspicious activity with greater vigilance and accuracy. Edge computing enhances AI surveillance by allowing solutions to analyze video feeds in real-time, potentially catching shoplifters in the act and preventing inventory shrinkage.

2. Localized, real-time insights

Retailers have a brief window to meet a customer’s needs before they get frustrated and look elsewhere, especially in a brick-and-mortar store. A retail store can use an edge computing application to learn about customer behavior and purchasing activity in real-time. For example, they can use this information to rotate the products featured on aisle endcaps to meet changing demand, or staff additional personnel in high-traffic departments at certain times of day. Stores can also place QR codes on shelves that customers scan if a product is out of stock, immediately alerting a nearby representative to provide assistance.

3. Enhanced inventory management

Effective inventory management is challenging even for the most experienced retail managers, but ordering too much or too little product can significantly affect sales. Edge computing applications can improve inventory efficiency by making ordering recommendations based on observed purchasing patterns combined with real-time stocking updates as products are purchased or returned. Retailers can use this information to reduce carrying costs for unsold merchandise while preventing out-of-stocks, improving overall profit margins.
.

4. Building management

Using IoT devices to monitor and control building functions such as HVAC, lighting, doors, power, and security can help retail organizations reduce the need for on-site facilities personnel, and make more efficient use of their time. Data analysis software helps automatically optimize these systems for efficiency while ensuring a comfortable customer experience. Running this software at the edge allows automated processes to respond to changing conditions in real-time, for example, lowering the A/C temperature or routing more power to refrigerated cases during a heatwave.

5. Warehouse automation

The retail industry uses warehouse automation systems to improve the speed and efficiency at which goods are delivered to stores or directly to users. These systems include automated storage and retrieval systems, robotic pickers and transporters, and automated sortation systems. Companies can use edge computing applications to monitor, control, and maintain warehouse automation systems with minimal latency. These applications also remain operational even if the site loses internet access, improving resilience.

The benefits of edge computing for retail

The benefits of edge computing in a retail setting include:
.

Edge computing benefits

Description

Reduced latency

Edge computing decreases the number of network hops between devices and the applications they rely on, reducing latency and improving the speed and reliability of retail technology at the edge.

Real-time insights

Edge computing can analyze data in real-time and provide actionable insights to improve the customer experience before a sale is lost or reduce waste before monthly targets are missed.

Improved resilience

Edge computing applications can continue functioning even if the site loses Internet or WAN access, enabling continuous operations and reducing the costs of network downtime.

Risk mitigation

Keeping sensitive internal data like personnel records, sales numbers, and customer loyalty information on the local network mitigates the risk of interception and distributes the attack surface.
Edge computing can also help retail companies lower their operational costs at each site by reducing bandwidth utilization on expensive MPLS links and decreasing expenses for cloud data storage and computing. Another way to lower costs is by using consolidated, vendor-neutral solutions to run, connect, and secure edge applications and workloads.

For example, the Nodegrid Gate SR integrated branch services router delivers an entire stack of edge networking, infrastructure management, and computing technologies in a single, streamlined device. The open, Linux-based Nodegrid OS supports VMs and Docker containers for third-party edge computing applications, security solutions, and more. The Gate SR is also available with an Nvidia Jetson Nano card that’s optimized for AI workloads to help retail organizations reduce the hardware overhead costs of deploying artificial intelligence at the edge.

Consolidated edge computing with Nodegrid

Nodegrid’s flexible, scalable platform adapts to all edge computing use cases in retail. Watch a demo to see Nodegrid’s retail network solutions in action.

Watch a demo

Why Securing IT Means Replacing End-of-Life Console Servers

by | Jul 25, 2024 | Data Center Management, Data Center Resilience, Improve Network Security, Increase Productivity, Micro-segmentation, Minimize Impact of Disruptions, Network Automation, Out of Band Management, Remote Network Management, Serial Consoles, User Management, Vendor Neutral Platform, Zero Touch Provisioning (ZTP), Zero Trust Security

Rene Neumann – Why Securing IT Means Replacing End of Life Console Servers

 

The world as we know it is connected to IT, and IT relies on its underlying infrastructure. Organizations must prioritize maintaining this infrastructure; otherwise, any disruption or breach has a ripple effect that takes services offline for millions of users (take 2024’s CrowdStrike outage, for example). A big part of this maintenance is ensuring that all hardware components, including console servers, are up-to-date and secure. Most console servers reach end-of-life (EOL) and need to be replaced, but for many reasons, whether budgetary concerns or the “if it isn’t broken” mentality, IT teams often keep their EOL devices. Let’s look at the risks of using EOL console servers, and why replacing them goes hand-in-hand with securing IT.

The Risks of Using End-of-Life Console Servers

End-of-life console servers can undermine the security and functionality of IT systems. These risks include:

1. Lack of Security Features and Updates

Aging console servers lack adequate hardware and management security features, meaning they can’t support a zero trust approach. On top of this, once a console server reaches EOL, the manufacturer stops providing security patches and updates. The device then becomes vulnerable to newly discovered CVEs and complex cyberattacks (like the MOVEit and Ragnar Locker breaches). Cybercriminals often target outdated hardware because they know that these devices are no longer receiving updates, making them easy entry points for launching attacks.

2. Compliance Issues

Many industries have stringent regulatory requirements regarding data security and IT infrastructure. DORA, NIS2 (EU), NIST2 (US), PCI 4.0 (finance), and CER Directive are just a few of the updated regulations that are cracking down on how organizations architect IT, including the management layer. Using EOL hardware can lead to non-compliance, resulting in fines and legal repercussions. Regulatory bodies expect organizations to use up-to-date and secure equipment to protect sensitive information.

3. Prolonged Recovery

EOL console servers are prone to failures and inefficiencies. As these devices age, their performance deteriorates, leading to increased downtime and disruptions. Most console servers are Gen 2, meaning they offer basic remote troubleshooting (to address break/fix scenarios) and limited automation capabilities. When there is a severe disruption, such as a ransomware attack, hackers can easily access and encrypt these devices to lock out admin access. Organizations then must endure prolonged recovery (like the CrowdStrike outage, or 2023’s MGM attack) because they need to physically decommission and restore their infrastructure.

 

The Importance of Replacing EOL Console Servers

Here’s why replacing EOL console servers is essential to securing IT:

1. Modern Security Approach

Zero trust is an approach that uses segmentation across IT assets. This ensures that only authorized users can access resources necessary for their job function. This approach requires SAML, SSO, MFA/2FA, and role-based access controls, which are only supported by modern console servers. Modern devices additionally feature advanced security through encryption, signed OS, and tampering detection. This ensures a complete cyber and physical approach to security.

2. Protection Against New Threats

New CVEs and evolving threats can easily take advantage of EOL devices that no longer receive updates. Modern console servers benefit from ongoing support in the form of firmware upgrades and security patches. Upgrading with a security-focused device vendor can drastically shrink the attack surface, by addressing supply chain security risks, codebase integrity, and CVE patching.

3. Ease of Compliance

EOL devices lack modern security features, but this isn’t the only reason why they make it difficult or impossible to comply with regulations. They also lack the ability to isolate the control plane from the production network (see Diagram 1 below), meaning attackers can easily move between the two in order to launch ransomware and steal sensitive information. Watchdog agencies and new legislation are stipulating that organizations follow the latest best practice of separating the control plane from production, called Isolated Management Infrastructure (IMI). Modern console servers make this best practice simple to achieve by offering drop-in out-of-band that is completely isolated from production assets (see Diagram 2 below). This means that the organization is always in control of its IT assets and sensitive data.

A network diagram showing Gen 2 out-of-band is vulnerable to the internet

Diagram 1: Though an acceptable approach, Gen 2 out-of-band lacks isolation and leaves management interfaces vulnerable to the internet.

A network diagram showing how Gen 3 out-of-band secures network and management interfaces.

Diagram 2: Gen 3 out-of-band fully isolates the control plane to guarantee organizations retain control of their IT assets and sensitive info.

4. Faster Recovery

New console servers are designed to handle more workloads and functions, which eliminates single-purpose devices and shrinks the attack surface. They can also run VMs and Docker containers to host applications. This enables what Gartner calls the Isolated Recovery Environment (IRE) (see Diagram 3 below), which is becoming essential for faster recovery from ransomware. Since the IMI component prohibits attackers from accessing the control plane, admins retain control during an attack. They can use the IMI to deploy their IRE and the necessary applications — remotely — to decommission, cleanse, and restore their infected infrastructure. This means that they don’t have to roll trucks week after week when there’s an attack; they just need to log into their management infrastructure to begin assessing and responding immediately, which significantly reduces recovery times.

A diagram showing the components of an isolated recovery environment.

Diagram 3: The Isolated Recovery Environment allows for a comprehensive and rapid response to ransomware attacks.

Watch How To Secure The Network Backbone

I recently presented at Cisco Live Vegas on how to secure the network’s backbone using Isolated Management Infrastructure. I walk you through the evolution of network management, and it becomes obvious that end-of-life console servers are a major security concern, both from the hardware perspective itself and their lack of isolation capabilities. Watch my 10-minute presentation from the show and download some helpful resources, including the blueprint to building IMI.

Cisco Live 2024 – Securing the Network Backbone
Watch My Presentation

The CrowdStrike Outage: How to Recover Fast and Avoid the Next Outage

by | Jul 23, 2024 | Consolidation, Data Logging, Improve Network Security, Increase Productivity, Micro-segmentation, Minimize Impact of Disruptions, Network Automation, Out of Band Management, Remote Network Management, Serial Consoles, User Management, Virtualization, Zero Touch Provisioning (ZTP), Zero Trust Security

CrowdStrike Outage BSOD

 

On July 19, 2024, CrowdStrike, a leading cybersecurity firm renowned for its advanced endpoint protection and threat intelligence solutions, experienced a significant outage that disrupted operations for many of its clients. This outage, triggered by a software upgrade, resulted in crashes for Windows PCs, creating a wave of operational challenges for banks, airports, enterprises, and organizations worldwide. This blog post explores what transpired during this incident, what caused the outage, and the broader implications for the cybersecurity industry.

What happened?

The incident began on the morning of July 19, 2024, when numerous CrowdStrike customers started reporting issues with their Windows PCs. Users experienced the BSOD (blue screen of death), which is when Windows crashes and renders devices unusable. As the day went on, it became evident that the problem was widespread and directly linked to a recent software upgrade deployed by CrowdStrike.

Timeline of Events

  1. Initial Reports: Early in the day, airports, hospitals, and critical infrastructure operators began experiencing unexplained crashes on their Windows PCs. The issue was quickly reported to CrowdStrike’s support team.
  2. Incident Acknowledgement: CrowdStrike acknowledged the issue via their social media channels and direct communications with affected clients, confirming that they were investigating the cause of the crashes.
  3. Root Cause Analysis: CrowdStrike’s engineering team worked diligently to identify the root cause of the problem. They soon determined that a software upgrade released the previous night was responsible for the crashes.
  4. Mitigation Efforts: Upon isolating the faulty software update, CrowdStrike issued guidance on how to roll back the update and provided patches to fix the issue.

What caused the CrowdStrike outage?

The root cause of the outage was a software upgrade intended to enhance the functionality and security of CrowdStrike’s Falcon sensor endpoint protection platform. However, this upgrade contained a bug that conflicted with certain configurations of Windows PCs, leading to system crashes. Several factors contributed to the incident:

  1. Insufficient Testing: The software update did not undergo adequate testing across all possible configurations of Windows PCs. This oversight meant that the bug was not detected before the update was deployed to customers.
  2. Complex Interdependencies: The incident highlights the complex interdependencies between software components and operating systems. Even minor changes can have unforeseen impacts on system stability.
  3. Rapid Deployment: In the cybersecurity industry, quick responses to emerging threats are crucial. However, the pressure to deploy updates rapidly can sometimes lead to insufficient testing and quality assurance processes.

We need to remember one important fact: whether software is written by humans or AI, there will be mistakes in coding and testing. When an issue slips through the cracks, the customer lab is the last resort to catch it. Usually, this can be done with a controlled rollout, where the IT team first upgrades their lab equipment, performs further testing, puts in place a rollback plan, and pushes the update to a less critical site. But in a cloud-connected SaaS world, the customer is no longer in control. That’s why they sign waivers stating that if such an incident occurs, the company that caused the problem is not liable. Experts are saying the only way to address this challenge is to have an infrastructure that’s designed, deployed, and operated for resilience. We discuss this architecture further down in this article.

How to recover from the CrowdStrike outage

CrowdStrike gives two options for recovering:

  • Option 1: Reboot in Safe Mode – Reboot the affected device in Safe Mode, locate and delete the file “C-00000291*.sys”, and then restart the device.
  • Option 2: Re-image – Download and configure the recovery utility to create a new Windows image, add this image to a USB drive, and then insert this USB drive into the target device. The utility will automatically find and delete the file that’s causing the crash.

The biggest obstacle that is costing organizations a lot of time and money is that with either of these recovery methods, IT staff need to be physically present to work on each affected device. They need to go one by one manually remediating via Safe Mode or physically inserting the USB drive. What makes this more difficult is that many organizations use physical and software/management security controls to limit access. Locked device cabinets slow down physical access to devices, and things like role-based access policies and disk encryption can make Safe Mode unusable. Because this outage is affecting more than 8.5 million computers, this kind of work won’t scale efficiently. That’s why organizations are turning to Isolated Management Infrastructure (IMI) and the Isolated Recovery Environment (IRE).

How IMI and IRE help you recover faster

IMI is a dedicated control plane network that’s meant for administration and recovery of IT systems, including Windows PCs affected by the CrowdStrike outage. It uses the concept of out-of-band management, where you deploy a management device that is connected to dedicated management ports of your IT infrastructure (e.g., serial ports, IPMI ports, and other ethernet management ports). IMI also allows you to deploy recovery services for your digital estate that is immutable and near-line when recovery needs to take place.

IMI does not rely at all on the production assets, as it has its own dedicated remote access via WAN links like 4G/5G, and can contain and encrypt recovery keys and tools with zero trust.

IMI gives teams remote, low-level access to devices so they can recover their systems remotely without the need to visit sites. Organizations that employ IMI are able to revert back to a golden image through automation, or deploy bootable tools to all the computers at the site to rescue them without data loss.

The dedicated out-of-band access to serial/IPMI and management ports gives automation software the same abilities as if a physical crash cart was pulled up to the servers. ZPE Systems’ Nodegrid (now a brand of Legrand) enables this architecture as explained next. Using Nodegrid and ZPE Cloud, teams can use either option to recover from the CrowdStrike outage:

  • Option 1: Reboot in Pre-Execution Environment Software – Nodegrid gives low-level network access to connected Windows as if teams were sitting directly in front of the affected device. This means they can remote-in, reboot to a network image, remote into the booted image, delete the faulty file, and restart the system.
  • Option 2: Re-image – ZPE Cloud serves as a file repository and orchestration engine. Teams can upload their working Windows image, and then automatically push this across their global fleet of affected devices. This option speeds up recovery times exponentially.
  • Option 3: – Run Windows Deployment server on the IMI device at the location and re-image servers and workstations if a good backup of the data has been located. This backup can be made available through the IMI after the initial image has been deployed. The IMI can provide dedicated secure access to the InTune services in your M365 cloud, and the backups do not have to transit the entire internet for all workstations at the time, speeding up recovery many times over.

All of these options can be performed at scale or even automated. Server recovery with large backups, although it may take a couple of hours, can be delivered locally and tracked for performance and consistency.

But what about the risk of making mistakes when you have to repeat these tasks? Won’t this cause more damage and data loss?

Any team can make a mistake repeating these recovery tasks over a large footprint, and cause further damage or loss of data, slowing the recovery further. Automated recovery through the IMI addresses this, and can provide reliable recording and reporting to ensure that the restoration is complete and trusted. 

What does IMI look like?

Here’s a simplified view of Isolated Management Infrastructure. You can see that ZPE’s Nodegrid device is needed, which sits beside production infrastructure and provides the platform for hosting all the tools necessary for fast recovery.

A diagram showing how to use Nodegrid Gen 3 OOB to enable IMI.

What you need to deploy IMI for recovery:

  1. Out-of-band appliance with serial, USB, ethernet interfaces (e.g., ZPE’s Nodegrid Net SR)
  2. Switchable PDU: Legrand Server Tech or Raritan PDU
  3. Windows PXE Boot image

Here’s the order of operations for a faster CrowdStrike outage recovery:

  • Option 1 – Recover
    1. IMI deployed with a ZPE Nodegrid device that will start Pre-Execution Environment (PXE) which are Windows boot images that the Nodegrid will push to the computers when they boot up
    2. Send recovery keys from Intune to IMI remote storage over ZPE Cloud’s zero trust platform easily available in cloud or air-gapped through Nodegrid Manager
    3. Enable PXE service (automated across entire enterprise) and define the PXE recovery image
    4. Use serial or IP control of power to the computers, or if possible Intel vPro or IPMI capable machines, to reboot all machines
    5. All machines will boot and check in to a control tower for PXE, or be made available to remote into using stored passwords on the PXE environment, Windows AD, or other Privileged Access Management (PAM)
    6. Delete Files
    7. Reboot

 

  • Option 2 – Lean re-image
    1. IMI deployed with a Windows Pre-Execution boot image running PXE service
    2. Enable access to cloud and Azure Intune to the IMI remote storage for the local image for the PC
    3. Enable PXE service (automated across entire enterprise) and define the PXE recovery image
    4. Use serial or IP control of power to the computers, or if possible, Intel vPro or IPMI capable machines, to reboot all machines
    5. Machines will boot and check in to Intune either through the IMI or through normal Internet access and finish imaging
    6. Once the machine completes the InTune tasks, InTune will signal backups to come down to the machines. If these backups are offsite, they can be staged on the IMI through backup software running on a virtual machine located on the IMI appliance to speed up recovery and not impede the Internet connection at the remote site
    7. Pre-stage backups onto local storage, push recovery from the virtual machine on the IMI

 

  • Option 3 – Windows controlled re-image
    1. Windows Deployment Server (WDS) installed as a virtual machine running on the IMI appliance (offline to prevent issues or online but under a slowed deployment cycle in case there was an issue) 
    2. Send recovery keys from Intune to IMI remote storage over a zero trust interface in cloud or air-gapped
    3. Use serial or IP control of power to the computers, or if possible, Intel vPro or IPMI capable machines, to reboot all machines
    4. Machines will boot and check in to the WDS for re-imaging
    5. Machines will boot and check in to Intune either through the IMI or through normal Internet access and finish imaging
    6. Once the machine completes the InTune tasks, InTune will signal backups to come down to the machines. If these backups are offsite, they can be staged on the IMI through backup software running on a virtual machine located on the IMI appliance to speed up recovery and not impede the Internet connection at the remote site
    7. Pre-stage backups onto local storage, push recovery from the virtual machine on the IMI

Deploy IMI to avoid the next outage

Get in touch for help choosing the right size IMI deployment for your organization. Nodegrid and ZPE Cloud are the drop-in solution to recovering from outages, with plenty of device options to fit any budget and environment size. Contact ZPE Sales now or download the blueprint to help you begin implementing IMI.

Contact ZPE sales
Download blueprint