Let’s clear the cloud visibility haze with app awareness

Increasingly, organisations are heading for the cloud, initiating new born-in-the-cloud architectures and migrating existing applications to Infrastructure-as-a-Service (IaaS) providers and hybrid clouds via 'lift and shift' or refactoring.

With this transition, they scale deployments with more servers and VMs, run high-capacity links, leverage containers and routinely add new observability, security and monitoring tools.

Further, they're often running hundreds or even thousands of apps, which, unknown to IT, could include rogue software such as crypto mining or BitTorrent. With ever-increasing volumes of application-oriented data, it's hard for IT teams and tools to focus on the most actionable activity and avoid wasting resources processing irrelevant traffic.

We often inundate security, observability, compliance and network monitoring tools with low-risk, low-value traffic, making them less effective and requiring needless scaling. Additionally, false positives and alerts can overwhelm NetOps, CloudOps, and SecOps teams, obscuring the root causes of network and application performance issues and the real threats buried in volumes of undifferentiated traffic.

Traditionally, IT teams have taken laborious steps to identify applications based on network traffic by either hardwiring ports to specific applications or writing regular expressions to inspect traffic patterns and identify apps.

Such manual workarounds bring their own challenges. For example, when change occurs, such as growth in an application's usage or the introduction of new applications, NetOps teams must update network segmentation.

And app updates can change traffic patterns and behaviour, meaning IT must constantly test and update their homegrown regex signatures. For the cloud, implementing such stopgap measures is difficult, if not impossible.

Until now, it's been hard to isolate cloud traffic by application type and specify whether or not it gets inspected by tools. Visibility has been siloed, and filtering options often only go up to Layer 4 elements, forcing organisations to pass all traffic through their tools or risk missing potential threats.

However, having each tool (intrusion detection system, data loss prevention, advanced threat detection, network analytics, forensics and so on) inspect packets to filter irrelevant traffic is inefficient and costly, as most tool pricing is based on traffic volume and processing load.

While packet brokering can reduce traffic, it requires programming knowledge to maintain complex rules. And although some systems provide a level of application filtering, it's hard to use, identifies a limited number of applications, and doesn't typically share this insight. Furthermore, the filters require ongoing maintenance to keep up with changing application behaviour.

A cloud suite with application filtering intelligence (AFI) can bring application awareness to multi-cloud environments. Public cloud (AWS and Azure) or private (VMware and Nutanix) are all covered.

A cloud suite will automatically extend Layer 7 visibility to identify more than 3,500 common business and network applications traversing the network and lets IT select and deliver only high-value or high-risk data based on application, location, and activity.

Seek an AFI that classifies applications into categories that automatically update as the landscape evolves. This allows a team to take actions on a 'family' of applications versus setting policies on individual apps. Examples of application families include antivirus, audio/video, database, ERP, gaming, messenger, peer-to-peer, telephony, webmail and dozens more.

Each tool is more efficient since it no longer needs to store and process large volumes of irrelevant traffic. NetOps can apply existing tools across a larger area by prioritising only core business applications and accelerating the investigation of network and application performance issues with easier data isolation.

SecOps teams can extend current tools to a larger attack surface, securing more of the network and preventing sensitive data, such as personally identifiable information (PII), from being routed to monitoring and recording tools. For more on how to benefit from AFI, check out this paper.

Not only is identifying applications a serious challenge in the cloud, but obtaining even basic metadata, such as NetFlow, is problematic in public IaaS. IT can derive basic details, such as which IP addresses are used and by whom, along with port and protocol details.

But what is required is summarised, context-aware information about raw packets, based on Layers 4–7, that provides insights into user behaviour, security breaches, customer experience and infrastructure health.

Advanced metadata attributes expand app layer visibility and support a comprehensive approach to obtaining application behaviour. Especially when deploying workloads in the cloud, IT can acquire critical flow details, reduce false positives by separating signal from noise, identify nefarious data extraction, and accelerate threat detection through proactive, real-time traffic monitoring and troubleshooting forensics.

Observability and SIEM solutions use this information to correlate and analyse log data from servers and security appliances. Network security and monitoring tools leverage this metadata to deliver the insight and analytics needed to manage the opportunities and risks associated with cloud deployments.

And administrators can automate anomaly detection, stop cyber threats that overcome perimeter or end-point protection, identify bottlenecks, and understand latency issues.

Based on Layers 4–7, application metadata intelligence (AMI) supplies network and security tools with more than 5,000 metadata characteristics that shed light on the application's performance, customer experience, and security. Leading vendor apps extract and append these elements to NetFlow and IPFIX. Records include:

• Identification: Social media user, file and video names, SQL requests.
• HTTP: URL identification, command response codes.
• DNS parameters: 39 elements, including request/response, queries, and device identifiers.
• IMAP and SMTP email-based communications with sender and receiver addresses
• Service identification: Audio, video, chat, and file transfers for VoIP and messaging.
• Customer/network awareness: VoIP (SIP, RTP) and mobile (GTP, HTTP/2) control/signalling and user/data plane sessions.

Advanced L7 metadata can be applied in a variety of use cases. AMI's principal deployment is in providing metadata to SIEM and observability tools for security analysis. This can help to:

• Identify the use of weak ciphers and expired TLS certificates.
• Investigate suspicious network activity by detecting unauthorised remote connections, bandwidth usage, connection longevity, or an unusual quantity of SSH, RDP, or Telnet sessions.
• Detect data exfiltration by monitoring the volume and types of DNS requests implying DNS tunnelling and evaluating the legitimacy of the domains.
• Pinpoint security breach origins with time-window analysis of Kerberos, SMB, and HTTP use to isolate the prior and post protocol activities that lead up to an incident.
• Find suspicious behaviours that suggest compromised credentials or brute force attacks, such as high-privilege user activity, logins from unauthorised systems or multiple hosts, and HTTP client errors.

While IaaS and private cloud orchestration and management platforms are remarkably resilient, dynamic and infinitely scalable, they don't offer next-generation network packet brokers (NGNPB) with a deep observability pipeline provided by advanced solutions.

Not only do these brokers aggregate, filter and distribute all traffic to the proper security and networking tools, they also provide the compute power behind AFI and AMI.