IT Brief Australia logo
Technology news for Australia's largest enterprises
Story image

How IoT data collection & aggregation with local event processing work

By Julia Gabel
Sat 2 Dec 2017
FYI, this story is more than a year old

Trying to predict how many Internet of Things (IoT) devices will go online over the next decade is like trying to predict the growth rate of rabbits in the wild.

It suffices to say that 2017 could be the year in which IoT devices exceed the total human population, based on a Gartner forecast of 8.4 billion IoT connected devices, or one device for each of the 7.5 billion people, plus just under a billion more to spare.

As companies wrap their brains around the loads of data coming from billions of IoT devices, they are finding that collecting, processing and analysing IoT data is more complex than for traditional data. Consider that each model of automotive vehicle on the road represents a mobile IoT device.

For driverless cars alone, that could translate into generating 47 megabytes per sec per car!

IoT traffic is intermittent and highly unstructured. It is diverse in frequency, function and bidirectionality. To complicate matters further, businesses typically need the insights it holds delivered in real-time.

Making real-time analytics a reality becomes increasingly difficult to achieve when you take into account the slow transfer of larger, more complex data flows to the cloud over less flexible, long-haul networks.

Such networks can’t scale to match need due to limited bandwidth and high-latency connectivity. Increasing interconnection to IoT devices at the edge and providing proximate, high-bandwidth, low-latency connections can address the issues that this “data diversity” present.

What’s happening with this data deluge?

The distributed sources of all this data, which are remote from the majority of corporate data centers, are creating forces that every business must come to terms with. These include:

  • The growth of IoT workflow traffic and data at the edge is creating “data gravity,” where more data attracts more data, as well as applications and analytics that need to interact with that data. This is driving the need to reduce the distance between data, applications and analytics to gain more timely results.
  • Latency is the biggest problem most businesses face. Eventually, adding more, costly bandwidth no longer increases data flow velocity, according to the laws of physics.
  • The ubiquitous nature of the IoT and related field use cases are driving greater simplicity out at the edge. It’s easier to put automated mechanisms for system updates in 10 places for 10,000,000 things, rather than in 10,000,000 places.
  • Not all actions that are required to process or analyse data are equal. Investigation into staging data processing allows a more balanced approach (e.g., Level 2 data does not need as heavy processing as Level 5 data).
  • The bidirectional nature of IoT workflows means aggregated data can be sent to an edge device instead of backhauling it to a centralised staging repository.

These would be pretty basic concepts to address if it weren’t for the inherent constraints within legacy IT infrastructures when dealing with IoT data, including:

  • Data from the field isn’t the same as prepared data—it tends to be messy, intermittent, unstructured and somewhat dynamic (especially if the source device is mobile).
  • The volume of devices, variety of sources, frequency of samples and the data itself are all growing and will not scale using traditional approaches to computing, storing and transporting the resulting data files.
  • In many use cases, latency matters. Delays between a data event and a reaction must be near real-time. Throughput is impacted along with data growth, and delays will become unacceptable to users.
  • Traditional approaches to centralising all data and running analytics in the cloud are unsustainable for real-time use cases. Cloud-based data and analytics can pose security challenges as they are physically outside of the data center’s security perimeter.
  • As machine learning intelligence capabilities become more commonplace in devices out in the field, those devices become more complex, requiring greater CPU and memory and drawing more power. Increasing complexity slows processing down and leads to data results being discarded since by the time results from the device are gathered, more recent data is desired.

Companies need a way to combine data collection and aggregation with local event processing. This would optimise streaming data flows and edge analytics resulting in faster time to insight and greater overall value. In addition, they need to participate in new and expanding IoT ecosystems by interconnecting with partners and exchanging data for optimal business results.

How IoT workflows and analytics work

The IoT is the new normal to enterprise data processing environments. Where software is fast, “things” are slow. An IoT solution must satisfy many requirements as a collective system, for example, interoperability and openness are two key design elements. IoT workflows are separated into two general types of functions:

  1. Device management that includes device-centric requirements such as registering devices, updating operating shells, and systems and authenticating identity and access.
  2. Event processing that involves data events or data polling events. This includes of the actual delivery of targeted data points produced by the IoT devices to their ultimate destinations.

The IoT is also heavily dependent on interconnected partnerships and ecosystems that rely on direct, private and secure connections between counterparties.

In essence, the IoT offers a balance between the cloud and the edge with two distinctly different frequencies of data flows that have two distinct paths: Hot (i.e., real-time telemetry) and Cold (i.e., finding patterns within a data warehouse) workflows, as illustrated in the diagram below.

An IoT Edge-to-Cloud Workflow

IoT analytics combines insights obtained from a traditional approach of finding patterns within streaming data by using data warehouse mining and real-time telemetry of data points provided by individual IoT devices in the field.

Device management design patterns are also dependent on the frequency and source of interconnected components. 

A by-product of the edge IoT device’s bi-directional data flow is the opportunity to process analytics locally with high proximity to the workload.

As a result, you can gain aggregated insights from the patterns found within the information produced by cloud-based analytics capabilities, along with the data from other devices and systems that are close to and interoperate with those IoT devices that are creating the data points.

Enterprise-grade scaling can be achieved by adding digital edge node aggregation points (aka., geographical distributed interconnection hubs) and placing them in regions where local cloud compute resources reside.

If leveraging cloud resources is not an option, an alternative would be to add more distributed data processing units within a digital edge node to create regionally-focused, distributed analytics engines that contribute compute power to IoT data in greater proximity to the workload. 

Optimising streaming data flows and edge analytics

As mentioned above, placing IoT event processing and device management (firmware and updates) in one or more digital edge nodes brings IoT data and analytics closer together.

Since the digital edge node is located at the intersection point of consolidated access (e.g., cellular, broadband, internet), it is as close as possible both to data sources in the field and to the clouds.

Placing device management and IoT analytical capabilities at the edge solves latency, bandwidth and device complexity constraints.

It also provides multi-destination control and choice in the types of networks/providers you want to use, as well as a selection of cloud analytic platforms.

Participating in IoT ecosystems by connecting with partners and exchanging data also provides fast, efficient access to a wide variety of different data sources and types.

As the data comes in, it is validated, authenticated, inspected, pre-processed, stored in the global namespace, and then delivered to the next downstream processing step (see diagram below).

Data Collection and Aggregation, with Local Event Processing

Leveraging an IOA strategy for data collection and aggregation with local event processing allows you to optimise data streaming flows and edge analytics. Follow these steps:

  1. Establish segmentation flows from field area networks. Messages from IoT gateways (in the field) get published on the message bus; otherwise a local IoT gateway processes it first.
  2. Boundary control validates and authenticates the source and message.
  3. Valid messages pass through an inspection zone and policy enforcement.
  4. Messages are persisted to the data repository, then delivered via IoT event processing. Downstream messages are published to the cloud analytic platform(s) of choice, or your own cloud-agnostic repository.
  5. IoT gateway “device requests” follow the same flow but are subscribed to by the device management function. Appropriate payload(s) (firmware, etc.) are loaded from the data repository and published back to the IoT gateway (or to the device directly). IoT data aggregation points in the field can also perform batch processing of the data.

Implementing this streaming data flows and edge analytics design pattern enable the following benefits:

  • IoT edge capabilities can scale to billions of devices, at each metro location globally.
  • Collected data can be validated, authenticated and inspected before being pre-processed.
  • All data can be stored in the global namespace — no need to discard.
  • Bandwidth is provided with the greatest efficiency and lowest latency for real-time event reactions and systematic scaling.
  • A choice of cloud IoT platforms provide as much processing as needed, localised at the edge.
  • Data can be monetised and its access sold on a data exchange, generating valuable insights and revenue.

Article by Ben Towner, Equinix Blog Network

Related stories
Top stories
Story image
Customer experience
Phone, internet complaints increase for first time in five quarters
Phone and internet complaints have increased for the first time in five quarters, according to a new telco report from the Ombudsman.
Story image
Consumer relationships with digital services continues to change
Two years of pandemic-induced reliance on technology for work has altered our relationship with digital apps and services, new research has found.
Story image
Check Point
Check Point and CCTV expert join forces to boost protection
The partnership will involve Check Point Quantum IoT Protect Nano Agent being embedded in Provision-ISR’s CCTV cameras for on-device runtime protection.
Story image
Red Sift
Entrust expands strategic partnership with Red Sift
Entrust has expanded its strategic partnership with Red Sift to make it easier for businesses to adopt Brand Indicators for Message Identification (BIMI) standards for email identification and security.
Story image
Data Protection
Information management capabilities to meet privacy requirements
Organisations with customers or operations across more than one country face a spate of new and proposed privacy and data protection laws.
Story image
Cyber attacks
Devastating cyber attacks expected to hit energy sector
Energy executives anticipate life, property, and environment-compromising cyber attacks on the sector within the next two years.
Story image
Cybersecurity prompts upgrade for 1.3 billion electricity meters
ABI Research finds Advanced Metering Infrastructure (AMI) and cybersecurity concerns are prompting the upgrade of 1.3 billion electricity meters by 2027.
Story image
New Relic
New Relic launches vulnerability management platform
New Relic has introduced New Relic Vulnerability Management to help organisations find and address security risks faster and with greater precision.
Story image
APAC organisations fail to disclose ransomware breaches
85% of organisations in APAC were breached by ransomware at least once in the past five years, but only 28% publicly disclosed the incident.
Story image
SoftIron named global leader for efficient DC infrastructure solutions
SoftIron has been named a global leader for supplying energy-efficient data infrastructure solutions for core-to-edge data centers after an assessment by Earth Capital Ltd.
Story image
NetApp launches Spot PC, a new Desktop-as-a-Service solution
This is a new managed cloud DaaS solution with security, automation, observability and optimisation capabilities, designed for the needs of today.
Story image
Silver Peak
The path to an adaptive, modern network
Managing and securing the network looks different than it did just two years ago—especially given that most of these networks are made up of multi-generations of infrastructure stitched together over time.
Story image
MEC spending for private cellular networks to reach $5.8b
A new ABI Research report finds the revenue for Multi Access Edge Compute (MEC) deployments for private cellular networks will reach USD$5.8 billion by 2030.
Story image
GapMaps Live to improve brand decisions on physical locations
GapMaps has released its latest service GapMaps Live, giving more insights and features to help brands make better decisions about physical locations.
Story image
Aligned Data Centers increases sustainability-linked loan
Aligned Data Centers has increased its sustainability-linked loan from $375 million to $1.75 billion to speed up the next phase of its strategic growth.
Story image
Comcast to use ThreatQuotient for cybersecurity operations
Comcast, the parent company of NBC Universal and SKY Group, has chosen ThreatQ Platform and ThreatQ Investigations to meet their cybersecurity needs.
Story image
Equinix announces milestones on sustainability commitments
Equinix has released its 2021 Sustainability Report which outlines progress, innovation and accomplishments on key ESG commitments.
Story image
Sift shares crucial advice for preventing serious ATO breaches
Are you or your business struggling with Account Takeover Fraud (ATO)? One of the latest ebooks from Sift can provide readers with the tools and expertise to help launch them into the new era of account security.
Story image
Australian space tech startup secures SpaceX support
Space Machines Company has secured the support of SpaceX as a launch partner in carrying SMC's Optimus Orbital Transfer Vehicle to space next year.
Story image
Sysdig unveils new Kubernetes troubleshooting and cloud innovations
Sysdig has introduced two new innovations that look to help bolster cloud services and simplify Kubernetes troubleshooting.
Story image
Skills shortages hold orgs back from capitalising on cloud 2.0
Organisations are becoming more comfortable with sophisticated 'cloud 2.0' technologies, even as they confront difficulties in hiring and retaining IT talent.
Story image
Digital Signage
MAXHUB's Digital Signage range to bolster boardroom productivity
The new MAXHUB Digital Signage technology is purpose-built to make every kind of team meeting more effective.
Story image
Digital Marketing
Similarweb acquires SEO and rank tracking company Rank Ranger
Digital intelligence company Similarweb, which specialises in analysing web traffic, has acquired Rank Ranger, a market leader in SEO and rank tracking.
Story image
Digital Marketing
Getty Images delves into the world of NFTs with Candy Digital
Getty Images and Candy Digital, the next-generation digital collectible company, have announced a new multi-year partnership agreement.
Story image
Trojan cyber attacks hitting SMBs harder than ever - Kaspersky
In 2022 the number of Trojan-PSW detections increased by almost a quarter compared to the same period in 2021 to reach 4,003,323.
Story image
The most common online scams in Australia
No one is safe from online scammers, and many of these scammers have capitalised on the pandemic, using this confusing time to attack more people than ever.
Story image
Digital Transformation
Digital transformation increasing business complexities
A new survey suggests businesses must re-examine their digital transformation approach to better help employees adapt to change.
Story image
Data Center
Preventing downtime costs and damage with Distributed Infrastructure Management
Distributed Infrastructure Management (DIM) can often be a lifeline for many enterprises that work with highly critical ICT infrastructure and power sources.
Story image
Let’s clear the cloud visibility haze with app awareness
Increasingly, organisations are heading for the cloud, initiating new born-in-the-cloud architectures and migrating existing applications via ‘lift and shift’ or refactoring.
Story image
Multi Cloud
Red Hat updates dev tools, empowers use of hybrid and multicloud
Red Hat has unveiled updates across its portfolio of developer tools, designed to help organisations build and deliver applications faster.
Story image
Boost Mobile
Boost Mobile now makes up over 7% of Aussie prepaid market
Boost Mobile has doubled its market share over the past four years and now has more than 7% of Australia's prepaid market, equivalent to over 700,000 people.
Story image
More than 40% of banks worried about cloud security - report
Publicis Sapient's new report finds security and the lack of cloud skills and internal understanding of business benefits are big obstacles for banks moving to the cloud.
Story image
Hybrid Cloud
Barracuda expands cloud-native SASE platform
"The expansion of Barracuda's cloud-native SASE platform for hybrid deployment models and IIoT environments solves a number of challenges."
Find out how a behavioural analytics-driven approach can transform security operations with the new Exabeam commissioned Forrester study.
Link image
Story image
Vectra AI
Understanding the weight on security leader’s shoulders, and how to shift it
Millions of dollars of government funding and internal budgets are being funnelled into cybersecurity to build resilience against sophisticated threats, indicating how serious this issue has become.
Story image
SPS network now available to CrescoData eCommerce customers
CrescoData, a Pitney Bowes Company and PaaS business in the commerce space, says its customers can now connect to the SPS Commerce Retail Network.
Story image
MailGuard warns of new scam targeting Telstra customers
Telstra customers in Australia are being warned of a new scam involving "Unsuccessful Payment" messages.
Story image
Alteryx releases updates, empowers data insights for enterprise
Alteryx has released new advancements designed to aid enterprises with cloud analytics, democratise insights and ensure data governance.
Story image
Infosec unveils role-guided cybersecurity training roadmaps
Infosec Skills Roles maps hands-on training and certifications to the 12 most in-demand cybersecurity roles to maximise training efficiency.
Story image
Tech job moves - Forcepoint, Malwarebytes, SolarWinds & VMware
We round up all job appointments from May 13-20, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
Vectra AI
Vectra’s inaugural Partner of the Year Awards revealed
APAC companies Baidam, Firmus, ShellSoft and Macnica have been recognised in Vectra AI's inaugural Partner of the Year Awards.
Story image
i-PRO releases smallest AI-based surveillance camera on the market
The new i-PRO mini network camera is now available, with a pocket-sized form factor and full AI analytics functionality.
Story image
Ponemon Institute
Email revealed to be riskiest channel for data loss
More than half (60%) of organisations experienced data loss or exfiltration caused by an employee mistake on email in the last 12 months.
Story image
Artificial Intelligence
Frost & Sullivan recognises Genesys as leader in new reports
Frost & Sullivan has recognised Genesys as a leader in the cloud contact centre market for its robust cloud and digital capabilities.