IT Brief Australia logo
Technology news for Australia's largest enterprises
Story image

New AI capabilities enabled with LiveData now automating Apache metadata into Databricks

By Ryan Morris-Reade
Tue 25 May 2021
FYI, this story is more than a year old

WANdisco, a LiveData company, has announced its LiveData Migrator platform can now automate the migration of Apache Hive metadata directly into Databricks, enabling users to quickly enable new AI and machine learning capabilities.

The LiveData Migrator platform, which automates the migration and replication of Hadoop data from on-premises to the cloud, can help users save time and reduce costs. WANdisco says for the first time, enterprises wanting to migrate on-premises Hadoop and Spark content from Hive to Databricks, can do so at scale and efficiently, while managing the risks associated with large-scale cloud migrations.

It says datasets don’t need to be fully migrated before they are converted into the Delta format. LiveData Migrator automates an incremental transformation to Delta Lake.

Ongoing changes to source metadata are instantly reflected in the Lakehouse platform, and on-premises data formats used in Hadoop and Hive are automatically made available in Delta Lake on Databricks. 

By combining data and metadata and making on-premises content immediately usable in Databricks, users can eliminate migration tasks that previously required constructing data pipelines to transform, filter, and adjust data, along with up-front planning and staging. This means work otherwise required for setting up auto-load pipelines to identify newly-landed data then and converting it into its final form is no longer required.

“This new feature brings together the power of Databricks and WANdisco LiveData Migrator,” says WANdisco CTO, Paul Scott-Murphy. 

“Data and metadata are migrated automatically without any disruption or change to existing systems. Teams can implement their cloud modernisation strategies without risk, immediately employing workloads and data that were locked up on-premises and are now in the cloud, by using the Lakehouse platform offered by Databricks.”

Databricks VP of product partnerships, Pankaj Dugar, says enterprises often want to break silos and bring all their data into a lakehouse for analytics and AI, but they've been constrained by their on-premises infrastructure. He says with the new Hive metadata capabilities in WANdiscos LiveData Migrator, it will now be easier to take advantage of the Databricks Lakehouse platform.

LiveData Migrator automates cloud data migration at any scale by enabling companies to migrate data from on-premises Hadoop-oriented data lakes to any cloud within minutes, even while the source data sets are under active change. 

Businesses can migrate their data without the need of engineers or other consultants to enable their digital transformation. LiveData says its Migrator works without any production downtime or business disruption while ensuring the migration is complete and continuous, and any ongoing data changes are copied to the target cloud environment.

Users can choose to convert content to the Delta Lake format when they create the Databricks metadata target. The data they choose to migrate is then set by defining a migration rule, and then selecting the Hive databases and tables that require migration.

Related stories
Top stories
Story image
Cybersecurity
CISOs need to consider a risk-based cybersecurity strategy
Rather than talking in terms of attack vectors and vulnerabilities, CISOs and security decision-makers must look at actual business risk.
Story image
MarTech
CM Group recognised as Best Overall MarTech Company
MarTech Breakthrough has ranked CM Group as the Best Overall MarTech Company for the third year running and Cheetah Digital as Best Overall Marketing Campaign Management Solution.
Story image
Unified Communications
Gold Coast private hospital improves comms with Alcatel-Lucent Enterprise solutions
With further demands placed on health workers as a result of the pandemic, they need efficient and adaptable solutions that help them effectively serve their communities.
Story image
Data
Workday research finds A/NZ organisations becoming leaders in digital agility
New research from Workday and IDC has revealed that Australia and New Zealand are becoming leaders in digital agility.
Story image
University of Melbourne
The University of Melbourne implements data ecosystem with Talend
Talend has announced that The University of Melbourne (UoM) has selected Talend Data Fabric for its cloud data platform.
Story image
Contact Centre
Treasure Data launches new customer experience functionality
Treasure Data has introduced a new strategic vision and product positioning for Customer Data Cloud, reflecting the company's focus.
Story image
Low-code
Video: 10 Minute IT Jams - An update from Mendix
Mendix is a low-code platform used by businesses to develop mobile and web apps at scale, and Jornt joins us today to discuss how these offerings work, and what benefit they have in the development process.
Story image
Lucid Software
Lucid Software expands enterprise offerings with enhanced slack apps
Lucid Software has expanded its enterprise offerings with enhanced slack apps for its Lucidspark and Lucidchart technology.
Wiise
Discover why a localised cloud ERP is the way to go for Australian businesses.
Link image
Story image
Microsoft
Avast reveals zero-day exploits targeting Chrome and Microsoft
Avast, released its Q2/2022 Threat Report today, revealing a significant increase in global ransomware attacks, up 24% from Q1/2022.
Story image
Hybrid Cloud
The essential guide to digital transformation by SolarWinds
Digital transformation is a buzzword thrown around all the time by companies, but what does it actually mean and why is it important? SolarWinds breaks it down.
Story image
Compliance
Why security needs to shape your journey to the cloud
It's estimated that 80% of workloads could be in the cloud in the next few years. How can you make all that data secure?
Story image
Digital Transformation
Top tips for making your finance transformation program a resounding success
Planning to make 2023 the year you embark on a wholesale finance transformation program? It’s a move that will stand your enterprise in excellent stead as you navigate the complexities of the post-Covid business landscape.
Story image
JLL
Investment in APAC cold storage to reach $5 in next decade
Investment in Asia Pacific’s cold storage market is expected to grow fivefold in the next decade, according to JLL.
Story image
Artificial Intelligence
Appier achieves historically high growth rate of 56% YoY
"Our strong momentum over the past two quarters underscores Appier's significant growth alongside our customers."
Story image
Digital Transformation
Why it is imperative to keep up with technology advancements
Technology is constantly evolving, which is why digital transformations can often be complicated. What legacy items stay? Will the new and old work together? Or is there a whole new system that we need to introduce?
Story image
Digital Transformation
Federated change is the best path to digital evolution
Businesses that can successfully manage the exponentially expanding masses of data produced by modern consumers will be the businesses that survive and prosper.
Story image
Artificial Intelligence
Is your chatbot bringing down the customer satisfaction score?
The top 10 reasons why chatbots are failing to meet customer expectations and what you must do to avoid that.
Story image
Data Protection
Advancing genomic sequencing and public health with digital infrastructures
Right before our eyes, we've witnessed the development of the COVID-19 vaccine in record time. An enormous achievement in an otherwise lengthy task that previously took, on average, 10-15 years.
Story image
consumers
Data protection: How SMEs can secure online consumers’ trust
Consumers believe how a company treats their data reflects how they treat their customers, according to a new survey.
Story image
Cybersecurity
Education sector seeing highest volumes of cyber attacks
When breaking down the numbers to education attacks by region in July 2022, A/NZ was the most heavily attacked.
Story image
Ransomware
Sophos reveals latest ransomware trend impacting orgs
Sophos has announced in a new whitepaper that Hive, LockBit and BlackCat, three prominent ransomware gangs, consecutively attacked the same network.
AWS Marketplace
Learn how security orchestration, automation, and response (SOAR) enhances your security strategy.
Link image
Story image
Artificial Intelligence
Concentric AI protects sensitive data with new capabilities
The new capability reveals sensitive data shared across email and business messaging platforms and highlights who has inappropriate access to content.
Story image
Malware
Kaspersky uncovers new attacks by advanced persistent threat group
The attacks involved modifications of the well-known malware, DTrack, as well as the use of a brand-new Maui ransomware.
Story image
Data breach
Weak breach data disclosure laws for IP theft leaves vital Australian industries vulnerable
Infoblox has seen unprecedented levels of demand for cyber protection from companies outside of the Critical Infrastructure Act who have discovered security events and those who fear falling victim to IP theft.
Story image
Indusface
Why enhancing bot protection for web and API endpoints matters
The trouble with bots is that they aren’t all bad. Unfortunately, this can make it challenging to detect malicious bots that find their way into your system and threaten your business.
Story image
Cloud Security
Tenable makes additions to Cloud Security portfolio
Tenable has announced additions to Tenable Cloud Security that represent the next step in assessing threats related to cloud vulnerabilities.
Story image
Biometrics
Can biometrics help? 123% increase in Gen Zs scammed online
In the three years leading up to 2022, the number of Gen Zs who fell victim to online scams rose by 123%, according to Ping Identity.
Story image
Cybersecurity
Australian IT security concerns higher than before pandemic
Australian organisations are more concerned about cyberattacks than they were prior to the COVID-19 pandemic, according to a new survey.
Story image
Gartner Magic Quadrant
Gartner names Lookout a Visionary in 2022 Magic Quadrant
Gartner has recognised Lookout as a Visionary in the 2022 Magic Quadrant for Security Service Edge (SSE) and one of the top three offerings in the 2022 Gartner Critical Capabilities for SSE report.
Story image
Artificial Intelligence
Gartner unveils key emerging tech to watch in 2022
"Such technologies present greater risks for deployment, but potentially greater benefits for early adopters," says Gartner.
Story image
Tech job moves
Tech job moves - Fastly, INX, Kinly, SmartBear & Vectra AI
We round up all job appointments from July 29 - August 12, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
Data analytics
Pressure on orgs to up their data analytics game - study
A recent report from Sisense highlights data transmission, analysis, and risk management remain top concerns for data professionals in APAC.
Story image
Enterprise Resource Planning / ERP
Why the right ERP (and partner) is crucial to an innovative and successful business
Enterprise Resource Planning (ERP) is a foundational step to ensuring a robust business model; here's why choosing the right one could be vital to ensuring long-term success and innovative results.
Story image
Customer experience
Scope Australia selects NICE CXone platform to streamline operations
One of Australia’s largest providers of disability support services, has chosen the NICE CXone platform to help streamline its contact centre operations.
Story image
Collaboration
Lacework launches new capabilities for better threat detection
Lacework has announced new capabilities that enable organisations to uncover more critical threats to their infrastructure and empower teams.
Story image
DevOps
Dynatrace extends application security capabilities for runtime environments
Dynatrace has announced that it has extended its Application Security Module to detect and protect against vulnerabilities in runtime environments.
Story image
Cyber attacks
Dramatic uptick in threat activity with exploits growing nearly 150%
"While it’s not a surprise given increased attack opportunities like remote work, it’s still a worrying development and one we cannot ignore."
Story image
Australian Defence Force
Cloudera and ADF partner for e-health management system
The Australian Defence Force (ADF) has selected Cloudera as a strategic data partner to help transform its e-health management system.
Story image
Augmented Reality
TeamViewer remote access software integrated into RealWear Cloud
TeamViewer has announced a major expansion of its partnership with RealWear, a leading provider of assisted reality wearable solutions for frontline industrial workers.