IT Brief Australia logo
Technology news for Australia's largest enterprises
Story image

Databricks announces new offering for Unity Catalog

By Zach Thompson
Mon 13 Jun 2022

Databricks has significantly expanded data governance capabilities on the lakehouse by unveiling data lineage for Unity Catalog.

Data lineage describes how data flows throughout an organisation, and the data and AI company’s newest feature will allow customers to gain visibility into areas such as where data in their lakehouse came from, who created it and when, and how it has been modified over time and how it is being used.

Databricks notes that businesses deal with large amounts of data from a range of sources, and understanding these different areas can be extremely difficult but having the understanding is crucial to ensuring trust and assessing risk.

Data lineage for Unity Catalog allows data teams to view every downstream consumer affected by data changes to gain a straightforward understanding of how severe the impact is and quickly notify the relevant stakeholder of changes.

This includes applications, dashboards, machine learning models and data sets.

In addition, the offering allows data consumers such as data scientists, data engineers and data analysts to be context-aware as they carry out their work, resulting in stronger outcomes.

Data stewards will also be able to see which data sets are no longer accessed or have become obsolete so they can retire unnecessary data, reducing risk and ensuring end users only use high-quality data.

These new capabilities in Unity Catalog offer organisations a complete view of the entire data lifecycle, so data leaders can understand how data is being collected, if it was updated, and the processes used.

“Governance capabilities such as data lineage are critical as we work to build the industry’s most robust lakehouse platform,” Databricks co-founder and chief technologist Matei Zaharia says.

“Without good data lineage, it is challenging to track the business and verification processes that data-driven organisations need to be successful.

“Our goal is to ensure our customers can focus on insights, and move toward proactive data management practices through a unified, transparent view of their entire data ecosystem.”

One of the key features of Unity Catalog is automated run-time lineage to capture all lineage generated in Databricks, enabling greater accuracy and efficiency than tagging it manually.

This information is captured for tables, views, and columns to give a granular picture of upstream and downstream data flows.

Lineage also works across all workloads supported by Databricks, including SQL, Python, R and Scala, meaning that all data personas can build on their tools with data intelligence and more substantial insights such as capturing lineage for entries like notebooks, workflows and dashboards.

Further, data lineage also assists businesses in meeting compliance standards, making it easier to track data flows that are subject to compliance regulations, including the General Data Protection Regulation, California Consumer Privacy Act, or Health Insurance Portability and Accountability Act.

Databricks says this aspect of data traceability is an important component of a modern data architecture that allows customers to meet their legal requirements.

Data lineage for Unity Catalog is now available for preview on AWS and Microsoft Azure.

Related stories
Top stories
Story image
Lucid Software
Lucid Software expands enterprise offerings with enhanced slack apps
Lucid Software has expanded its enterprise offerings with enhanced slack apps for its Lucidspark and Lucidchart technology.
Story image
Biometrics
Can biometrics help? 123% increase in Gen Zs scammed online
In the three years leading up to 2022, the number of Gen Zs who fell victim to online scams rose by 123%, according to Ping Identity.
Story image
consumers
Data protection: How SMEs can secure online consumers’ trust
Consumers believe how a company treats their data reflects how they treat their customers, according to a new survey.
Story image
Digital Transformation
Why it is imperative to keep up with technology advancements
Technology is constantly evolving, which is why digital transformations can often be complicated. What legacy items stay? Will the new and old work together? Or is there a whole new system that we need to introduce?
Story image
Artificial Intelligence
Is your chatbot bringing down the customer satisfaction score?
The top 10 reasons why chatbots are failing to meet customer expectations and what you must do to avoid that.
Story image
Tech job moves
Tech job moves - Fastly, INX, Kinly, SmartBear & Vectra AI
We round up all job appointments from July 29 - August 12, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
Data Protection
Advancing genomic sequencing and public health with digital infrastructures
Right before our eyes, we've witnessed the development of the COVID-19 vaccine in record time. An enormous achievement in an otherwise lengthy task that previously took, on average, 10-15 years.
Story image
Customer experience
Scope Australia selects NICE CXone platform to streamline operations
One of Australia’s largest providers of disability support services, has chosen the NICE CXone platform to help streamline its contact centre operations.
Story image
ExtraHop
Organisations exposing highly sensitive protocols to public internet
More than 60% of organisations expose remote control protocol SSH to the public internet, while 36% of organisations expose the insecure FTP protocol.
Story image
Marmalade
Exclusive: Marmalade empowers businesses to take control of their cash flow
Marmalade has been making waves in the invoicing space across Australia with its world-first invoice payments platform.
AWS Marketplace
Learn how security orchestration, automation, and response (SOAR) enhances your security strategy.
Link image
Story image
Cybersecurity
Datacom research explores reality of zero trust in A/NZ
Zero trust is fast emerging as global best practice in cybersecurity and local leaders are on board, with 83% considering it essential to security.
Story image
Privileged Access Management / PAM
The importance of stopping identity sprawl for cybersecurity
The 2021 Data Breach Investigations Report (DBIR) shows that 61% of all breaches involve malicious actors gaining unauthorised, privileged access to data by using a compromised credential. Unfortunately, it is often too late when the misuse of a credential is detected.
Story image
Network Management
Superloop helping Aussie K12 schools with latest offering
Superloop has launched CyberEdge, a new cybersecurity platform that will give K12 schools in Australia the tools to optimise, secure and manage their network.
Story image
Infrastructure
IBM expands Power10 server line for business modernisation
IBM has recently announced a significant expansion of its Power10 server line with the introduction of mid-range and scale-out systems.
Story image
SaaS
Exclusive: The Access Group shares the benefits of embracing SaaS
In today's rapidly changing working environments, efficiency and productivity are surefire ways to create business growth and success.
Story image
IDC
High level of Customer Identity & Access Management adoption
The study from Okta revealed that the pandemic has either accelerated or highlighted the need for digital-first strategies.
Story image
Enterprise Resource Planning / ERP
Why the right ERP (and partner) is crucial to an innovative and successful business
Enterprise Resource Planning (ERP) is a foundational step to ensuring a robust business model; here's why choosing the right one could be vital to ensuring long-term success and innovative results.
Story image
Indusface
Why enhancing bot protection for web and API endpoints matters
The trouble with bots is that they aren’t all bad. Unfortunately, this can make it challenging to detect malicious bots that find their way into your system and threaten your business.
Story image
Remote Working
Cybersecurity concerns higher than before the pandemic - report
BeyondTrust's new survey shows Australian organisations are more concerned about cyberattacks than they were before the COVID-19 pandemic.
Story image
Developers
Snyk announces plans to expand partner network in APJ
Recognising that partnerships are critical for growth, Snyk is building an entire partner ecosystem that will drive its expansion across APJ.
Story image
Gartner Magic Quadrant
Gartner names Lookout a Visionary in 2022 Magic Quadrant
Gartner has recognised Lookout as a Visionary in the 2022 Magic Quadrant for Security Service Edge (SSE) and one of the top three offerings in the 2022 Gartner Critical Capabilities for SSE report.
Story image
Digitalisation
Aus Steel selects SYSPRO Cloud ERP in digitalisation journey
Aus Steel has selected global ERP specialist SYSPRO to help the company drive efficiencies in its build process and improve the scheduling of its precision engineered frames.
Story image
Digital Transformation
Top tips for making your finance transformation program a resounding success
Planning to make 2023 the year you embark on a wholesale finance transformation program? It’s a move that will stand your enterprise in excellent stead as you navigate the complexities of the post-Covid business landscape.
Story image
Augmented Reality
TeamViewer remote access software integrated into RealWear Cloud
TeamViewer has announced a major expansion of its partnership with RealWear, a leading provider of assisted reality wearable solutions for frontline industrial workers. 
Story image
Compliance
Why security needs to shape your journey to the cloud
It's estimated that 80% of workloads could be in the cloud in the next few years. How can you make all that data secure?
Story image
Hybrid Cloud
The essential guide to digital transformation by SolarWinds
Digital transformation is a buzzword thrown around all the time by companies, but what does it actually mean and why is it important? SolarWinds breaks it down.
Story image
Data analytics
Pressure on orgs to up their data analytics game - study
A recent report from Sisense highlights data transmission, analysis, and risk management remain top concerns for data professionals in APAC.
Story image
Cybersecurity
Education sector seeing highest volumes of cyber attacks
When breaking down the numbers to education attacks by region in July 2022, A/NZ was the most heavily attacked.
Story image
Charity
SnapLogic teams up with meetmagic for charity and children
SnapLogic has announced its partnership with meetmagic, an online Australian platform that combines business and philanthropy.
Story image
Cloud Security
Tenable makes additions to Cloud Security portfolio
Tenable has announced additions to Tenable Cloud Security that represent the next step in assessing threats related to cloud vulnerabilities.
Story image
Microsoft
Microsoft, NSW partnership to accelerate digital transformation
The renewed partnership is designed to maximise the value of Microsoft solutions to various NSW Government agencies.
Story image
Data breach
Weak breach data disclosure laws for IP theft leaves vital Australian industries vulnerable
Infoblox has seen unprecedented levels of demand for cyber protection from companies outside of the Critical Infrastructure Act who have discovered security events and those who fear falling victim to IP theft.
Story image
VMware
Latest VMware threat report reveals truth about deepfakes
"Cyber criminals have evolved. Their new goal is to use deepfake technology to compromise organisations and gain access to their environment."
Story image
Ransomware
Big business in cryptocurrencies and cybercrime
As of June this year, about a quarter (26%) of Australians considered cryptocurrency as a good investment, and over 36% have at one point considered buying cryptocurrency as an investment.
Story image
Application Performance Monitoring / APM
New Relic integrates offering with Atlassian’s Jira Software
New Relic has integrated errors inbox with Jira Software to allow developers to easily access and set up complete stack error tracking and software performance monitoring from within the tool.
Story image
JLL
Investment in APAC cold storage to reach $5 in next decade
Investment in Asia Pacific’s cold storage market is expected to grow fivefold in the next decade, according to JLL.
Story image
Microsoft
Avast reveals zero-day exploits targeting Chrome and Microsoft
Avast, released its Q2/2022 Threat Report today, revealing a significant increase in global ransomware attacks, up 24% from Q1/2022.
Story image
Digital Transformation
Common challenges in the evolving digital world, and how to overcome them
Five technology challenges that will help highlight potential pain points and suggest how to circumvent or overcome them on the path to a smooth digital transformation.
Story image
Cyber attacks
Dramatic uptick in threat activity with exploits growing nearly 150%
"While it’s not a surprise given increased attack opportunities like remote work, it’s still a worrying development and one we cannot ignore."
Story image
Artificial Intelligence
Gartner unveils key emerging tech to watch in 2022
"Such technologies present greater risks for deployment, but potentially greater benefits for early adopters," says Gartner.
Wiise
Discover why a localised cloud ERP is the way to go for Australian businesses.
Link image
Story image
Cybersecurity
Australian IT security concerns higher than before pandemic
Australian organisations are more concerned about cyberattacks than they were prior to the COVID-19 pandemic, according to a new survey.