IT Brief Australia logo
Technology news for Australia's largest enterprises
Story image

Exclusive interview: What happens when artificial intelligence is racist?

By Julia Gabel
Fri 1 Dec 2017
FYI, this story is more than a year old

Dr. Rob Walker is the vice president of decision management at Pegasystems, and also an artificial intelligence (AI) expert. 

This exclusive interview delves deep into two strands of artificial intelligence - opaque AI and transparent AI. Walker discusses the differences between the two, situations where one is more appropriate than the other, and what happens when we the use of AI becomes unethical. 

Opaque AI vs. transparent AI – what’s the difference?

Opaque AI uses algorithms that are not explainable to humans.  

Not just because, in some cases, it’s like ‘alien’ thinking, but also because of the sheer complexity of the resulting (prediction, classification, or decision models).

Examples of opaque models are the multi-layered neural networks used for deep learning, which roughly mimics the inner workings of the human brain; or some of the results of genetic algorithms, a technique that evolves solutions to problems using variants of a simulated survival of the fittest context.

In contrast, transparent AI relies on techniques that can be successfully explained.

Examples are modestly sized scorecards or decision trees that explicitly show how they use data to come to a prediction, classification, or decision.

However, insisting on transparency is a severe constraint on the algorithms used and, as a consequence, opaque AI can be expected to be more powerful, for instance, make better predictions.

Opaque versus transparent is not a delineation between good and bad (after all, human thinking is often quite opaque as well), it’s a matter of organisations choosing where understanding trumps performance and where it’s the other way around.

Artificial intelligence in the enterprise

The use of AI in business, especially opaque algorithms like deep learning, is only now starting.

Such algorithms have been in use for a while, but were mostly used in research and applied to cognitive categories involving games (go, poker), recognition (images, speech, emotions), language (translation, generation), robotics (autonomous vehicles, robots), and a few others.

But with the rise of (big) data and far-reaching automation, AI categories like predictive analytics have ventured into the opaque area.

Better predictions are worth a lot of money: approve loans with a lower probability of default, diagnose diseases more accurately, make more relevant offers.

So, the question then becomes, are organisations (and the general public) prepared to trade more transparency for better decisions?

It’s safe to say, in general, that the opaque/transparent discussion will be had first in industries that are heavily regulated.

For instance, a bank that could potentially use (opaque) deep learning to better select those applicants that are likely to repay a loan (plus interest), may find themselves struggling explaining their risk exposure.

After all, their opaque algorithms may be more effective, but they can’t explain why certain applicants are accepted and others rejected.

If an opaque algorithm tells a bank that customer X is a 0.9 possibility of getting into default that’s one thing; if it can’t explain why, that’s a very different thing.

The dark side of AI

Within the scope of customer engagement and CRM, the dark side of AI would manifest itself by making decisions (to yes/no grant a loan, make a certain offer, increase a priority, invest in retention) using sensitive customer traits (gender, religion, race, age, etc.) without explaining that to human supervisors.

Therefore, customer strategies can go into effect with profound negative, perhaps even illegal, consequences.

Because opaque AI, even when it’s better at its task, cannot explain why or how it comes to a decision, the risk is that those decisions do not comply with legislation or even corporate policies.

What if an algorithm is racist?

In an opaque algorithm, it may be hard to find that bias by inspecting the model. A complicated factor is that modern algorithms (opaque and transparent) can easily make undesirable inferences from the vast amount of (big) data they are given.

In other words, to be racist, an AI doesn’t require a customer’s race to be held in any database.

It can infer race (or gender, age, sexual orientation, intelligence, mental health, etc.) through innocent looking attributes and subsequently use it for a selection or decision.

If the algorithm inferring any of those traits is opaque to boot, it’s impossible to know how it’s using them to come to a decision. It’s a decision made by a black box that can’t explain itself.

Now, even though that sounds very undesirable, humans are often black boxes as well.

They will use words like intuition or gut feel to describe opaque thinking, or, more often, are not even aware their ideas or insights are not ‘logical’.

So – which is better?

Obvious scenarios where opacity should be considered more than just carefully are those in which AI make decisions (or even just recommendations) where such decisions have a meaningful impact, especially if they’re incorrect.

Sending someone an irrelevant offer may not be the end of the world, but incorrectly rejecting a mortgage or job application is something where a black box is likely not acceptable.

In the EU, the General Data Protection Regulation (GDPR) will got into effect in May 2018, affecting not just EU companies but any company with EU customers.

Among many other things, GDPR requires an explanation for any automated decision that has ‘legal significance’.

Lawsuits will soon define the legal boundaries of ‘significance’ but it’s a safe bet that under GDPR, opaque algorithms are a liability where such decisions are concerned.

The balance between machine and human decision making in the enterprise

Tools like Decision Management support very explicit collaboration between AI and human decision making.

The AI provides, either through transparent algorithms, opaque algorithms, or a mix, higher order input for a human decision maker.

For instance, the AI would determine whether a marketing offer is relevant, but a human can decide how much irrelevance is acceptable to the brand; are they going to spam a lot of prospects to harvest a higher number of responders, or is relevance important to them so that their customers continue to read their messages and feel the company knows them?

Alternatively, AI takes over all of this and maximises ROMI (return on marketing investment), but humans responsible for the brand promise will decide whether selling is appropriate in the first place, versus risk management, retention, or service.

These are examples at the core of the organisation, but in the front-office AI will often recommend a course of action to a human agent.

However, one should not be naïve about such collaboration.

If the AI is clearly superior, human agents will be expected to simply do as recommended (or just do it to make their targets); if the AI is not, it will be ignored.

In the front-office humans are probably still marginally better at detecting emotions or non-obvious intents.

While that’s still the case, collaboration between artificial and original intelligence may still beat either.

And of course, not every human is comfortable with chatting with or talking to something evidently artificial; or, if not evident, feel tricked into thinking they’re conversing with a human agent.

In the near future, that distinction will become hard to make while at the same time customers will likely become more tolerant, especially if an AI delivers superior advice, has full context, and is more patient.

The best collaboration is where AI and human judgment each play to their own strengths.

AI is undisputedly better in finding complex patterns in vast amounts of data and thus often supplies higher quality probabilities and predictions.

Humans can take that automated insight and apply strategy. That boundary, too, will shift over time, and shift fast, but for now it’s a workable and effective balance.

Unchecked AI: The ethical sign-off

Opaque AI is less safe than transparent AI because it’s a black box.

Its decisions, meanwhile, can be superior. Therefore, it’s not a straightforward matter of banning opaque algorithms.

Superior decisions, even by a black box, can increase profits, reduce risk, or save lives (in a clinical setting, for instance).

What’s required are two things:

1) a setting in any decisioning system relying on AI to restrict the use of opaque algorithms to areas of the business and to use cases where an inability to explain is not critical.

And 2), where allowed, to test for ‘rogue’ biases in the decisions made.

Because there’s no way, in opaque AI, to reverse-engineer the internal decision-making process, we can only judge the black box by what it does (versus how it does it).

That’s where the ethical sign-off comes into play. The AI is asked to make decisions about a selection of customers/prospects for who certain sensitive traits are known (gender, age, etc.) and those decisions, rather than the underlying algorithm, are then analyzed for undesirable biases.

Related stories
Top stories
Story image
Hawaiki Cable
BW Digital completes acquisition of Hawaiki Submarine Cable
BW Digital has completed its full acquisition of Hawaiki Submarine Cable, with all applicable regulatory filings and approvals now received.
Story image
Adyen expands partnership with Afterpay as BNPL payments increase
Adyen has expanded its partnership with AfterPay allowing more of Adyen’s merchants in more countries worldwide to use the BNPL provider.
Story image
Ivanti and Lookout bring zero trust security to hybrid work
Ivanti and Lookout have joined forces to help organisations accelerate cloud adoption and mature their zero trust security posture in the everywhere workplace.
Story image
Atturra partners with Focus HQ to support Aus organisations
Atturra has executed a partnership agreement with Focus HQ, to resell and support the company's Australian developed SaaS-based portfolio management platform.
Story image
Aussie data & analytic execs not confident in data strategy
Less than half of Australian data and analytics leaders are confident in their data strategy as siloes and lack of culture prevent innovation.
Story image
Employee Retention
Company values increasingly important in post-pandemic world
New research released today reveals that company values are the new currency when it comes to employee retention in a post-covid economy.
Story image
Avaya OneCloud sees 118% ARR growth for second quarter 2022
Avaya Holdings has reported $750 million annual recurring revenue (ARR) for its OneCloud offering, up 21% sequentially and 118% from the same period last year.
Story image
Tech innovation crucial to growth, but barriers remain
Businesses in the A/NZ region believe tech innovation is crucial to drive future growth, but 76% say they’re being held back by complex data architecture.
Story image
Hands-on review: STM laptop bags
The advent of hybrid working has meant we need laptop bags. We got our hands on two of the most popular laptop bags from STM.
Story image
Digital Transformation
Why enterprise records management should be part of any digital transformation strategy
Modern organisations create and rely upon an enormous volume of content, and digital records make up a significant proportion of that content.
Story image
Tech job moves
Tech job moves - Datacom, Micro Focus, SnapLogic and VMware
We round up all job appointments from May 6-12, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
Power / Energy
SmartCIC report reveals top five 5G carriers in the world
The Global Cellular Performance Survey also found that 5G networks are delivering high download speeds but lagging in upload speeds.
Story image
Amazon Web Services / AWS
Databricks strengthens AWS partnership with new Lakehouse offering
Customers will experience faster onboarding and unified account administration to make building a Databricks Lakehouse on AWS easier.
Story image
SmartCIC, BICS partner to expand wireless service options
SmarCIC has partnered with BICS to increase choice for organisations using fixed wireless services, expanding existing carrier relationships for its CELLSMART division.
Story image
Grasping the opportunity to rethink the metrics of a sustainable data centre
A data centre traditionally has two distinct operations teams: the Facility Operations team, and the IT Operations team. Collaboration between them is the key to defining, measuring, and delivering long-term efficiency and sustainability improvements.
Story image
Could your Excel practices be harming your business?
While Excel has been the de-facto standard for budgeting, planning, and forecasting, is it alone, enough to support organisations in the global marketplace that’s facing rapid changes due to digital transformation?
Story image
HINDSITE wins Aerospace Xelerated Pitch Challenge with solution to support Boeing
Brisbane-based startup HINDSITE was the winner of the first ever Pitch Challenge organised by Aerospace Xelerated in partnership with Queensland XR Hub. 
Story image
Employee Experience
Zendesk launches customer service and employee experience offering
"Zendesk is helping businesses embrace this new generation of conversational customer relationship management and turn customer service into growth.”
Story image
Google reveals new safety and security measures for users
Google's new measures include automatic two step verification, virtual cards and making it easier to remove contact information on Google Search results.
Story image
BeyondTrust integrates Password Safe solution with SailPoint
BeyondTrust has announced the integration of BeyondTrust Password Safe with SailPoint identity security offerings.
Story image
Absolute Software expands Secure Access product offering
Absolute Software is enhancing its Secure Access product portfolio, enabling minimised risk exposure and optimised user experiences in the hybrid working environment.
Find out how a behavioural analytics-driven approach can transform security operations with the new Exabeam commissioned Forrester study.
Link image
Story image
Sift shares crucial advice for preventing serious ATO breaches
Are you or your business struggling with Account Takeover Fraud (ATO)? One of the latest ebooks from Sift can provide readers with the tools and expertise to help launch them into the new era of account security.
Story image
Power at the edge: the role of data centers in sustainability
The Singaporean moratorium on new data center projects was recently lifted, with one of the conditions being an increased focus on power efficiency and sustainability.
For every 10PB of storage run on HyperDrive vs. comparable alternatives, an estimated 6,656 tonnes of CO₂ are saved by reduced energy consumption alone over its lifespan. That’s the equivalent of taking nearly 1,500 cars off the road for a year.
Link image
Story image
Kodari Securities (KOSEC)
NFT trends and opportunities: expert reveals all
The NFT market is growing at an exponential rate, with unprecedented liquidity. Here we explore how businesses can profit.
Story image
HackerOne launches Attack Resistance Management solution
HackerOne has launched Attack Resistance Management - a new category of security solution that targets the root causes of the attack resistance gap. 
Story image
Prophecy International migrates VMware environment with Oracle Cloud Solution
The Adelaide-based global provider is using the solution to eliminate the need to re-write applications, therefore allowing the company to enhance its business operations.
Story image
Jabra investigates what makes an ideal hybrid work model
“The way we work has changed forever and the current state of knowledge work requires access to digital platforms and technologies to be successful."
Story image
Ingram Micro Cloud adds Bitdefender solutions to marketplace
Ingram Micro Cloud has announced the expanded availability of Bitdefender solutions on the Ingram Micro Cloud Marketplace.
Story image
Application Security
What are the DDoS attack trend predictions for 2022?
Mitigation and recovery are vital to ensuring brand reputation remains solid in the face of a Distributed Denial of Service (DDoS) attack and that business growth and innovation can continue.
Story image
Zerto unveils updates to ransomware recovery capabilities
"Organisations face increased risks from the volume and sophistication of ransomware attacks prevalent today."
Threat actors are exploiting weaknesses in interconnected IT/OT ecosystems. Darktrace illuminates your entire business and takes targeted action to stop emerging attacks.
Link image
Story image
Energy storage demand momentum continues, says BYD
BYD has announced an expansion of its production capacities and will deliver 250,000 units of its energy storage system, BYD Battery-Box Premium.
Story image
Avaya expands Microsoft partnership to deliver OneCloud on Azure
The joint technology and go to market agreement will help customers accelerate their digital transformation initiatives in the cloud.
Story image
Cybersecurity starts with education
In 2021, 80% of Australian organisations responding to the Sophos State of Ransomware study reported being hit by ransomware. 
Story image
Digital Transformation
Unlocking the next digital frontier for educational institutions
Understanding where to invest in technology can be challenging for education institutions, especially after the COVID-19 disruptions.
Story image
Wasabi Technologies
Wasabi opens new cloud storage in Australia with Equinix
Wasabi Technologies has opened a new hot cloud storage region in Sydney, Australia, using Equinix services. This is the company's 12th global storage region.
Story image
Data Center
Preventing downtime costs and damage with Distributed Infrastructure Management
Distributed Infrastructure Management (DIM) can often be a lifeline for many enterprises that work with highly critical ICT infrastructure and power sources.
Story image
Siemens showcases new automated solutions for data centers
Siemens has implemented new automated solutions and AI in the Baltic region's largest data center, providing insight into the future of data center management.
Story image
Power / Energy
Keysight Technologies introduces new next-gen DPT solution
Keysight Technologies has announced its new next-generation Double-Pulse Tester (DPT) with the PD1550A Advanced Dynamic Power Device Analyser.
Story image
Digital Transformation
Physical security systems guide the hybrid workplace to new heights
Organisations are reviewing how data gathered from their physical security systems can optimise, protect and enhance their business operations in unique ways.
Story image
Telstra and Silver Trak Digital delivers 5G to the cinemas
Telstra and Silver Trak Digital say they've launched Australia's fastest and most secure delivery of content over 5G for cinemas.
Story image
A10 Networks finds over 15 million DDoS weapons in 2021
A10 Networks notes that in the 2H 2021 reporting period, its security research team tracked more than 15.4 million Distributed Denial-of-Service (DDoS) weapons.