Q&A: SnapLogic CTO makes the case for investment in machine learning

SnapLogic field chief technology officer Brad Drysdale discusses the roadblocks to a successful machine learning implementation and the ways they can overcome them.

The ML process can seem daunting for many organisations due to its unpredictable and experimental nature.

When IT teams initially start going through the data to deploy ML algorithms, most probably won't know the type of data that is required. A lot of exploration must be done before IT decision-makers have an idea of what data will be useful, and which ML algorithms will work best to solve a particular problem.

Other technical challenges include being able to automate data access. When organisations have formulated clear policies that allow easy access to real-time data, they need to consider how to set up a channel or a pipeline to access the data.

Organisations also need to ensure that there is availability of constant real-time data. ML models should not be trained on a single fixed set of data, so organisations need to set them up so that they can retrain their models to adapt to the changing behaviour of the data and the systems that they're working with.

Additionally, there is a significant talent shortage. While the number of qualified data scientists is growing, there are only a small number of data scientists that we can produce each year.

There are concerns about access to certain types of data, particularly when you have different groups of employees or other stakeholders coming in at different times to work on projects. So organisations should consider filtering out any potentially sensitive information from the data first so that the rest can be used to deploy ML algorithms.

Another issue to overcome is how to fulfil the demand for data scientists. While it's great to see that more data scientists are emerging in the workplace, a lot of time still goes into training them, so supply is still not keeping up with the rising demand.

However, more people who have been trained in other areas, such as senior business analysts and software engineers, are increasingly expanding their knowledge of data science and ML, which can help bridge that gap.

Additionally, organisations will have IT business analysts who have experience with handling databases, even if they're not programmers, they're still analytically minded, so they can take advantage of ML through self-training too.

All of these developments are following a positive trend, as tools and platforms are beginning to allow a broader range of users to engage with ML and make it useful for them.

I see two main misconceptions about ML that relate to its complexity and capabilities. First, businesses often think that ML is very complex and requires PhD's to get value out of an implementation.

Many relatively simple ML algorithms can be applied to business data to provide predictions or classifications.  On the other hand, there is an unrealistic conception that ML is a panacea to all business problems. The sweet spot is understanding the realistic capabilities of different, well-understood ML algorithms and match them with the right business data to derive real value.

Businesses of all sizes should be working with universities or creating apprenticeship programmes to bring in fresh talent.

For example, the Computer Science department at the University of San Francisco offers a project course for both undergraduates and graduates for one term, where they typically do small project work with an industry sponsor. This not only allows students to work with a variety of different companies, but it also makes the recruitment process for businesses far easier.

Another way to help bridge the skills gap is through investment in technology that will lift the burden off IT professionals.

Low-code/no-code platforms are a prime example of this, as they can enable data tasks to be undertaken by people outside of the IT department working in the lines of business.

Currently, having the right data to deploy ML algorithms is an incredibly time-consuming process. A lot of time is spent trying to get access to and sift through vast volumes of disorganised data with manual coding, leaving IT professionals little time to focus on higher-value tasks.

By investing in the right low-code/no-code technology, businesses can easily automate data pipelines, giving all departments regular access to real-time data, and make ML processes as seamless as possible with little to no coding required.

Businesses can look at how investment in emerging technologies will benefit them in two ways – either to get ahead of their competition or to prevent their organisation from becoming obsolete.

Businesses need to follow and even move ahead of technology trends to not only offer a better experience and more effective utilisation of their resources but continue to provide the services that their users and customers expect.

Eventually, all organisations will need to adopt ML simply because that will become an expectation, so that applications and services can better anticipate what their users are attempting to do and to provide recommendations or predictions that enable them to achieve their goals more rapidly.

This doesn't just apply to just investment in the technologies, but the skills training as well. Businesses need to ensure that people are trained well to utilise these technologies, but also continue to help expand their skill set to harness the full potential of these emerging technologies.