As noted in my previous article, automation throughout the analytics pipeline is the backbone of a system capable of real-time, hyper-contextual information and analysis.
But beyond enabling faster, smarter acquisition-to-insight, automation holds the promise of even greater things: it can extend the intelligent data pipeline right into the realm of actions.
As we saw in the evolution of DevOps, technology alone cannot change the entire environment. People and processes need to change too.
As DataOps and automated data analytics pipelines change the face of data analytics, the people and processes within this environment will have to change.
To support this, automation must extend into the people and processes component of data analytics, to trigger alerts and actual actions without the need for manual intervention.
Driving dynamic actions
Insights that don’t just inform decisions, but also trigger action are what businesses today need if they are to stay ahead of the competition.
Automated actions − or application automation − takes advantage of modern web application openness. Because most ERP systems or line of business systems are available as SaaS or via modern web browser bases, APIs allow us to link easily to those systems.
This extends the automation of data management and analysis right through to prodding someone to take action in terms of workflow. It also presents the opportunity to bypass manual intervention completely, and simply make an automated change to the workflow where necessary.
Waiting a day for fresh data is seldom appropriate in the modern business.
In practice, this could mean that where in the past a regional sales manager might review the sales data and identify a problem, then pick up the phone to call a region and order certain changes, it now becomes possible to automate these steps.
The regional sales manager can simply put a rule in place stating that when a certain condition has been met, a set of tasks will automatically be assigned to the sales team in Salesforce. In this way, we cut out a whole human workflow process and automate the creation of tasks or tickets.
By triggering immediate action from real-time, up-to-date and trusted data, we can accelerate business value across the entire data and analytics supply chain.
The benefit of automation is that it changes and enables all the personas in the data pipeline.
Application automation cuts down the time needed for programming of repetitive back-office tasks and gives data teams more time to deliver compelling real-time analytics.
In this new automated pipeline, the business analyst’s world also looks different: he still sets meetings with data and business teams, but now has a catalogue of trusted, real-time data at hand, so each meeting is much more fruitful.
Trusted, real-time data is crucial in these environments. If an organisational culture has successfully embraced automation, it is important to look carefully at the data sources and catalogues to ensure continuous governance takes place throughout the data delivery lifecycle, and that the entire pipeline enables real-time data.
Waiting a day for fresh data is seldom appropriate in the modern business. Without intelligent automation, data teams keep getting dragged back because of changes that happen on source systems, which slows delivery and access to real-time insights on the business side.
Challenges of SaaS-related data
A Statista May 2021 report shows SaaS applications accounted for 70% of the software used globally in 2020, with an average of 80 SaaS applications per organisation, up from an estimated eight in 2015.
The explosion in SaaS-related data, along with the worldwide push to become more data-driven that was accelerated by COVID-19, has dramatically intensified IT and data team workloads.
The significant hours and manual tasks required to integrate the universe of hundreds of potential SaaS applications into a coherent analytics process has limited the ability of internal teams to deliver on the promise of real-time data-driven decision-making through active intelligence.
This is where active intelligence comes to the fore: it is based on continuous intelligence from real-time, up-to-date information that establishes an intelligent data analytics pipeline with dynamic business content and logic.
More precisely, application automation drives dynamic actions delivering the last mile in active intelligence through a no-code, cloud native user interface that easily automates and streamlines workflows between SaaS applications and the cloud.
Moving toward the end-to-end automated data environment
DataOps to streamline data delivery is still emerging and is just one component of this new automated end-to-end data environment. In many organisations, data teams have a concept of modernising the DataOps pipeline, but they are stuck in the detail of just getting data through to the data warehouse or data lake.
We still see armies of extract, transform and load developers challenged in enabling the level of data analysis and insight organisations need today. This is costing organisations time and money.
As we have seen with most IT problems, as soon as you automate processes, people quickly realise it’s a better way to do things. It doesn’t reduce jobs, it clears IT backlogs.
