With the outbreak of Coronavirus in South Africa, a leading retailer – which also provides financial products and services – wanted to offer customers payment relief options.
However, it still follows a mostly waterfall approach when developing and deploying new services. This was negatively impacting business processing workflows, especially when it came to transferring and processing large transaction files between it and its joint venture partner.
The retailer’s legacy approach to development meant that it would take months before it could offer its customers financial relief. By then, it might have lost a lot of money to defaulting customers who have fallen on tough financial times.
Business challenge
To be able to respond quickly to COVID-19 and future challenges, the retailer needed easy, reliable access to customer and other data. Under the existing framework, any time the company needed to access data, it had to wait for its holding company to develop an Application Programming Interface (API). But this time, there was no time to waste.
So, it partnered with iOCO to build a data lake on Amazon Web Services (AWS), where all relevant data could be securely stored and easily accessed by the DevOps team when deploying new features.
We followed an Extract, Transform, Load (ETL) approach when moving the data into the cloud, using AWS Glue – a serverless data integration service – and AWS Lambdas, which split large files into smaller files and shifted them into the cloud using AWS’s SFTP Transfer Family.
This approach not only reduced the retailer’s reliance on its holding company to access data as and when needed, but having data stored in the cloud also eliminated the need to integrate into the holding company’s’ environment through APIs.
With data instantly accessible in the AWS data lake, our client was empowered to offer value-added services, like payment relief during COVID-19.
Solution
Management had determined that implementing an AWS cloud-based file transfer and storage solution was the way to go.
However, the retailer did not have the in-house experience or skills to design and implement this solution. But through its partnership with iOCO, it could access valuable expertise, knowledge, and experience in next-generation digital infrastructure.
iOCO assisted with:
- Streamlining business processes by adopting an agile design and implementation of AWS cloud-based technologies
- Driving new business capabilities through the ability to rapidly scale the AWS capacity
- Empowering the business and its customers by designing and implementing the right cloud-based solution.
Following a DevOps approach and using the AWS File Transfer Family, we provided fully managed support for file transfers directly into and out of AWS S3. With support for Secure File Transfer Protocol (SFTP), File Transfer Protocol over SSL (FTPS), and File Transfer Protocol (FTP), the AWS Transfer Family helps the retailer’s financial services arm to seamlessly migrate its workflows to AWS by integrating with existing authentication systems and providing DNS routing.
As soon as files are dropped into AWS S3, they’re immediately and automatically added to a transfer queue, which is deployed using AWS CloudFormation. AWS Lambdas then processes the files and stores them in the data lake, for easy, secure access.
We also used AWS CloudWatch and AWS CloudFormation script templates to create data transfer pipelines and dashboards, to provide complete visibility and active monitoring of the environment, and to alert the DevOps team to any issues.
When an issue is detected, AWS CloudTrail and X-Rays provide insight into where the bottlenecks or errors are occurring. The system then either automatically runs pre-created tools and scripts to fix the problem, or notifies a team member to take action. Issues can either be put into a backlog or, if urgent, into a Sprint to create hot fixes.
AWS Athena was used to run analytics on the data and to enable the retailer to in future create AI models for applications such as fraud detection and compliance reporting.
Finally, active monitoring solutions were implemented to determine when to scale the compute power and data storage capacity up or down.
Practically, queues, Lambda, and step functions were used to create this technology stack.
Business outcomes
With an AWS-hosted data lake, the retailer no longer has to wait for its holding company to develop APIs in order to access critical data.
The HTTPS-based front-end and the APIs linking it to the rest of the architecture provide a responsive interface that employees can use to access customer records and action requests.
Benefits include:
- Improved customer management
- The ability to provide customers with payment relief measures
- Reduced monthly premiums through improved customer management processes
- Reduced financial losses through the effective management of defaulting customers
- The effective transfer of data between internal and external software systems.
Additionally, the technological advantages include:
- DevOps can control their own way of working through the implementation of agile methodologies and cloud-native technology.
- ETL processes were decoupled as much as possible to suit the robustness of the AWS architecture.
- The development, deployment, and maintenance of the ETL pipeline and data lake processes is done via AWS CloudFormation templates where it’s easily implemented and corrected if necessary.
In a fast-paced operating environment, businesses need to think on their feet if they are to remain competitive. With a robust AWS environment, our client now has this capability and can also add more capabilities as and when the business needs them.