How Symfa transformed a Fortune 500 insurer’s data platform with cloud migration
A Fortune 500 insurance carrier recently embarked on an ambitious journey to transition its on-prem big data ETL platform to the cloud. This initiative was spearheaded by the client’s largest Azure service partner. The promise? Enhanced speed and scalability for data transfer, mapping, and verification. Ivan Sokolov, BI and ETL developer at Symfa, explained its role in the transition.
The migration aimed to enable faster data processing compared to on-prem solutions. For example, loading test data to mimic production-phase requirements demonstrated remarkable speed, reducing tasks that once took a week to just 12 hours.
While the cloud infrastructure delivered on speed, the cost implications for ramping up server capacity were significant, highlighting a trade-off between efficiency and expense.
Initial challenges with cloud infrastructure
The transition was not without challenges. The cloud vendor’s initial proof of concept (PoC) was promising, but scaling the solution to process 500 insurance businesses revealed limitations. Delays, miscommunication, and infrastructure shortcomings resulted in an extended development timeline.
Throughout late 2023 and into 2024, collaborative efforts between Symfa, the client, and the cloud vendor were necessary to refine and optimise the system.
The arrival of new project leaders brought fresh scrutiny to the cloud-based solution. While technologically advanced, the platform was criticised for being cumbersome and lacking flexibility.
For instance, running specific data processes required executing the entire pipeline, consuming unnecessary resources. Symfa has since been involved in addressing these inefficiencies, aiming to enhance usability and reduce costs.
Why move to the cloud despite the hurdles?
Despite these issues, the shift to the cloud offers undeniable benefits. The scalability and speed of Azure’s infrastructure enable processing volumes that would be impractical on local servers.
The flexibility of the cloud environment also supports advanced data modelling and reconciliation, laying the groundwork for future innovation.
The project underscored important lessons for organisations considering similar migrations. Effective vendor selection, clear communication, and domain expertise are critical to success.
Copying on-prem solutions directly into the cloud is rarely feasible; instead, cloud-specific architectures must align with the organisation’s business logic and processes.
Delivering value to the client
Two and a half years into the initiative, the platform has achieved significant milestones:
- 350+ businesses processed, with 120 more to be added.
- 28 years of data reconciled and standardised.
- Enhanced reporting and analytics capabilities enabled by a cloud-based data warehouse.
The platform is set to go live in August, combining the Preprocessor’s local capabilities with the cloud’s advanced features. Once fully operational, it is expected to deliver long-term value across various business functions, from reconciliation to forecasting.
