Over time, the typical business acquires more and more applications. Some will be bespoke, some off the shelf, some cloud hosted SaaS tools and some in the data centre.
Over time, business requirements will arise where these applications need to be integrated. For instance, every time an order is placed on our eCommerce website, a customer record should be created in the CRM system, the marketing system, and the ERP system.
The problem with this from a data and analytics perspective is that many of these integrations are batch based. For instance, every 24 hours or so, information is extracted from one system, uploaded to another as a data file, and imported into the destination. This is referred to as extract, transform and load (ETL).
Of course businesses realise the downsides of this, and have spent years implementing service buses and point to point APIs for more immediate data exchange. However, the fact remains that batch data exchange is still the predominant method of moving data between systems in an enterprise setting.
The biggest implication of this is the impact on the customer and employee experience. For instance, most of us will have had the following experiences when interacting with our banks:
- Online banking taking hours to update after a transaction has occured;
- Delays before a requested statement is sent out or a letter issued;
- Delays in opening accounts and being onboarded as a customer;
- Being passed between different call centre departments and having to explain your request over again.
Slow and unreliable batch data exchange is likely to be at the heart of these frustrating customer experiences.
Now compare this to modern neo banks such as Monzo or Starling, where push notifications arrive on your phone the instant you swipe your debit card in a store.
The immediate push notification shows that the payment event is instantly flowing through the system from front to back. The confirmation is then issued to immediately enhance the user experience and make a measurable impact against fraud. I think the fact that startup banks can do this seemingly simple thing should be concerning for the incumbents as it demonstrates that they are built on real time data foundations which can power amazing customer experiences.
To build these user experiences, companies need to implement an event driven architecture and move away from batch processing towards real time streaming and data exchange.
This can be a complex evolution, but I believe that it is absolutely at the heart of the transformation that enterprises need to make in order to become more data driven. If we are building on batch integration, it is very hard to build personalised and responsive customer experiences, and hard to introduce intelligent automation and proactive use of data into business processes.