Monday, December 2, 2013

Why In-Memory Data Grids Are A Must-Have For Enterprise Integration

In-Memory Data Grid computing (IMDG) is increasingly used by enterprises to deliver fast, robust compute capability across the organisation. Having previously only been used by large companies such as financial institutions and internet giants, the cost of the technology has now reduced to the point that it should underpin your business application integration. Why? IMDGs offer seamless clustering and failover capabilities, out of the box; guaranteed message delivery and error recovery; and elastic scalability with parallel processing, all of which mean your application will run faster and more reliably.


How they work

In-memory data grid technology essentially uses memory as a storage area while a task is being processed, rather than writing it to a disk. This capability builds on the traditional in-memory database technology that has been used for many years, not least in SMS text messages.

The data grid takes the technology a step further, by distributing the data and storing it in servers, whereeach server works in an active mode. The data model is object-oriented and non-relational, and needs to be scalable in order to hold the necessary data in memory while also processing it.

The three key challenges in creating commercially viable IMDG systems were reliability, capacity and cost: cost has been met by the inexorable decline in memory prices; the IMDG’s elastic scalability and distributed architecture handles the capacity challenge; and the system uses replication to address reliability. Working entirely in the memory means that a true IMDG is also faster than a cache system. How much faster? Consider that an SSD with a SATA interface has a data transfer rate of approximately 500MB/s, a HDD around 150MB/s, and even the fastest SSD with a PCI Express connection can only reach 3,000MB/s, while RAM can reach 20,000MB/s. Since the data transfer is a key bottleneck, reducing this with IMDGs can greatly improve processing time. Added to the very low latency transaction processing is high availability capability from parallelisation, and the ability to work directly with document objects.


Why it’s a must-have

I wrote recently about reasons why many integration projects fail before they are even complete, but equally problematic is the number of projects that just don’t work as well as expected once they have gone live. Many of those are caused by one problem: messages aren’t delivered between systems. Here’s why In-Memory Data Grids are an ideal way to guarantee message delivery:


1) Transmission errors are a thing of the past

If a message needs to be delivered from System A to System B, it’s fairly simple to deal with System B being down: System A receives a failure notice and tries again later. Granted, you need to determine how that message will be cached for re-sending, but it’s relatively unproblematic. On the other hand, if System A fails during message transmission, the message is lost.

However, if the sending or receiving system in a grid crashes, part of the grid sends the message to the memory and retains it until the systems are fixed and ready to re-send. The grid’s management facilities allow it to know when the systems are ready. Clustering and failover are available out of the box, so the system has built-in reliability.



2) Elastic scalability and failover respond to your changing environment

It’s a truth of IT systems that scale changes over time, and this can result in the original data pipe no longer being large enough to serve the system demand. I worked with a customer which had a very long-running and successful integration between their ERP and CRM systems, which worked perfectly until someone added a Google Maps image to every transaction thereby greatly increasing the transaction size, whereupon the machine ran out of memory and crashed. IMDGs can automatically cache transmissions that cannot be sent, and grant extra resources to deal with sudden increases in demand. Integration projects are not static, so you need a dynamic solution (“pipe”).

High availability is needed to meet today’s user and business expectations for fast, consistent application performance even during heavy traffic periods and in usage spikes. This is met with elastic scalability, allowing more system resources to be made available when necessary, and parallel processing to accomplish tasks faster and reliably.


3) Increasing transaction size and frequency

Business and consumer transactions are becoming larger and more frequent, and we are increasingly demanding real-time analytics of activities. As a result, read/write operations are becoming bottlenecks, so IMDGs’ ability to dissociate data from the back end systems for processing and retention, along with asynchronous write behind capabilities where the application can continue to work while the changes are filled in afterwards) is an ideal solution.

4) The move from centralised to distributed architectures

Applications are becoming ever more integrated into processes and that requires them to talk to increasingly complex and distributed environments, where data could be on-premise or in cloud services; in legacy applications or up to date ones. Abstracting data from the applications provides the flexibility to work seamlessly across environments, as well as helping to easily upgrade or deploy applications, and go to market faster.

As users, we are demanding access to more data (both more varied and larger quantities) and a larger range of systems, and we want that access from any location. The rise of cloud services means we are more likely to switch between services such as CRM and ERP, and the new service has to seamlessly integrate into our existing workflows. From an IT management perspective, the ability to recruit more machines to participate in overnight and weekend processing tasks as needed is also valuable; and with a true IMDG developers barely need to be aware of the technology as it will ‘just work’.


In-memory data grids provide accurate, real-time information to the business, allowing users to make informed decisions. Fast, low-latency processing helps provide access to business critical information as and when it’s needed, and the clustering and recovery functions provide fault-tolerance.

This combination of high availability, increased application performance, improved failover, recovery and fault-tolerance along with elastic scalability and parallel processing helps your business applications run faster and makes In-Memory Data Grids a must-have in your enterprise integration.

In-Memory Data Grids now offer bullet-proof and dynamic capability out of the box without the cost and configuration complexity traditionally associated with clustering. Today these solutions need only half a day of configuration work, and when your integration platform has IMDG technology built in and ready to use, this offers a new era of grid-based solutions.




David Akka is the Managing Director of Magic Software (UK) Ltd and have worked with Magic Software for over 16 years.
Visit his blog at: http://www.davidakka.com/