LSE Goes Boom!

"No other exchange is undertaking such an ambitious technology refresh programme based on next-generation Microsoft technology. We have always provided a first-class service, but now we can claim to be the fastest in the world as well."
- David Lester, CIO, London Stock Exchange

Now I wonder why no other exchange uses Microsoft technology for high performance messaging then...

The LSE was a big coup for Microsoft - one of the worlds leading exchanges which, until the Infolect platform based on .Net technology came along, had no outages since 1987. Even more embarassing are the anti-linux rhetoric contained in the "Get The Facts" campaign quoting the above David Lester quote - a nice analysis of which you can find here from Paul Murphy. He concludes that the so called high performance message delivery is the same as a "relatively slow LAN".

This won't be the last time this occurs - in my operational experience, solutions based on .Net exhibited mysterious freezes which were never solved.

The Transition from N-Tier to High Performance Architecture

This is a little story of Anybank, who have a classic three-tier architecture using in-house developed solutions. They are embarking on a plan to improve the architecture, aiming for scalability, globalisation and integration in order to meet predicted growth in volumes.

In order to achieve this, the architecture will be transitioned from a transaction-orientated platform to an event-driven, peer-to-peer, asynchronous, persistent messaging-based architecture. Anybank have recognised that in order to achieve this, they must move away from in-house developed solutions and take advantage of open source and third-party products.

Architectural Components

Anybank have a classic three-tier architecture with a moderate level of fan out/in.

At the front end, there are a series of internal trading and control applications and several external client applications and gateways to systems such as FIX.

The communication from tier 3 to tier 2 is TCP based, carrying FIX or HTTP traffic to series of middle tier servers. Additionally, there is a custom protocol used by the thick client to talk to the middle tier servers. This is an efficient ASCII based protocol with pipe-delimited fields.

External Applications

The system has FIX connections to several exchanges and markets. Clients have a choice of a thick client or a web front end. Quotes are sent over FIX at a rate of 4/second.

Internal Applications and Services

There are several internal applications that are used by trading, credit and risk control and operations. The connections to these are managed by a series of front office servers and talk to the database tier for information. They run a generic set of applications that would be found in any other bank:

• Market Making, Trading Station and Sales Trader Workbench: Prop. trading tools
• Session Management: Routing, STP gateways, reception of execution reports.
• Order Moniting: provides snapshot requests and streaming prices
• Client Margin Calculation Engine: Limit checking, credit check, event publishing service
• Exposure and Barrier Monitoring
• Front Office Server: Authentication, Authorisation and Access control (AAA), Event notification.
• Trade Server: Visual trading, large trades and limit trading.
• Market Data Servers: Takes streaming market prices from feed handlers and distributes Market Data
• Metrics and SLA monitoring

The current system handles thousands of daily trades with a peak rate of 200/second. Latency across several systems is in the order of .5 – 1 second with a desire to improve this figure across the board.

Anybank Tiered Architecture Analysis

Several key components have been targeted in an attempt to reduce potential bottlenecks in the system. Serial activities are being replaced with parallel activities, single threaded code is being rewritten to be multhreaded.

However, by applying HPC techniques and chosing the right products, we can gain significant performance quickly and robustly.

Tier 1 - Database Persistence Layer

This is the database layer, which is currently a replicated SQL Server. As with all relational databases, the maximal transaction rate is a function of the IO subsystem, log file transaction rate, number of indexes, data volumes and access patterns. Typically, one can expect a transaction rate of between 1-5 thousand transactions per second, which translates into 5-1 milliseconds per transaction.

Business level transactions may consist of many database transactions, which results in the potential for adding considerable latency to client and API calls.

Tier 2 - Application Services and Access Control Layer

The application logic layers combines authentication, access control and authorisation with business logic, static and derived data, streaming market data and database transactions. It is traditionally at this layer where fan-in/out is facilitated.

Tier 3 - Application Layer

The application layer consists of traditional thick, web and FIX clients, each with their own particular data access profile. Thick clients typically are used when streaming data is consumed by, for example, a pricing application. They may consume data delineated by topic or routed by function.

Web clients typically consume http/https/javascript and/or xml data to drive form-based, browser applications.

FIX clients can potentially deliver 100K messages per second when using FAST, the FIX compression protocol.

HPC Tools Applied to Anybank

The architecture is in transition from client server to an event-based and message orientated structure in order to increase resilience and enable scalability. The architecture has already some elements of fan out via multiple services connected via TCP.

There are opportunities to augment the current architecture and improve transaction capacity whilst reducing latency in each of the three tiers of the current Anybank architecture using Open Source messaging and caching products which have proven robustness in real-world applications.

Tier 1 and 2 - Peer-to-peer Message Bus

The publish and subscribe paradigm is ideal to use where multiple senders and receivers or combinations of the two are present and is common pattern for trading applications where prices, curves, derived data, immediate messages and topic based mechanisms need to be catered for.

In Tier 1 and 2, it is proposed to use the MantaRay peer-to-peer messaging product to provide message caching to tier 2 applications and services, effectively decoupling tier 1 and 2. MantaRay Messaging persistence logic guarantees delivery to unavailable peers once they come back online. Sending nodes need not await responses from a broker before resuming
other tasks. Such capabilities are critical issues in high availability environments. This is considerably faster than a synchronous database access thereby accelerating application performance.

Also In tier 2, application/service fan out will be added using the pub/sub capabilities of the MantaRay solution.

Reliable Multicast over the WAN

MantaRay also has a WAN Bridge capability which accepts connections from other MantaRay peers thereby getting around multi-cast over the WAN issues. This would allow the mixing of the benefits of multicast with the recoverability of MantaRay.

This could be used to distribute real-time market data globally. Peer-to-peer is ideal for this purpose. It can be configured to deliver topic orientated messages over multiple multicast streams, layering the topics in order to use bandwidth effectively using source paced delivery which would be ideal for Anybank.

to be continued...