Waters European Trading Architecture Summit 2011

Some feedback from this event which I attended today.

Event http://events.waterstechnology.com/etas

Infrastructure Management: Reducing costs, Improving performance, Professor Roger Woods, Queens University, Belfast

Prof Woods gave an impassioned talk about a tool that he has developed which takes c++, allows you to navigate the code and identify subsystems which you can target to run on hardware or emulation of hardware.

• He worked on the JP Morgan collaboration with Maxellor and was bullish about the technology.
• Two years from pilot to production.
• Developed at tool that allows identification of sections that are suitable for FPGA
• Key issue: programming FPGA bitstreams (http://en.wikipedia.org/wiki/Bitstream) - took six months
• C++ is translated into C (manually) before being cross compiled into Java which is what the Maxellor compiler requires.
• This is to remove c++ abstraction which "kills parellisation" (see slides)
• Focus was hardware FFT - all other logic in software - comms via FPGA bitstream

In summary:

• ideal for risk calculation and monte carlo where algorithm does not change.
• C++ legacy code does not parallelise easily and is not a candidate for FPGA
• Three year dev cycle.
• Complex, manual process
• JPM own 20% of Maxellor

Resources

• http://www.ecit.qub.ac.uk/Card/?name=r.woods
• eFutures Portal - http://efutures.ac.uk/

This continued to a panel hosted by Chris Skinner

Panel: The Parallel Paradigm Shift: are we about to enter a new chapter in the algorithmic arms race

Moderator: Chris Skinner, Panel: Prof Woods, Steven Weston, Global Head of Analytics, JPM. Andre Nedceloux, Sales guy, Excelian

• FPGA Plant needs to be kept hot to achieve best latency. To keep FPGA busy you need a few regular host cores loading work onto them.
• Programming/debugging directly in VHDL is ‘worse than a nightmare’, don’t try.
• Isolate the worst performing pieces, (Amdahl’s law) de-abstract and place on FPGA, they call each of the isolated units a ‘kernel’ .
• Compile times are high for Maxeler compiler to output VHDL, 4 hours for a model on a 4 core box.
• Iterative model for optimisation and implementation. They improved both the mathematics in the models and the implementation onto FPGA – ie, consider it not just a programming problem, but also a maths modeling one.
• They use python to manage the interaction with the models (e.g pulling reports)
• Initially run a model on the FPGA hosts and then incrementally update it through the day - when market data or announcements occur.
  
• No separate report running phase – it is included in the model run and report is kept in memory. Data only written out to a database at night time, if it is destroyed then it can be re-created.
• Low-latency is no longer a competitive advantage but now a status quo service for investment banking.
• Requires specialist non-general/outsourced programmers required who can understand hardware and algorithms who work alongside the business.

Panel

How low can you go? Ultra-low-latency trading

Moderator: David Berry

Members: Jogi Narain, CTO, FGS Capital LLP, Benjamin Stopford, Architect, RBS. Chris Donan, Head of Electronic Trading - Barcap.

This was a well run panel with some good insights from Chris Donan in particular:

• Stock programmers don't understand the stack from network to nic to stack to application and the underlying hardware operations
• Small teams of experienced engineers produce the best results
• Don't develop VHDL skills in house - use external resources.
• Latency gains correlate to profitability
• FPGA is good for market data (ie fixed problem) and risk
• Software parallelism is the future.
  

HPC=Grid+Data+FPGA

Another presentation from the past. Presented at Platform's Grid conference in Paris in 2006.

Monte Carlo Using COTS FPGA

A few years ago I put together this presentation based on some work I did with some guys at Southampton University on embedding Monte Carlo onto FPGA. At the time we used System C and the C code from the Monte Carlo implementation in the SCIMARK 2.0 Benchmark, something I've used to great effect in the past. The implementation was niaive and simplistic but it was used to start the HPC movement that's been a la mode in the City for a while...

CUDA - GPU Programming Framework from nVidia

Catching up with some reading this morning, I picked up a series of articles from the Mar/April edition of ACM Queue. In particular, CUDA which was released by Nvidia last year. I read the article "Scalable Parallel Programming with CUDA" which can be found here.

The article identifies three key abstractions, hierarchical thread groups, shared memories and barrier synchronisation. CUDA is based on C with extensions for parallelisation - much like Handel-C. The difference is that Handel-C was FPGA based whilst CUDA is for GPU with its built-in floating point capability. There are simple and straightforward code examples showing parallel threading and memory sharing which was always an issue in my mind with FPGA: the leap of faith with Handel-C was what to do with the data set you generated in a Monte Carlo simulation.

This question has been perplexing developers on the CUDA forums at Nvidia too - but it looks like there's been progress as outlined in this presentation on Monte Carlo Options Pricing paper on the Nvidia developer site. However, the algorithm outlined in the paper is trivial, the secret being the generation of quasi-random numbers enabling quick convergence. Then filtration close to the data so you're not schlepping large lumps of data unnecessarily.

Then the next logical step is to make this a service. The appetite is reckoned to be about 5 trillion simulations per day in the average organisation according to a quant chum of mine. Combine this with S3 for asynchronous storage and you have the makings of a nice little business I think.

Fork/Join - The End of the Road for FPGA/GPU/ASIC Gang?

Steve W sent this article from IBM - Java Theory and Practise: Stick a fork in it

"One of the additions to the java.util.concurrent packages coming in Java™ 7 is a framework for fork-join style parallel decomposition. The fork-join abstraction provides a natural mechanism for decomposing many algorithms to effectively exploit hardware parallelism."

Steve says "Looks like java 7 will have built-in support for the Fork/Join pattern (i.e. similar to what Erlang has and what the Squeak guys have been talking about)"

Is this the end for the short-lived FPGA/GPU bandwagon? It could well be - with massively multi-core chips round the corner, they are considerably more attractive an option with their on core FPU and low-latency cache etc. Solves a whole lot of issues around buying and supporting esoteric hardware it would seem.

HPC=Grid+Data+FPGA

I was kindly invited by Platform Computing to give a presentation at the their European Grid Conference PGC06 last October. I've just made this deck and others available on the Enhyper subversion share under decks. You can download the powerpoint presentation here HPC=Grid+Data+FPGA. The abstract for the pitch is below:

High Frequency Finance demands an infrastructure that can supply large quantities of computational resource plus the delivery of multidimensional, arbitrary precision data to applications at scalable rates, in a platform independent manner.

Statistical analysis techniques for pricing and risk measurement also require significant algorithmic performance. This is being delivered by multi-core processors, however, the quest for low-latency is driving the emergence of algorithms in silicon using Field Programmable Gate Array techniques.

The traditional approach to application and data engineering has been client/server glued together with delimited files and relational databases. These technologies are increasingly being surpassed by grid enabled; in-memory applications combined with FPGA based algorithms.

This was immediately after some friends at Celoxica had run a trial of BGM in one of the tier 1's - a trial that has since been emulated in academia by Wayne Luk et al as outlined in this paper entitled Reconfigurable Acceleration for Monte Carlo based Financial Simulation. The problem with academics is that, to quote Dr Duncan Shaw, "they have 98% of the time but only 2% of the data, whereas it's the reverse for the practitioner". There are better ways of skinning this particular cat which could have significantly improved the performance...