Waters European Trading Architecture Summit 2011

Some feedback from this event which I attended today.

Event http://events.waterstechnology.com/etas

Infrastructure Management: Reducing costs, Improving performance, Professor Roger Woods, Queens University, Belfast

Prof Woods gave an impassioned talk about a tool that he has developed which takes c++, allows you to navigate the code and identify subsystems which you can target to run on hardware or emulation of hardware.

• He worked on the JP Morgan collaboration with Maxellor and was bullish about the technology.
• Two years from pilot to production.
• Developed at tool that allows identification of sections that are suitable for FPGA
• Key issue: programming FPGA bitstreams (http://en.wikipedia.org/wiki/Bitstream) - took six months
• C++ is translated into C (manually) before being cross compiled into Java which is what the Maxellor compiler requires.
• This is to remove c++ abstraction which "kills parellisation" (see slides)
• Focus was hardware FFT - all other logic in software - comms via FPGA bitstream

In summary:

• ideal for risk calculation and monte carlo where algorithm does not change.
• C++ legacy code does not parallelise easily and is not a candidate for FPGA
• Three year dev cycle.
• Complex, manual process
• JPM own 20% of Maxellor

Resources

• http://www.ecit.qub.ac.uk/Card/?name=r.woods
• eFutures Portal - http://efutures.ac.uk/

This continued to a panel hosted by Chris Skinner

Panel: The Parallel Paradigm Shift: are we about to enter a new chapter in the algorithmic arms race

Moderator: Chris Skinner, Panel: Prof Woods, Steven Weston, Global Head of Analytics, JPM. Andre Nedceloux, Sales guy, Excelian

• FPGA Plant needs to be kept hot to achieve best latency. To keep FPGA busy you need a few regular host cores loading work onto them.
• Programming/debugging directly in VHDL is ‘worse than a nightmare’, don’t try.
• Isolate the worst performing pieces, (Amdahl’s law) de-abstract and place on FPGA, they call each of the isolated units a ‘kernel’ .
• Compile times are high for Maxeler compiler to output VHDL, 4 hours for a model on a 4 core box.
• Iterative model for optimisation and implementation. They improved both the mathematics in the models and the implementation onto FPGA – ie, consider it not just a programming problem, but also a maths modeling one.
• They use python to manage the interaction with the models (e.g pulling reports)
• Initially run a model on the FPGA hosts and then incrementally update it through the day - when market data or announcements occur.
  
• No separate report running phase – it is included in the model run and report is kept in memory. Data only written out to a database at night time, if it is destroyed then it can be re-created.
• Low-latency is no longer a competitive advantage but now a status quo service for investment banking.
• Requires specialist non-general/outsourced programmers required who can understand hardware and algorithms who work alongside the business.

Panel

How low can you go? Ultra-low-latency trading

Moderator: David Berry

Members: Jogi Narain, CTO, FGS Capital LLP, Benjamin Stopford, Architect, RBS. Chris Donan, Head of Electronic Trading - Barcap.

This was a well run panel with some good insights from Chris Donan in particular:

• Stock programmers don't understand the stack from network to nic to stack to application and the underlying hardware operations
• Small teams of experienced engineers produce the best results
• Don't develop VHDL skills in house - use external resources.
• Latency gains correlate to profitability
• FPGA is good for market data (ie fixed problem) and risk
• Software parallelism is the future.
  

Global Connectivity Vendor Selection

TeleGeography produce a topological map of submarine cables which is of interest to hft firms. There are many aspects to consider when procuring global connectivity.

Before you choose a suppler, perhaps some important questions to ask are:

1. What are the fibre miles versus physical miles of long haul links
   
2. How do you measure your quoted latency
3. What is your underlying network technology and how is it provisioned
   
4. What network equipment do you use for your core network
   
5. How many hardware queues on the above equipment do you allocate to our service
6. When using third parties, what is their underlying technology and distribution methodology
   
7. What hand-off equipment do you use
   
8. What can the above deliver in terms of serialisation capability
   
9. What traffic shaping do you perform
10. How often do you sample cdr violation
11. Do you report CDR violation
12. Do you offer QoS
13. Do you offer IGMP snooping

Not an exhaustive list, but a good boilerplate for vendor assessment. I'll expand on each point in due course.

HIFREQ 2011 Panel Discussion Input

Here's my input to the panel discussion for HIFREQ 2011.

• What are the different set ups and combinations for HFT architecture?

I have experience of four different architectures:

• traditional monolithic event queue and broadcast
• reflective memory, distributed processing
• dma, shared memory, multi-process and multicast
• trading engine on a card

The latter appears to be the dream set up. It consists of an FPGA enabled network card with the strategy running on the card itself. This has been implemented by several large prop trading outfits who have arb strats. Up until June last year, we were able to compete with an aggressive arb strat - but we our fill rates have dropped off dramatically and we're consistently being beaten on speed. So we've moved the goalposts - we now focus on market making, news and global multi-venue trading.

I'd be interested in other approaches

• Is massive multicore or specialist silicon (FPGA, GPU etc.) the next frontier?

Multicore is attractive for a multi strategy play, but it requires careful design to avoid data races and performance issues like tlb cache misses and memory barriers. FPGA has always been attractive for dealing with FIX and conversion of ascii to binary (ie parsing). GPU has promise in the equities world where dynamic pricing and portfolio analysis are required. What's been widely overlooked is DSP. There are some very interesting things you can do with DSP.

• Which solutions provide maximum scalability, configuration, and customisation to ensure continual upgrade and development of your
systems to survive in tough technology race?

It has to be a message orientated multicast pure layer 2 architecture with software routing.

• Managing microbursts: the art and engineering of data capacity management

Again, a high performance messaging system combined with high resolution time and accurate traffic analysis are a must. Combine this with knowledge of the underlying network hardware in order to utilise multiple hardware queues on the switches, judicious use of QoS and correctly configuring the messaging system to utilise multiple channels yields good results.

• Taming the data torrent: conflating direct and aggregate market data feeds

Here's where an FPGA enabled network card helps e.g. by coalescing multiple keystations or A and B feeds, hashing messages and dropping duplicates and translating ascii to binary help greatly. Combine this with multicast for efficient data transport.

• Is asynchronous processing inevitable? What are the implications?

This has been used in HFT for many years and are necessary for effective parallelisation. One pattern I use is an "asynchronous n-slot put-take connector" which is a way of joining different processes in a way that allows each process to utilise its full timeslice. The implications of not using it are latency...

• What is next for complex event processing and tick databases? Will they be able to keep up?

CEP is a necessary evil, however, any strategy that uses it is almost impossible to debug.

With regards to tick databases I fail to see why people store stuff in databases at all - all our
market data is captured on an electrically connected secondary machine using HDF5 in date order directories. it's then shipped up to a disk array overnight. Backtest generally uses only three months worth of data.

• Surviving the technical glitch at high speed: designing robust architectures

The ability to run multiple strategies and services on a multicast message bus means recovery
from failure is straightforward.

Mankoff Company 2nd Annual Ultra Low Latency

I've been kindly invited to speak at another HFT conference in London. The 2nd Annual Ultra Low Latency: Trading Opportunities and Development in FX and other asset classes run by the Mankoff Company to be held at the Grange Holborn, London on the 23rd March 2011.