Financial Data Infrastructure with HDF5

Presentation given to the FITE Club in 2004.

Erlang for Five Nines Talk

Erlang for Five Nines Talk

I attended the above talk given at Skillsmatter.com on the 13th December. There were 37 attendees, two of which were women (women in technology take note.) I'm afraid I was virtually the only suit in the room apart from Francesco (the speaker) so I stuck out like the proverbial sore thumb.

It was a good talk, well received and well presented. Not particulary technical, so I'm afraid I may have asked too many questions, but I couldn't let the opportunity to get a real understanding of practical Erlang slip by. I was particularly interesting in support for multicast but as no one else had heard of it I kept quiet.

Shared Memory Support

One area I was particulary interested in was no shared memory support - all comms has to go through a tcp/ip software stack apparently. I can understand the aesthetic reasons behind this but the implications for high performance messaging are significant: from one Erlang process to another on a remote machine there's an extra two stacks to cross. It would be interesting to compare with the high performance messaging providers.

Real-world Erlang - Yaws Performance

One rather amusing piece was on the performance of Yaws, the Erlang web server, which shows very high throughput compared to Apache. Yaws manages to run 80,000 parallel sessions, serviing 2 x 20KB pages whilst poor old Apache dies after 4,000.

One unfortunate side effect of this would be to consume your organisation's entire bandwidth so I'll stick with Apache for now. If I want to ship that much (presumably) random data, the last tech I'll use is a web server. HDF5, Lustre, GFS, Gigaspaces all have a better handle on this sort of data architecture.

Conclusion

I like Erlang as a concept but they have a mountain to climb with OTP (the function library) if they're going to be more than niche. New features are being added all the time - the latest being a "posix like" threading library and support for SMP. Faster ASN.1 support too with the latest version. Erlang is growing up fast.

For those of you interested in learning Erlang, I'd recommend starting with Joe Armstrong's PhD.

Mining Massive Data Sets for Security

Semiophore points me to the forthcoming two week workshop on the above to be held in mid-September 2007 in Italy.

"It is the purpose of this workshop to review the various technologies available (data mining algorithms, social networks, crawling and indexing, text-mining, search engines, data streams) in the context of very large data sets."

I'd love to attend as this is an area I think is crucial for High Frequency Finance. Whilst working on a high performance trade order router for a tier 1, I did some research which I was allowed to present publicly at the Fiteclub, a forum which meets occasionally in London. I presented two papers of note - Financial Data Infrastructure with HDF5 which concentrated on high performance data delivery and analysis. In this presentation I proposed a machine which could be built for around $25K that could eat 20TB of data in 90 minutes - using COTS components. This was inspired by the seminal article on disk technology amusingly entitled "You don't know jack about disks" published by the ACM.

The second presentation, also at Fiteclub, was entitled Open Source Intelligence in Finance and was inspired by the techniques used in open source intelligence applied to finance. Here I build the case for news analysis applied to program trading.

HPC=Grid+Data+FPGA

I was kindly invited by Platform Computing to give a presentation at the their European Grid Conference PGC06 last October. I've just made this deck and others available on the Enhyper subversion share under decks. You can download the powerpoint presentation here HPC=Grid+Data+FPGA. The abstract for the pitch is below:

High Frequency Finance demands an infrastructure that can supply large quantities of computational resource plus the delivery of multidimensional, arbitrary precision data to applications at scalable rates, in a platform independent manner.

Statistical analysis techniques for pricing and risk measurement also require significant algorithmic performance. This is being delivered by multi-core processors, however, the quest for low-latency is driving the emergence of algorithms in silicon using Field Programmable Gate Array techniques.

The traditional approach to application and data engineering has been client/server glued together with delimited files and relational databases. These technologies are increasingly being surpassed by grid enabled; in-memory applications combined with FPGA based algorithms.

This was immediately after some friends at Celoxica had run a trial of BGM in one of the tier 1's - a trial that has since been emulated in academia by Wayne Luk et al as outlined in this paper entitled Reconfigurable Acceleration for Monte Carlo based Financial Simulation. The problem with academics is that, to quote Dr Duncan Shaw, "they have 98% of the time but only 2% of the data, whereas it's the reverse for the practitioner". There are better ways of skinning this particular cat which could have significantly improved the performance...