Showing posts with label data architecture. Show all posts
Showing posts with label data architecture. Show all posts

Sunday, May 11, 2008

HPC=Grid+Data+FPGA

Another presentation from the past. Presented at Platform's Grid conference in Paris in 2006.

Saturday, June 16, 2007

Mining Massive Data Sets for Security

Semiophore points me to the forthcoming two week workshop on the above to be held in mid-September 2007 in Italy.

"It is the purpose of this workshop to review the various technologies available (data mining algorithms, social networks, crawling and indexing, text-mining, search engines, data streams) in the context of very large data sets."

I'd love to attend as this is an area I think is crucial for High Frequency Finance. Whilst working on a high performance trade order router for a tier 1, I did some research which I was allowed to present publicly at the Fiteclub, a forum which meets occasionally in London. I presented two papers of note - Financial Data Infrastructure with HDF5 which concentrated on high performance data delivery and analysis. In this presentation I proposed a machine which could be built for around $25K that could eat 20TB of data in 90 minutes - using COTS components. This was inspired by the seminal article on disk technology amusingly entitled "You don't know jack about disks" published by the ACM.

The second presentation, also at Fiteclub, was entitled Open Source Intelligence in Finance and was inspired by the techniques used in open source intelligence applied to finance. Here I build the case for news analysis applied to program trading.

Friday, June 15, 2007

HPC=Grid+Data+FPGA

I was kindly invited by Platform Computing to give a presentation at the their European Grid Conference PGC06 last October. I've just made this deck and others available on the Enhyper subversion share under decks. You can download the powerpoint presentation here HPC=Grid+Data+FPGA. The abstract for the pitch is below:

High Frequency Finance demands an infrastructure that can supply large quantities of computational resource plus the delivery of multidimensional, arbitrary precision data to applications at scalable rates, in a platform independent manner.

Statistical analysis techniques for pricing and risk measurement also require significant algorithmic performance. This is being delivered by multi-core processors, however, the quest for low-latency is driving the emergence of algorithms in silicon using Field Programmable Gate Array techniques.

The traditional approach to application and data engineering has been client/server glued together with delimited files and relational databases. These technologies are increasingly being surpassed by grid enabled; in-memory applications combined with FPGA based algorithms.

This was immediately after some friends at Celoxica had run a trial of BGM in one of the tier 1's - a trial that has since been emulated in academia by Wayne Luk et al as outlined in this paper entitled Reconfigurable Acceleration for Monte Carlo based Financial Simulation. The problem with academics is that, to quote Dr Duncan Shaw, "they have 98% of the time but only 2% of the data, whereas it's the reverse for the practitioner". There are better ways of skinning this particular cat which could have significantly improved the performance...