Monday, June 11, 2007

Skillsets for the HFF Future

So which languages are going to succeed in the world of High Frequency Finance? Well you can bet your bottom dollar it's not going to be C# or C++. Both generate buggy and leaky solutions and are too reliant on third party libraries of unknown veracity. You can also rule out the raft of scripting languages - no matter how much the developers like them: Perl, Python, Ruby et al have good geek factor, however they don't cut it in production systems.

So we're left with C and the functional languages. C is considered as a weird throwback to the 70's by most nascent programmers. What most of them do not realise is that the C# CLR, Java Virtual Machine (parts of), Ruby, Perl, Python, Apache, Linux, Solaris, C++ etc etc are all written in C. The main reason for this is performance, simplicity, robustness. C has it's share of problems, but in general it's a fairly good language.

So, why are all the quantitative libraries in Investment Banks written in C++? And why are they mostly single threaded? The answer is hubris on the part of the programmers and, as previously outlined, the difficulty in coding thread safe libraries.

Well, times have changed I'm afraid. C++ has to die because it does not translate into hardware - a route where tremendous performance gains are to be had. C is looking weak in light of the scalability models which can take advantage of multi-core within the functional languages. The future in the short term is C but watch for the rise of Erlang and Haskell. Their time has come.


Anonymous said...

An interesting discussion but I think the criticisms are not substantiated by experienced quant teams in the City.

C++ provides good mechanisms for flexible code reuse. Consider global variables in C and the difficulty of adding these to existing projects (and managing, etc). In C++ these can be subsumed into a class.
I've worked on parallel multi-threaded C++ libraries. C++ does a good job of hiding the nasty details from most development work (recall mutexes accessed by the constructor/destructor) and significant safety features such as guarded pointers (try using CRITICAL SECTIONs without these).

The object orientated features allow a good description of the concepts and can save a considerable amount of rewriting common functionality. Exception handling can give more natural processing of errors rather than checking return values at every step.

I can see your criticism carries weight from a software engineering point of view. C++ is an interesting beast, one of the most challenging to parse with context free parsers, and based on my own limitations, takes years to learn.
As a project manager with no experience and a new team would one choose it?
For experienced teams and banks with quant developers it is the natural choice at the moment.
My summary: When the team works well together C++ is a powerful tool!

Anonymous said...

Actually, you seem to have omitted to explain why Java is not a candidate:

- it strikes a reasonable middle ground in complexity between C++ and scripting languages; Java developers are certainly about 3-4 times as productive as their C++ counterparts.

- the core API is pretty feature-rich and if you need a 3rd party lib in general the situation is not that bad from "veracity" point of view.

- except perhaps for intense floating point calculation, performance should be acceptable. Hotspot + NIO + Reactor pattern embedded in architectures like SEDA make for high throughput event-driven architectures (SEDA has been shown to outperform servers written in C or C++).

I have substantial experience with everything in {C, C++, Java} set and could easily see building a solution where Java is used as the "systems programming" language and JNI is used to link to C/C++ floating-point engines.

Although the world of enterprise software cannot claim to have the absolute edge in high performance computing, a large fraction of that world moved to Java years ago and has never looked back.