Mining Massive Data Sets for Security
Semiophore points me to the forthcoming two week workshop on the above to be held in mid-September 2007 in Italy.
"It is the purpose of this workshop to review the various technologies available (data mining algorithms, social networks, crawling and indexing, text-mining, search engines, data streams) in the context of very large data sets."
I'd love to attend as this is an area I think is crucial for High Frequency Finance. Whilst working on a high performance trade order router for a tier 1, I did some research which I was allowed to present publicly at the Fiteclub, a forum which meets occasionally in London. I presented two papers of note - Financial Data Infrastructure with HDF5 which concentrated on high performance data delivery and analysis. In this presentation I proposed a machine which could be built for around $25K that could eat 20TB of data in 90 minutes - using COTS components. This was inspired by the seminal article on disk technology amusingly entitled "You don't know jack about disks" published by the ACM.
The second presentation, also at Fiteclub, was entitled Open Source Intelligence in Finance and was inspired by the techniques used in open source intelligence applied to finance. Here I build the case for news analysis applied to program trading.
Saturday, June 16, 2007
The Rise and Rise of Software Patents
James Robertson points to a New York Times op-ed piece by Timothy Lee.
Microsoft has changed their position on software patents since 1991 – what was different then?
In 1991 most personal computer users were still using DOS, not Windows.
Windows was welcome because it provided a standard way for PC applications to take advantage of RAM over 640K. Applications like Lotus and WordPerfect required incompatible and unstable high memory drivers (want to use a different app? Edit AUTOEXEC.BAT and reboot. Repeat). While Mac users were able to use Excel and Word without these limitations, the vast majority of PC users were unaware of the advantages of standardised graphical user interfaces, postscript printing or built-in networking. Apple, however, saw Windows as a threat, and had already been litigating against Microsoft for three years by this time.
When Bill Gates was stating his concerns about patents held by "some large company", he may have talking about Apple. But the lawsuit was related to copyrights, not patents.
Apple had the tables (briefly) turned on them, when Xerox filed suit, also alleging copyright infringement against work they had done on the Star.
So where were the patents?
Some people feel that Apple, Microsoft, and others plagiarized the GUI and other innovations from the Xerox Star, and believe that Xerox didn't properly protect its intellectual property. The truth is a bit more complicated. Many patent disclosures were in fact submitted for the innovations in the Star; however, at the time the 1975 Xerox Consent Decree, an FTC
antitrust action, placed restrictions on what the company was able to patent.[4] In addition, when the Star disclosures were being prepared, the Xerox patent attorneys were busy with several other new technologies such as laser printing. Finally, patents on software, particularly those relating to user interfaces, were an untested legal area at that time.
The law is clearer now, notwithstanding differences between Europe and the US.
Apple wasn't able to patent these things because they didn't invent them.
Xerox wasn't able to patent them because of an antitrust ruling (and there may be some truth in the fact that the business was oblivious to the incredible work being done at PARC).
While I don't like to play the alternative history game (the real thing is confusing enough, thanks), the rise of Microsoft and Apple might have turned out differently had it not been for this ruling.
Investment banks have an interest in software patents for two reasons. First, they have a responsibility to their clients to invest in companies with legally identifiable market advantages. Patents are a big part of that, especially for technology companies. Right or wrong, if you have a technology company and you want someone to invest in your firm, patents can increase your market value.
Second, financial patents are becoming more widespread, and software is a big part of that. Investment banks are actively encouraging their (very large) development teams to work with their legal departments to protect their inventions. Again, right or wrong, this is partly defensive (e.g. if firms have a portfolio of patents they can enter into cross-licensing agreements with potential competitors, or they can use patents to obtain a monopoly on innovative business ideas), and partly opportunistic. It is possible that even banks might find the revenue streams from IP licensing attractive.
While not everyone is happy about software patents (they increase the cost of doing business, and can act as barriers to new entrants into a market), they can also be useful. They can help to create and sustain advantages for people who are careful and clever enough to spot the opportunities.
Either way, ignoring them is not advisable.
Update: Donald Knuth on Software Patents
Friday, June 15, 2007
Functional Programming meets #crypto
We met up last night as mentioned in for a Functional Programming Beer in the Evening event and had a thoroughly enjoyable evening. Dr Dominic Steinitz, Nigel Stuckey of Systemgarden, Steve Wart and I spent several hours talking about a variety of topics ranging from FP through to Cryptography.
Nigel, Steve and I met at a tier 1 we all worked for through an internal chat channel I ran on cryptography called #crypto which was a hot bed of financial crypto interest. I had been following the great Robert Hettinga and his rants on bearer security and his promotion of FC in general. Through this I met and became friends with Ian Grigg who runs the Financial Cryptography website which is always worth a read as Ian challenges conventional wisdom and crypto practise from a more business related angle.
Dominic is an active member of the Haskell community and hopefully will be joining the enhyper blog and contributing his expertise. Dominic wrote the cryptographic functions (sha1, rsa, blowfish etc) in Haskell and maintains the library on one of the Haskell webservers. I'm going to create a subversion respository with public access for Dominic's work to allow the community to contribute to the Haskell crypto effort so watch this space.
Nigel will also be contributing to the Enhyper blog - his expertise in platform management is world-class, gained through the development of Harvest at System Garden.
We talked in detail about the evolution of specialist languages in Finance, in particular K and Smalltalk. Steve Wart is a smalltalk programmer and outlined a fantastic sounding system he worked on for a Candadian power company so I hope he's going to expand on that here. Dominic and I discussed stream fusion at length and I think I understand it - however, I'm not going to steal his thunder as he promises to write an article on data parallelism. Nigel and Steve had a lively debate on Second Life and discussed the additon of models of ancient buildings as a mechanism for historical education. We finished the evening with a sausage and sauerkraut at Kurz and Lang in Smithfield which was reminiscent of Zurich night life.
We met up last night as mentioned in for a Functional Programming Beer in the Evening event and had a thoroughly enjoyable evening. Dr Dominic Steinitz, Nigel Stuckey of Systemgarden, Steve Wart and I spent several hours talking about a variety of topics ranging from FP through to Cryptography.
Nigel, Steve and I met at a tier 1 we all worked for through an internal chat channel I ran on cryptography called #crypto which was a hot bed of financial crypto interest. I had been following the great Robert Hettinga and his rants on bearer security and his promotion of FC in general. Through this I met and became friends with Ian Grigg who runs the Financial Cryptography website which is always worth a read as Ian challenges conventional wisdom and crypto practise from a more business related angle.
Dominic is an active member of the Haskell community and hopefully will be joining the enhyper blog and contributing his expertise. Dominic wrote the cryptographic functions (sha1, rsa, blowfish etc) in Haskell and maintains the library on one of the Haskell webservers. I'm going to create a subversion respository with public access for Dominic's work to allow the community to contribute to the Haskell crypto effort so watch this space.
Nigel will also be contributing to the Enhyper blog - his expertise in platform management is world-class, gained through the development of Harvest at System Garden.
We talked in detail about the evolution of specialist languages in Finance, in particular K and Smalltalk. Steve Wart is a smalltalk programmer and outlined a fantastic sounding system he worked on for a Candadian power company so I hope he's going to expand on that here. Dominic and I discussed stream fusion at length and I think I understand it - however, I'm not going to steal his thunder as he promises to write an article on data parallelism. Nigel and Steve had a lively debate on Second Life and discussed the additon of models of ancient buildings as a mechanism for historical education. We finished the evening with a sausage and sauerkraut at Kurz and Lang in Smithfield which was reminiscent of Zurich night life.
HPC=Grid+Data+FPGA
I was kindly invited by Platform Computing to give a presentation at the their European Grid Conference PGC06 last October. I've just made this deck and others available on the Enhyper subversion share under decks. You can download the powerpoint presentation here HPC=Grid+Data+FPGA. The abstract for the pitch is below:
High Frequency Finance demands an infrastructure that can supply large quantities of computational resource plus the delivery of multidimensional, arbitrary precision data to applications at scalable rates, in a platform independent manner.
Statistical analysis techniques for pricing and risk measurement also require significant algorithmic performance. This is being delivered by multi-core processors, however, the quest for low-latency is driving the emergence of algorithms in silicon using Field Programmable Gate Array techniques.
The traditional approach to application and data engineering has been client/server glued together with delimited files and relational databases. These technologies are increasingly being surpassed by grid enabled; in-memory applications combined with FPGA based algorithms.
This was immediately after some friends at Celoxica had run a trial of BGM in one of the tier 1's - a trial that has since been emulated in academia by Wayne Luk et al as outlined in this paper entitled Reconfigurable Acceleration for Monte Carlo based Financial Simulation. The problem with academics is that, to quote Dr Duncan Shaw, "they have 98% of the time but only 2% of the data, whereas it's the reverse for the practitioner". There are better ways of skinning this particular cat which could have significantly improved the performance...
I was kindly invited by Platform Computing to give a presentation at the their European Grid Conference PGC06 last October. I've just made this deck and others available on the Enhyper subversion share under decks. You can download the powerpoint presentation here HPC=Grid+Data+FPGA. The abstract for the pitch is below:
High Frequency Finance demands an infrastructure that can supply large quantities of computational resource plus the delivery of multidimensional, arbitrary precision data to applications at scalable rates, in a platform independent manner.
Statistical analysis techniques for pricing and risk measurement also require significant algorithmic performance. This is being delivered by multi-core processors, however, the quest for low-latency is driving the emergence of algorithms in silicon using Field Programmable Gate Array techniques.
The traditional approach to application and data engineering has been client/server glued together with delimited files and relational databases. These technologies are increasingly being surpassed by grid enabled; in-memory applications combined with FPGA based algorithms.
This was immediately after some friends at Celoxica had run a trial of BGM in one of the tier 1's - a trial that has since been emulated in academia by Wayne Luk et al as outlined in this paper entitled Reconfigurable Acceleration for Monte Carlo based Financial Simulation. The problem with academics is that, to quote Dr Duncan Shaw, "they have 98% of the time but only 2% of the data, whereas it's the reverse for the practitioner". There are better ways of skinning this particular cat which could have significantly improved the performance...
Wednesday, June 13, 2007
Time to Embrace New Data Parallel Programming Models?
In "The rise and fall of High Performance Fortran: An Historical Object Lesson" there's a several lessons to be learned but of significant relevance to the development and adoption of Functional Programming Languages in finance was the promise offered by data parallel architectures.
Data parallelism is the ability to run sequences of instructions which are broadcast to processing units in lock-step, on different elements of data in parallel. This was first implemented in hardware in the 1960's and was classifed as SIMD - Single Instruction Multiple Data
This is the mechanism is now in implemented in software and part of the Glasgow Haskell Compiler and is explored in detail in the presentation "Nested Data Parallelism in Haskell" where they state that it's the only way to take advantage of 100's of cores and that it can even be run on GPUs. Like Handel-C, Haskell has par and seq statements allowing the programmer to parallelise and sequentialise instructions. They also introduce a new concept called stream fusion which is non-recursive allowing data to be "bused"from one stream to another inline - now I'm not sure I fully understand the implications of this but I'm sure going to find out.
Data parallelism is the ability to run sequences of instructions which are broadcast to processing units in lock-step, on different elements of data in parallel. This was first implemented in hardware in the 1960's and was classifed as SIMD - Single Instruction Multiple Data
This is the mechanism is now in implemented in software and part of the Glasgow Haskell Compiler and is explored in detail in the presentation "Nested Data Parallelism in Haskell" where they state that it's the only way to take advantage of 100's of cores and that it can even be run on GPUs. Like Handel-C, Haskell has par and seq statements allowing the programmer to parallelise and sequentialise instructions. They also introduce a new concept called stream fusion which is non-recursive allowing data to be "bused"from one stream to another inline - now I'm not sure I fully understand the implications of this but I'm sure going to find out.
Functional Programming Beer In The Evening Event Redux
Following our last meet up last week to speak about Functional Programming in the Finance. I'm setting up another beer around Smithfield in London, 14th June 2007, with Dominic Steinitz who has made considerable contribution Functional Programming via his Haskell crypto library and paper on Trends in Functional Programming. We'll be talking about Haskell in particular but also about Erlang and its use in service based analytics.
Following our last meet, Someone sent me a link to a pdf entitled Caml Trader: Adventures of a Functional Programmer on Wall Street by Yaron Minsky of Jane Street Capital which confirms the rise of FP in finance. Worth a read.
So if you fancy joining us, we'll be in the Long Lane Pub in Long Lane near Smithfield from 6pm onwards. Drop me a note (rgb at enhyper.com) or call my mobile +44 791 505 5 three eight zero.
Following our last meet up last week to speak about Functional Programming in the Finance. I'm setting up another beer around Smithfield in London, 14th June 2007, with Dominic Steinitz who has made considerable contribution Functional Programming via his Haskell crypto library and paper on Trends in Functional Programming. We'll be talking about Haskell in particular but also about Erlang and its use in service based analytics.
Following our last meet, Someone sent me a link to a pdf entitled Caml Trader: Adventures of a Functional Programmer on Wall Street by Yaron Minsky of Jane Street Capital which confirms the rise of FP in finance. Worth a read.
So if you fancy joining us, we'll be in the Long Lane Pub in Long Lane near Smithfield from 6pm onwards. Drop me a note (rgb at enhyper.com) or call my mobile +44 791 505 5 three eight zero.
Monday, June 11, 2007
Skillsets for the HFF Future
So which languages are going to succeed in the world of High Frequency Finance? Well you can bet your bottom dollar it's not going to be C# or C++. Both generate buggy and leaky solutions and are too reliant on third party libraries of unknown veracity. You can also rule out the raft of scripting languages - no matter how much the developers like them: Perl, Python, Ruby et al have good geek factor, however they don't cut it in production systems.
So we're left with C and the functional languages. C is considered as a weird throwback to the 70's by most nascent programmers. What most of them do not realise is that the C# CLR, Java Virtual Machine (parts of), Ruby, Perl, Python, Apache, Linux, Solaris, C++ etc etc are all written in C. The main reason for this is performance, simplicity, robustness. C has it's share of problems, but in general it's a fairly good language.
So, why are all the quantitative libraries in Investment Banks written in C++? And why are they mostly single threaded? The answer is hubris on the part of the programmers and, as previously outlined, the difficulty in coding thread safe libraries.
Well, times have changed I'm afraid. C++ has to die because it does not translate into hardware - a route where tremendous performance gains are to be had. C is looking weak in light of the scalability models which can take advantage of multi-core within the functional languages. The future in the short term is C but watch for the rise of Erlang and Haskell. Their time has come.
So we're left with C and the functional languages. C is considered as a weird throwback to the 70's by most nascent programmers. What most of them do not realise is that the C# CLR, Java Virtual Machine (parts of), Ruby, Perl, Python, Apache, Linux, Solaris, C++ etc etc are all written in C. The main reason for this is performance, simplicity, robustness. C has it's share of problems, but in general it's a fairly good language.
So, why are all the quantitative libraries in Investment Banks written in C++? And why are they mostly single threaded? The answer is hubris on the part of the programmers and, as previously outlined, the difficulty in coding thread safe libraries.
Well, times have changed I'm afraid. C++ has to die because it does not translate into hardware - a route where tremendous performance gains are to be had. C is looking weak in light of the scalability models which can take advantage of multi-core within the functional languages. The future in the short term is C but watch for the rise of Erlang and Haskell. Their time has come.
Sunday, June 10, 2007
News Analysis for Program Trading
As previously posted, I'm writing a paper on news based program trading for the KM stream at the Operational Research Society's Annual Conference in September. This paper is the culmination of many years research and interest in the area of new analysis and I hope to show that the application of statistical techniques combined with visualisation can lead to an effective intelligence system which solves some of the conundrums facing traders, and for that matter, intelligence analysts.
The goal is to greatly shorten the time to disseminate events to the people who need to consume them, allowing them to act on this information. However, there's also an intention to analyse the likely outcome of this interaction and put in place a strategy to take advantage of this event. Another hypothetical outcome is that event "signatures" will be recognised and effects correlated in different sectors.
News Analysis
The first goal is to simplify the elements of news which we will analyse. To do this, I propose to model the way that people tend to read newspapers and select stories which interest them. Some read from front to back, others select favourite sections first, others, and I include myself here, read from back to front.
When we read, the first element to be considered is either the title or a picture. The writer of the article has to aphoristically state the contents of the news in an attempt to get the readers interest.
The title also contains other information like people, places, sectors, amounts, therefore this is the key piece which is used for presentation to the end user.
The rest of the story consists of a series of sentences arranged into paragraphs. Within the story will also be the information we are interested in. The relativity of people can be used to build a Social Network Analysis graph based on proximity. If two people are mentioned in the same sector (e.g. FX trading) they are related. If they are mentioned in the same publication they are related more closely. The same story, closer still. Same paragraph, even closer. Same sentence, the closest. From this we can draw a graph showing the individuals "social network". There's a very good example of this at www.namebase.org where you can perform useful searches on people involved in the intelligence world from their appearance in related publications.
Topographical Mapping
News also contains physical places. Mapping individuals, companies, sectors, amounts to physical location can reveal useful information and is a technique much used in policing.
Categorisation
Categorisation is something humans do every day and is fundamental to our heuristic judgement. Humans are very good at it, however, what they're not so good at is dealing with something which falls into multiple categories.
To be continued shortly...
As previously posted, I'm writing a paper on news based program trading for the KM stream at the Operational Research Society's Annual Conference in September. This paper is the culmination of many years research and interest in the area of new analysis and I hope to show that the application of statistical techniques combined with visualisation can lead to an effective intelligence system which solves some of the conundrums facing traders, and for that matter, intelligence analysts.
The goal is to greatly shorten the time to disseminate events to the people who need to consume them, allowing them to act on this information. However, there's also an intention to analyse the likely outcome of this interaction and put in place a strategy to take advantage of this event. Another hypothetical outcome is that event "signatures" will be recognised and effects correlated in different sectors.
News Analysis
The first goal is to simplify the elements of news which we will analyse. To do this, I propose to model the way that people tend to read newspapers and select stories which interest them. Some read from front to back, others select favourite sections first, others, and I include myself here, read from back to front.
When we read, the first element to be considered is either the title or a picture. The writer of the article has to aphoristically state the contents of the news in an attempt to get the readers interest.
The title also contains other information like people, places, sectors, amounts, therefore this is the key piece which is used for presentation to the end user.
The rest of the story consists of a series of sentences arranged into paragraphs. Within the story will also be the information we are interested in. The relativity of people can be used to build a Social Network Analysis graph based on proximity. If two people are mentioned in the same sector (e.g. FX trading) they are related. If they are mentioned in the same publication they are related more closely. The same story, closer still. Same paragraph, even closer. Same sentence, the closest. From this we can draw a graph showing the individuals "social network". There's a very good example of this at www.namebase.org where you can perform useful searches on people involved in the intelligence world from their appearance in related publications.
Topographical Mapping
News also contains physical places. Mapping individuals, companies, sectors, amounts to physical location can reveal useful information and is a technique much used in policing.
Categorisation
Categorisation is something humans do every day and is fundamental to our heuristic judgement. Humans are very good at it, however, what they're not so good at is dealing with something which falls into multiple categories.
To be continued shortly...
The Case for Asynchronous Logging
It is common practise for federated systems to maintain seperate logfiles to assist in fingerpointing should an error in production occur. However, this duplication of effort is unsustainable in the world of High Frequency Finance (HFF) where messaging volumes are approaching the 400K messages per second leading to a rethink and perhaps a spirit of cooperation between data sinks and sources.
I propose that it's time to abrogate responsibility to one party not both, to log asynchronously. Deciding who carries the responsibility however, is not obvious. To understand the problem, lets look at the issues involved (or if you're of the half-empty glass persuasion - who gets the blame). Here's an example of a possible route between two applications:
Sender Logging
If we rely on the sender, there's the immediate advantage that the sender will have to account for the log file space, access control and maintenance. However, from a consumer's point of view - that means there is a lack of control and potentially the case where you require a log and it's been deleted or is offline. From an audit point of view, you have increased the external dependency and hence the risk.
From the senders perspective, consumer lifecycle mangement also becomes slighty more difficult as you now have to poll your customers to see if they are still consuming your data, as it's not unknown for applications to be turned off without turning the feeds off due to lack of knowledge on who to contact.
Receiver Logging
With receiver logging, we have, effectively, a forensic record of the transfer across the stack and have control over the logfile lifecycle. It seems strange to state the obvious, but for higher performance, you should log to local disk, not NFS or SAN storage then back up the log files to resiliient storage.
Service Level Monitoring
A nice addition is to write service level monitoring as part of our productionised system. In this way you can monitor the normal performance of the system and build a predictive capability on applicaiton performance.
Conclusion
Asynchronous logging has the potential to save considerable disk space and processor time whilst reducing maintenance overhead. The receiver/data sink is the right place to log as it tests the circuit between server and receiver and puts the management of log files in the domain of the application which is where it belongs from a resource, audit and service level management perspective.
I propose that it's time to abrogate responsibility to one party not both, to log asynchronously. Deciding who carries the responsibility however, is not obvious. To understand the problem, lets look at the issues involved (or if you're of the half-empty glass persuasion - who gets the blame). Here's an example of a possible route between two applications:
- Memory/Disk/SAN
- Sender Application
- Application proxy
- TCP/IP Stack
- Software Firewall
- Hardware NIC
- Network infrastructure (various routers/switches/firewalls, lan/wan etc)
- Receivers NIC
- Software Firewall
- TCP/IP Stack
- Application proxy
- Receiver Application
- Memory/Disk/SAN
Sender Logging
If we rely on the sender, there's the immediate advantage that the sender will have to account for the log file space, access control and maintenance. However, from a consumer's point of view - that means there is a lack of control and potentially the case where you require a log and it's been deleted or is offline. From an audit point of view, you have increased the external dependency and hence the risk.
From the senders perspective, consumer lifecycle mangement also becomes slighty more difficult as you now have to poll your customers to see if they are still consuming your data, as it's not unknown for applications to be turned off without turning the feeds off due to lack of knowledge on who to contact.
Receiver Logging
With receiver logging, we have, effectively, a forensic record of the transfer across the stack and have control over the logfile lifecycle. It seems strange to state the obvious, but for higher performance, you should log to local disk, not NFS or SAN storage then back up the log files to resiliient storage.
Service Level Monitoring
A nice addition is to write service level monitoring as part of our productionised system. In this way you can monitor the normal performance of the system and build a predictive capability on applicaiton performance.
Conclusion
Asynchronous logging has the potential to save considerable disk space and processor time whilst reducing maintenance overhead. The receiver/data sink is the right place to log as it tests the circuit between server and receiver and puts the management of log files in the domain of the application which is where it belongs from a resource, audit and service level management perspective.
Saturday, June 09, 2007
Parsi Lamb
this is a fantastic curry and really easy to make. It looks surprisingly ordinary - but the results are fantastic. I've presented the recipe in a way which is designed for the cook - dividing the recipe into lots in sequence. I find this easier than rereading the list of ingredients and the recipe to find out what goes where.
Lot 1
4 Large Onions sliced
4 Green Cardoman pods
3 cloves
4 cm cinnamon stick
Lot 2
50ml of water
2 garlic cloves
thumb sized piece of ginger
(or two tablespoons of ginger and garlic paste)
1.5 tsp of chilli powder
1 tbsp ground cumin (I freshly grind mine)
1.5 tsp of turmeric
1.5 tsp salt
Lot 3
3 tomatoes chopped
750g of cubed leg of lamb
1 dried red chilli
4 green chillies sliced lengthways
200ml water
Lot 4
3 tbsp chopped fresh coriander
Method
1. Heat 6 tbsps of Vegetable oil over a medium heat then add Lot 1, stirring occasionally for 20-25 minutes until the onions are golden.
2. Add the water from Lot 2 which drops the temperature then add Lot 2 and cook for 1 minute
3. Add Lot 3 then simmer for 40 minutes - or until he lamb is tender
4. Add Lot 4 and serve or you can freeze for up to 6 weeks
Serve with boiled basmati rice.
this is a fantastic curry and really easy to make. It looks surprisingly ordinary - but the results are fantastic. I've presented the recipe in a way which is designed for the cook - dividing the recipe into lots in sequence. I find this easier than rereading the list of ingredients and the recipe to find out what goes where.
Lot 1
4 Large Onions sliced
4 Green Cardoman pods
3 cloves
4 cm cinnamon stick
Lot 2
50ml of water
2 garlic cloves
thumb sized piece of ginger
(or two tablespoons of ginger and garlic paste)
1.5 tsp of chilli powder
1 tbsp ground cumin (I freshly grind mine)
1.5 tsp of turmeric
1.5 tsp salt
Lot 3
3 tomatoes chopped
750g of cubed leg of lamb
1 dried red chilli
4 green chillies sliced lengthways
200ml water
Lot 4
3 tbsp chopped fresh coriander
Method
1. Heat 6 tbsps of Vegetable oil over a medium heat then add Lot 1, stirring occasionally for 20-25 minutes until the onions are golden.
2. Add the water from Lot 2 which drops the temperature then add Lot 2 and cook for 1 minute
3. Add Lot 3 then simmer for 40 minutes - or until he lamb is tender
4. Add Lot 4 and serve or you can freeze for up to 6 weeks
Serve with boiled basmati rice.
Friday, June 08, 2007
Time for some Knowledge - The Enhyper Library
http://enhyper.com/lib
Way back in 1998 I had a job looking after the development environment for a tier 1 while back. One of the things I did which proved useful was put together a library of links using software from Gossamer Threads. as a first pass at Knowledge Management.
This was a pre-wiki tool which allowed you to arrange a series of links hierarchically and let users rate them. It also had a newsletter feature which I used to run the Enhyper Newletter, which was a weekly/monthly list of the links I found of note. I emailed this to about 400 people all over the world for a period of several years. These days I use del.icio.us which is pretty neat - but I still use the library occasionally to refer to articles which had seminal influence, like:
Bob Hettinga's Reading List for Financial Cryptographers - I bought most of the books and read them (yes - sad I know)
SET Grid - a comparison between SSL + Credit card and bearer cash - which is a simple but effective analysis of the advantages of bearer cash against SSL/Credit card transaction.
In Operational Research, the Maestro - Conductor of Multimedia Analysis Technologies paper taught me a lot about news analysis and multi-factor target identification.
One thing the tool taught me was that levels of hierachy are an average of 3-4 levels deep - with about 7 being the maximal. Something which is observable in some of the other KM tools I've been involved in over the years.
Anyway, there's around 1400 interesting URL's to browse - so enjoy.
Way back in 1998 I had a job looking after the development environment for a tier 1 while back. One of the things I did which proved useful was put together a library of links using software from Gossamer Threads. as a first pass at Knowledge Management.
This was a pre-wiki tool which allowed you to arrange a series of links hierarchically and let users rate them. It also had a newsletter feature which I used to run the Enhyper Newletter, which was a weekly/monthly list of the links I found of note. I emailed this to about 400 people all over the world for a period of several years. These days I use del.icio.us which is pretty neat - but I still use the library occasionally to refer to articles which had seminal influence, like:
Bob Hettinga's Reading List for Financial Cryptographers - I bought most of the books and read them (yes - sad I know)
SET Grid - a comparison between SSL + Credit card and bearer cash - which is a simple but effective analysis of the advantages of bearer cash against SSL/Credit card transaction.
In Operational Research, the Maestro - Conductor of Multimedia Analysis Technologies paper taught me a lot about news analysis and multi-factor target identification.
One thing the tool taught me was that levels of hierachy are an average of 3-4 levels deep - with about 7 being the maximal. Something which is observable in some of the other KM tools I've been involved in over the years.
Anyway, there's around 1400 interesting URL's to browse - so enjoy.
Wednesday, June 06, 2007
Time for some Code - Shell Script Collection
I started programming in 1986 and largely owe my scripting expertise to Chris Bertin, who I believe works for HP. I found a script of his on a XePIX Gator-L on which I learned to program in C and Shell and it greatly inspired me due to it's technical content and beauty. So here's a collection of scripts which I've written over the years - you can find them on the Enhyper subversion server
There's some useful scripts - the biggest and most sophisticated is envbuild which automated a three day piece of work down to minutes. It automated the building of a sophisticated database schema and the underlying disk placement. There's some neat techniques in there - one where bc(1) is used to perform an iterative calculation of Informix data spaces.
I started programming in 1986 and largely owe my scripting expertise to Chris Bertin, who I believe works for HP. I found a script of his on a XePIX Gator-L on which I learned to program in C and Shell and it greatly inspired me due to it's technical content and beauty. So here's a collection of scripts which I've written over the years - you can find them on the Enhyper subversion server
There's some useful scripts - the biggest and most sophisticated is envbuild which automated a three day piece of work down to minutes. It automated the building of a sophisticated database schema and the underlying disk placement. There's some neat techniques in there - one where bc(1) is used to perform an iterative calculation of Informix data spaces.
Back to Basics in the Tier 1's?
There's a wind of change sweeping the CIty which I believe is being driven by the technology arms race. I'm beginning to hear rumours of the project manager culture being dismantled in some of the tier 1's - there seems to be a realisation that the people that matter are those who write code and deliver projects - not the project managers and paper architects which litter organisational structures.
The career path in a typical IB starts with graduate recruitment - which is a two year slog usually involving coding of peripheral functionality (if you're lucky) then jump ship to get more cash to another IB where you do more coding. But coding is hard and you need to get out of it so you buy one of those blue shirts with polo logo on it and a pair of chinos and hey presto - instant project manager. You spend your time in meetings and making technology decisions, lunching with the vendors, put on three stone in weight and develop a bad blackberry habit. You build teams, manage the politics and deliver what you think the business wants.
Deep down though, you know that the guys really controlling the show are the developers - they hold the key and you know it - so the last thing you do is let them talk to the business because as soon as they do - people will start to question what you do and boom - you're out the door.
It's an all too familiar pattern I'm afraid - but it wasn't always like this. When I started in IT back in 1986, Unix was rattling the cage of the mainframes. Everyone in software development at that time was competent scripters and programmers and the industry was quite small. However, back in the early nineties, I was the only guy sitting on the train with a computer book then I started to notice a lot of other people reading "dummies guide to whatever". This was the rise of the supply led consultancies who made a killing by overstaffing IT projects.
Suddenly everyone as getting into IT - this was the new way to make money. In reality, projects were delivered by small teams or even individuals. I remember working on one large project where I was the only guy in a team of 20 who could program - I delivered the whole data migration piece whilst the rest of the team wrote docs - an no, it didn't make me feel important - I just felt sorry for the poor client who was paying through the nose.
So the tier 1's are apparently making a strong effort to hire hybrid type lead developers - people who combine hands-on development / lead small teams and are able to run day-to-day delivery of projects. They want 'innovators' - something the consultancy led culture which still blights our industry seeks to deprecate.
If they're serious about hiring talent, then get rid of the non-programmers, abolish the title of architect and send them back to the coding front, replace project managers with tools which automate their function - tools like xProcess which builds the project plan in real-time and enables capture of processes so that you can do project estimation based on empirical data - not on invented deadlines.
There's a wind of change sweeping the CIty which I believe is being driven by the technology arms race. I'm beginning to hear rumours of the project manager culture being dismantled in some of the tier 1's - there seems to be a realisation that the people that matter are those who write code and deliver projects - not the project managers and paper architects which litter organisational structures.
The career path in a typical IB starts with graduate recruitment - which is a two year slog usually involving coding of peripheral functionality (if you're lucky) then jump ship to get more cash to another IB where you do more coding. But coding is hard and you need to get out of it so you buy one of those blue shirts with polo logo on it and a pair of chinos and hey presto - instant project manager. You spend your time in meetings and making technology decisions, lunching with the vendors, put on three stone in weight and develop a bad blackberry habit. You build teams, manage the politics and deliver what you think the business wants.
Deep down though, you know that the guys really controlling the show are the developers - they hold the key and you know it - so the last thing you do is let them talk to the business because as soon as they do - people will start to question what you do and boom - you're out the door.
It's an all too familiar pattern I'm afraid - but it wasn't always like this. When I started in IT back in 1986, Unix was rattling the cage of the mainframes. Everyone in software development at that time was competent scripters and programmers and the industry was quite small. However, back in the early nineties, I was the only guy sitting on the train with a computer book then I started to notice a lot of other people reading "dummies guide to whatever". This was the rise of the supply led consultancies who made a killing by overstaffing IT projects.
Suddenly everyone as getting into IT - this was the new way to make money. In reality, projects were delivered by small teams or even individuals. I remember working on one large project where I was the only guy in a team of 20 who could program - I delivered the whole data migration piece whilst the rest of the team wrote docs - an no, it didn't make me feel important - I just felt sorry for the poor client who was paying through the nose.
So the tier 1's are apparently making a strong effort to hire hybrid type lead developers - people who combine hands-on development / lead small teams and are able to run day-to-day delivery of projects. They want 'innovators' - something the consultancy led culture which still blights our industry seeks to deprecate.
If they're serious about hiring talent, then get rid of the non-programmers, abolish the title of architect and send them back to the coding front, replace project managers with tools which automate their function - tools like xProcess which builds the project plan in real-time and enables capture of processes so that you can do project estimation based on empirical data - not on invented deadlines.
Tuesday, June 05, 2007
Strategy for migrating to new messaging middlware
I was chatting to an old accomplice the other day who headed up FX architecture for a tier 1 in the US about my new position with 29West - apart from saying they were a great company with great tech, he said they has decided not to use the LBM product because of the pain of migration from the existing Tibco RV middleware. So part of my new role will be to create a migration path which is as pain free for our customers.
This is exactly the problem we had to solve in my current assignment (I don't start with 29West until late July btw) - how do we integrate Wombat (which runs over LBM) into an organisation which relies on another product successfully? The strategy we adopted was to use a straightforward design pattern to implement a piece of throw-away middleware combined with a touch of perception management - always important.
Project Evangelism Framework
As important as the technical feasibility and superiority of a candidate solution is the management of the change process. I've amalgamated some techniques which are based on open source tools, allowing debate and development of a community of interest around the solution. The idea is not new and was inspired by reading the Cluetrain Manifesto as a younger man.
Basically we have several chat channels - one for the project where the public can ask questions and get expert answers, one for the technology, so that we can fight our technollectual battles. A blog for a record of meetings, decision, vendor liaison and general project karma and lastly a wiki where we can disseminate project documents and build a support knowledgebase. I'll blog again about this as I have some theories about the roles people play which I'll elaborate.
Using this setup has surprising effects and advantages - meetings are now a matter of record for public consumption which modifies people behaviour towards cooperation and congeniality. Vendors are treated fairly and openly, decisions vindicated or challenged thereby using the wisdom of crowds. Project documentation is completed and of a higher standard. People behave more professionally. Business customers can see the real state of the project.
Technological Framework
First thing is to get the new infrastructure up to near production level on dev hardware with the minimum of fuss and footprint. Then productionise it so that there's autostart/stop, logfile monitoring and maintenance etc. Next you need to develop some examples or tailor the ones supplied to assist with onboarding.
It helps if you can develop the code to fill patterns of use within the organisation - this way the developers need minimum effort to integrate. Be ready to answer support from developers and sort out issues quickly. This way the community will begin to trust you because if you lose people early through poor infra, support of lousy code, it's going to be all the harder to get your new solution adopted.
You need to run the two infrastructures in parallel but new development needs to use the new infrastructure and existing code phased over to the new infrastructure once it has been proven.
To help here, I implemented daemon process which implemented a Facade/Proxy/Adaptor pattern. This can be written in whatever language you like, but I'd recommend C because it just so happens that it's the lowest common denominator. This allows you to take data from any source (databases, flat files, memory maps, MQ, Tibco RV, Emma, Vhayu, FAME etc). and provide it to the existing and new clients. In time, the new clients bypass the adaptor. This allows you to maintain the client functionality whilst developing new interfaces.
I was chatting to an old accomplice the other day who headed up FX architecture for a tier 1 in the US about my new position with 29West - apart from saying they were a great company with great tech, he said they has decided not to use the LBM product because of the pain of migration from the existing Tibco RV middleware. So part of my new role will be to create a migration path which is as pain free for our customers.
This is exactly the problem we had to solve in my current assignment (I don't start with 29West until late July btw) - how do we integrate Wombat (which runs over LBM) into an organisation which relies on another product successfully? The strategy we adopted was to use a straightforward design pattern to implement a piece of throw-away middleware combined with a touch of perception management - always important.
Project Evangelism Framework
As important as the technical feasibility and superiority of a candidate solution is the management of the change process. I've amalgamated some techniques which are based on open source tools, allowing debate and development of a community of interest around the solution. The idea is not new and was inspired by reading the Cluetrain Manifesto as a younger man.
Basically we have several chat channels - one for the project where the public can ask questions and get expert answers, one for the technology, so that we can fight our technollectual battles. A blog for a record of meetings, decision, vendor liaison and general project karma and lastly a wiki where we can disseminate project documents and build a support knowledgebase. I'll blog again about this as I have some theories about the roles people play which I'll elaborate.
Using this setup has surprising effects and advantages - meetings are now a matter of record for public consumption which modifies people behaviour towards cooperation and congeniality. Vendors are treated fairly and openly, decisions vindicated or challenged thereby using the wisdom of crowds. Project documentation is completed and of a higher standard. People behave more professionally. Business customers can see the real state of the project.
Technological Framework
First thing is to get the new infrastructure up to near production level on dev hardware with the minimum of fuss and footprint. Then productionise it so that there's autostart/stop, logfile monitoring and maintenance etc. Next you need to develop some examples or tailor the ones supplied to assist with onboarding.
It helps if you can develop the code to fill patterns of use within the organisation - this way the developers need minimum effort to integrate. Be ready to answer support from developers and sort out issues quickly. This way the community will begin to trust you because if you lose people early through poor infra, support of lousy code, it's going to be all the harder to get your new solution adopted.
You need to run the two infrastructures in parallel but new development needs to use the new infrastructure and existing code phased over to the new infrastructure once it has been proven.
To help here, I implemented daemon process which implemented a Facade/Proxy/Adaptor pattern. This can be written in whatever language you like, but I'd recommend C because it just so happens that it's the lowest common denominator. This allows you to take data from any source (databases, flat files, memory maps, MQ, Tibco RV, Emma, Vhayu, FAME etc). and provide it to the existing and new clients. In time, the new clients bypass the adaptor. This allows you to maintain the client functionality whilst developing new interfaces.
Friday, June 01, 2007
Fun with sort(1)
There are some unix utilities which give it a bad name - prime culprits are sed(1) - just read the man(1) page and you'll understand why. I think sort(1) is pretty abstruse too - I've been using it to manipulate log files which monitor market data info being pushed in and out of wombat.
The trouble with log files is that they usually are full of everything - which is fine if you have the time or patience to extract the information you require, however, now that we're shoving hundreds of trades through the algo system, this generates hundreds of thousands of log messages, I can no longer use vi(1) - the unix editor, to view them, as it runs out of space for the temp file. This mans resorting to all sorts of sed/awk/grep nonsense in order to extract the info we need. The criteria for this embryonic scriptette was to order entries according to a suffix alphabetically, then order numerically ascending within that suffix. Here's a script which does the job. The input data looks like this:
10:23:34.323 : 5 3,2760.MAIN-EXC.Dx.LT {[1]=24 [2]=2 [3]=7360 [4]=36.730340 [5]=28.032340 [6]=2007-05-29 09:11:00Z}
10:23:34.541 : 6 3,2760.MAIN-EC.DM.SS {[1]=24 [2]=1 [3]=7260 [4]=34.730000 [5]=28.000000 [6]=2007-05-29 10:23:34.095576000Z}
All 598344 lines of it. The first line sorts on the field "LT" and "SS" above and gives us a list of subsets that we need to process:
FIELDS="`sort -u -t '.' -k 5,5.2 MarketDataServer0.log | sed 's/.*\.\(.*\) {.*/\1/'`"
Now we create a file callled out
> out
for CODE in $FIELDS
do
sed -n '/.*\.'"$CODE"' {.*/p' MarketDataServer0.log | sort -n -t '>' -k 2,2 >> out
done
Then we cut out the entries for each "code" then pass them to our sort command which uses the > as a field delimiter and sorts numerically on the second field - ugly but necessary. No error handling or parameter passing yet - but this saves a whole lot of pain. Looks painful? sure but it's the sort of thing you just can't do on windows (well without Cygwin anyway)
There are some unix utilities which give it a bad name - prime culprits are sed(1) - just read the man(1) page and you'll understand why. I think sort(1) is pretty abstruse too - I've been using it to manipulate log files which monitor market data info being pushed in and out of wombat.
The trouble with log files is that they usually are full of everything - which is fine if you have the time or patience to extract the information you require, however, now that we're shoving hundreds of trades through the algo system, this generates hundreds of thousands of log messages, I can no longer use vi(1) - the unix editor, to view them, as it runs out of space for the temp file. This mans resorting to all sorts of sed/awk/grep nonsense in order to extract the info we need. The criteria for this embryonic scriptette was to order entries according to a suffix alphabetically, then order numerically ascending within that suffix. Here's a script which does the job. The input data looks like this:
10:23:34.323 : 5 3,
10:23:34.541 : 6 3,
All 598344 lines of it. The first line sorts on the field "LT" and "SS" above and gives us a list of subsets that we need to process:
FIELDS="`sort -u -t '.' -k 5,5.2 MarketDataServer0.log | sed 's/.*\.\(.*\) {.*/\1/'`"
Now we create a file callled out
> out
for CODE in $FIELDS
do
sed -n '/.*\.'"$CODE"' {.*/p' MarketDataServer0.log | sort -n -t '>' -k 2,2 >> out
done
Then we cut out the entries for each "code" then pass them to our sort command which uses the > as a field delimiter and sorts numerically on the second field - ugly but necessary. No error handling or parameter passing yet - but this saves a whole lot of pain. Looks painful? sure but it's the sort of thing you just can't do on windows (well without Cygwin anyway)
Thursday, May 31, 2007
Red Pepper and Coconut Chutney
Outside of the American psyche we have other programmer food than pizza which fuels software development - a lot of people who work in the Investment Banking community are addicted to curry and there's a good choice of hard core restaurants nearby like one of my favourites, the Lahore Kebab House. So as well as software wisdom, we'll be posting the occasional recipe - here's a start. This chutney only keeps for about a week in a cold fridge but is wonderful - if you can get it - use Kashmiri chilli powder for a more mellow flavour...
Ingredients
Two large red peppers (Bell or Capsicum)
Two teaspoons of roasted cumin seeds, finely ground
Two cloves of garlic
Two tablespoons of dessicated coconut
Half a teaspoon of salt
Half a teaspoon of hot chilli powder
Two tablespoons of water
Method
Dry roast the cumin seeds in a frying pan, colouring them to your taste, the darker the
roast, the stronger the taste. Grind in a spice mill or coffee grinder to a fine powder
Deseed the peppers and cut into small chunks suitable to put into a hand blender jug, then add
all the ingredients. Blend with a handblender and decant to a sterilised jar which should be refrigerated.
This is an excellent accompaniment to cold meats, dosas and any curry, particularly fish/shellfish.
Outside of the American psyche we have other programmer food than pizza which fuels software development - a lot of people who work in the Investment Banking community are addicted to curry and there's a good choice of hard core restaurants nearby like one of my favourites, the Lahore Kebab House. So as well as software wisdom, we'll be posting the occasional recipe - here's a start. This chutney only keeps for about a week in a cold fridge but is wonderful - if you can get it - use Kashmiri chilli powder for a more mellow flavour...
Ingredients
Two large red peppers (Bell or Capsicum)
Two teaspoons of roasted cumin seeds, finely ground
Two cloves of garlic
Two tablespoons of dessicated coconut
Half a teaspoon of salt
Half a teaspoon of hot chilli powder
Two tablespoons of water
Method
Dry roast the cumin seeds in a frying pan, colouring them to your taste, the darker the
roast, the stronger the taste. Grind in a spice mill or coffee grinder to a fine powder
Deseed the peppers and cut into small chunks suitable to put into a hand blender jug, then add
all the ingredients. Blend with a handblender and decant to a sterilised jar which should be refrigerated.
This is an excellent accompaniment to cold meats, dosas and any curry, particularly fish/shellfish.
OR49 Keynote Speech in the Knowledge Management Stream
I'm giving the keynote in the KM stream at this year's Operational Research Society Conference OR49 based on a stream of research which started about 8 years ago after reading a paper on newsgroup cluster analysis called telltale. Here's the abstract:
I'm giving the keynote in the KM stream at this year's Operational Research Society Conference OR49 based on a stream of research which started about 8 years ago after reading a paper on newsgroup cluster analysis called telltale. Here's the abstract:
"It is proposed to summarise and statistically categorise multiple public and private information feeds to produce centroids directed by a combination of user constructed keywords and analysis of previously archived or disseminated knowledge. Social and physical networks will be extracted for temporal analysis and association projection. Comprehensive analysis of centroid relationships across sectors, categories and physical location will give a statistical event prediction capability and lead to the discovery of hidden relationships and associated events. End-users will construct a hierarchical keyword tree which will contain individual articles, summarisations, centroids or sets of related centroids. Users will also participate in a community of interest which they may form inter or intra-federation in order to disseminate emerging events or explicit knowledge. The system has applicability to financial market analysis, law enforcement and intelligence analysis."
Tuesday, May 29, 2007
Why Events are a Bad Idea (for high concurrency servers)
And Why Mixing Events and Threads is Even Worse
I've just been badly burned by mixing two paradigms - threading and events. All was working fine until the day before go live; the testers started to pour 400 stock baskets through the system rather than 40 which resulted in one of those issues which make your heart sink as a programmer. Between 5 and 15 stocks would go into pending which meant that there was a threading issues somewhere and it was go live that evening. Tracking it down was to prove difficult due to poor separation of duties between the threads resulting from the design having its origins as a single threaded, serial set of calls which took data from an event generated on one side and generated a modified message dispatched to the receiver and vice versa. In retrospect, the problem would have been solved by having single queue between the two threads, following the asynchronous put/take connector pattern. This would have ensured complete separation and higher throughput.
Mutex Spaghetti
As it happens, it was impossible in the time available, to redesign the solution and the simplest course of action was to go back, reluctantly, to a single threaded implementation. Time to test was a major factor here, however, not before some time was spent playing the mutex game. I started mutexing at a very low granularity which appeared to fix the issue (or break things completely) almost - I was down to one or two trades pending - which was not good enough.
Adding additional mutexes, it quickly became apparent that it's very easy to get in a mess either by blocking on an already locked mutex or by adding too many mutexes which means you end up in a mess anyway. After four hours of mutex soup - I made the decision to remediate the code back to single-threaded code. Performance, at the moment is not an issue.
So the moral of the story is look for a design pattern which fits your code - understand it and think the design through, we seldom have time, but if you can, use UML to build a sequence diagram - then you'll see the call chain and understand conflict between threads.
A friend of mine related the story of Sun's attempts to make the solaris kernel mt safe - this was a much harder task than they anticipated. Most of their effort was centred around protecting the key data structures rather and changing the programming paradigm so that users took the responsibility for data allocation.
Another pointed out an article on slashdot this morning "Is Parallel Programming Just Too Hard?" which raises some concerns which, as we can see from the above, seem to be valid. If you're going to write parallel threads, you have to spend time on the design - it takes three times the effort and you really need to use patterns, example code and sequence diagrams. I hacked it and got away with it - to a point. If performance proves an issue, and you can bet it will at some stage soon, then it will be back to the drawing board - and this time, the design will come first.
Interestingly, threading versus events is an interesting debate which this paper: "Why Events Are A Bad Idea (for high concurrency servers)" argues well and threading comprehensively wins the day as a paradigm over events. I've always been of the opinion that separation of duties via a thread is intuitively faster than event dispatch - this paper goes a way to prove my intuition right.
Finally, there's a well deserved mention for Functional Programming Languages such as Erlang and Haskell, which has much promise for multi-core programming as outlined in these excellent slides: Data Parallel for Haskell.
And Why Mixing Events and Threads is Even Worse
I've just been badly burned by mixing two paradigms - threading and events. All was working fine until the day before go live; the testers started to pour 400 stock baskets through the system rather than 40 which resulted in one of those issues which make your heart sink as a programmer. Between 5 and 15 stocks would go into pending which meant that there was a threading issues somewhere and it was go live that evening. Tracking it down was to prove difficult due to poor separation of duties between the threads resulting from the design having its origins as a single threaded, serial set of calls which took data from an event generated on one side and generated a modified message dispatched to the receiver and vice versa. In retrospect, the problem would have been solved by having single queue between the two threads, following the asynchronous put/take connector pattern. This would have ensured complete separation and higher throughput.
Mutex Spaghetti
As it happens, it was impossible in the time available, to redesign the solution and the simplest course of action was to go back, reluctantly, to a single threaded implementation. Time to test was a major factor here, however, not before some time was spent playing the mutex game. I started mutexing at a very low granularity which appeared to fix the issue (or break things completely) almost - I was down to one or two trades pending - which was not good enough.
Adding additional mutexes, it quickly became apparent that it's very easy to get in a mess either by blocking on an already locked mutex or by adding too many mutexes which means you end up in a mess anyway. After four hours of mutex soup - I made the decision to remediate the code back to single-threaded code. Performance, at the moment is not an issue.
So the moral of the story is look for a design pattern which fits your code - understand it and think the design through, we seldom have time, but if you can, use UML to build a sequence diagram - then you'll see the call chain and understand conflict between threads.
A friend of mine related the story of Sun's attempts to make the solaris kernel mt safe - this was a much harder task than they anticipated. Most of their effort was centred around protecting the key data structures rather and changing the programming paradigm so that users took the responsibility for data allocation.
Another pointed out an article on slashdot this morning "Is Parallel Programming Just Too Hard?" which raises some concerns which, as we can see from the above, seem to be valid. If you're going to write parallel threads, you have to spend time on the design - it takes three times the effort and you really need to use patterns, example code and sequence diagrams. I hacked it and got away with it - to a point. If performance proves an issue, and you can bet it will at some stage soon, then it will be back to the drawing board - and this time, the design will come first.
Interestingly, threading versus events is an interesting debate which this paper: "Why Events Are A Bad Idea (for high concurrency servers)" argues well and threading comprehensively wins the day as a paradigm over events. I've always been of the opinion that separation of duties via a thread is intuitively faster than event dispatch - this paper goes a way to prove my intuition right.
Finally, there's a well deserved mention for Functional Programming Languages such as Erlang and Haskell, which has much promise for multi-core programming as outlined in these excellent slides: Data Parallel for Haskell.
Tuesday, May 22, 2007
From CORBA to Command and Control Web Services
I was inspired to write a paper on "web services" back in 1999, when people in the banking community had heard of XML but had no idea what they could do with it. My inspiration had been the IATA protocol which I had seen used to automate the aircraft industry and heard rumours that the US Navys' supply line was also managed by these straightforward ascii messages which began with ZCZC and ended with NNNN.
I had just spent a couple of years boning up on CORBA - reading several books, trying to get a copy of Iona's Orbix product so that I could teach myself a new set of skills which would keep me fed for the foreseeable future. I liked the idea of CORBA - in particular the trader service - whereby you could look up a "service" and invoke dynamically. However, when I finally moved to a bank which could afford the technology - it was clear that all was not well. I had expected some form of centrally managed infrastructure and a set of services which I could use/add to. I wholly expected to find analytics (maths libraries) and a range of business "services" - instead I found a ORB on every desk. This meant that any solution developed would, in essence be, be standalone - thereby defeating any advantage of ORB technology. True to form, I discovered that it was indeed the developers who had mandated ORB technology, for the same reasons I had - career sustainment.
But this is not the real reason why ORB technology failed - the principle two were unreliability and complexity. You see the thing most banks don't tell you is that they use an awful lot of perl to "munge" data - perl is fast, quick to develop, easy to understand. ORB's on the other hand were developed by a small (in open source terms) team -who had a limited bandwidth - not quick enough to test the solution or fix the bugs quickly. To use an ORB for a solution costs money in licences, support and requires clever (i.e. expensive) programmers to implement. Once engineered, it was by no means performant - needing big metal to run - but the worst of all was the bugs - it just wasn't reliable enough for serious production use. Combine this with a long fix lifecycle and you'll understand why very few ORB solutions are to be found in any organisation.
So web services to the rescue - well - not quite yet - not that there's anything wrong with the techology - especially the security aspects. It has been possible to tunnel web services over ssh for a long time - combine this with X.509 certs and you have authenticated end points. No, the problem with implementing web services is mindset - people just don't understand the issues. Two of the most common complaints are security (see above) and verbosity (payloads increase by 30% - gasp!) This misses the point. The protagonists seem to want to design a web service that is meant to handle a substantial amount of data and be invoked repetitively. In the real world, web services should be used for command and control. The analogy of the data feed (i.e. comma seperated file sent somewhere for processing) still exists - only this way you have a way of programmatically orchestrating the solution inter enterprise - and that's what web services are all about.
References
I had just spent a couple of years boning up on CORBA - reading several books, trying to get a copy of Iona's Orbix product so that I could teach myself a new set of skills which would keep me fed for the foreseeable future. I liked the idea of CORBA - in particular the trader service - whereby you could look up a "service" and invoke dynamically. However, when I finally moved to a bank which could afford the technology - it was clear that all was not well. I had expected some form of centrally managed infrastructure and a set of services which I could use/add to. I wholly expected to find analytics (maths libraries) and a range of business "services" - instead I found a ORB on every desk. This meant that any solution developed would, in essence be, be standalone - thereby defeating any advantage of ORB technology. True to form, I discovered that it was indeed the developers who had mandated ORB technology, for the same reasons I had - career sustainment.
But this is not the real reason why ORB technology failed - the principle two were unreliability and complexity. You see the thing most banks don't tell you is that they use an awful lot of perl to "munge" data - perl is fast, quick to develop, easy to understand. ORB's on the other hand were developed by a small (in open source terms) team -who had a limited bandwidth - not quick enough to test the solution or fix the bugs quickly. To use an ORB for a solution costs money in licences, support and requires clever (i.e. expensive) programmers to implement. Once engineered, it was by no means performant - needing big metal to run - but the worst of all was the bugs - it just wasn't reliable enough for serious production use. Combine this with a long fix lifecycle and you'll understand why very few ORB solutions are to be found in any organisation.
So web services to the rescue - well - not quite yet - not that there's anything wrong with the techology - especially the security aspects. It has been possible to tunnel web services over ssh for a long time - combine this with X.509 certs and you have authenticated end points. No, the problem with implementing web services is mindset - people just don't understand the issues. Two of the most common complaints are security (see above) and verbosity (payloads increase by 30% - gasp!) This misses the point. The protagonists seem to want to design a web service that is meant to handle a substantial amount of data and be invoked repetitively. In the real world, web services should be used for command and control. The analogy of the data feed (i.e. comma seperated file sent somewhere for processing) still exists - only this way you have a way of programmatically orchestrating the solution inter enterprise - and that's what web services are all about.
References
[1] The Role of XML in Enabling Business Solutions Built from Collaborative Web-Based Services. Burnett, Papiani, Dec 1999
Friday, May 11, 2007
Grid Computing Slowly Falls From Grace
I spoke at the The 451 Groups ECS Summit, London Regional Roundtable on Thursday, May 10, 2007 at the Brewery at Chiswell Street in London where the focus was supposed to be on Grid computing, the focus of much research by the 451 Group over the last year.
I noticed the trend in early 2006 that term Grid was losing it market appeal. It started disappearing from job titles, to be replaced by High Performance Computing (HPC) - this has now become mainstream with at least two IB's having the post. Another two trends were noted - data architecture is now high on the agenda and low-latency messaging is seen as the crucial to meeting the challenges of high transaction volumes.
So where's Grid computing heading? The general concensus were that the technology had not lived up to its promises in terms of performance and in particular: managability, security and lack of data architecture seemed to be the prime areas of concern. Grid in IB needs to grow up - in essence, despite the marketing hype, current grid offerings are little more than cluster computing applied to a few niche areas in IB such as CDO/CDO squared (btw check out www.cdo2.com - run by an acquaintance of mine - a real web service with real customers running on BLAST
The reality is that most jobs can be done now on multi-core, large memory machines. An 8-core/4 CPU machine with 64GB memory are a lot cheaper than an investment in grid - and you get a lot of processing power now for not a lot of cash. Nodes can be chained together with 1GE nics and switch to produce an effective HPC cluster and you can either use your own threaded app with memcached or have a play with jini.
I noticed the trend in early 2006 that term Grid was losing it market appeal. It started disappearing from job titles, to be replaced by High Performance Computing (HPC) - this has now become mainstream with at least two IB's having the post. Another two trends were noted - data architecture is now high on the agenda and low-latency messaging is seen as the crucial to meeting the challenges of high transaction volumes.
So where's Grid computing heading? The general concensus were that the technology had not lived up to its promises in terms of performance and in particular: managability, security and lack of data architecture seemed to be the prime areas of concern. Grid in IB needs to grow up - in essence, despite the marketing hype, current grid offerings are little more than cluster computing applied to a few niche areas in IB such as CDO/CDO squared (btw check out www.cdo2.com - run by an acquaintance of mine - a real web service with real customers running on BLAST
The reality is that most jobs can be done now on multi-core, large memory machines. An 8-core/4 CPU machine with 64GB memory are a lot cheaper than an investment in grid - and you get a lot of processing power now for not a lot of cash. Nodes can be chained together with 1GE nics and switch to produce an effective HPC cluster and you can either use your own threaded app with memcached or have a play with jini.
Subscribe to:
Posts (Atom)
View Graeme Burnett's profile