Tuesday, July 08, 2008
If they follow my logic, they'll use community expertise to do the analysis, annotation, recategorisation and dissemination to peer interest groups. Then it will be possible to build a reputation based analytics platform to summarise the trend, relate it to previous events and capture side-effects...
Tuesday, May 27, 2008
Friday, May 23, 2008
Last night's get together was attended by about 15 men (women in technology take note) from a variety of backgrounds (students, cryptographers, investment bank staff.) Beers were consumed and nonsense talked - then we toddled off to Kurz and Lang for some bratwurst around eight as is our tradition (from our days in Zurich with a well known Swiss bank.)
The conclusion of the evening was that there is a place for Haskell in the city - why? Well here's two extracts from and article on Haskell and Performance on Planet Haskell written by Neil Mitchell:
Haskell's multi-threaded performance is amazing
A lot of clever people have done a lot of clever work on making multi-threaded programming in Haskell both simple and fast. While low-level speed matters for general programming, for multi-threaded programming there are lots of much higher-level performance considerations. Haskell supports better abstraction, and can better optimise at this level, outperforming C.
The trend is for higher-level optimisation
As time goes buy, higher-level programs keep getting faster and faster. The ByteString work allows programmers to write high-level programs that are competitive with C. Performance enhancements are being made to the compiler regularly, pointer tagging, constructor specialisation etc. are all helping to improve things. More long term projects such as Supero and NDP are showing some nice results. Optimisation is a difficult problem, but progress is being made, allowing programs to be written in a higher-level.
and that it would be a good idea to form a Functional Programming get together on a reasonably regular basis - so watch this space.
Wednesday, May 21, 2008
I gave this presentation at IISyG in 2005 to an audience of Infosec specialists. The talk was inspired by Dr Sally Leivesly of Newrisk Ltd who regularly appears on the BBC after terrorist incidents. She outlined combinatorial threat and area denial scenarios.
Another influence was the amount of fuel required to power centralised data centers which, in times of crisis, may be hard to supply regularly. Then there is the strange notion that staff will have the desire and ability to travel to some remote data center regularly which frankly beggars belief. The whole BCP/DR scenario is, sadly, driven by regulators, auditors and policy wonks.
So my take on the answer is that we need fabric/cloud computing, micro data centers and UWB MAN networks which can be up and running in an instant. Combined with the "crisis desktop", a bootable OS on a memory stick which can form a secure platform to access your virtual desktop. The parts are all there - all we need is the strategists to catch up...
Monday, May 19, 2008
Wednesday, May 14, 2008
The event is being described as: "Graeme’s rant will be looking into how the Security policy landscape is changing and how policies can no longer afford to be a loose collection of word documents seldom referred to. He will talk about how it is the business logic and rules of the organisation which are now becoming executable thanks to web services."
Details can be found here
Monday, May 12, 2008
Risk assessment is a mandatory activity under all the Information Security policy frameworks e.g. COBIT, ISO27001. However, risk assessment methodologies are point-in-time and don't deal with incremental risk as the above process does.
We will be meeting on the 22nd May - details are here
To participate, please add your name to the Doodle Poll entitled Haskell + Beer. This is so we know how many buckets of chips to order...
As All Bar One has now closed, we will be in the Bishop's Finger, Smithfield. I'm planning on getting there around 530pm. My mobile number is zero seven eight three zero three six eight zero two four - when you get there give me a call. There's a good chance we'll get some of the beers paid for too but no promises yet.
Topics for discussion/contention
- Crypto in Haskell (Dominic is Mr Haskell Crypto btw). Opportunities for parallelism (e.g.)
- Is Haskell and multi-core the best solution to meet the projected demand for simulation in finance or has FPGA yet to have it's day?
- Challenges of Haskell and high performance messaging integration. 2.4 million messages per second (not transactions) are now possible using tools like LBM from 29West.
- The K language (an APL derivative), Fortran, Objective-C, LISP and Smalltalk gained a small foothold in the eighties/nineties in quantitative analytics but C++ remained the main language of implementation due to availability of skillset (K programmers are like hen's teeth.) Java and now C# are now the de facto implementation languages but have serious issues for high frequency finance - will Haskell be able to make an impact or will it be another marginal language used by quants.
- Is Haskell capable of high performance?
- Haskell and distributed memory (memcached) performance aspects
- Haskell Data Parallelism versus state of the art FPGA approaches. Check out HPCPlatform's offering btw.
- I hear rumours that Haskell is being rolled out to the HPC team of a major investment bank in Canary Wharf - watch this space...
Sunday, May 11, 2008
I've been invited onto the security panel by Ben Laurie of Google (Caja) and Apache SSL fame. I'm going to be talking about beancounter-led regulation in financial markets and how it has killed security engineering...
Thursday, May 08, 2008
The article identifies three key abstractions, hierarchical thread groups, shared memories and barrier synchronisation. CUDA is based on C with extensions for parallelisation - much like Handel-C. The difference is that Handel-C was FPGA based whilst CUDA is for GPU with its built-in floating point capability. There are simple and straightforward code examples showing parallel threading and memory sharing which was always an issue in my mind with FPGA: the leap of faith with Handel-C was what to do with the data set you generated in a Monte Carlo simulation.
This question has been perplexing developers on the CUDA forums at Nvidia too - but it looks like there's been progress as outlined in this presentation on Monte Carlo Options Pricing paper on the Nvidia developer site. However, the algorithm outlined in the paper is trivial, the secret being the generation of quasi-random numbers enabling quick convergence. Then filtration close to the data so you're not schlepping large lumps of data unnecessarily.
Then the next logical step is to make this a service. The appetite is reckoned to be about 5 trillion simulations per day in the average organisation according to a quant chum of mine. Combine this with S3 for asynchronous storage and you have the makings of a nice little business I think.
Wednesday, May 07, 2008
"You will have previous experience of designing and building distributed, fault tolerant systems in a commercial environment. Experience of multi threading, socket programming, network programming and functional programming languages (Haskell, Ocaml, F#) will be an advantage."
"Experience with functional languages such as Haskell, Erlang, F#, Scheme, LISP, etc., are greatly appreciated."
Bit of a scattergun approach in the last example perhaps? I wonder who writes the job specs - I guess the bizerati analystas high on the latest marketing speak. I'm still confused about is the insistence on C++ with it's late binding and poor library coverage (compared to Java.) As illustrated by this graph from the paper below, C++ is slower than C - so why would you want to use it when speed is the ultimate criterion? Beats me.
An empirical comparison of C, C++, Java, Perl, TCL and REXX for search/string processing
I'm also bemused at the use of C# - in light of the recent debacles at the LSE and TSE.
One wonders who is in charge of algo and program trading strategy. I do hope they realise the advantages of a monadic language are not without performance implications and that without stream fusion and massively multi-core processors (with FPUs) the performance gains they seek are going to be rather elusive. Then there's the data issue - you have to crack that particular nut - and here's a clue - the answer's not xml or any of its bloated siblings.
Friday, May 02, 2008
Each virus is rated using the particular manufacturer's rating mechanism giving an independent benchmark. Escalation for high impact viruses is immediate and possibly would result in the security incident management process being invoked. Cumulative threat is also reported if there are more than 50% of medium threat.
Trend and Pattern analysis
I've instituted a qualitative and quantitative approach to virus analysis which delivers a modicum of business intelligence and facilitates escalation based on threat level.
Trending is twofold: year to date and historical highlighting monthly and annual trends. Seasonality is present in the year-to-date statistics whilst long term trend can be gauged over five years.
Patterns in virus targeting are analysed per geographic location, business area, workstation and user. A history of infection is maintained so that repeated infection can be analysed further.
An adaptive approach is taken to monitoring frequency. If an uplift in activity of 20% incurs, the monitoring frequency is doubled to a minimum of daily monitoring. If the trend is reversed for three monitoring periods, the frequency is halved to a maximum of monthly sampling.
Sunday, April 27, 2008
- Half a cup of Scott's Porage Oats
- Two handfuls of Holland and Barrett Omega Sprinkle
- One handful of Holland and Barrett dried goji berries, blueberry and cranberries mixed in equal proportions
- Tablespoon of Good Oil
- Quarter of a pomegranate seeded
Monday, April 21, 2008
Thursday, April 17, 2008
We have a diverse workforce who speak many languages including: Hindi, Finnish, Russian, Bulgarian etc. Deciding which languages should cover was a simple conundrum to the in-house legal expert - we only need to cover the languages in which we conduct our business, (in the clients case) namely English, French, German and Japanese.
The stock English dictionary supplied with MIMESweeper contained mainly anatomical parts and the usual canonical profanity. The strategy of this particular stop list seemed to be male, heterosexual, white, American and lower class. I believe this is where the fecund British mind can add value: there are several online dictionaries are sources of expletives and profanity (e.g. The Alternative Dictionaries, Roger's Profanisaurus etc.) Other interesting sources are SMS acryonyms which are increasingly being used by the younger generation. There are even translation sites which can change English into "text speak". and several words used by the slightly more educated such as onanist. In the latter case, it would be hard to argue that any of these words, phrases and acronyms have any place in business communications.
UK Legislative Changes
The legislative environment in the area or employment law, company regulation, data protection and human rights has changed so radically of late that there is a significant gap between practitioner perception and legal reality. Therefore, it may be wise to address address this chasm in order to "provide a safe working environment" for staff. To that end, I've tried to cover all bases here in a succinct yet comprehensive statement in the Email and Instant messaging policy to reflect this intent:
- the creation or distribution of any disruptive or offensive messages, including offensive comments about race, statements about gender, hair colour, disabilities, age, religious beliefs and practise, political beliefs, national origin, sexual orientation, sexist remarks, sexual candour or voyeurism
Contrary to the widely held belief, employees do have rights under the Data Protection and Human Rights Acts in the UK. Therefore, an explicit governance framework must be in place to control access to email, user data and calendar information:
Email Analysis Using MIMESweeper
And lastly, once we have our email, we need to be able to analyse it in a way which does not necessarily mean we need to read the contents.
To be continued
Thursday, April 10, 2008
The user controlled web portal allows you to upload your staff mobile phone database then categorise them accordingly. When an incident occurs they will receive a pre-recorded message. Of considerable further use is the ability to record update messages on the move from a mobile phone. It's a slightly better alternative that SMS (see below.)
In times of crisis
Only fly in the ointment is what happens if they switch the mobile phone networks off? This happened during 7/7 for whatever reason - presumably in an attempt to block a control route to a yet unexploded devices. Perhaps this is something that needs to be taken into consideration in any emergency scenario? However, I'm surprised that so many organisations seem to devolve the responsibility for DR/BCP to the least able of the IT groups and usually to their least able members i.e. Infrastructure.
SMS comes with no quality of service guarantees - in fact it's store and forward at its worst. As an example, people frequently encounter difficulties with SMS traffic each New Year or in crowded venues. Occasionally, you'll get an SMS message that has been significantly delayed too.
Wednesday, April 02, 2008
Conflicts of interest
Honest and ethical conduct
Treating stakeholders fairly
Protection and proper use of assets
Compliance with laws, rules and regulations
Compliance with this code
Interpretation of this code
Policy class hierarchy
Ownership and maintenance
New policy - written 2008
All employed in the company must not only comply with the applicable laws, rules and regulations but should also promote honest and ethical conduct of the business. They must abide by the polices and procedures that govern the conduct of the company's business. Their responsibilities include helping to create and maintain a culture of high ethical standards and commitment to compliance, and to maintain a work environment that encourages stakeholders to raise concerns for the attention of management.
This code is indented to provide guidance and help in recognising and dealing with ethical issues, provide a mechanism to report unethical conduct and help foster a culture of honesty and accountability.
These can arise from:
- the receipt of improper personal benefits as a result of one's position in the company (including benefits for family members)
- an outside business activity that detracts from an individuals ability to devote appropriate time and attention to their responsibilities in the company.
- The receipt of non-nominal gifts or excessive entertainment from any person or company with which
has current or prospective business
- any significant ownership interest in in a supplier, customer, development partner or competitor of the company.
- any consulting or employment relationship with any suppler, customers, development partner or competitor.
All employed by the company should be scrupulous in avoiding conflicts of interest. Where there is likely to be a conflict of interest, the person concerned should make full disclosure of all the facts and circumstances to the board of directors or the the committee or officer nominated for this purpose and prior written approval be obtained.Honest and Ethical Conduct
The directors, officers and employees shall act in accordance with the highest standards of personal and professional integrity not only on the company's premises but also at company-sponsored business, social events and in their personal lives.
They shall be free from fraud and deception.
They shall always conform to the best standards of ethical conduct.
All have a duty to the company to advance its legitimate interests when the opportunity arises.
Directors, officers and employees are expressly prohibited from
- taking for themselves personally, opportunities that are discovered through the use of the company's property, information or position.
- competing directly with the current business of the company or its likely future business.
- using company property, information or position for personal gain.
If the company has made a final decision not to pursue an opportunity, an individual may follow it up only after disclosing the same to the board of directors or to the nominated committee or individual.
All shall respect confidential information on the company, any of its customers, suppliers or business associates. Disclosure of such information should only be made where authorised or required by law.
The use of confidential information for personal gain is strictly forbidden.
Treating stakeholders fairly
The company will not tolerate discrimination on any grounds whatsoever. All employees will be treated fairly and given the opportunity to grow and develop within the company. Promotion will be on merit. Bullying or harassment is regarded as a serious offence and will not be tolerated.
Everyone has a duty to protect the company's assets and ensure their proper use. Theft, carelessness and waste of the company assets damage profitability. These assets should only be used for legitimate business purposes.
All employed by the company shall comply with all relevant laws, rules and regulations. All employees at all levels are expected to know how these laws and rules apply to their area of decision-making. In the event of any uncertainty, the employee concerned should consult the company legal department before taking action.
If any person suspects or knows of a violation of this code, they must immediately report the same to the board of directors or the designated committee or person. The company has a "whistle blowing policy" or to give its formal title, Public Interest Disclosure Policy that will protect their anonymity. Details of the policy are available on the intranet via the preceding link.
Violations of this code will result in disciplinary action and, in some cases, dismissal. Full details of the disciplinary procedures and the appeal processes are included on the intranet.
Interpretation of this code is reserved for the board. The board may appoint a designated committee or designated person to act on its behalf in interpreting and clarifying this code.
Any employee found to have violated this policy may be subject to disciplinary action, up to and including termination of employment.
This policy is owned by the Head of Information Security at
Tuesday, March 25, 2008
Seems to me that the ideas and the technology have been around for a while - it's the time it takes to change the strategy juggernaut that's the issue...
A conversation with Ian Grigg of Systemics from 27th November 2003
I have an idea called Notelets - an arbitrary lightweight data storage mechanism with good
security and highly redundant. Access is via key or x509 cert. Notelets can be "chained" - linked together and "nested" - heirarchical, distributed - stored on at least n nodes (where n > 1.)
This is a bit like freenet project (freenet.sourceforge.net - forget the "freedom" rhetoric - this is the incentive mechanism) - a cryptographic file storage system which is highly redundant and highly distributed. However, there's no element of revenue in the model - and therefore no incentive to run a freenet node. Notelets is different in that the revenue model is a key part system - allowing those who host the service to earn a revenue and build a reputation.
The idea is to "give away" the software to whoever would host data storage - instant and pervasive infrastructure. One of the principal usages would be that of a semaphore between one-to-one or one-to-many parties.
The service could facilitate high levels of security (I won't bore you with the details - but a good analogy is the "tradecraft" symbols used to coordinate drop points as detailed in the Mitrokhin Archive)
A p2p management layer would handle server reputation, notelet distribution and would be geolegislatively sensitive.
[GB] Here's an idea. What about a "notes" service (a la post it notes.)
- user/application issues request for access
- cert issued via email/soap call
- cert presented when service accessed.
[IG] Hmmm.... OK. We are talking website - browser access with client cert here, right? That infrastructure exists which is a big plus. It has had mixed reports, which is a minus. I suppose we'd have to test it to see if the promise is kept.
[GB] - User can create up to 20 blobs of info/text/binary - up to 20KB each - blobs can be linked together - chained, deleted, modified, searched etc
[IG] OK, so this is a sort of user management desktop thing. As close as possible to one's personal PC capabilities, but over the web, right? An all-powerful web services thingie.
This would be sort of like PeopleSoft I imagine, not that I have the faintest clue what PeopleSoft do.
[GB] - blobs persist for 7 days then recycled. These blobs are free - alternatively, blobs cost 2cents per 20K (based on a dollar a gig at present - this will drop though.) One time fee - for five years (based on MTBF of disk) - Plus running costs of say 5.47 dollars a day for a 120GB mirrored server (5 years @ 2000 bucks for rent and power and bandwidth - based on uk prices)
works out at (120 * 1024 * 1024) / 20 * 1024 = 6144 "notes" per day is 5.47/6144 = .9 millicents per note per day - Then add our profit margin - 100%?- you get 4cents setup - 1.8 millicents per day.
[IG] Ok, so someone comes up with a pricing model and revenue and all that.
[GB] So who would use it? Well - you ever tried to build an application in a bank? ha ha - nightmare - power and control are exerted by restricting access to machines and storage.
[IG] Hmm, ok. But - to cast my devil's advocate spears at the idea - anyone who is working in a financial institution like that will not want to pay for the access to a "great tool." They would only use it if it was free to them, which means either there is some con going on and the bank pays for it without knowing, or, the bank is paying for it up front, and it is therefore part of the application.
Are you saying the latter, as that's ok, because the bank itself wants it anyway?
Where I would go a little weak at the knees is that most places I have been get all fuzzy and crazy when they hear that substantial amounts of information are off-site, out of their circle of "physical control."
But let's keep going I'm the plant, recall, I criticise and others grumble and groan until they figure out how to slip one past...
[GB] Second users would be lightweight coordination services - inter organisation - like as suggested in The Tentative Hold Protocol.
Nice idea I think matey
[IG] That's heading in an important direction. Whoever cracks the way to automate the pre-contract parts of the deal process is going to be big.
Personally, my current view is think of it this way:
1. build the payment
2. add the bits that make people use the payment...
2. a hence the IM addition to WebFunds which is "nearly there (tm)".
I.e., build one tool, and then build out from there.
One thing I'm very leary of is people who say that they've worked out a way to automate purchasing and contracting. They never have clue. But, this loose/tentative approach might be such that someone has realised how hard this is...
(Haven't time to read it. Oops, it was by Intel
But I agree, a nice idea. I doubt that we would be able to "build a protocol" .... It's a big job.
[GB] What is it that the large institutions want? In my experience, they want something sold to them by the big 5 (IBM, SUN, Microsoft, HP etc.) Of course, selling to them is next to impossible.
[GB] They want to minimise risk and cost and maximise profit (for them) of course
[IG] Yeah I heard that too If I knew how to do that, I'd be tempted to keep it to myself though
[GB] Sure - that's the crux - they want to buy from the big 5 so they can sue them if anything goes wrong. But the big five need to take advantage of the bazaar effect - millions of programmers rather than just a couple of thousand.
[IG] Well, if that's what you want, then do this:
That we can do. I have the tech, and the knowledge. There is a fair bit of programming to be done, but, I know the traps. Several companies have tried, and have all failed as far as I know. There is one company doing it currently, something with "brain" in the title.
Mind you it is a fair way from J&S. However, maybe it can be turned to legal work. "Project to research a brief on effects of digsig law on payment systems. Must include cases, cite precedences. Supports case to defend Twinkies Tokens v. Crown."
Now, confidentiality of the attack before the judge is an issue in litigation, so it might not be so obvious that we could use open research projects. But, it can be used for deception as well. And,
many projects would be obvious.
[GB] They need to consume services which are reliable and secure - and they will pay the price. The Big 5 need to peer content/services so they can have a comprehensive offering.
[IG] Um. OK. So, a characteristic is reliability. And, security. But, that's all something that we can consider when we've figured out the service.
[GB] This is the point of a directory approach - reputation is key to the success of a service - fast, reliable and cheap - that happy medium. Doesn't stop you downgrading when a service fails - and this gives you realtime failover which is kinda cute.
[IG] Right. But, you cannot have reputation unless you have success. First, you have to succeed, then you can create the brand to protect that success. It is improbable to create a reputation without first having a presence, which means a service or product that has already made it.
Catch 22, of course, which is why the big 5 like buying from big companies and paying the price
[IG] OK. If they are into the legal field... one thing that did arise in my thoughts was a thought that discovery could become a web service.
Here's how it works - in a case there are hundreds of documents flying back and forth. Discovery is the first phase, then there is an intermediate phase where documents are listed in preparation for hearings. Finally, there is the "lifting into evidence" phase, where certain of the previous set are presented.
Now. All the documents have to be fairly available. But, both sides have an incentive to futz with their responsibilities. So, the whole thing is ripe for intermediation. If, in the preliminary proceedings, someone suggests that "we go with S&J Discovery", then all documents/transactions would go to these guys instead. And, S&JD would mount a website, and a huge data repository, such that as documents come in, they are webbed. Each side gets their account, and their password. Access is private, so each side can't see what the other is doing.
In the hearings, S&JD deliver the sets, so the docs are standardised. Both sides get the same quality, so there are fewer arguments.
Hmm, having written the above, I see that it has nothing to do with FC Oh well, maybe S&J can make something of it. Ideas are cheap.
[GB] It dovetails nicely with my idea for a notes services - serendipity. And mine is very much about FC - because we're dealing at the engine level - not the car - the ladiemight make a truck out of it for the legal profession - and on the back of it we make an infra biz...
[IG] It does. Yes. What suprised me was that the legal services world is very very bad at technical stuff. They are very vulnerable to a service bureau taking over large parts of their enteprise.
So, it doesn't have to be the above idea. It could be any hook really, but, once you can get into the legal services field with a net based support service, that combines multiple firms, you can then expand out, using either your post-it notes idea, or the above discovery notion.
That is, eventually, you become the para-legals department. In a sense. You do everything for the lawyers, bar making the coffee.
Have you seen the movie "Minority Report"? Imagine the setup they had, and compare that to a law library.
Big barrier is the law library - it has to be purchased by a new attorney trying to set up practice. I asked my guy - a real good guy - what the good book to get for contracts was, so I could understand how to do Ricardian contracts properly. He said the reference is "Chitty on Contracts."
So I checked, and it is $500 at amazon :-/
Now, imagine S&J Legal Services, puts a big screen into the spare room that is now called the law library, but with no books... Lawyers drive the tech, rent if for $X per month, and scan and search and ...
But, the point is, it doesn't matter what the bait is, in principle, lawyers are vulnerable to this service dependency, I postulate. So the task is to find the best cheap easy bait, and the best set of hooks 1 thru N and roll them out.
[IG] I'll think of something more FC. Perhaps a legal settlement currency, where cases are settled and escrowed in advance, and held in a sort of funny money, in Ricardo of course, and thus less money changes hands?
[GB] Sounds like a great idea. We've been dealing with some legal stuff recently - and one of the things people have been asking for is for us to deposit 30K with a solicitor - it would be better to indicate intent by depositing a virutal currency - which would indicate intent and effect contractual obligation?
[IG] There is certainly a market for independant escrow services. Placing it in a currency would be one way. It might be easier to find someone already doing it, and sell them an FC system.
Now that I think about it, if you ask the solicitor this, he'll probably say, "yes, this is what you do.
You go to J&S Escrow Partners in London, place the money with them, and they give you a piece of paper, which you then hand to the solicitor. They are the firm that everyone uses. But your solicitor won't ever tell you about them because he needs your funds in his account because he uses the float to cover his loan to the book seller......."
Now, the thing is to find that firm and sell them on the benefits of going digital. If they were to issue units in Ricardian, instead of paper, they could save some small amount on costs. But, the real future is that once they have got all the lawyers using their Ricardian escrow paper, they can launch a legal currency for other purposes.
[GB] Also it could be used as a proof service so that you could ask whether counterparty has deposited or performed some action?
Tuesday, March 18, 2008
Friday, March 14, 2008
"MUSING is an Integrated Project co-financed by the 6th Framework Programme of the European Union. Its mission is to shape next generation (Basel II & beyond) semantic-based Business Intelligence, bringing benefits particularly for the financial sector and SMEs by ensuring they have more effective access to credit. MUSING will impact positively on the processes of internationalisation, and empower companies to manage their operational risks more effectively" - they're running the above event details of which you can find here.
I'm interested that a bunch of text miners are focusing on BASEL II compliance and lauding XBRL which seems to be much in the news this week. I read an article in CACM entitled Costs and Benefits of XBRL Adoption: Early Evidence which had good empirical evidence, quoting reporting speed ups of 4x.
If you're interested in this area, there's a good read called Intelligence and Security Informatics for International Security which is now a Google book. My take on the book was that it showed just how ineffective semantic analysis was (70% success rate). I infer that summarisation would fare no better but what do I know.
Sign up - come along - should be a hoot.
Wednesday, March 12, 2008
Depth Aggregator Market
Of real interest is pricing which used to be done the mid-market average and was becoming market-depth aware about 4-5 years ago. With internalisation and MiFID, we now, in theory, have multiple "exchanges" which we need to consult to ensure we are offering best price. This means depth from each exchange needs to be aggregated as point-to-point won't scale. This could be achieved by depth aggregation services - a nice little business opportunity for someone out there. Perhaps we'll see this sort of service being offered by a smart hosting provider?
Topic-based Multicast Architectures
Specific techniques to calculate in real time could be facilitated by topic based, multi-cast architectures with n of m threshold schemes ensuring data quality.
Visualisation, Social Networking, Idea Networks
Then perhaps looking at the part humans play in this (from trader to trading "intelligence" analyst who builds her idea based social networks) and what visualisation is needed.
Accelerated Hardware Market
Friday, February 29, 2008
As with many of his articles Robert X Cringely started off with the breathless tones that got me thinking, "how could I have missed this?", but his post goes on for too long with too many unsubstantiated claims to maintain that emotion. As I've maintained over many pints, this is a fascinating time to be in IT. All we need is funding to check these claims, because nothing rankles like unsubstantiated claims.
For example, NPTL sounds really cool. Graeme may have some insights, but this statement seems like fuzzy thinking to me:
"My e-mail application runs on a four-core Opteron server," says a techie friend of mine, "but I've seen it have over 4,000 simultaneous connections - 4,000 separate threads (where I'm using "thread" to describe a lightweight process) competing for those four CPU's. And looking at the stats, my CPUs are running under five percent almost all the time. This stuff really has come a long way."Is that good? Why do we need 4,000 threads if they aren't actually keeping the CPUs busy? I get the impression that most of these "lightweight processes" are waiting for something. NTPL may be a good thing, but I don't understand how this illustrates the benefits.
In his conclusion he says "If an Azul box were installed on that network, my little app would instantly and mysteriously run up to 50 times faster." Really? I can't follow his argument here at all.
I must have missed the point of his article. If anyone can clarify I'd love to find out more.
Friday, February 15, 2008
This, I feel, reflects the information security world of the past - blighted with too many people who write abstruse policies which no one reads and are heard to continually say "no you can't do that".
I feel it's time for a change so I proposed the following three areas:
- Security Architecture
- Policy framework, policy and process design
- Security architecture: security architecture patterns
- Security API design and analysis
- Operational infrastructure design and implementation
- Product selection procurement
- Process design
- Information Security Operations
- Process implementation and operation
- SLA, compliance monitoring and reporting
- Information Security Analytics
- KRI production, trends, outlier analysis, event correlation, ad hoc reports, event visualisation
- Forensic evidence capture, expert witness
- Risk Assessment
- He discusses the "Von Neumann mistake" which has led to problems with scheduling due to shared state in computer programs, and ways to solve these problems
- Software transactional memory is a potential solution but currently has issues with scalability
- He is most enthusiastic about functional programming languages. Interestingly he points out that the most commonly used functional languages are Excel and SQL.
- Existing programming languages cannot be "tweaked" - an entirely new way of writing programs is needed to obtain the necessary levels of productivity
Saturday, February 09, 2008
This article caught my eye when Google Readering yesterday.
Back in November,I noted that the LSE went Boom! and that this won't be the last time it falls over. It seems a coincidence that the Tokyo Stock Exchange is suffering performance problems and system crashes like it's technology partner the LSE. The pair "are developing jointly traded products and share technology" - oh really? that's a good idea? Same security vulnerabilities, same performance problems, same mysterious crashes then...
Word on the street also about other products based on dot net having serious, unsolvable bugs: mysterious freezes, runaway memory growth, threading issues and performance headaches, just like I experienced when building my last algo trading platform.
Mind you, this is nothing specific against dot net. I remember when I first worked in the city in 1996 doing C++ and using Roguewave libraries - what a disaster that was. Well I did promise to tell you about the eek bug back in the Haskell post so here goes: I was at my first investment bank - sigh - the good old days. One of the developers had a bug that he couldn't solve after a week - we tried everything - every debugger known - they all pointed to threading problems in the Roguewave libraries but the manufacturer was responding slowly. One day, the programmer put the string "eeek" as a printf debug. Miraculously, the bug disappeared. We tested other strings - non worked - only the eeek string did the trick.
That hardened my heart against third party libraries developed by private companies - when the bugs hit - you often have nowhere to turn. My guess is there's an eeek bug or two in the dot net stuff. If it were open source, perhaps there would be a hope of fixing it...
Wednesday, January 30, 2008
Interesting offering from zeroMQ who seem to be pals with Intel. I've checked out the code briefly and am surprised by the use of dynamic memory allocation at such a low level of granularity. I usually allocate a large lump and rely on paging to weed out the empty pages. Anyway, from the very brief look I've had, it's written in C++ and has AQMP compatibility and for some reason I get a whiff of Erlang. More analysis coming soon.
Tuesday, January 29, 2008
Steve just sent this through:
Order execution dynamics in a global FX market
"Very nice overview. This is key: "First, you need to understand what you mean by end-to-end latency (it will be different for different types of organization)".
One of the key metrics for FX turns out to be time to cancel a trade - on EBS over 70% of $1M trades are cancelled. Nobody wants to talk about the games they are playing so it's a bit tricky to know what someone means when they say they want sub-millisecond execution."
An intuitive paper detailing the clustering, spread and timings of limit orders on EBS. It's argued that impatient traders can drop below current price for faster execution. Sitting above the price takes longer as one would imagine. There's also a correlation with order size, showing smaller volumes trade quicker. Volumes are based on 2006 data and are not particularly high.
What's missing though is the underlying need for fast price and market data. The logical implication is reduction in latency - and therefore the necessity for physical proximity. Perhaps it's time to replicate LINX idea for high performance trading applications?
Thursday, January 17, 2008
I gave a talk this week on Object Orientated Security Architecture to a Infosec management audience outside of the normal financial sectors that I work. Hopefully it was well received and made some sense. You can download the original presentation here.
- Invisible policy framework
- •Loose collection of abstract policies
- •Rarely if ever read, understood or referred to
- •Complex & technical v. functional & clear
- •No business rules (BPML)
- •Monolithic & centralised v. global and federated
- •Uncertain legal status
- •Audit/accountancy driven v. business focused
- •Control rather than functionality
- •Inheritance applied to Security Policy
- •General and abstract to specific and detailed
- •Data, event and process centred
- • Detailed and technical to minimal and clear
- • Navigable framework
- •Process catalogue
- •human readable (ie not just by policy wonks)
- •machine executable
- Visual Policy framework – (BITS SIG++)
- Hierarchal process catalogue (BITS AUP++)
- •Security Architecture Capture
- subset of BITS SIG
- concentrates on data flows
- asset, data, risk classes
- business value and application complexity
- operational impact analysis
- Standardised Information Gathering (SIG) -
- Agreed Upon Procedures (AUP)
- Employment Law
Sunday, January 13, 2008
Wednesday, January 02, 2008
I attended the above talk given at Skillsmatter.com on the 13th December. There were 37 attendees, two of which were women (women in technology take note.) I'm afraid I was virtually the only suit in the room apart from Francesco (the speaker) so I stuck out like the proverbial sore thumb.
It was a good talk, well received and well presented. Not particulary technical, so I'm afraid I may have asked too many questions, but I couldn't let the opportunity to get a real understanding of practical Erlang slip by. I was particularly interesting in support for multicast but as no one else had heard of it I kept quiet.
Shared Memory Support
One area I was particulary interested in was no shared memory support - all comms has to go through a tcp/ip software stack apparently. I can understand the aesthetic reasons behind this but the implications for high performance messaging are significant: from one Erlang process to another on a remote machine there's an extra two stacks to cross. It would be interesting to compare with the high performance messaging providers.
Real-world Erlang - Yaws Performance
One rather amusing piece was on the performance of Yaws, the Erlang web server, which shows very high throughput compared to Apache. Yaws manages to run 80,000 parallel sessions, serviing 2 x 20KB pages whilst poor old Apache dies after 4,000.
One unfortunate side effect of this would be to consume your organisation's entire bandwidth so I'll stick with Apache for now. If I want to ship that much (presumably) random data, the last tech I'll use is a web server. HDF5, Lustre, GFS, Gigaspaces all have a better handle on this sort of data architecture.
I like Erlang as a concept but they have a mountain to climb with OTP (the function library) if they're going to be more than niche. New features are being added all the time - the latest being a "posix like" threading library and support for SMP. Faster ASN.1 support too with the latest version. Erlang is growing up fast.
For those of you interested in learning Erlang, I'd recommend starting with Joe Armstrong's PhD.