Temporal News Signatures Used to Track Disease

Nice to see application of the techniques I described in my paper "Community-of-Interest Predicated Program Trading" used to track disease.

If they follow my logic, they'll use community expertise to do the analysis, annotation, recategorisation and dissemination to peer interest groups. Then it will be possible to build a reputation based analytics platform to summarise the trend, relate it to previous events and capture side-effects...

Open Source Intelligence in Finance

Presentation delivered to the FITE Club in 2005.

Business Driven Platform Architecture

Old presentation from 2005 where I attempted to introduce infrastructure people to the pressures the business were facing then as now...

Haskell, HPC and Crypto Get Together 22nd May 2008

Last night's get together was attended by about 15 men (women in technology take note) from a variety of backgrounds (students, cryptographers, investment bank staff.) Beers were consumed and nonsense talked - then we toddled off to Kurz and Lang for some bratwurst around eight as is our tradition (from our days in Zurich with a well known Swiss bank.)

The conclusion of the evening was that there is a place for Haskell in the city - why? Well here's two extracts from and article on Haskell and Performance on Planet Haskell written by Neil Mitchell:

Haskell's multi-threaded performance is amazing

A lot of clever people have done a lot of clever work on making multi-threaded programming in Haskell both simple and fast. While low-level speed matters for general programming, for multi-threaded programming there are lots of much higher-level performance considerations. Haskell supports better abstraction, and can better optimise at this level, outperforming C.

The trend is for higher-level optimisation

As time goes buy, higher-level programs keep getting faster and faster. The ByteString work allows programmers to write high-level programs that are competitive with C. Performance enhancements are being made to the compiler regularly, pointer tagging, constructor specialisation etc. are all helping to improve things. More long term projects such as Supero and NDP are showing some nice results. Optimisation is a difficult problem, but progress is being made, allowing programs to be written in a higher-level.

and that it would be a good idea to form a Functional Programming get together on a reasonably regular basis - so watch this space.

Financial Data Infrastructure with HDF5

Presentation given to the FITE Club in 2004.

Crisis Resilient Architecture

I gave this presentation at IISyG in 2005 to an audience of Infosec specialists. The talk was inspired by Dr Sally Leivesly of Newrisk Ltd who regularly appears on the BBC after terrorist incidents. She outlined combinatorial threat and area denial scenarios.

Another influence was the amount of fuel required to power centralised data centers which, in times of crisis, may be hard to supply regularly. Then there is the strange notion that staff will have the desire and ability to travel to some remote data center regularly which frankly beggars belief. The whole BCP/DR scenario is, sadly, driven by regulators, auditors and policy wonks.

So my take on the answer is that we need fabric/cloud computing, micro data centers and UWB MAN networks which can be up and running in an instant. Combined with the "crisis desktop", a bootable OS on a memory stick which can form a secure platform to access your virtual desktop. The parts are all there - all we need is the strategists to catch up...

Object-Orientated Security Policy

RANT Talk - Object Orientated Security Policy

I'm speaking at RANT on Object Orientated Security Policy. This is a post implementation talk which will highlight the benefits of having a visual policy framework with executable procedures arranged in an inheritance hierarchy.

The event is being described as: "Graeme’s rant will be looking into how the Security policy landscape is changing and how policies can no longer afford to be a loose collection of word documents seldom referred to. He will talk about how it is the business logic and rules of the organisation which are now becoming executable thanks to web services."

Details can be found here

Information Security - Risk Management Process

Risk assessment is a mandatory activity under all the Information Security policy frameworks e.g. COBIT, ISO27001. However, risk assessment methodologies are point-in-time and don't deal with incremental risk as the above process does.

Haskell, HPC and Crypto - Beer in the Evening

We're organising another "beer in the evening" event somewhere round about Smithfield in London to discuss Haskell, multi-core, HPC, Security and all sorts really - so if you fancy having a beer or two with a gang of gnarly, middle-aged architects, programmers and scientists (attractive huh?) then you'll be very welcome.

We will be meeting on the 22nd May - details are here

To participate, please add your name to the Doodle Poll entitled Haskell + Beer. This is so we know how many buckets of chips to order...

As All Bar One has now closed, we will be in the Bishop's Finger, Smithfield. I'm planning on getting there around 530pm. My mobile number is zero seven eight three zero three six eight zero two four - when you get there give me a call. There's a good chance we'll get some of the beers paid for too but no promises yet.

Topics for discussion/contention

• Crypto in Haskell (Dominic is Mr Haskell Crypto btw). Opportunities for parallelism (e.g.)
  
• Is Haskell and multi-core the best solution to meet the projected demand for simulation in finance or has FPGA yet to have it's day?
  
• Challenges of Haskell and high performance messaging integration. 2.4 million messages per second (not transactions) are now possible using tools like LBM from 29West.
  
• The K language (an APL derivative), Fortran, Objective-C, LISP and Smalltalk gained a small foothold in the eighties/nineties in quantitative analytics but C++ remained the main language of implementation due to availability of skillset (K programmers are like hen's teeth.) Java and now C# are now the de facto implementation languages but have serious issues for high frequency finance - will Haskell be able to make an impact or will it be another marginal language used by quants.
• Is Haskell capable of high performance?
  
• Haskell and distributed memory (memcached) performance aspects
• Haskell Data Parallelism versus state of the art FPGA approaches. Check out HPCPlatform's offering btw.
• I hear rumours that Haskell is being rolled out to the HPC team of a major investment bank in Canary Wharf - watch this space...
  

More topics very welcome.

Collaborative Web Services

Collaboration was a theme that underlay some of fundamental concepts behind the Trade Ideas platform from youDevise. This presentation highlights some of the many threads which prevailed at the time.

Open Source Intelligence in Finance

A presentation given in 2005 to the FITE Club. This was the foundations to my paper on Community-of-Interest predicated Program Trading which was presented at the Operational Research Society Conference, OR49 in September 2007.

HPC=Grid+Data+FPGA

Another presentation from the past. Presented at Platform's Grid conference in Paris in 2006.

Monte Carlo Using COTS FPGA

A few years ago I put together this presentation based on some work I did with some guys at Southampton University on embedding Monte Carlo onto FPGA. At the time we used System C and the C code from the Monte Carlo implementation in the SCIMARK 2.0 Benchmark, something I've used to great effect in the past. The implementation was niaive and simplistic but it was used to start the HPC movement that's been a la mode in the City for a while...

OpenTech 2008

OpenTech2008 is being held in London on the 5th July 2008. The conference is "an informal, low cost one-dayer on technology, society and low-carbon living, featuring Open Source ways of working and technologies that anyone can have a go at."

I've been invited onto the security panel by Ben Laurie of Google (Caja) and Apache SSL fame. I'm going to be talking about beancounter-led regulation in financial markets and how it has killed security engineering...

CUDA - GPU Programming Framework from nVidia

Catching up with some reading this morning, I picked up a series of articles from the Mar/April edition of ACM Queue. In particular, CUDA which was released by Nvidia last year. I read the article "Scalable Parallel Programming with CUDA" which can be found here.

The article identifies three key abstractions, hierarchical thread groups, shared memories and barrier synchronisation. CUDA is based on C with extensions for parallelisation - much like Handel-C. The difference is that Handel-C was FPGA based whilst CUDA is for GPU with its built-in floating point capability. There are simple and straightforward code examples showing parallel threading and memory sharing which was always an issue in my mind with FPGA: the leap of faith with Handel-C was what to do with the data set you generated in a Monte Carlo simulation.

This question has been perplexing developers on the CUDA forums at Nvidia too - but it looks like there's been progress as outlined in this presentation on Monte Carlo Options Pricing paper on the Nvidia developer site. However, the algorithm outlined in the paper is trivial, the secret being the generation of quasi-random numbers enabling quick convergence. Then filtration close to the data so you're not schlepping large lumps of data unnecessarily.

Then the next logical step is to make this a service. The appetite is reckoned to be about 5 trillion simulations per day in the average organisation according to a quant chum of mine. Combine this with S3 for asynchronous storage and you have the makings of a nice little business I think.

Functional Programming Creeps into Job Specs

As predicted in this article from June 2007 "Haskell - the Next Mainstream Programming Language?" - functional programming is getting into job specs...

http://jobview.monster.com/GetJob.aspx?JobID=70786548

http://jobview.monster.com/GetJob.aspx?JobID=70153611

http://jobview.monster.com/GetJob.aspx?JobID=70575524

http://jobview.monster.com/GetJob.aspx?JobID=67440522

http://jobview.monster.com/GetJob.aspx?JobID=70311202

"You will have previous experience of designing and building distributed, fault tolerant systems in a commercial environment. Experience of multi threading, socket programming, network programming and functional programming languages (Haskell, Ocaml, F#) will be an advantage."

"Experience with functional languages such as Haskell, Erlang, F#, Scheme, LISP, etc., are greatly appreciated."
Bit of a scattergun approach in the last example perhaps? I wonder who writes the job specs - I guess the bizerati analystas high on the latest marketing speak. I'm still confused about is the insistence on C++ with it's late binding and poor library coverage (compared to Java.) As illustrated by this graph from the paper below, C++ is slower than C - so why would you want to use it when speed is the ultimate criterion? Beats me.

An empirical comparison of C, C++, Java, Perl, TCL and REXX for search/string processing

I'm also bemused at the use of C# - in light of the recent debacles at the LSE and TSE.

One wonders who is in charge of algo and program trading strategy. I do hope they realise the advantages of a monadic language are not without performance implications and that without stream fusion and massively multi-core processors (with FPUs) the performance gains they seek are going to be rather elusive. Then there's the data issue - you have to crack that particular nut - and here's a clue - the answer's not xml or any of its bloated siblings.

Information Security - Adaptive Virus Trend Analysis Process

Virus Classification

Each virus is rated using the particular manufacturer's rating mechanism giving an independent benchmark. Escalation for high impact viruses is immediate and possibly would result in the security incident management process being invoked. Cumulative threat is also reported if there are more than 50% of medium threat.

Trend and Pattern analysis

I've instituted a qualitative and quantitative approach to virus analysis which delivers a modicum of business intelligence and facilitates escalation based on threat level.

Trending is twofold: year to date and historical highlighting monthly and annual trends. Seasonality is present in the year-to-date statistics whilst long term trend can be gauged over five years.

Patterns in virus targeting are analysed per geographic location, business area, workstation and user. A history of infection is maintained so that repeated infection can be analysed further.

Adaptive Sampling

An adaptive approach is taken to monitoring frequency. If an uplift in activity of 20% incurs, the monitoring frequency is doubled to a minimum of daily monitoring. If the trend is reversed for three monitoring periods, the frequency is halved to a maximum of monthly sampling.

Geek Breakfast

Maintaining concentration all day every day when coding is no mean feat. It helps to have a good breakfast:

• Half a cup of Scott's Porage Oats
• Two handfuls of Holland and Barrett Omega Sprinkle
• One handful of Holland and Barrett dried goji berries, blueberry and cranberries mixed in equal proportions
• Tablespoon of Good Oil
• Quarter of a pomegranate seeded
  

Mix it all together and soak in milk and/or yoghurt overnight - delicious warm or cold. Finish off with a nice with glass of organic beetroot juice.

Graphduplex - Graph Coupling by Shared Attributes

Over at Semiosys, Elie has been working on a new social network analysis piece called GraphDuplex. GraphDuplex allows you to couple N graphs by shared attributes. The use of petals and histograms is an alternative to the traditional pie chart where sectors can easily be missed due to relative segment sizes. The thickness of the line denotes volume and can be easily also denote flow if the that were relevant. Acceleration could also be represented by a rotating decal with rotational speed indicated rate of change.

Information Security - Email Management

I've just finished a fascinating tranche of work on mail management, in my particular clients case, using MIMESweeper which works by a "stop" list, i.e. a list of banned words/phrases/regular expressions. First issue to understand was legal compliance, secondly, effective operational procedures leveraging the features of the tool. Below is a Freemind map of the overall process:

Language Coverage

We have a diverse workforce who speak many languages including: Hindi, Finnish, Russian, Bulgarian etc. Deciding which languages should cover was a simple conundrum to the in-house legal expert - we only need to cover the languages in which we conduct our business, (in the clients case) namely English, French, German and Japanese.

The stock English dictionary supplied with MIMESweeper contained mainly anatomical parts and the usual canonical profanity. The strategy of this particular stop list seemed to be male, heterosexual, white, American and lower class. I believe this is where the fecund British mind can add value: there are several online dictionaries are sources of expletives and profanity (e.g. The Alternative Dictionaries, Roger's Profanisaurus etc.) Other interesting sources are SMS acryonyms which are increasingly being used by the younger generation. There are even translation sites which can change English into "text speak". and several words used by the slightly more educated such as onanist. In the latter case, it would be hard to argue that any of these words, phrases and acronyms have any place in business communications.

UK Legislative Changes

The legislative environment in the area or employment law, company regulation, data protection and human rights has changed so radically of late that there is a significant gap between practitioner perception and legal reality. Therefore, it may be wise to address address this chasm in order to "provide a safe working environment" for staff. To that end, I've tried to cover all bases here in a succinct yet comprehensive statement in the Email and Instant messaging policy to reflect this intent:

• the creation or distribution of any disruptive or offensive messages, including offensive comments about race, statements about gender, hair colour, disabilities, age, religious beliefs and practise, political beliefs, national origin, sexual orientation, sexist remarks, sexual candour or voyeurism

Email Governance - Data Protection, Human Rights and RIPA

Contrary to the widely held belief, employees do have rights under the Data Protection and Human Rights Acts in the UK. Therefore, an explicit governance framework must be in place to control access to email, user data and calendar information:



Email Analysis Using MIMESweeper

And lastly, once we have our email, we need to be able to analyse it in a way which does not necessarily mean we need to read the contents.


To be continued

Security Incident Management - Event Notification

Deep in policy land of late, finishing off a BITS compliant set of Infosec polices, I noticed a service which may prove useful in Information Security Incident management (or any other incident management scenario for that matter) from Grasswhispers (http://grasswhispers.co.uk.)

The user controlled web portal allows you to upload your staff mobile phone database then categorise them accordingly. When an incident occurs they will receive a pre-recorded message. Of considerable further use is the ability to record update messages on the move from a mobile phone. It's a slightly better alternative that SMS (see below.)

In times of crisis

Only fly in the ointment is what happens if they switch the mobile phone networks off? This happened during 7/7 for whatever reason - presumably in an attempt to block a control route to a yet unexploded devices. Perhaps this is something that needs to be taken into consideration in any emergency scenario? However, I'm surprised that so many organisations seem to devolve the responsibility for DR/BCP to the least able of the IT groups and usually to their least able members i.e. Infrastructure.



SMS comes with no quality of service guarantees - in fact it's store and forward at its worst. As an example, people frequently encounter difficulties with SMS traffic each New Year or in crowded venues. Occasionally, you'll get an SMS message that has been significantly delayed too.

Policy.HR.Ethics

Index

Overview
Purpose
Scope
Policy
Conflicts of interest
Honest and ethical conduct
Corporate opportunities
Confidentiality
Treating stakeholders fairly
Employees
Protection and proper use of assets
Compliance with laws, rules and regulations
Compliance with this code
Interpretation of this code
Policy class hierarchy
Enforcement
Ownership and maintenance
Copyright

Document Status and Revision History

New policy - written 2008

Overview

"Your Company Name" is committed to conducting its business in accordance with the applicable laws, rules and regulations and with the highest standards of business ethics.

All employed in the company must not only comply with the applicable laws, rules and regulations but should also promote honest and ethical conduct of the business. They must abide by the polices and procedures that govern the conduct of the company's business. Their responsibilities include helping to create and maintain a culture of high ethical standards and commitment to compliance, and to maintain a work environment that encourages stakeholders to raise concerns for the attention of management.

Purpose

This code is indented to provide guidance and help in recognising and dealing with ethical issues, provide a mechanism to report unethical conduct and help foster a culture of honesty and accountability.

Policy

Conflicts of Interest

These can arise from:

• the receipt of improper personal benefits as a result of one's position in the company (including benefits for family members)
• an outside business activity that detracts from an individuals ability to devote appropriate time and attention to their responsibilities in the company.
• The receipt of non-nominal gifts or excessive entertainment from any person or company with which has current or prospective business
• any significant ownership interest in in a supplier, customer, development partner or competitor of the company.
• any consulting or employment relationship with any suppler, customers, development partner or competitor.

All employed by the company should be scrupulous in avoiding conflicts of interest. Where there is likely to be a conflict of interest, the person concerned should make full disclosure of all the facts and circumstances to the board of directors or the the committee or officer nominated for this purpose and prior written approval be obtained.

Honest and Ethical Conduct

The directors, officers and employees shall act in accordance with the highest standards of personal and professional integrity not only on the company's premises but also at company-sponsored business, social events and in their personal lives.

They shall be free from fraud and deception.

They shall always conform to the best standards of ethical conduct.

Corporate Opportunities

All have a duty to the company to advance its legitimate interests when the opportunity arises.

Directors, officers and employees are expressly prohibited from

• taking for themselves personally, opportunities that are discovered through the use of the company's property, information or position.
• competing directly with the current business of the company or its likely future business.
• using company property, information or position for personal gain.

If the company has made a final decision not to pursue an opportunity, an individual may follow it up only after disclosing the same to the board of directors or to the nominated committee or individual.

Confidentiality

All shall respect confidential information on the company, any of its customers, suppliers or business associates. Disclosure of such information should only be made where authorised or required by law.

The use of confidential information for personal gain is strictly forbidden.

Treating stakeholders fairly

All employees at all levels should deal fairly with those we do business with. No-one should take unfair advantage of anyone through manipulation, concealment, abuse of confidential information, misrepresentation of facts or any other unfair dealing practices.

Employees

The company will not tolerate discrimination on any grounds whatsoever. All employees will be treated fairly and given the opportunity to grow and develop within the company. Promotion will be on merit. Bullying or harassment is regarded as a serious offence and will not be tolerated.

Protection and proper use of company assets

Everyone has a duty to protect the company's assets and ensure their proper use. Theft, carelessness and waste of the company assets damage profitability. These assets should only be used for legitimate business purposes.

Compliance with laws, rules and regulations

All employed by the company shall comply with all relevant laws, rules and regulations. All employees at all levels are expected to know how these laws and rules apply to their area of decision-making. In the event of any uncertainty, the employee concerned should consult the company legal department before taking action.

Compliance with the code

If any person suspects or knows of a violation of this code, they must immediately report the same to the board of directors or the designated committee or person. The company has a "whistle blowing policy" or to give its formal title, Public Interest Disclosure Policy that will protect their anonymity. Details of the policy are available on the intranet via the preceding link.

Violations of this code will result in disciplinary action and, in some cases, dismissal. Full details of the disciplinary procedures and the appeal processes are included on the intranet.

Interpretation of this code

Interpretation of this code is reserved for the board. The board may appoint a designated committee or designated person to act on its behalf in interpreting and clarifying this code.

Any employee found to have violated this policy may be subject to disciplinary action, up to and including termination of employment.

Policy Class Hierarchy

Class

Policy.HR.Ethics

Procedures

None

Standards

None

See Also

None

Ownership and Maintenance

This policy is owned by the Head of Information Security at and is revised on an annual basis.

Notelets

I've been looking with interest at Amazon's Web Service stuff, particularly S3, for several years though never really saw it as being a realistic part of the Investment Banking scene. However, I wonder if that's going to change with great deal of interest in SOA and now Web 2.0 from the marketeers which is feeding into IT strategy. Here's an idea I bandied around with Ian Grigg of Systemics, a leading light in the financial cryptography community and author of the seminal paper Financial Cryptography in 7 Layers.

Seems to me that the ideas and the technology have been around for a while - it's the time it takes to change the strategy juggernaut that's the issue...

A conversation with Ian Grigg of Systemics from 27th November 2003

I have an idea called Notelets - an arbitrary lightweight data storage mechanism with good
security and highly redundant. Access is via key or x509 cert. Notelets can be "chained" - linked together and "nested" - heirarchical, distributed - stored on at least n nodes (where n > 1.)

This is a bit like freenet project (freenet.sourceforge.net - forget the "freedom" rhetoric - this is the incentive mechanism) - a cryptographic file storage system which is highly redundant and highly distributed. However, there's no element of revenue in the model - and therefore no incentive to run a freenet node. Notelets is different in that the revenue model is a key part system - allowing those who host the service to earn a revenue and build a reputation.

The idea is to "give away" the software to whoever would host data storage - instant and pervasive infrastructure. One of the principal usages would be that of a semaphore between one-to-one or one-to-many parties.

The service could facilitate high levels of security (I won't bore you with the details - but a good analogy is the "tradecraft" symbols used to coordinate drop points as detailed in the Mitrokhin Archive)

A p2p management layer would handle server reputation, notelet distribution and would be geolegislatively sensitive.

[GB] Here's an idea. What about a "notes" service (a la post it notes.)

- user/application issues request for access
- cert issued via email/soap call
- cert presented when service accessed.


[IG] Hmmm.... OK. We are talking website - browser access with client cert here, right? That infrastructure exists which is a big plus. It has had mixed reports, which is a minus. I suppose we'd have to test it to see if the promise is kept.

[GB] - User can create up to 20 blobs of info/text/binary - up to 20KB each - blobs can be linked together - chained, deleted, modified, searched etc

[IG] OK, so this is a sort of user management desktop thing. As close as possible to one's personal PC capabilities, but over the web, right? An all-powerful web services thingie.

This would be sort of like PeopleSoft I imagine, not that I have the faintest clue what PeopleSoft do.

[GB] - blobs persist for 7 days then recycled. These blobs are free - alternatively, blobs cost 2cents per 20K (based on a dollar a gig at present - this will drop though.) One time fee - for five years (based on MTBF of disk) - Plus running costs of say 5.47 dollars a day for a 120GB mirrored server (5 years @ 2000 bucks for rent and power and bandwidth - based on uk prices)
works out at (120 * 1024 * 1024) / 20 * 1024 = 6144 "notes" per day is 5.47/6144 = .9 millicents per note per day - Then add our profit margin - 100%?- you get 4cents setup - 1.8 millicents per day.

[IG] Ok, so someone comes up with a pricing model and revenue and all that.

[GB] So who would use it? Well - you ever tried to build an application in a bank? ha ha - nightmare - power and control are exerted by restricting access to machines and storage.

[IG] Hmm, ok. But - to cast my devil's advocate spears at the idea - anyone who is working in a financial institution like that will not want to pay for the access to a "great tool." They would only use it if it was free to them, which means either there is some con going on and the bank pays for it without knowing, or, the bank is paying for it up front, and it is therefore part of the application.

Are you saying the latter, as that's ok, because the bank itself wants it anyway?

Where I would go a little weak at the knees is that most places I have been get all fuzzy and crazy when they hear that substantial amounts of information are off-site, out of their circle of "physical control."

But let's keep going I'm the plant, recall, I criticise and others grumble and groan until they figure out how to slip one past...

[GB] Second users would be lightweight coordination services - inter organisation - like as suggested in The Tentative Hold Protocol.

Nice idea I think matey

[IG] That's heading in an important direction. Whoever cracks the way to automate the pre-contract parts of the deal process is going to be big.

Personally, my current view is think of it this way:

1. build the payment
2. add the bits that make people use the payment...
2. a hence the IM addition to WebFunds which is "nearly there (tm)".

I.e., build one tool, and then build out from there.

One thing I'm very leary of is people who say that they've worked out a way to automate purchasing and contracting. They never have clue. But, this loose/tentative approach might be such that someone has realised how hard this is...

(Haven't time to read it. Oops, it was by Intel

But I agree, a nice idea. I doubt that we would be able to "build a protocol" .... It's a big job.

[GB] What is it that the large institutions want? In my experience, they want something sold to them by the big 5 (IBM, SUN, Microsoft, HP etc.) Of course, selling to them is next to impossible.

[GB] They want to minimise risk and cost and maximise profit (for them) of course

[IG] Yeah I heard that too If I knew how to do that, I'd be tempted to keep it to myself though

[GB] Sure - that's the crux - they want to buy from the big 5 so they can sue them if anything goes wrong. But the big five need to take advantage of the bazaar effect - millions of programmers rather than just a couple of thousand.

[IG] Well, if that's what you want, then do this:

http://www.iang.org/papers/task_market.html

That we can do. I have the tech, and the knowledge. There is a fair bit of programming to be done, but, I know the traps. Several companies have tried, and have all failed as far as I know. There is one company doing it currently, something with "brain" in the title.

Mind you it is a fair way from J&S. However, maybe it can be turned to legal work. "Project to research a brief on effects of digsig law on payment systems. Must include cases, cite precedences. Supports case to defend Twinkies Tokens v. Crown."

Now, confidentiality of the attack before the judge is an issue in litigation, so it might not be so obvious that we could use open research projects. But, it can be used for deception as well. And,
many projects would be obvious.

[GB] They need to consume services which are reliable and secure - and they will pay the price. The Big 5 need to peer content/services so they can have a comprehensive offering.


[IG] Um. OK. So, a characteristic is reliability. And, security. But, that's all something that we can consider when we've figured out the service.

[GB] This is the point of a directory approach - reputation is key to the success of a service - fast, reliable and cheap - that happy medium. Doesn't stop you downgrading when a service fails - and this gives you realtime failover which is kinda cute.


[IG] Right. But, you cannot have reputation unless you have success. First, you have to succeed, then you can create the brand to protect that success. It is improbable to create a reputation without first having a presence, which means a service or product that has already made it.

Catch 22, of course, which is why the big 5 like buying from big companies and paying the price

[IG] OK. If they are into the legal field... one thing that did arise in my thoughts was a thought that discovery could become a web service.

Here's how it works - in a case there are hundreds of documents flying back and forth. Discovery is the first phase, then there is an intermediate phase where documents are listed in preparation for hearings. Finally, there is the "lifting into evidence" phase, where certain of the previous set are presented.

Now. All the documents have to be fairly available. But, both sides have an incentive to futz with their responsibilities. So, the whole thing is ripe for intermediation. If, in the preliminary proceedings, someone suggests that "we go with S&J Discovery", then all documents/transactions would go to these guys instead. And, S&JD would mount a website, and a huge data repository, such that as documents come in, they are webbed. Each side gets their account, and their password. Access is private, so each side can't see what the other is doing.

In the hearings, S&JD deliver the sets, so the docs are standardised. Both sides get the same quality, so there are fewer arguments.

Hmm, having written the above, I see that it has nothing to do with FC Oh well, maybe S&J can make something of it. Ideas are cheap.

[GB] It dovetails nicely with my idea for a notes services - serendipity. And mine is very much about FC - because we're dealing at the engine level - not the car - the ladiemight make a truck out of it for the legal profession - and on the back of it we make an infra biz...

[IG] It does. Yes. What suprised me was that the legal services world is very very bad at technical stuff. They are very vulnerable to a service bureau taking over large parts of their enteprise.

So, it doesn't have to be the above idea. It could be any hook really, but, once you can get into the legal services field with a net based support service, that combines multiple firms, you can then expand out, using either your post-it notes idea, or the above discovery notion.

That is, eventually, you become the para-legals department. In a sense. You do everything for the lawyers, bar making the coffee.

Have you seen the movie "Minority Report"? Imagine the setup they had, and compare that to a law library.

Big barrier is the law library - it has to be purchased by a new attorney trying to set up practice. I asked my guy - a real good guy - what the good book to get for contracts was, so I could understand how to do Ricardian contracts properly. He said the reference is "Chitty on Contracts."

So I checked, and it is $500 at amazon :-/

Now, imagine S&J Legal Services, puts a big screen into the spare room that is now called the law library, but with no books... Lawyers drive the tech, rent if for $X per month, and scan and search and ...

But, the point is, it doesn't matter what the bait is, in principle, lawyers are vulnerable to this service dependency, I postulate. So the task is to find the best cheap easy bait, and the best set of hooks 1 thru N and roll them out.

[IG] I'll think of something more FC. Perhaps a legal settlement currency, where cases are settled and escrowed in advance, and held in a sort of funny money, in Ricardo of course, and thus less money changes hands?

Comments?

[GB] Sounds like a great idea. We've been dealing with some legal stuff recently - and one of the things people have been asking for is for us to deposit 30K with a solicitor - it would be better to indicate intent by depositing a virutal currency - which would indicate intent and effect contractual obligation?


[IG] There is certainly a market for independant escrow services. Placing it in a currency would be one way. It might be easier to find someone already doing it, and sell them an FC system.

Now that I think about it, if you ask the solicitor this, he'll probably say, "yes, this is what you do.
You go to J&S Escrow Partners in London, place the money with them, and they give you a piece of paper, which you then hand to the solicitor. They are the firm that everyone uses. But your solicitor won't ever tell you about them because he needs your funds in his account because he uses the float to cover his loan to the book seller......."

Now, the thing is to find that firm and sell them on the benefits of going digital. If they were to issue units in Ricardian, instead of paper, they could save some small amount on costs. But, the real future is that once they have got all the lawyers using their Ricardian escrow paper, they can launch a legal currency for other purposes.

[GB] Also it could be used as a proof service so that you could ask whether counterparty has deposited or performed some action?

[IG] yep!

Subprime Explained

A learned colleague sent me a presentation called The Subprime Primer which cynically explains the fine mess that the finance sector are in and how the RSG's bosses are all to blame - mild profanity - you have been warned...

MUSING Workshop - Turning Data Into Risk Knowledge - 21st May 2008

Of note is the MUSING Executive Workshop "Turning data into risk knowledge: Implementation of semantic-based risk management processes in the financial services industry". I've signed up as the agenda looks good. Lots of academics - in particular a guy from Sheffield who is working for GATE.

"MUSING is an Integrated Project co-financed by the 6th Framework Programme of the European Union. Its mission is to shape next generation (Basel II & beyond) semantic-based Business Intelligence, bringing benefits particularly for the financial sector and SMEs by ensuring they have more effective access to credit. MUSING will impact positively on the processes of internationalisation, and empower companies to manage their operational risks more effectively" - they're running the above event details of which you can find here.

I'm interested that a bunch of text miners are focusing on BASEL II compliance and lauding XBRL which seems to be much in the news this week. I read an article in CACM entitled Costs and Benefits of XBRL Adoption: Early Evidence which had good empirical evidence, quoting reporting speed ups of 4x.

If you're interested in this area, there's a good read called Intelligence and Security Informatics for International Security which is now a Google book. My take on the book was that it showed just how ineffective semantic analysis was (70% success rate). I infer that summarisation would fare no better but what do I know.

Sign up - come along - should be a hoot.

Enterprise Computing Strategies Summit

The 451 Group have kindly invited me to speak at the above event on low latency and security. This will give me the opportunity to summarise the High Performance Computing journey that started in 2005 and put in perspective what has evolved over the last three years as we move towards increasingly automated electronic trading.

My approach will be to start with message orientated architectures and argue that they're superior to event driven when it comes to high volumes. Electronic trading mandates the move from end-of-day to real-time risk calculation and analysis - no mean feat. This led my forays into FPGA based Monte Carlo and the whole accelerated simulation idea being followed by various Investment Banks.

Depth Aggregator Market

Of real interest is pricing which used to be done the mid-market average and was becoming market-depth aware about 4-5 years ago. With internalisation and MiFID, we now, in theory, have multiple "exchanges" which we need to consult to ensure we are offering best price. This means depth from each exchange needs to be aggregated as point-to-point won't scale. This could be achieved by depth aggregation services - a nice little business opportunity for someone out there. Perhaps we'll see this sort of service being offered by a smart hosting provider?

Topic-based Multicast Architectures

Specific techniques to calculate in real time could be facilitated by topic based, multi-cast architectures with n of m threshold schemes ensuring data quality.

Visualisation, Social Networking, Idea Networks

Then perhaps looking at the part humans play in this (from trader to trading "intelligence" analyst who builds her idea based social networks) and what visualisation is needed.

Accelerated Hardware Market

I'm then going to summarise the whole accelerated hardware scene, looking at why it's not really going anywhere (apathy, lack of easy integration, no standards, esoteric languages etc) then show the true path (IMHO) - ie monadic languages that are highly parallelisable.

That should be a hoot and quite unique...

Has Cringely been drinking the Kool-aid?

As with many of his articles Robert X Cringely started off with the breathless tones that got me thinking, "how could I have missed this?", but his post goes on for too long with too many unsubstantiated claims to maintain that emotion. As I've maintained over many pints, this is a fascinating time to be in IT. All we need is funding to check these claims, because nothing rankles like unsubstantiated claims.

For example, NPTL sounds really cool. Graeme may have some insights, but this statement seems like fuzzy thinking to me:

"My e-mail application runs on a four-core Opteron server," says a techie friend of mine, "but I've seen it have over 4,000 simultaneous connections - 4,000 separate threads (where I'm using "thread" to describe a lightweight process) competing for those four CPU's. And looking at the stats, my CPUs are running under five percent almost all the time. This stuff really has come a long way."

Is that good? Why do we need 4,000 threads if they aren't actually keeping the CPUs busy? I get the impression that most of these "lightweight processes" are waiting for something. NTPL may be a good thing, but I don't understand how this illustrates the benefits.

In his conclusion he says "If an Azul box were installed on that network, my little app would instantly and mysteriously run up to 50 times faster." Really? I can't follow his argument here at all.

I must have missed the point of his article. If anyone can clarify I'd love to find out more.

Hail the New Information Security World

As it happens I'm deep in Information Security management at the moment which is an interesting diversion from HPC. A young turk is trying to set up an Information Security practise at the firm of some chums and has been asking my advice on where Information Security is at and where it's heading. His hypothesis was to structure it along three lines: architecture (application and networks), implementation and compliance.

This, I feel, reflects the information security world of the past - blighted with too many people who write abstruse policies which no one reads and are heard to continually say "no you can't do that".

I feel it's time for a change so I proposed the following three areas:

• Security Architecture
• Strategy
• Governance
  
• Policy framework, policy and process design
• Security architecture: security architecture patterns
  
• Security API design and analysis
  
• Operational infrastructure design and implementation
• Product selection procurement
• Process design
  
• Information Security Operations
• Process implementation and operation
  
• SLA, compliance monitoring and reporting
• Information Security Analytics
• KRI production, trends, outlier analysis, event correlation, ad hoc reports, event visualisation
  
• Forensic evidence capture, expert witness
• Risk Assessment
  

But there's still something missing - the process of cumulative risk management. Click the image to expand:

General-purpose parallelism

Via Joe Duffy's weblog I just watched an interesting interview with Burton Smith, which is definitely worth sharing. It's about an hour long but here are some key points (many of which we have discussed here in the past):

• He discusses the "Von Neumann mistake" which has led to problems with scheduling due to shared state in computer programs, and ways to solve these problems

• Software transactional memory is a potential solution but currently has issues with scalability

• He is most enthusiastic about functional programming languages. Interestingly he points out that the most commonly used functional languages are Excel and SQL.

• Existing programming languages cannot be "tweaked" - an entirely new way of writing programs is needed to obtain the necessary levels of productivity

• He also says that if Microsoft doesn't solve this problem, nobody will. Maybe he's paid to say that, but they are doing some interesting things with .NET, in particular Smith notes that F# is being "productized" and they are working on parallel LINQ
  

Tokyo Exchange goes Boom! Is there a link to the LSE? Could be...

This article caught my eye when Google Readering yesterday.

http://www.finextra.com/fullstory.asp?id=18058

Back in November,I noted that the LSE went Boom! and that this won't be the last time it falls over. It seems a coincidence that the Tokyo Stock Exchange is suffering performance problems and system crashes like it's technology partner the LSE. The pair "are developing jointly traded products and share technology" - oh really? that's a good idea? Same security vulnerabilities, same performance problems, same mysterious crashes then...

Word on the street also about other products based on dot net having serious, unsolvable bugs: mysterious freezes, runaway memory growth, threading issues and performance headaches, just like I experienced when building my last algo trading platform.

Mind you, this is nothing specific against dot net. I remember when I first worked in the city in 1996 doing C++ and using Roguewave libraries - what a disaster that was. Well I did promise to tell you about the eek bug back in the Haskell post so here goes: I was at my first investment bank - sigh - the good old days. One of the developers had a bug that he couldn't solve after a week - we tried everything - every debugger known - they all pointed to threading problems in the Roguewave libraries but the manufacturer was responding slowly. One day, the programmer put the string "eeek" as a printf debug. Miraculously, the bug disappeared. We tested other strings - non worked - only the eeek string did the trick.

That hardened my heart against third party libraries developed by private companies - when the bugs hit - you often have nowhere to turn. My guess is there's an eeek bug or two in the dot net stuff. If it were open source, perhaps there would be a hope of fixing it...

zeroMQ - Fastest Messaging Ever

Interesting offering from zeroMQ who seem to be pals with Intel. I've checked out the code briefly and am surprised by the use of dynamic memory allocation at such a low level of granularity. I usually allocate a large lump and rely on paging to weed out the empty pages. Anyway, from the very brief look I've had, it's written in C++ and has AQMP compatibility and for some reason I get a whiff of Erlang. More analysis coming soon.

What do you mean by end-to-end latency?

Steve just sent this through:

Order execution dynamics in a global FX market

"Very nice overview. This is key: "First, you need to understand what you mean by end-to-end latency (it will be different for different types of organization)".

One of the key metrics for FX turns out to be time to cancel a trade - on EBS over 70% of $1M trades are cancelled. Nobody wants to talk about the games they are playing so it's a bit tricky to know what someone means when they say they want sub-millisecond execution."

An intuitive paper detailing the clustering, spread and timings of limit orders on EBS. It's argued that impatient traders can drop below current price for faster execution. Sitting above the price takes longer as one would imagine. There's also a correlation with order size, showing smaller volumes trade quicker. Volumes are based on 2006 data and are not particularly high.

What's missing though is the underlying need for fast price and market data. The logical implication is reduction in latency - and therefore the necessity for physical proximity. Perhaps it's time to replicate LINX idea for high performance trading applications?

Object Orientated Security Policy

I gave a talk this week on Object Orientated Security Architecture to a Infosec management audience outside of the normal financial sectors that I work. Hopefully it was well received and made some sense. You can download the original presentation here.

Current State

• Invisible policy framework
• •Loose collection of abstract policies
• •Rarely if ever read, understood or referred to
• •Complex & technical v. functional & clear
• •No business rules (BPML)
• •Monolithic & centralised v. global and federated
• •Uncertain legal status
• •Audit/accountancy driven v. business focused
• •Control rather than functionality

New Ideas

• •Inheritance applied to Security Policy
• •General and abstract to specific and detailed
• •Data, event and process centred
• • Detailed and technical to minimal and clear
• • Navigable framework
• •Process catalogue
• •human readable (ie not just by policy wonks)
• •machine executable

• Components

• Visual Policy framework – (BITS SIG++)
• Hierarchal process catalogue (BITS AUP++)
• •Security Architecture Capture
• subset of BITS SIG
• concentrates on data flows
• asset, data, risk classes
• business value and application complexity
• operational impact analysis

Compliance

• Standardised Information Gathering (SIG) -

- ISO27002:2005, COBIT, PCI-DSS 1.1

• Agreed Upon Procedures (AUP)

• - GLB, HIPPA, COSO, SysTrust, SOX

• Employment Law