Temporal News Signatures Used to Track Disease

Nice to see application of the techniques I described in my paper "Community-of-Interest Predicated Program Trading" used to track disease.

If they follow my logic, they'll use community expertise to do the analysis, annotation, recategorisation and dissemination to peer interest groups. Then it will be possible to build a reputation based analytics platform to summarise the trend, relate it to previous events and capture side-effects...

Applied Infoviz and Knowledge Re-injection

The Infoviz toolkit is used at Project Seven by a friend of mine et al. They're working on an intelligence analysis tool which supports reinjection of explicit knowledge earlier in the categorisation/discovery chain to guide discovery.

This is a different approach to the one I took in my signature based approach in my paper "Community-of-interest predicated program trading" where I suggested using centroid categorisation augmented with off-center categories. The Project Seven techinque relies on heuristically directed iteration (or as it's known in lay terms - trial and error) where my technique relies on the visualisation of the categorisation centroid. I think both approaches have merit and would produce good results.

Community of Interest Predicated Program Trading

I've had my paper on the above reviewed and accepted for publication and presentation to the Knowledge Management stream of this year's Operational Research Society conference, OR49 in Edinburgh. I'll be attending the conference for the three days and hanging out with Dr Duncan Shaw, an expert in Soft Systems Methodologies at Aston University.

You can download the paper from the Enhyper site here.

Mining Massive Data Sets for Security

Semiophore points me to the forthcoming two week workshop on the above to be held in mid-September 2007 in Italy.

"It is the purpose of this workshop to review the various technologies available (data mining algorithms, social networks, crawling and indexing, text-mining, search engines, data streams) in the context of very large data sets."

I'd love to attend as this is an area I think is crucial for High Frequency Finance. Whilst working on a high performance trade order router for a tier 1, I did some research which I was allowed to present publicly at the Fiteclub, a forum which meets occasionally in London. I presented two papers of note - Financial Data Infrastructure with HDF5 which concentrated on high performance data delivery and analysis. In this presentation I proposed a machine which could be built for around $25K that could eat 20TB of data in 90 minutes - using COTS components. This was inspired by the seminal article on disk technology amusingly entitled "You don't know jack about disks" published by the ACM.

The second presentation, also at Fiteclub, was entitled Open Source Intelligence in Finance and was inspired by the techniques used in open source intelligence applied to finance. Here I build the case for news analysis applied to program trading.

News Analysis for Program Trading

As previously posted, I'm writing a paper on news based program trading for the KM stream at the Operational Research Society's Annual Conference in September. This paper is the culmination of many years research and interest in the area of new analysis and I hope to show that the application of statistical techniques combined with visualisation can lead to an effective intelligence system which solves some of the conundrums facing traders, and for that matter, intelligence analysts.

The goal is to greatly shorten the time to disseminate events to the people who need to consume them, allowing them to act on this information. However, there's also an intention to analyse the likely outcome of this interaction and put in place a strategy to take advantage of this event. Another hypothetical outcome is that event "signatures" will be recognised and effects correlated in different sectors.

News Analysis

The first goal is to simplify the elements of news which we will analyse. To do this, I propose to model the way that people tend to read newspapers and select stories which interest them. Some read from front to back, others select favourite sections first, others, and I include myself here, read from back to front.

When we read, the first element to be considered is either the title or a picture. The writer of the article has to aphoristically state the contents of the news in an attempt to get the readers interest.
The title also contains other information like people, places, sectors, amounts, therefore this is the key piece which is used for presentation to the end user.

The rest of the story consists of a series of sentences arranged into paragraphs. Within the story will also be the information we are interested in. The relativity of people can be used to build a Social Network Analysis graph based on proximity. If two people are mentioned in the same sector (e.g. FX trading) they are related. If they are mentioned in the same publication they are related more closely. The same story, closer still. Same paragraph, even closer. Same sentence, the closest. From this we can draw a graph showing the individuals "social network". There's a very good example of this at www.namebase.org where you can perform useful searches on people involved in the intelligence world from their appearance in related publications.

Topographical Mapping

News also contains physical places. Mapping individuals, companies, sectors, amounts to physical location can reveal useful information and is a technique much used in policing.

Categorisation

Categorisation is something humans do every day and is fundamental to our heuristic judgement. Humans are very good at it, however, what they're not so good at is dealing with something which falls into multiple categories.


To be continued shortly...

OR49 Keynote Speech in the Knowledge Management Stream

I'm giving the keynote in the KM stream at this year's Operational Research Society Conference OR49 based on a stream of research which started about 8 years ago after reading a paper on newsgroup cluster analysis called telltale. Here's the abstract:

"It is proposed to summarise and statistically categorise multiple public and private information feeds to produce centroids directed by a combination of user constructed keywords and analysis of previously archived or disseminated knowledge. Social and physical networks will be extracted for temporal analysis and association projection. Comprehensive analysis of centroid relationships across sectors, categories and physical location will give a statistical event prediction capability and lead to the discovery of hidden relationships and associated events. End-users will construct a hierarchical keyword tree which will contain individual articles, summarisations, centroids or sets of related centroids. Users will also participate in a community of interest which they may form inter or intra-federation in order to disseminate emerging events or explicit knowledge. The system has applicability to financial market analysis, law enforcement and intelligence analysis."

This paper is the crystalisation of several themes and our experience into a system which we hope to build into an operational system. Many of the components already exist and over a series of articles I'll be discussing the philosophy behind the system. I'm joined on the enhyper blog by two experts in data visualisation whom you'll meet in due course. One is Dr Elie Naulleau from Semiophore. We'll propose how we can use the system for expert trading, algo trading and on the flip side, intelligence analysis.