By Peter Cochrane, 31 March 2008 14:53
COMMENT
Written at home on a wet evening and dispatched to silicon.com via personal LAN.
Over the past few years I must have seen a dozen new search engines claiming to be the answer to a maiden's prayer.
Each has been based on single, or combined, parameter search and techniques involving: bots, crawlers, directories, indexing, inference, mathematics, nets, phrase, spiders, statistics, wiki, words and more.
And in each case the operational attributes have been in some way advantageous for some niche application or information space, but nothing could be held in overall superiority.
Today, the top 10 most used search engines include Google, Yahoo!, MSN, Ask, AltaVista and Yandex. There are hundreds of lesser-known examples, plus those amalgamating and federated search engines of search engines.
%20(3).jpg)
So you might think we had the search topic cracked. Not so.
By and large, today's search engines are brilliant and useless at the same time. Finding 63,800,000 results is both confounding and worrying.
Have I missed something vital and have I got the full picture, the latest and most relevant document, and perhaps more important, is what I am reading true?
Bluntly, I have no idea, and neither do you. What we really need is a search-engine filter based on validity. In short, a truth engine.
What we want to know is: are these statistics accurate and up to date? Are these historical events in order? Do I have the best available data? Is that politician correct? Is he lying, or is he bending the truth to his own advantage? And so on.
Our entire history of growth and success has been founded on a high-wire act that teeters between misconception and deception. Truth is at once vital, valuable and tenuous. And until recently I feel it was more identifiable and easier to establish.
The explosion of information seems to have made us richer and poorer at the same time. And while the baby boomers tend to question the validity of things, probe and research, generations X and Y see no such requirement.
A quick visit to Urban Legends and other debunking sites such as Snopes.com and Truthorfiction.com reveal a wealth of riches and illustrates my point about validity and truth. Just search 'debunking' for much more. Searching in sequence gave:
- 'Truth' gave 3,060,000 results.
- 'Truth engine' gave 991,000 results.
- 'Truth engine development' gave 213,000 results.
- 'Truth engine research' gave 1,860,000 results.
The general irrelevance of these search returns offers me little reassurance that this mega problem is being investigated to the point where we might expect a commercial offering in the near future.
This is all a bit of a blunder. It threatens the generation of content and the provision of a valued service. So, what's the answer? One thing is certain is that it cannot be policed or stopped.
As far as I can see the most formidable truth engine available to date is based on wiki technology. And while it is easy to see how we might automate the truth filtering of established factually based data, it is not clear how we might go a lot further without cognitive computation complete with contextual awareness. That is, a machine intelligence to match - or excel - our own.
When will this happen? In my view - soon. Web 2.0 is the starting point. Bandwidth, connectivity and sensors everywhere are the vital components - without which there will be no collective intelligence.
And the next big step is adaptability of software and ideally, but not absolutely necessary, hardware. These aspects are being born of the evolutionary developments in artificial life and artificial intelligence.
How soon is soon? I reckon the next 20 years will be really interesting, and we might just be able to find what we are looking for first time.
Comments
There are 6 comments. Join the discussion
1. David Flint
This really is asking for the automation of thought.
The general problem of deciding the truth of an assertion requires logic, analogy and a very wide knowledge base - itself of variable reliability.
The largest effort on this known to me is Project Cyc - now in its 24th year - and that clearly hasn't solved the problem. Cyc used to use stories from the National Inquirer as test cases.
Let us know when you find the Philosopher's Stone too, Peter.
2. Karen Challinor
A truth engine ?
Hmm. We could use it to tell when politicians are lying to us - sorry couldn't resist that one. Anyway, we can always tell when they are lying - their mouths are open.
The problem with truth is that it is relative and open to interpretation.
Two people observing the same event will see that event from two different perspectives and one may see something totally different to the other.
If this is subsequently reported then both will be telling the truth but the stories would be radically different. How is the truth engine going to decide which is more valid?
At best it will tell us what it decides is the truth. But who says it's going to be correct?
I may disagree and I frequently do. I would like to see the raw evidence for myself and make up my own mind.
Let the truth engine between your ears decide what the truth is. Yes it's a subjective and personal truth but it's worked pretty well so far.
3. John H Woods
Sounds ambitious but there are small improvements that could be made quickly.
I'm fed up with being presented with ended-auction listings from eBay when I search for things to buy, or looking for product reviews and getting either ancient previews or sites that say, 'No review is available for this item'.
4. Peter Cochrane
David: You are quite correct. I know Doug Lenat and Cycorp and his stirling efforts. He is getting partial success, but like decoding the genome this will take bigger machines, more intelligence, and more time. Peter
5. Peter Cochrane
Karen: On some topics we might find we have to settle for an 'agreed or concensus truth', but on a lot of factual information or recorded statements there can be little doubt when they are misquoted or distorted. Peter
6. Karen Challinor
"Karen: On some topics we might find we have to settle for an 'agreed or concensus truth', but on a lot of factual information or recorded statements there can be little doubt when they are misquoted or distorted. Peter"
But the hypothetical truth engine must have access to correct as well as incorrect information, so how should it decide which to report as the truth?
What will it check results against?
if it uses a definitive or authoritative source then definitive sources can occasionally make mistakes, if it uses consensus then you need a really large number of sources and even then there's still a possibility of being wrong.
I think you'll have to settle for the truth as you percieve it, based on the evidence that you have witnessed because even if you do make an engine to do this it will only be able to tell you the truth as it perceives it. It won't be infallible and if it isn't there's little point having it.