Showing posts with label knowledge corruption. Show all posts
Showing posts with label knowledge corruption. Show all posts

Sunday, August 02, 2009

A case of knowledge corruption and its remedies

Information these days is flowing from all directions. Traditional sources like books, magazines, newspapers and articles have provided information for centuries. New forms like websites, blogs, emails and sms have emerged and newer ones are breeding. The delivery mechanism in itself has seen a tremendous technological enhancement ranging from text to on-demand services. Whether it is communicated officially or casually, is structured or unstructured, documented or loose, information is found in abundance these days.
Abundance is good and welcomed by all. Unlike commodities, information does not perish with consumption. It combines with the existing knowledge and gives new dimensions to think and work with. Hence, authenticity and correctness of information is of paramount importance for its recipients. Incorrect data cumulated over a period of time leads to knowledge corruption. This is a disaster in a world where knowledge is considered power.
Consider, for example, you receive an email containing statistics about the bandwidth consumption and the behavior pattern of an average Internet user. You find it interesting, believe it, and forward it to your friends and colleague. What you have just done is the beginning of knowledge corruption cycle, unless you were vigilant enough and verified the text you received.

This is a very trivial case with a low impact. Imagine the state of an organization or a team that is working on building a product and follows similar practices for its knowledge management.
In this connected world, the rate at which information can spread is exponential. The process of publishing is also very easy and quick. No certification or authenticity check, not even a simple sanity check is done or is in place. Anyone can publish and circulate anything and there is no accountability. There are no filters that can save you.
How can one ensure that he is not contributing in knowledge corruption? Here are a few simple points to keep in mind:
  • Always quote the source: This helps in building confidence. That’s how scientific journals work. Referencing is an important parameter used in Google’s PageRank algorithm.
  • Control excitement: Excitement leads you to skim information and this leads to half baked knowledge. Emotions make thinking bias. Don’t get carried away and don’t start cooking your own stories.
  • Verify before communicating any dubious information: If you are unsure about the things you are communicating, verify them. Use any search engine and keep searching unless you are convinced.
  • Do not make assumptions: If you are unsure about something, then tell that you don't know. Don't speak just for the sake of speaking.
  • Establish pattern once you have understood the information correctly, whenever feasible try to stay away from patterns during learning phase.
Role of technology and policies
Technology acts as an instrument for enforcing policies. Policies provide rules and filters for publishing information. Technology can also be used to check the authenticity. Here is how we can use a combination of technology and policies to ensure correctness:
  • Information publishing policy: Enforce policies that permit publishing only after authenticity is established. This might be difficult and time consuming for the world information, but is suited for use within organizations. How can this be done when we are not yet very close to machine learning and understanding? Till then, people will have to substitute for machines. This will enable a selected group of people to first validate that the information they are providing (existing information) is correct. Everything above it will be based on and will refer this verified information. How will we add a totally new knowledge base? And how do we add contradicting information (for example, people have different theories and beliefs which are completely opposite to each other)? What about thoughts and vague ideas? These can be categorized as personal opinions and best left to the readers’ perception. If followers increase beyond a certain number, a new religion is formed. This article should hence be categorized as personal beliefs.
  • Rating and collective intelligence: Fix ownership and build a rating system. People will have complete ownership of what they are publishing, circulating, or promoting. Trust-levels can be established by cashing on social networks. People can associate trust-levels to different entities. Say, for example, a person can set that he trusts an entity A (person, source, or information) 100% and trusts another entity B just 75%. The chain continues and the trust-levels are calculated for each entity by using the cumulative ratings. Two types of trust-levels can be established for each entity:
    • Global: This is the trust-level computed using the ratings provided by all the people who have rated the entity.
    • My circle: Trust-level computed using the ratings provided by only the people who are part of the reader’s circle.
    This is similar to other applications of collective intelligence that are already in use for features like “Find similar items” or “You might also like these”.

Monday, November 24, 2008

Why not to use search engines

During my college days I was not aware of search engine clientele. Consequently, the area/domain where Google search was being used was an unfamiliar territory. For me, it was a new search engine that was continuously crawling net and used the PageRank algorithm to bring better search results as compared to other technologies. I believed it was being used by casual surfers. I was still betting more on Semantic web (studying OWL was something new that I did at that time), but I do believe that the ultimate search engine will be based on algorithms that use Artificial Intelligence to understand text available on the Internet and provide accurate, meaningful, and context-sensitive results.

Few years back, I was shocked to find how widely search engines were being exploited by the developer community. I saw many people depending on search engines for code snippets. Personally, I started hating search engines for corrupting developers’ mindset and mounting reliance. Hate is no solution, so I embarked on a new project. The aim was straight “Get rid of developers”. Initially, the task looked simple -- write a program that fetches the correct code snippet from the net, filter the results based on the task at hand, and customize it depending on user’s input.

The results were not convincing, following were the primary causes:

  1. Authenticity was not guaranteed
  2. Incomplete results
  3. Addressed either the simplest (the most common) or a single scenario
  4. Incorrect
  5. Not ready for direct consumption

Solution to this exists: trust services, semantic web, investment in organizing and building appropriate code snippets, ETL. But the scope is limited. Some IT firms maintain a database that can be queried for commonly used code, but the process is not fully automated. It is an aid, not a replacement.

Although, priorities changed and I completely forgot about this project, looks like I need to restart the engine :)
Intent was much bigger – give software creation directly in the hands of its user. The idea was to develop an application that given the requirements will break these down into atomic tasks, maintain a set of domain specific components, integrate components by guiding the flow of data and control from one component to other (glue), map task to functionality, and relies on net (shared repository) for missing code.
In short build an ecosystem that helps user build what he wants.

I believe in a continuous knowledge acquisition process. A business expands by investing in the development of new product lines, expanding the service areas and acquiring other companies. A developer grows by reading more and more (on a regular basis) and venturing into new domains. This is a slow process, but results in fewer hurdles, timely completion of tasks, confidence of going in the right direction and doing it right.

Search engine provides a quick solution, is addictive, but hollows the intellect. Areas worst hit are product quality and developer's confidence.

When to use search engines:

  • for awareness on what’s already there, so that you don't reinvent the wheel
  • to look what others are doing, so that you don't do a parallel development (did someone mention competition ;))
  • to find online projects, so that you can contribute your expertise
  • for new and upcoming technologies -- this is the right time to search for parallelism and cloud computing
  • find recipe of paw bhaji :)
  • to check whether your internet connection is working :)

Don’t hit the search engines to find answers within a domain where you are supposed to be an expert, use it to find knowledge banks that provide comprehensive theory coverage. Use it to find authentic source of information. There are plenty of these on the net. Here are some of the sites that I visit:

Look for magazines and journals that are published (some are given out for free)

  • MSDN
  • Dr. Dobb
  • Architecture Journal
  • CIO
  • PCQuest

Subscribe to feeds and newsletters published by infosites. At times, you will be bogged down by the number of emails in your inbox, so choose judiciously.

And most importantly, never forget your textbooks.

So surf well, surf wise.