System oriented Information Retrieval
- Some cnocepts important to system oriented information retrieval
- Types of systems / models of SOIR
- Evaluation model of SOIR and its components
- Critique of SOIR
- Some techniques
- .............................................Order a bit intermixed
Information and knowledge
- Knowledge: perceptions, observations and beliefs (inside humans)
- Information (independent of humans) (Losee, 1997)
- Information can change the state of knowledge
- Assumption: subject based information retrieval
(not "known item search"
- Information retrieval is the process of acquiring information, to change knowlege state
- How good is our system to help in this process?
What types of systems do we have?
Retrieval (matching) models
- Set based (boolean, or exact match)
- rank based (partial match)
- Vector model
- calculates "similarity" between document and query
Retrieval (matching) models (cont.)
- rank based (partial match)(cont.)
- Probabilistic model
- measures the probability that a document is relevant to a query (enkel formel)
- Language modeling: Similar to the prob. model, but with a different starting point
==> How do we evaluate the performance of a retrieval system?
Basic model of the retrieval evaluation situation
The Cranfield paradigm, developed in the 50-60s, still heavily used
(Kekäläinen & Järvelin, 2002)
Characteristics of the evaluation model
- "Hard", deterministic, retrieval evaluation
- Simple user-model (user isn't there ...)
- Documents either "relevant" or irrelevant to topic
- (Later: Documents can be partially relevant, but they are equally "partially relevant" for everybody)
Components of the evaluation
What is being evaluated
- Indexes the document collection
- The system should give a result for any query posed to it
"System" can be "anyting" (evidence* for relevance)
- A physical system
- An indexing method
- Preprocessing (Stemming, lemmatization, structure (tagging?))
- A variation over a retrieval model
The document collection, Test Queries and "Facit"
- A number of "test queries" are prepared (representative?)
- "experts" or laymen, intellectually assess each document in the collection against the queries
- generate "recall base" or a "facit" for each query.
- Test queries are "sent" to the system
- Results are recorded for each query
- Measure correspondence between the "FACIT" and the "RESULT"
- Average result over queries
Measurement of performance
Recall: How many of the relevant documents did the system find?
Precision: To what extent are relevant documents "leading" in the ranking
- Both recall and precision change along the ranked list
- High recall correlated with low precision (eks. stemming versus phrase indexing)
- Characterize a system by a graph, rather than by a number.
- (+) good for overall performance
- (-) difficult to compare systems
Characterize aspects of system performance
- precision at 10 documents
- reciprocal rank (when do we, in average, meet the first relevant document? the earlier the better ...)
- recall at 1000 retrieved documents
- MAP = Mean average precision
- NDCG at 10 documents
- "Relevance is not primarily an
issue involving human–computer interaction, but an issue
involving human interaction with recorded knowledge and
represented in discourses, documents, and languages." (Hjørland, 2010)
|Types of relevance (Saracevic, 1996)
- Situational relevance or utility
- Cognitive relevance, or pertinence
- Topical or Subject Relevance
- System or Algorithmic relevance
Mizzaro's "problem set" (Mizzaro,1996)
- No direct correspondence ... but Saracevic's relevances are at different levels of Mizzaro's "problem set"
Where does relevance come into play in the evaluation model?
Critique against hard retrieval evaluation
- Unclear, non- (or only partly)- expressible, dynamic (in time)
- Clear, an expression, static.
- ”A document, after all, is supposed to be a statement of what its author knows about a topic...[whereas] the expression of an information need is a general statement of what the user does not know” (Belkin et.al., 1982)
Techniques for being more "user-oriented"
Relevance feedback (user welcome inn ...)
- The user is asked to click on the first document(s) conceived relevant
- System modifies the query with words from these documents, producing a new ranked list.
Pseudo relevance feedback (away with the user again ...)
- The system assumes the first documents in the list are relevant
- System modifies the query with words from these document, producing a new ranked list.
- Belkin, N. J., Oddy, R. & Brooks, H. (1982). Ask for information retrieval:
Part i. background and theory. , 38 (2), 61–71.
- Hjørland, Birger (2010). The Foundation of the Concept of Relevance. Journal of the American Society of Information Science, 61(2), 217-237.
- Kekäläinen, J. & Järvelin, K. (2002a). Evaluating information retrieval systems under the challenges of interaction and multidimensional dynamic
relevance. In H. Bruce, R. Fidel, P. Ingwersen & P. Vakkari (Eds.), Proceedings of 4th CoLIS conference (pp. 253–270). Greenwood Village,
CO.: Libraries Unlimited.
- Losee, R. M. (1997). A discipline independent definition of information.
Journal of the American Society of Information Science, 48 (3), 254-
- Mizzaro, S. Relevance: The whole history. Journal of the American Society for Information Science, 48(9):810-832
- Saracevic, T. Relevance reconsidered. In Information Science:
Integration in Perspective, Proceedings of the 2nd
International Conference on Conceptions of Library and
Information Science (CoLIS 2), Copenhagen, Denmark,
Oct 13–16, 1996; Ingwersen, P., Pors, N.O., Eds.; Royal
School of Librarianship: Copenhagen, Denmark, 1996;
Linguistics / "semantic ..."
Human information retrieval? Labor theories?
Pseudo relevance feedback?