System oriented Information Retrieval

Michael Preminger

Outline

Information and knowledge

What types of systems do we have?

Retrieval (matching) models

Retrieval (matching) models (cont.)

==> How do we evaluate the performance of a retrieval system?

Basic model of the retrieval evaluation situation

The Cranfield paradigm, developed in the 50-60s, still heavily used

kek jar(Kekäläinen & Järvelin, 2002)

Characteristics of the evaluation model

Components of the evaluation

Simpler view

The system

The system

What is being evaluated

"System" can be "anyting" (evidence* for relevance)

The document collection, Test Queries and "Facit"

Evaluation

evaluation

Measurement of performance

Recall: How many of the relevant documents did the system find?

Precision: To what extent are relevant documents "leading" in the ranking

Problems

measurement

Summary measures

Characterize aspects of system performance

Relevance

Types of relevance (Saracevic, 1996)
  • Situational relevance or utility
  • Cognitive relevance, or pertinence
  • Topical or Subject Relevance
  • System or Algorithmic relevance

Mizzaro's "problem set" (Mizzaro,1996)

 

Mizzaro

 

Where does relevance come into play in the evaluation model?

 

kekalainen jarvellin 2002

Critique against hard retrieval evaluation

Problems:

User need

Query:

Document

Techniques for being more "user-oriented"

Relevance feedback (user welcome inn ...)

Pseudo relevance feedback (away with the user again ...)

References

 

 

Other things:

 

Linguistics / "semantic ..."

Human information retrieval? Labor theories?

Pseudo relevance feedback?

Linked data?