links

Establishes a probabilistic weighting framework that quantifies term specificity in document retrieval by formalising the inverse relationship between collection frequency and retrieval value. The paper derives Inverse Document Frequency (IDF) - calculated as the log of total documents divided by documents containing a term - demonstrating that rare terms carry disproportionately higher discriminatory power for isolating relevant documents from noise. Search ranking systems applying IDF-weighted term scoring achieve measurably superior precision over raw term-frequency matching, forming the mathematical foundation for TF-IDF signals that are useful for content relevance scoring, anchor text evaluation, and keyword targeting models.

The vector space model represents documents and queries as weighted term vectors in high-dimensional space, enabling similarity computation via cosine measures rather than Boolean exact-match retrieval. Experiments on the SMART system demonstrate that term weighting schemes combining term frequency (TF) with inverse document frequency (IDF) consistently outperform binary indexing, with IDF-weighted vectors producing superior recall-precision tradeoffs across multiple test collections. This mechanism directly powers ranked retrieval systems by scoring documents against queries through continuous similarity values, replacing brittle keyword matching with a scalable, corpus-aware relevance signal that underlies modern inverted index scoring functions including BM25 and learning-to-rank feature generation.

BM25 (Best Match 25) operationalises the Probabilistic Relevance Framework (PRF) by modelling document relevance as a probability estimate derived from term frequency, inverse document frequency, and document length normalisation, combining these signals through a tuneable saturating TF component (controlled by parameters k1 and b) to score documents against queries. The critical mechanism is the non-linear TF saturation curve, which prevents high-frequency terms from dominating relevance scores disproportionately, while the b parameter normalises document length against corpus averages, penalising verbose documents that accumulate term counts artificially. BM25 provides a computationally efficient, parameter-interpretable baseline that outperforms raw TF-IDF by handling term redundancy and document length bias - making it the de facto retrieval function for inverted-index architectures where lexical matching must approximate probabilistic relevance without requiring training data or vector embeddings.