links

Establishes a probabilistic weighting framework that quantifies term specificity in document retrieval by formalising the inverse relationship between collection frequency and retrieval value. The paper derives Inverse Document Frequency (IDF) - calculated as the log of total documents divided by documents containing a term - demonstrating that rare terms carry disproportionately higher discriminatory power for isolating relevant documents from noise. Search ranking systems applying IDF-weighted term scoring achieve measurably superior precision over raw term-frequency matching, forming the mathematical foundation for TF-IDF signals that are useful for content relevance scoring, anchor text evaluation, and keyword targeting models.

The vector space model represents documents and queries as weighted term vectors in high-dimensional space, enabling similarity computation via cosine measures rather than Boolean exact-match retrieval. Experiments on the SMART system demonstrate that term weighting schemes combining term frequency (TF) with inverse document frequency (IDF) consistently outperform binary indexing, with IDF-weighted vectors producing superior recall-precision tradeoffs across multiple test collections. This mechanism directly powers ranked retrieval systems by scoring documents against queries through continuous similarity values, replacing brittle keyword matching with a scalable, corpus-aware relevance signal that underlies modern inverted index scoring functions including BM25 and learning-to-rank feature generation.