Average Document Length in Okapi BM25

Question

I am studying the Okapi BMS25 model. I understand everything but two confusion. While calculating document length (dl) and average document length (avdl). I found the document length is

So it is a summation of my keywords/terms in a particular document. But when I see wiki's def:

So |D| is the length of the document D in words (i.e. is summation of total words count). Now, the question what is dl actually?

Now, second question how to calculate avdl? (just calculating (doc1+doc2+...N)/N where N is my total no documents in collection? (and avdl is fixed for whole collection?)

Answer 1

According the Joaquín Pérez-Iglesias in Integrating the Probabilistic Model BM25/BM25F into Lucene, the score function R should be defined as followed :

such as

• occurs_t^d is the term frequency of t in d,
• l_d is the document d length.
• avl_d is the document average length along the collection
• k_1 is a free parameter usually 2 and b in [0,1] (usually 0.75).

Assigning 0 to b is equivalent to avoid the process of normalisation and therefore the document length will not affect the final score.

If b takes 1, we will be carrying out a full length normalisation.

where N is the number of document in the collection and df is the number of documents where appears the term t.