Yioop - PHP Search Engine

2012-12-10

Practice final solution for Problem 4.

Originally Posted By: fchan

4) What is document eliteness? How is it estimated in DFR? How is DFR modified to handled document-length normalization

document eliteness - A document is considered elite for a term, if the document is about the topic associated with the term.

In divergence from randomness (DFR), a generalized formula can be of the following...

Code: (1 - P2)(-logP1), where P2 is the part associated with "eliteness".
To calculate P2, we can use a modified version of "LaPlace's Theorum"

Code: (m/m+1), where we substitute m -> f(t,d)
to give us...

Code: P2 = f(t,d)/(f(t,d) + 1)
The formula above assumes all documents are of the same length so we can apply some form of normalization to factor in documents of different lengths within our corpus.

Therefore, we can modify our version of "LaPlace's Theroum" with the following...

Code: f'(t,d) = f(t,d) * (log(1+ (lavg/ld)))
this will accommodate fluctuations in document lengths.

Team members:
Frank Chan
Ranjith Jidigam
Hardik Rana
Nitin Tenali

'''Originally Posted By: fchan''' 4) What is document eliteness? How is it estimated in DFR? How is DFR modified to handled document-length normalization document eliteness - A document is considered elite for a term, if the document is about the topic associated with the term. In divergence from randomness (DFR), a generalized formula can be of the following... Code: (1 - P2)(-logP1), where P2 is the part associated with "eliteness". To calculate P2, we can use a modified version of "LaPlace's Theorum" Code: (m/m+1), where we substitute m -> f(t,d) to give us... Code: P2 = f(t,d)/(f(t,d) + 1) The formula above assumes all documents are of the same length so we can apply some form of normalization to factor in documents of different lengths within our corpus. Therefore, we can modify our version of "LaPlace's Theroum" with the following... Code: f'(t,d) = f(t,d) * (log(1+ (lavg/ld))) this will accommodate fluctuations in document lengths. Team members: Frank Chan Ranjith Jidigam Hardik Rana Nitin Tenali