Tokenisation, bag of words, n-gram, tf-idf क्या है?

Tokenisation splits a text into units (words, word pieces, punctuation), usually after normalising case. A bag of words represents a document by the counts of its tokens, forgetting their order. An n-gram is a sequence of $n$ consecutive tokens; bags of n-grams keep some order. Term frequency–inverse document frequency (tf-idf) weights the count of term $w$ in document $d$ by $\log(N/n_w)$, where $n_w$ of the $N$ documents contain $w$, so that words found everywhere count for little; the rows are then scaled to unit length.