Function tokenize

Split text into index terms. Tokens are runs of letters and digits; single characters are dropped (they carry no selectivity and bloat the index).