Your task in this assignment is to improve the performance of the deterministic methods you used in Assignment 1 through the use of a probabilistic language model (the details of what you try are up to you).
Your primary resource in tackling this problem is a large list of bigrams, with frequency counts, derived from the Google N-gram corpus. The most obvious approach is to use these counts to derive a bigram language model and then use that model to find the most probable segmentation given some hashtag.
Improving performance in this context, means reducing the error rate from the previous HW, where error rate is WER (length normalized minimum edit distance).
How much credit you get for this will be based in large part on how much your system improves things (reduces WER) and how complete/clever you were in your approach. Note that I am specifically interested in approaches that employ a probabilistic language model.
This assignment is due on Thursday, February 24.