The effect of regularization is even more dramatic than the effects of noise and surgery. SAM supports both Dirichlet and Dirichlet mixture regularization. To gauge the effects of regularization, we compared no regularization to four other choices: the default simple regularizer, the original 9-component mixture [Brown et al., 1993], and more recent 9- and 21-component regularizers [Karplus, 1995]. Histograms of the test set NLL scores for these experiments, which used the noise settings determined in the previous experiments, are shown in Figure 6. Clearly, regularization is needed. With 50 training sequences, however, the distinction between the different regularizers is not high, as expected, since the model is built from a reasonably-sized training set. Although the original 9-component mixture appears to be the best, this mixture was in part based on a globin alignment, and thus is at an advantage in this experiment.
Where regularization truly shines is with small training sets. Figure 7 shows the performance of the regularization methods for small training sets. In this case, each test point represents the average test set NLL score over 20 training runs, each based on a different random training set of the indicated size
The Dirichlet mixture priors require additional running time, but have somewhat improved performance over the single-component prior. All four of these regularizers are available with the SAM distribution.
