4 Calculating p-values

The main feature of the myFitter library is the numerical computation of p-values in likelihood ratio tests of nested and non-nested models. As discussed in [arXiv:1207.1446v2], p-values in likelihood ratio tests have to be computed numerically in cases where Wilks’ theorem is not applicable. Such cases include bounded parameters and models which are not nested, meaning that one model can not be obtained from the other by fixing some of its parameters.

In a likelihood ratio test one compares the performance of two models A and B in desribing observed data. The test is performed under the null hypothesis that one of the models, say, model B, is realised with certain parameters (usually its best-fit parameters for the measured data x_0. Then one considers a large ensemble of “toy measurements” which are randomly distributed about their true values (as predicted by model B) according to their experimental errors and uses the difference \Delta\chi^2=\chi^2_B-\chi^2_A of the minimal chi-square values of the two models as test statistic. The p-value is the probability that a toy measurement leads to a \Delta\chi^2 value which is bigger (i.e. more in favour of model A) than the \Delta\chi^2 value obtained from the measured data x_0.