P-value analysis of the `statistical significance' of the data

P-value is the term preferred in modern statistics to describe what physicists call, in simple words, ``probability of the tail(s),'' or ``probability to observe the events actually observed, or rarer ones, given a null hypothesis'' (note `given': the probability of whatever has been observed, without the specification of a particular condition, is always unity). In the frequentistic approach, the null hypothesis is rejected with a significance level $\alpha$ if the p-value gets below $\alpha$, where $\alpha$ is typically chosen to be $5\%$ or $1\%$. Besides the recognized misinterpretation of the p-value result (see e.g. [2]), there are often disputes about how this reasoning should be applied, because it is easy to show that there is much arbitrariness in the kind of test to be performed (it is well known that practitioner often seeks for the test that tells what they like, moving for $\chi^2$-test, to run-test and to other tests with fancy names, if the previously tried tests were ``not sensitive to the effect'') and in the data to include in the test, as it is sketched in the following subsections.