Explore the vast range of titles available.

Questions often arise concerning when, whether, and how we should adjust our interpretation of the results from multiple hypothesis tests. Strong arguments have been put forward in the epidemiological literature against any correction or adjustment for multiplicity, but regulatory requirements (particularly for pharmaceutical trials) can sometimes trump other concerns. The formal basis for adjustment is often the control of error rates, and hence the problems of multiplicity may seem rooted in a purely frequentist paradigm, though this can be a restrictive viewpoint. Commentators may never wholly agree on any of these things. This article draws some of the key threads from the discussion and suggests further reading.

A sizeable literature exists on the use of frequentist power analysis in the null-hypothesis significance testing (NHST) paradigm to facilitate the design of informative experiments. In contrast, there is almost no literature that discusses the design of experiments when Bayes factors (BFs) are used as a measure of evidence. Here we explore Bayes Factor Design Analysis (BFDA) as a useful tool to design studies for maximum efficiency and informativeness. We elaborate on three possible BF designs, (a) a fixed- n design, (b) an open-ended Sequential Bayes Factor (SBF) design, where researchers can test after each participant and can stop data collection whenever there is strong evidence for either ℋ 1 or ℋ 0 , and (c) a modified SBF design that defines a maximal sample size where data collection is stopped regardless of the current state of evidence. We demonstrate how the properties of each design (i.e., expected strength of evidence, expected sample size, expected probability of misleading evidence, expected probability of weak evidence) can be evaluated using Monte Carlo simulations and equip researchers with the necessary information to compute their own Bayesian design analyses.

This article is devoted to the construction and asymptotic study of adaptive, group-sequential, covariate-adjusted randomized clinical trials analysed through the prism of the semiparametric methodology of targeted maximum likelihood estimation. We show how to build, as the data accrue group-sequentially, a sampling design that targets a user-supplied optimal covariate-adjusted design. We also show how to carry out sound statistical inference based on such an adaptive sampling scheme (therefore extending some results known in the independent and identically distributed setting only so far), and how group-sequential testing applies on top of it. The procedure is robust (i.e. consistent even if the working model is mis-specified). A simulation study confirms the theoretical results and validates the conjecture that the procedure may also be efficient.

We consider a multiple‐hypothesis testing setting where the hypotheses are ordered and one is only permitted to reject an initial contiguous block H1,…,Hk of hypotheses. A rejection rule in this setting amounts to a procedure for choosing the stopping point k. This setting is inspired by the sequential nature of many model selection problems, where choosing a stopping point or a model is equivalent to rejecting all hypotheses up to that point and none thereafter. We propose two new testing procedures and prove that they control the false discovery rate in the ordered testing setting. We also show how the methods can be applied to model selection by using recent results on p‐values in sequential model selection settings.

We consider optimal sequential allocation in the context of the so-called stochastic multi-armed bandit model. We describe a generic index policy, in the sense of Gittins [J. R. Stat. Soc. Ser. B Stat. Methodol. 41 (1979) 148—177], based on upper confidence bounds of the arm payoffs computed using the Kullback—Leibler divergence. We consider two classes of distributions for which instances of this general idea are analyzed: the kl-UCB algorithm is designed for one-parameter exponential families and the empirical KL-UCB algorithm for bounded and finitely supported distributions. Our main contribution is a unified finite-time analysis of the regret of these algorithms that asymptotically matches the lower bounds of Lai and Robbins [Adv. in Appl. Math. 6 (1985) 4—22] and Burnetas and Katehakis [Adv. in Appl. Math. 17 (1996) 122—142], respectively. We also investigate the behavior of these algorithms when used with general bounded rewards, showing in particular that they provide significant improvements over the state-of-the-art.

In this paper, we propose a class of monitoring statistics for a mean shift in a sequence of high-dimensional observations. Inspired by recent U-statistic based retrospective tests, we extend the U-statistic-based approach to the sequential monitoring problem by developing a new adaptive monitoring procedure that can detect both dense and sparse changes in real time. Unlike existing methods in retrospective testing that use self-normalization, we introduce a class of estimators for the q-norm of the covariance matrix and prove their ratio consistency. To facilitate fast computation, we further develop recursive algorithms to improve the computational efficiency of the monitoring procedure. The advantages of the proposed methodology are demonstrated using simulation studies and real-data illustrations.

Open data allows researchers to explore pre-existing datasets in new ways. However, if many researchers reuse the same dataset, multiple statistical testing may increase false positives. Here we demonstrate that sequential hypothesis testing on the same dataset by multiple researchers can inflate error rates. We go on to discuss a number of correction procedures that can reduce the number of false positives, and the challenges associated with these correction procedures.

The practice of sequentially testing a null hypothesis as data are collected until the null hypothesis is rejected is known as optional stopping . It is well known that optional stopping is problematic in the context of p value-based null hypothesis significance testing: The false-positive rates quickly overcome the single test’s significance level. However, the state of affairs under null hypothesis Bayesian testing, where p values are replaced by Bayes factors, has perhaps surprisingly been much less consensual. Rouder ( 2014 ) used simulations to defend the use of optional stopping under null hypothesis Bayesian testing. The idea behind these simulations is closely related to the idea of sampling from prior predictive distributions. Deng et al. ( 2016 ) and Hendriksen et al. ( 2020 ) have provided mathematical evidence to the effect that optional stopping under null hypothesis Bayesian testing does hold under some conditions. These papers are, however, exceedingly technical for most researchers in the applied social sciences. In this paper, we provide some mathematical derivations concerning Rouder’s approximate simulation results for the two Bayesian hypothesis tests that he considered. The key idea is to consider the probability distribution of the Bayes factor, which is regarded as being a random variable across repeated sampling. This paper therefore offers an intuitive perspective to the literature and we believe it is a valid contribution towards understanding the practice of optional stopping in the context of Bayesian hypothesis testing.

Active monitoring of safety outcomes following COVID-19 vaccination is critical to understand vaccine safety and can provide early detection of rare outcomes not identified in pre-licensure trials. We present findings from an early warning rapid surveillance system in three large commercial insurance databases including more than 16 million vaccinated individuals. We evaluated 17 outcomes of interest following COVID-19 vaccination among individuals aged 12–64 years in Optum, HealthCore, and CVS Health databases from December 11, 2020, through January 22, 2022, January 7, 2022, and December 31, 2021, respectively. We conducted biweekly or monthly sequential testing and generated rate ratios (RR) of observed outcome rates compared to historical (or expected) rates prior to COVID-19 vaccination. Among 17 outcomes evaluated, 15 did not meet the threshold for statistical signal in any of the three databases. Myocarditis/pericarditis met the statistical threshold for a signal following BNT162b2 in two of three databases (RRs: 1.83–2.47). Anaphylaxis met the statistical threshold for a signal in all three databases following BNT162b2 vaccination (RRs: 4.48–10.86) and mRNA-1273 vaccination (RRs: 7.64–12.40). Consistent with published literature, our near-real time monitoring of 17 adverse outcomes following COVID-19 vaccinations identified signals for myocarditis/pericarditis and anaphylaxis following mRNA COVID-19 vaccinations. The method is intended for early detection of safety signals, and results do not imply a causal effect. Results of this study should be interpreted in the context of the method’s utility and limitations, and the validity of detected signals must be evaluated in fully adjusted epidemiologic studies.

In many fields of science, we observe a response variable together with a large number of potential explanatory variables, and would like to be able to discover which variables are truly associated with the response. At the same time, we need to know that the false discovery rate (FDR)—the expected fraction of false discoveries among all discoveries—is not too high, in order to assure the scientist that most of the discoveries are indeed true and replicable. This paper introduces the knockoff filter, a new variable selection procedure controlling the FDR in the statistical linear model whenever there are at least as many observations as variables. This method achieves exact FDR control in finite sample settings no matter the design or covariates, the number of variables in the model, or the amplitudes of the unknown regression coefficients, and does not require any knowledge of the noise level. As the name suggests, the method operates by manufacturing knockoff variables that are cheap—their construction does not require any new data—and are designed to mimic the correlation structure found within the existing variables, in a way that allows for accurate FDR control, beyond what is possible with permutation-based methods. The method of knockoffs is very general and flexible, and can work with a broad class of test statistics. We test the method in combination with statistics from the Lasso for sparse regression, and obtain empirical results showing that the resulting method has far more power than existing selection rules when the proportion of null variables is high.