Explore the vast range of titles available.

Firms using online advertising regularly run experiments with multiple versions of their ads since they are uncertain about which ones are most effective. During a campaign, firms try to adapt to intermediate results of their tests, optimizing what they earn while learning about their ads. Yet how should they decide what percentage of impressions to allocate to each ad? This paper answers that question, resolving the well-known “learn-and-earn” trade-off using multi-armed bandit (MAB) methods. The online advertiser’s MAB problem, however, contains particular challenges, such as a hierarchical structure (ads within a website), attributes of actions (creative elements of an ad), and batched decisions (millions of impressions at a time), that are not fully accommodated by existing MAB methods. Our approach captures how the impact of observable ad attributes on ad effectiveness differs by website in unobserved ways, and our policy generates allocations of impressions that can be used in practice. We implemented this policy in a live field experiment delivering over 750 million ad impressions in an online display campaign with a large retail bank. Over the course of two months, our policy achieved an 8% improvement in the customer acquisition rate, relative to a control policy, without any additional costs to the bank. Beyond the actual experiment, we performed counterfactual simulations to evaluate a range of alternative model specifications and allocation rules in MAB policies. Finally, we show that customer acquisition would decrease by about 10% if the firm were to optimize click-through rates instead of conversion directly, a finding that has implications for understanding the marketing funnel. Data is available at https://doi.org/10.1287/mksc.2016.1023 .

We propose an alternative dynamic price experimentation policy that extends multiarmed bandit (MAB) algorithms from statistical machine learning to include microeconomic choice theory. Pricing managers at online retailers face a unique challenge. They must decide on real-time prices for a large number of products with incomplete demand information. The manager runs price experiments to learn about each product’s demand curve and the profit-maximizing price. In practice, balanced field price experiments can create high opportunity costs, because a large number of customers are presented with suboptimal prices. In this paper, we propose an alternative dynamic price experimentation policy. The proposed approach extends multiarmed bandit (MAB) algorithms from statistical machine learning to include microeconomic choice theory. Our automated pricing policy solves this MAB problem using a scalable distribution-free algorithm. We prove analytically that our method is asymptotically optimal for any weakly downward sloping demand curve. In a series of Monte Carlo simulations, we show that the proposed approach performs favorably compared with balanced field experiments and standard methods in dynamic pricing from computer science. In a calibrated simulation based on an existing pricing field experiment, we find that our algorithm can increase profits by 43% during the month of testing and 4% annually. Data files and the online appendix are available at https://doi.org/10.1287/mksc.2018.1129 .

The web provides an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called randomized experiments, A/B tests (and their generalizations), split tests, Control/Treatment tests, MultiVariable Tests (MVT) and parallel flights. Controlled experiments embody the best scientific design for establishing a causal relationship between changes and their influence on user-observable behavior. We provide a practical guide to conducting online experiments, where end-users can help guide the development of features. Our experience indicates that significant learning and return-on-investment (ROI) are seen when development teams listen to their customers, not to the Highest Paid Person’s Opinion (HiPPO). We provide several examples of controlled experiments with surprising results. We review the important ingredients of running controlled experiments, and discuss their limitations (both technical and organizational). We focus on several areas that are critical to experimentation, including statistical power, sample size, and techniques for variance reduction. We describe common architectures for experimentation systems and analyze their advantages and disadvantages. We evaluate randomization and hashing techniques, which we show are not as simple in practice as is often assumed. Controlled experiments typically generate large amounts of data, which can be analyzed using data mining techniques to gain deeper understanding of the factors influencing the outcome of interest, leading to new hypotheses and creating a virtuous cycle of improvements. Organizations that embrace controlled experiments with clear evaluation criteria can evolve their systems with automated optimizations and real-time analyses. Based on our extensive practical experience with multiple systems and organizations, we share key lessons that will help practitioners in running trustworthy controlled experiments.

We reframe A/B testing as a decision problem with the goal of maximizing profit and derive the optimal sample size formula. Marketers often use A/B testing as a tool to compare marketing treatments in a test stage and then deploy the better-performing treatment to the remainder of the consumer population. Whereas these tests have traditionally been analyzed using hypothesis testing, we reframe them as an explicit trade-off between the opportunity cost of the test (where some customers receive a suboptimal treatment) and the potential losses associated with deploying a suboptimal treatment to the remainder of the population. We derive a closed-form expression for the profit-maximizing test size and show that it is substantially smaller than typically recommended for a hypothesis test, particularly when the response is noisy or when the total population is small. The common practice of using small holdout groups can be rationalized by asymmetric priors. The proposed test design achieves nearly the same expected regret as the flexible yet harder-to-implement multi-armed bandit under a wide range of conditions. We demonstrate the benefits of the method in three different marketing contexts—website design, display advertising, and catalog tests—in which we estimate priors from past data. In all three cases, the optimal sample sizes are substantially smaller than for a traditional hypothesis test, resulting in higher profit.

The metaphor of technical debt (TD) has generated a conceptual framework on factors that weaken the quality of software and accumulate a repair cost. However, user-related aspects like user experience (UX) receive little consideration among TD types, for reasons like the substantial focus on code TD, some dynamics inherent to agile processes, and an apparent lack of cumulative cost over time. This article has two main goals: first, to present evidence of the existence of UXDebt as a type of TD, with a cumulative cost for the development team as well as stakeholders; second, to propose a definition and characterization of UXDebt that may serve as a frame for further research on methods and tools for continuous management within agile processes. For the first goal, we have compiled evidence on the current state of UXDebt from three sources: a literature review, a survey among software engineering professionals in agile teams, and the analysis of UX issues in GitHub. All sources have evidenced some form of UXDebt; surveyed practitioners have recognized its poor management with a cost for the entire team that accumulates over time. Moreover, issue tracking systems allow to visualize and measure a technical form of UXDebt. For the second goal, we have defined a conceptual model that characterizes UXDebt in terms of both technical and non-technical aspects. On the technical side, we propose the notion of UX smells which allows us to discuss concrete management activities.

ContextContinuous experimentation (CE) is used by many companies with internet-facing products to improve their business models and software solutions based on user data. Some companies deliberately adopt a systematic experiment-driven approach to software development while some companies use CE in a more ad-hoc fashion.ObjectiveThe goal of this study is to identify factors for success in CE that explain the variations in the utility and efficacy of CE between different companies.MethodWe conducted a multi-case study of 12 companies involved with CE and performed 27 interviews with practitioners at these companies. Based on that empirical data, we then built a theory of factors at play in CE.ResultsWe introduce a theory of Factors Affecting Continuous Experimentation (FACE). The theory includes three factors, namely 1) processes and infrastructure for CE, 2) the user problem complexity of the product offering, and 3) incentive structures for CE. The theory explains how these factors affect the effectiveness of CE and its ability to achieve problem-solution and product-market fit.ConclusionsOur theory may inspire practitioners to assess an organisation’s potential for adopting CE and to identify factors that pose challenges in gaining value from CE practices. Our results also provide a basis for defining practitioner guidelines and a starting point for further research on how contextual factors affect CE and how these may be mitigated.

Advances in smartphone technology have led to the strong emergence of mobile learning (m-learning) on the market to support foreign language learning purposes, especially for the Japanese language. No matter what kind of m-learning application, their goal should help learners to learn the Japanese language independently. However, popular Japanese m-learning applications only accommodate on enhancing reading, vocabulary and writing ability so that user experience issues are still prevalent and may affect the learning outcome. In the context of user experience, usability is one of the essential factors in mobile application development to determine the level of the application’s user experience. In this paper, we advocate for a user experience improvement by using the mental model and A/B testing. The mental model is used to reflect the user’s inner thinking mode. A comparative approach was used to investigate the performance of 20 high-grade students with homogenous backgrounds and coursework. User experience level was measured based on the usability approach on pragmatic quality and hedonic quality like effectiveness (success rate of task completion), efficiency (task completion time) and satisfaction. The results then compared with an existing Japanese m-learning to gather the insight of improvement of our proposed method. Experimental results show that both m-learning versions proved can enhance learner performance in pragmatic attributes. Nevertheless, the study also reveals that an m-learning that employs the conversational mental model in the learning process is more valued by participants in hedonic qualities. Mean that the proposed m-learning which is developed with the mental model consideration and designed using A/B testing is able to provide conversational learning experience intuitively.

In large-scale e-commerce, recommendation systems must overcome the shortcomings of conventional models, which often struggle to convert user interest into purchases. This study proposes a revenue-driven recommendation approach that explicitly incorporates user price sensitivity. This study introduces a hybrid recommendation engine that combines collaborative filtering (CF), best match 25 (BM25) for textual relevance, and a price-similarity algorithm. The system is deployed within a scalable three-tier architecture using Elasticsearch and Redis to maintain stability under high-traffic conditions. The system’s performance was evaluated through a large-scale A/B test against both a CF-only model and a popular-item baseline. Results showed that while the CF-only model reduced revenue by 5.10%, our hybrid system increased revenue by 5.55% and improved click-through rate (CTR) by 2.55%. These findings demonstrate that integrating price similarity is an effective strategy for developing commercially viable recommendation systems that enhance both user engagement and revenue growth on large online platforms.

PurposeThe aim of the paper is to test and demonstrate the potential benefits in applying reinforcement learning instead of traditional methods to optimize the content of a company's mobile application to best help travellers finding their ideal flights. To this end, two approaches were considered and compared via simulation: standard randomized experiments or A/B testing and multi-armed bandits.Design/methodology/approachThe simulation of the two approaches to optimize the content of its mobile application and, consequently, increase flights conversions is illustrated as applied by Skyscanner, using R software.FindingsThe first results are about the comparison between the two approaches – A/B testing and multi-armed bandits – to identify the best one to achieve better results for the company. The second one is to gain experiences and suggestion in the application of the two approaches useful for other industries/companies.Research limitations/implicationsThe case study demonstrated, via simulation, the potential benefits to apply the reinforcement learning in a company. Finally, the multi-armed bandit was implemented in the company, but the period of the available data was limited, and due to its strategic relevance, the company cannot show all the findings.Practical implicationsThe right algorithm can change according to the situation and industry but would bring great benefits to the company's ability to surface content that is more relevant to users and help improving the experience for travellers. The study shows how to manage complexity and data to achieve good results.Originality/valueThe paper describes the approach used by an European leading company operating in the travel sector in understanding how to adapt reinforcement learning to its strategic goals. It presents a real case study and the simulation of the application of A/B testing and multi-armed bandit in Skyscanner; moreover, it highlights practical suggestion useful to other companies.

With the development of society, users have increasing requirements for the high-quality experience of products. The pursuit of a high profit conversion rate also gradually puts forward higher requirements for product details in the competition. Product providers need to iterate products fast and with a high quality to enhance user viscosity and activity to improve the profit conversion rate efficiently. A/B testing is a technical method to conduct experiments on target users who use different iterative strategies, and observe which strategy is better through log embedding and statistical analysis. Usually, different businesses of the same company are supported by different business systems, and the A/B tests of different business systems need to be operated in a unified manner. At present, most A/B testing systems cannot provide services for more than one business system at the same time, and there are problems such as high concurrency, scalability, reusability, and flexibility. In this regard, this paper proposes an idea of dynamic strategy distribution, based on which a configuration-driven traffic-multiplexing A/B testing model is constructed and implemented systematically. The model solves the high-concurrency problem when requesting experimental strategies by setting message middleware and strategy cache modules, making the system more lightweight, flexible, and efficient to meet the A/B testing requirements for multiple business systems.