How We Support You
Shopper and marketing insights come into focus throughout Artestry’s five-layered process:
Shopper and marketing insights come into focus throughout Artestry's five-layered process:
- Data Intelligence
- Creative Ideation
- Test Design
- In-market Execution
- Analysis & Insights
Testing never begins with a blank canvas. usually the starting point is the current status quo (best practices or "control" condition). Then we dig into your existing transactional and customer data to find high-potential marketing-mix changes... What variables stand out as important, what changes in the past made a difference, where have competitors succeeded, and which people, products, and promotions have opportunities for improvement?
This data intelligence can come from your in-house marketing and analytical teams, or Artestry can analyze your sales and customer data to identify important variables that should be included in the in- market test. After in-depth statistical analysis, we step back to provide a clear summary of actionable intelligence: key variables with a clear impact on sales and where we see opportunities for testing and improvement.
Intelligence also comes from the marketing team and creative group. We look at the creative variables, campaigns, products and promotions that have been implemented or tested in the past. We leverage your team's knowledge and skills to identify variables within the "art" of marketing that have a clear impact or provide the potential for increasing sales. The scientific data and marketing experience then come together to set the foundation for the test.
Once we've gathered all intelligence from past programs, promotions, and data, then we open the door to your team's creative energy...
Ideas drive marketing. Good ideas, well executed, drive profits. An organization may have the brightest creative minds, but without feedback, bright ideas often burn out in the marketplace. Unfocused creativity is like moonlight in the night—shining everywhere but illuminating nothing. Artestry helps you focus creative energy like a laser—drawing the best ideas to burn a lasting impression.
The success of every test starts with the building blocks: the specific elements of the marketing mix we choose to test. Even with the most advanced techniques, we can't test everything. A successful test strategy identifies high-potential elements, defines bold and independent settings for each, and executes clear and consistent combinations of elements for each market, store, or group of customers.
Since the selection and execution of test elements – like the tiles in a mosaic artwork – ultimately determine the attractiveness of the outcome, Artestry offers clear guidance in these early steps, before the statistical test design is ever created. This is accomplished through a multi-step process: unrestricted creative freedom in the initial brainstormed, followed by scientific discipline to quickly concentrate the list of ideas down to a few – or few dozen – high-potential, well-defined test elements.
The science of mosaic testing is a large step beyond standard one-variable-at-a-time techniques like controlled-store or matched-market testing (you can read more about mosaic testing). The toolbox of techniques allows you to test many variables all at once. Each test store (or mailing, e-mail, contact strategy, or media market) is assigned a unique combination of all the test elements as defined by the statistical test design. This means that each test group provides one data point on every variable in the test, so test data is more robust, efficient, and actionable.
Benefits include efficiency, speed, accuracy, and flexibility:
- Freedom to test numerous marketing-mix variables at once
- Using a fraction of the sample size required for one-variable techniques
- With greater depth of insights to quantify the real-world relationship among elements of the marketing mix
- With versatile test designs and strategies to manage resource constraints while ensuring clear, actionable results
The in-market test is the central step of the process. Data intelligence and new ideas are the inputs, while the in-market execution and analysis determine the outputs. Test results can be rolled back into the statistical models to further refine models, increase consumer and market knowledge, and uncover new opportunities for continual improvement.
The test design – the canvas upon which the test elements are mixed – is the scientific structure that ensures clear, valid, actionable results. But the science in isolation has limited value. What goes in (the ideas) and how it's presented (in-market execution) link the science with the reality of the marketplace. The integration of all aspects of the art and science is key to driving success.
In-market execution is unique for each project, program, and company. The general steps include:
1. Sample size and selection of test units
For retail testing, this is the number of weeks, particular markets, and the stores within those markets that will be analyzed for the test. For a CRM direct marketing program, this is the customer segments and number of individuals to give sufficient data to see valid results.
Common controlled-store, matched-market, or split-run tests make use of a test group and control group – the control is the status quo to compare against the changes executed for the test group. With mosaic retail testing, the "improvement versus status quo" is often analyzed 2-3 ways: (i) pieces of the "control" are built into each combination in the mosaic test strategy – run at the same time in randomly assigned markets – so all results are comparative, (ii) sales for each store are compared to predicted sales (based on recent history), (iii) a separate control group of markets/stores can be used for comparison to confirm results versus the status quo. For direct marketing tests – like direct mail, Internet, and e-mail programs – the test units are simply randomly assigned names from a homogeneous sample.
2. Ensuring clear data tracking
If you can't measure it, you can't improve. In the early stages of every test, the key metrics are defined along with the method for tracking results. Even for a simple direct mail test, we must consider whether online or in-store sales can be tracked along with the DM or catalog orders (to avoid sub-optimizing one channel versus total sales).
Retail tests can vary: we may track sales directly for loyalty cardholders, or focus on the change in retail sales, transactions, and basket size for each test group, or we may look at the change in sales for one market area versus another. Generally, more granular measurements are better (like sales for each household versus geographical region), but the ultimate metrics and data tracking depends upon the objectives and scope of each test.
3. Assignment of test recipes to markets/stores/individuals
Each group of households, stores, or markets that will receive a specific combination of variables is one "test unit." Each test unit should be homogeneous and comparable to each other test unit, so the change in sales among test units is a direct measure of the impact of all the test elements. The statistics can be complex, but generally we need to:
- Keep all test units similar
- Have minimal variation within each test unit (where all people/stores are predicted to respond the same way to any change in the marketing mix)
- Reduce outside sources of variation that may add noise to the test (like removing exceptional stores and separating high- and low-value customers)
- Accurately calculate predicted sales (and response and sales for the control group) as a baseline during the test
The general rule is to control what you can and randomize to protect from the unknown and uncontrollable. This is easier said than done, but gives additional power to the test by reducing error and ensuring reliable cause-and-effect results.
4. Operational execution
One reason marketers don't take advantage of these powerful tests is because retail tests are inherently more difficult to manage than direct marketing tests. You lose some control the closer you get to the "front lines" of the marketplace – customers, competitors, in-store decisions, and even changes in weather mean that so many variables are outside of your control. But that's where Artestry's experience is a clear advantage. We don't ignore the challenges, but cut through the noise to find clear answers within unstable markets. By focusing on small, controllable units, we can run quick, concise tests before market forces drown out the insights. Maybe the term "guerilla testing" would be appropriate, since we move in quickly to learn all we can before customers and competitors know what's going on.
5. Data collection and outlier analysis
Early calculations determine how long each test should run – the numbers or people, stores, markets, and/or weeks required. With careful up-front planning, we have systems in place to begin gathering and analyzing data every few weeks. This process includes some data cleansing and outlier analysis to be sure data are accurate and we remove any results that add to the error (sometimes as simple as removing daily sales just before/after a blizzard, or the first week of data for a store where the display was incorrectly executed).
With powerful tests, every datapoint is valuable. So extra effort pays off to ensure all data reflect real-world results. For many tests, these early analytics include both the "test" and "control" data. The control may be built into the test recipes, a separate group of people or stores, or a prior or predicted sales level for each test unit. The goal is always to quantify reality. The art is how to simplify the analytics – often reducing the number of variables in the analytical model (since each adds some potential error) – to give a clear, accurate, actionable picture of results. We balance power and simplicity, statistical rigor with efficiency.
Analysis & Insights
Test elements are the raw material – the "potential energy" of the test. The test design defines the collection of "in-market mosaics" presented to the audience of consumers – the one picture of the marketing mix that each person sees and reacts to. And the test units – the people, stores, and markets who see each mosaic – are the source of information.
With well-defined test elements, a solid test structure, and sufficient market data, the analysis is straightforward, with clear and actionable insights. Complex analytical techniques can be valuable to dig useful information from unstructured historical data, but well-structured test data should provide clear insights without overly complex statistics. The challenges are how best to cut through the noise and eliminate outliers and sources of variation outside of the elements intentionally changed. This requires analytical techniques unique to mosaic testing, yet with results that are easily understood and readily actionable.
Three Waves of Strategic Testing
Strategic projects often include three waves of testing: one large test to illuminate important elements, followed by a focused test to refine learnings and quickly squeeze out greater improvement, and then the final implementation and confirmation of results. These three waves – illumination, focus, and confirmation – help maximize insights while confirming results and building confidence for rapid rollout.
All combined, Artestry's focus on real-world insights offers you numerous, specific, quantifiable new opportunities proven to increase sales. With speed and confidence, you can react to market changes, respond quickly with new products, pricing, and programs, and reap the rewards before your competitors have a chance. There are few easy answers, but there are always some opportunities for improvement. The challenge is finding the right tools that help you dig out the hidden gems. Artestry can be your guide.