# You Ask, I Answer - Setting Test Timeline Expectations > [!info] Deep-read port — Readwise notes field already contained a full deep-read (Refactor Appendix, 3 literature atoms + 1 permanent candidate). > Ported from the existing deep-read. YouTube video. https://www.youtube.com/watch?v=p9vljHYz9lM > [!question] Lingering Questions > - The video defines confidence level as "the percentage of repeat tests that would return the same result" — that's a misdefinition (conflates confidence, reproducibility, and power). Does that error propagate into the actual calculator recommendation, or is it just loose narration around a correct underlying tool? > - Where's the line between "document the stakeholder's chosen risk tolerance" (legitimate) and "use documentation to shift blame away from the analyst" (the video's own framing risks this)? > [!tip] Article Ideas > - A short corrective note on the confidence-interval misconception, usable as a standalone reference the next time this specific error shows up in other marketing-analytics content. > [!link] Cross Reference > - [[3. Resources/MOCs/Product & Startup Instincts MOC]] — data-driven timeline-setting as a check against instinct-driven deadline promises > - [[3. Resources/Permanent Notes/Instincts Are Reliable Ideas Are Not So Test Small Bets on Proven Foundations]] — same underlying tension between gut-feel commitments and what the data can actually support > [!quote] Vocabulary > - **confidence level** (as *correctly* defined, contra the video) — the long-run proportion of confidence intervals, constructed the same way across repeated sampling, that would contain the true population parameter — not "the percentage of repeat tests that return the same result." > - **stopping rule** — a pre-registered rule for when an experiment ends and its result is read, meant to prevent "peeking" at partial data and stopping early on a favorable-looking result. ## Author's thesis Christopher S. Penn argues that marketing-test timelines are calculable, not negotiable: baseline traffic, the chosen experiment design, and the desired confidence level together determine how long a test needs to run, and an interactive calculator can make that tradeoff visible to stakeholders instead of leaving it as an unexamined deadline argument. ## Key arguments (≤3) 1. **Test duration is a function of traffic and design, not a negotiable date** — a two-day deadline doesn't create two weeks of traffic; the achievable timeline is bounded by what the actual sample can support. 2. **An interactive calculator turns hidden assumptions into visible, adjustable inputs** — stakeholders can see the tradeoff between speed, traffic, and evidential strength directly, rather than treating a deadline as independent from what the data can prove. 3. **The video's own statistical explanation contains a real error** — describing a confidence interval as "the percentage of repeat tests that would return the same result" conflates confidence level, reproducibility, and statistical power; the underlying planning-calculator idea is sound even though this specific explanation needs correction before reuse. ## Claim atoms ### Let the available data constrain the timeline > [!quote] > It always comes back to how much traffic can you put through that to get the result that you want. Traffic and effect size set feasibility boundaries a deadline cannot override. ### Make the tradeoff visible > [!quote] > It can show you an estimate and say, "To get to this level of confidence interval, you will it will take this much traffic, which will take this much time based on existing data." The useful design move is the interactive connection among assumptions, sample, and duration — though the "confidence interval" language here carries the video's definitional error and needs correcting before reuse. ### Use the model for future planning > [!quote] > These tools then you can repurpose into planning tools. A reviewed calculator becomes more valuable used before a test than as a defense after a deadline dispute. ## Permanent candidates - **A test deadline is an output of traffic and decision requirements, not an independent input** — the video's central, defensible claim, cleanly separable from its statistical-definition error. Promoted below. ## Processing checklist - [x] Port the existing deep-read into Konik Literature Note format - [ ] Write (or find) a corrected, precise definition of confidence level to pair with any future reuse of this video's calculator idea - [x] Connect into [[3. Resources/MOCs/Product & Startup Instincts MOC]] ## Links - [[3. Resources/MOCs/Product & Startup Instincts MOC]] — data-constrained timeline-setting as a discipline against instinct-driven deadline commitments - [[3. Resources/Readwise/Articles/You Ask, I Answer Setting Test Timeline Expectations]] — source document, full deep-read - [[3. Resources/MOCs/Evidence & Claims Literacy MOC]] — applies-in: Joins notes on testing claims and reading statistics correctly.