Did the product fail, or did the test setup fail?

01

What should you do immediately after a failed run?

Freeze the result before cleaning, resetting, editing, returning, or discarding anything. A failed take remains part of the test record even if a later run succeeds.

Duplicate the original video and photos rather than overwriting them with the retest. Preserve images of instrument displays, product condition, placement, connections, packaging, instructions and identifying marks. Keep the tested unit and any remnants when it is safe to do so.

Record the exact result and the acceptance threshold. Add the date, time, location, relevant temperature or humidity, surface, lighting, voltage or water conditions, product variant, supplier order, lot markings and shipping condition. Note preparation, charging, curing, soaking, heating, cooling or waiting times. For instruments, record the model, units, range, resolution, mode, settings, battery state and calibration or reference-check status.

Also identify the operator and document interruptions or departures from the planned method. Filming can itself affect a test: a person may use an awkward angle, pause for camera coverage or apply force differently. These are possibilities to investigate, not automatic excuses.

FDA guidance for investigating unexpected pharmaceutical test results is not a general testing standard for dropshippers. It nevertheless illustrates a sound evidence-preservation principle: retain raw data, document obvious errors promptly and investigate before disregarding an unexpected result. A structured evidence brief makes that discipline easier to follow.

1. Stop changing the test area or product.
2. Save raw files under unique names and duplicate them.
3. Photograph the complete setup and identifying marks.
4. Write down the result, threshold, conditions and deviations.
5. Quarantine the sample if further handling could destroy evidence or create a hazard.

02

What claim was the test actually designed to examine?

Restate the test as a narrow, measurable hypothesis before diagnosing the failure: under specified conditions, a specified unit should produce a defined outcome within a defined tolerance and time.

Keep observation separate from interpretation. “The scale displayed 8.4 kilograms” is an observation. “This unit did not meet our 10-kilogram threshold under this setup” is a bounded interpretation. “Every unit in the supplier’s inventory can hold only 8.4 kilograms” is an unsupported expansion unless broader sampling justifies it.

One unit tells you what happened to that unit under the recorded conditions. NIST explains that facts observed in a sample are not automatically facts about the population; representativeness, sample size, variability and required precision affect the inference. A failed sample should not be ignored, but neither should it be used alone to calculate or imply an inventory-wide defect rate.

The same discipline applies to advertising. The FTC states that advertisers need a reasonable basis for objective express and implied claims before dissemination. If a creative communicates that tests prove a performance claim, the supporting evidence must be adequate for that claim. A demonstration also must not be staged so that it conveys a false impression of what the product can do.

1. Name the exact unit or sample tested.
2. Define the operating and environmental conditions.
3. Define the outcome, tolerance and measurement time.
4. Write the direct observation without causal language.
5. Draft only the narrow conclusion supported by that observation.

03

Which seven possible failure sources should you audit?

Audit preparation, environment, operator, timing, instrument, sample and procedure in a fixed order. Do not stop when you find the first plausible—but unverified—explanation.

Preparation: Check assembly, charging, mixing, washing, conditioning, priming and preheating against the instructions. Confirm that protective film, transport locks and packing inserts were removed. Verify quantities and ratios from records rather than memory.

Environment: Consider temperature, humidity, airflow, lighting, surface texture, voltage, water quality or signal strength when relevant. Record whether the condition was unusual or representative of foreseeable customer use. A demanding but realistic condition may reveal a genuine limitation rather than invalidate the test.

Operator: Check whether the tester understood the instructions and consistently controlled force, angle, speed, distance and positioning. Review the footage for interruptions, improvised steps or camera requirements that changed normal use.

Timing: Verify charging, resting, curing, soaking, heating, response and measurement intervals. A reading taken too early can be invalid, while a reading taken after expected deterioration may answer a different question.

Instrument: Confirm that the scale, timer, thermometer, meter or reference object was suitable for the expected range and precise enough for the threshold. Check zeroing, mode, units, configuration and performance against a known reference. NIST’s gauge-study guidance identifies repeatability, reproducibility, stability, resolution, drift, instrument differences and configuration differences as possible contributors to measurement uncertainty.

Sample: Inspect for visible damage, contamination, missing parts, expiration, previous stress, storage problems or the wrong size or variant. Compare order, package and lot information with other samples. Shipping damage can explain one unit without proving that the method was invalid.

Procedure: Compare the recorded run with the written protocol. Ask whether the method measures the claimed attribute directly or only a proxy. For example, appearance on camera may not measure durability, and a brief load demonstration may not establish long-term load capacity.

04

When is a setup failure genuinely established?

A setup failure requires positive evidence of an assignable cause. It cannot be inferred merely because a later attempt produced a better result.

Strong evidence includes an instrument failing a reference check, video showing the wrong quantity or setting, a required preparation step being omitted, the wrong variant being tested, a demonstrably incorrect calculation, or documented instructions showing that the test exceeded an operating limit.

Write the suspected cause in falsifiable terms: “The first reading was low because the scale was not zeroed,” not “something probably went wrong.” Then verify the scale’s initial state from footage or records, check it against a reference and repeat the run after correcting only that issue.

If you cannot demonstrate the error, keep the original run valid as an unresolved result. “The next attempt passed” does not prove why the first attempt failed. Repeating until a desirable result appears and then publishing only the pass discards inconvenient evidence rather than resolving it.

1. Name one suspected cause.
2. Identify the record or check that could confirm it.
3. Explain how that cause could produce the observed result.
4. Correct only that cause during the targeted retest.
5. Retain both the original and corrected results.

05

How should you plan the retest without moving the goalposts?

Write the retest plan before starting. Fix the threshold, number of attempts, reporting rule, sample choice and single intended change in advance.

Begin with a repeatability check: repeat the same procedure with the same operator, instrument, location, conditions and short time interval. NIST defines repeatability in terms of successive measurements under the same measurement conditions. If the result swings widely, investigate the method, instrument or naturally variable product response.

Next, probe reproducibility. NIST uses this term for agreement when specified conditions change, such as the observer, instrument, location, method or time. A test can be repeatable for one operator yet differ for another. That pattern may reveal operator sensitivity, ambiguous instructions or a condition-dependent product.

When several factors are plausible, do not fix all of them in one polished retake. Changing the operator, instrument, preparation and sample together may produce a pass, but it cannot show which change caused it. NIST describes design of experiments as planned factor variation used to support objective conclusions; factors that are not separated can become confounded.

Use a staged sequence: first repeat the same setup and unit; then correct one evidenced factor on that unit; then test a fresh unit with the original valid method; then vary one operator, day or instrument; finally test additional supplier samples if you need to explore unit-to-unit or batch variation. For a direct comparison, use a documented side-by-side protocol rather than adjusting each sample differently.

1. State the suspected cause and the one factor to change.
2. Keep unrelated factors fixed and recorded.
3. Set pass, fail and ambiguous thresholds in advance.
4. Choose the number of runs before seeing new results.
5. Specify which unit, order or supplier batch each run uses.
6. Preserve and report failures as well as passes.

06

How should you classify the final evidence?

Use four outcomes: product failure, setup failure, condition-specific, or inconclusive. The classification should state both what happened and how far the evidence can reasonably be generalized.

Product failure: The failure repeats under a verified procedure and no setup error is demonstrated. The safe initial conclusion is “this unit failed this test.” Broader wording requires additional representative samples and a method capable of supporting the broader inference.

Setup failure: A specific preparation, procedural, calculation or instrument error is demonstrated, and correcting only that cause resolves the result. Preserve the first run and explain why it was invalid; do not pretend it never happened.

Condition-specific: The product passes under one defined condition but fails under another relevant condition. Report the dependency. Do not compress it into a universal pass or failure. This outcome may matter especially when the failing condition resembles normal customer use.

Inconclusive: Results conflict, the suspected cause cannot be isolated, the instrument is unsuitable, or the available samples cannot answer the question. “The available test does not support a reliable conclusion” is a legitimate outcome. It is better than forcing uncertain evidence into an advertising claim.

07

When should you stop retesting for safety reasons?

Stop ordinary testing when a run reveals a credible risk such as overheating, smoke, electric shock, fire, hazardous breakage, chemical exposure or a choking hazard. Isolate the sample and related inventory rather than repeatedly recreating the event for footage.

Do not handle a potentially hazardous product merely to obtain a cleaner take. Preserve evidence from a safe distance, document the supplier and product identifiers, and check applicable recalls and reporting obligations. The CPSC advises online sellers to inspect products for safety and recalls; depending on their role and the facts, manufacturers, importers, distributors and retailers can also have reporting duties for potentially dangerous products.

Dropshipping does not remove product-safety responsibility. Shopify’s legal guidance tells dropshipping merchants to purchase products from suppliers to assess quality and states that merchants are responsible for ensuring the products they offer are safe. Obtain appropriate technical or legal advice when the evidence indicates a genuine hazard.

08

How can ATIYO preserve the investigation for future creative work?

ATIYO can keep the failed run, test conditions, diagnosis, retest plan, assets, iterations and resulting learning connected to the creative brief. It does not determine whether the scientific conclusion is correct; the team must record and evaluate the evidence.

A practical record should connect each raw asset to its unit, procedure, conditions and outcome. The final learning should state why a take was rejected, whether the rejection was evidentiary or merely editorial, and what later testing did or did not establish. This prevents a polished passing clip from becoming detached from contradictory source material.

Once the product evidence is sufficiently supported, that context can inform a demonstration brief and subsequent creative iterations. Media performance remains in the ad platform. ATIYO preserves creative context and learnings; it does not connect to ad accounts, buy media, calculate ROAS or know campaign performance unless a user records it.

Do not ask which result you prefer. Ask which explanation survives a documented, controlled retest.