How many times will Hypothesis run my test?

This is a trickier question than you might expect. The short answer is “exactly max_examples times”, with the following exceptions:

Read on for details.

Search space exhaustion

If Hypothesis detects that there are no more test cases left to try, it may stop generating test cases before it hits max_examples. For example:

from hypothesis import given, strategies as st

calls = 0

@given(st.integers(0, 19))
def test_function(n):
    global calls
    calls += 1

test_function()
assert calls == 20

This runs test_function 20 times, not 100, since there are only 20 unique integers to try.

The search space tracking in Hypothesis is good, but not perfect. We treat this more as a bonus than something to strive for.

assume() and .filter()

If a test case fails to satisfy an assume() or .filter() condition, Hypothesis will retry generating that test case and will not count it towards the max_examples limit. For instance:

from hypothesis import assume, given, strategies as st

@given(st.integers())
def test_function(n):
    assume(n % 2 == 0)

will run roughly 200 times, since half of the test cases are discarded from the assume().

Note that while failing an assume() triggers an immediate retry of the entire test case, Hypothesis will try several times in the same test case to satisfy a .filter() condition. This makes expressing the same condition using .filter() more efficient than assume().

Also note that even if your code does not explicitly use assume() or .filter(), a builtin strategy may still use them and cause retries. We try to directly satisfy conditions where possible instead of relying on rejection sampling, so this should be relatively uncommon.

Test cases which are too large

For performance reasons, Hypothesis places an internal limit on the number of choices required to generate any single test case. If a test case exceeds this limit, we will retry generating it and will not count it towards the max_examples limit. (And if we see too many of these large test cases, we will raise HealthCheck.data_too_large, unless suppressed with settings.suppress_health_check).

The specific value of this size limit is an undocumented implementation detail. The majority of Hypothesis tests do not come close to hitting it.

Failing test cases

If Hypothesis finds a failing test case, it stops generation early, and may call the test function additional times during the Phase.shrink and Phase.explain phases. Sometimes, Hypothesis determines that the initial failing test case was already as simple as possible, in which case Phase.shrink will not result in additional test executions (but Phase.explain might).

Regardless of whether Hypothesis runs the test during the shrinking and explain phases, it will always run the minimal failing test case one additional time to check for flakiness. For instance, the following trivial test runs with n=0 twice, even though it only uses the Phase.generate phase:

from hypothesis import Phase, given, settings, strategies as st

@given(st.integers())
@settings(phases=[Phase.generate])
def test_function(n):
    print(f"called with {n}")
    assert n != 0

test_function()

The first execution finds the initial failure with n=0, and the second execution replays n=0 to ensure the failure is not flaky.