Replaying failed tests
======================

Replaying failures found by your Hypothesis tests is almost as important as finding failures in the first place. Hypothesis therefore contains several ways to replay failures: they are automatically saved to (and replayed from) a local |ExampleDatabase|, and can be manually replayed via |@example| or |@reproduce_failure|.

The Hypothesis database
-----------------------

When a test fails, Hypothesis automatically saves the failure so it can be replayed later. For instance, the first time you run the following code, it will take up to a few seconds to fail:

.. code-block:: python

    import time

    from hypothesis import given, strategies as st

    @given(st.integers())
    def f(n):
        assert n < 50
        time.sleep(0.1)

    f()

But the next time you run this code, it will fail instantly. When Hypothesis saw the failure the first time, it automatically saved that failing input. On future runs, Hypothesis retries any failing inputs (in |Phase.explain|) before generating new inputs (in |Phase.generate|)

Hypothesis saves failures to the |settings.database| setting. By default, this is a |DirectoryBasedExampleDatabase| in the local ``.hypothesis`` directory.

Disabling the database
~~~~~~~~~~~~~~~~~~~~~~

You can disable the database by passing ``database=None`` to |@settings|:

.. code-block:: python

    @settings(database=None)
    def f(n): ...

Using |@example| to run a specific input
----------------------------------------

If you want Hypothesis to always run a specific input, you can use |@example|. |@example| adds an explicit input which Hypothesis will run every time, in addition to the randomly generated |test cases|. You can think of |@example| as combining unit-testing with property-based testing.

For instance, suppose we write a test using |st.integers|, but want to make sure we try a few special prime numbers every time we run the test. We can add these inputs with |@example|:

.. code-block:: python

    # two mersenne primes
    @example(2**17 - 1)
    @example(2**19 - 1)
    @given(st.integers())
    def test_integers(n):
        pass

    test_integers()

Hypothesis runs all |explicit examples| first, in the |Phase.explicit| phase, before generating additional random test cases in the |Phase.generate| phase.

Inputs from |@example| do not shrink
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Note that unlike examples generated by Hypothesis, examples provided using |@example| do not shrink. We can see this by adding a failing assertion:

.. code-block:: python

    @example(2**17 - 1)
    @given(st.integers())
    def test_something_with_integers(n):
        assert n < 100

Hypothesis will print ``Failing explicit example: test_something_with_integers(n=131071)``, instead of shrinking to ``n=100``.

Prefer |@example| over the database for correctness
---------------------------------------------------

.. TODO_DOCS: link to /explanation/database-keys

While the database is useful for quick local iteration, Hypothesis may invalidate it when upgrading (because e.g. the internal format may have changed). Changes to the source code of a test function may also change its database key, invalidating its stored entries. We therefore recommend against relying on the database for the correctness of your tests. If you want to ensure an input is run every time, use |@example|.

Replaying test cases with |@reproduce_failure|
----------------------------------------------

If |settings.print_blob| is set to ``True`` (the default in the ``ci`` settings profile), and a test fails, Hypothesis will print an |@reproduce_failure| decorator containing an opaque blob as part of the error message:

.. code-block:: pycon

    >>> from hypothesis import settings, given
    >>> import hypothesis.strategies as st
    >>> @given(st.floats())
    ... @settings(print_blob=True)
    ... def test(f):
    ...     assert f == f
    ...
    >>> test()

    ...
    Failing test case: test(
        f=nan,
    )
    You can reproduce this test case by temporarily adding @reproduce_failure('6.131.23', b'ACh/+AAAAAAAAA==') as a decorator on your test function


You can add this decorator to your test to reproduce the failure. This can be useful for locally replaying failures found by CI. Note that the binary blob is not stable across Hypothesis versions, so you should not leave this decorator on your tests permanently. Use |@example| with an explicit input instead.

Sharing failures with the database
----------------------------------

If you work with multiple developers, or want to share failures across environments (such as locally replaying a failure found by CI), another option is to share the Hypothesis database.

For instance, by setting |settings.database| to an instance of a networked database like |RedisExampleDatabase|, any developer connecting to that networked database will automatically replay any failures found by other developers.

Similarly, setting |settings.database| to |GitHubArtifactDatabase| will automatically replay any failures found by the connected CI artifact.