Better handling of tracebacks by DRMacIver · Pull Request #4696 · HypothesisWorks/hypothesis

DRMacIver · 2026-04-06T11:27:09Z

We were previously lying about what files things came from, in a way that ended up producing wrong tracebacks. This removes that, and then does a bunch of other traceback mangling to handle cases where this turned out to be load-bearing. Mostly it increases the number of places we strip Hypothesis frames, and uses pytest traceback hiding in more places.

This PR is on top of #4695 because the snapshot tests were very useful in tracking down output problems and specifying the right behaviour.

I am underconfident in this change because this logic is all extremely fragile (it's changing stupid hacks that date back to 2015), but the snapshot tests make me think it's at least mostly working, and the fact that the rest of the build passes is encouraging.

Fixes #4681

Snapshot tests capturing full pytest output for scenarios from issue #4681 where hypothesis's function impersonation (co_filename/co_firstlineno replacement in reflection.py) affects tracebacks. Uses pytester to run real pytest sessions and normalizes unstable elements (timing, seeds) while preserving test file line references as source content annotations. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Previously, get_trimmed_traceback only removed hypothesis frames from the beginning of the traceback chain. Now it also removes frames with __hypothesistracebackhide__ (or from hypothesis source files) from the middle of the traceback, so impersonated wrapper frames don't leak through as phantom intermediate frames. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add snapshot tests for the two examples from PR #1582 (the commit that introduced traceback trimming): a simple assertion failure and a multiple-failures case with different exception types per branch. Also normalize absolute paths in output for stability. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…racebackhide__ Remove the code object replacement in impersonate() that caused incorrect source lines in tracebacks (issue #4681). Instead, add __tracebackhide__ to the COPY_SIGNATURE_SCRIPT wrapper so pytest hides this pure-forwarding frame entirely. This fixes the phantom traceback frames showing wrong source lines for normal test failures. Two edge cases remain where all frames are hidden: FailedHealthCheck (no user code ran) and ExceptionGroup/multiple failures (the outer traceback has no user frames after hiding). These need the __tracebackhide__ in wrapped_test to be rethought separately. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The raise inside wrapped_test was hidden by __tracebackhide__ = True. Previously the co_filename impersonation made pytest treat this frame as user code, bypassing __tracebackhide__. With impersonation removed, the raise needs to be in a separate function without __tracebackhide__ so pytest shows it. Also remove the fragile line-counting healthcheck traceback test from test_capture.py — this is now covered by the snapshot test, and update test_traceback_elision.py frame count for the extra frame. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Update test_traceback_elision to expect 4 frames (not 5) since _reraise_trimmed_error is now only used for exception groups. Fix RET504 lint violation in test_impersonation_investigation.py. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Without co_filename impersonation, inspect.getsource() can't find the source of the wrapper function (co_filename is '<string>'). Setting __wrapped__ = target lets inspect.unwrap() follow to the original function, fixing the patching system which uses inspect.getsource(). Also fix traceback elision test frame count (4 not 5) and lint error. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ed__ Setting __wrapped__ on impersonated functions broke inspect.signature, annotations, and other inspect-based introspection throughout the codebase. Instead, use the existing __wrapped_target attribute (already set by impersonate) in the patching code to find the original function for inspect.getsource. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…line-number-impersonation

Liam-DeVoe

I'm broadly on board with the approach here 👍

Diff of before/after:

from hypothesis import given, strategies as st

@given(st.none())
def t(_) -> None:
    1/0

t()

~/Desktop/Liam/coding/hypothesis λ /opt/homebrew/bin/python3.12 /Users/tybug/Desktop/Liam/coding/hypothesis/sandbox.py
   Traceback (most recent call last):
     File "/Users/tybug/Desktop/Liam/coding/hypothesis/sandbox.py", line 7, in <module>                                                       
       t()                                                                                                                                    
  -  File "/Users/tybug/Desktop/Liam/coding/hypothesis/sandbox.py", line 4, in t                                                              
  -    def t(_) -> None:                                                                                                                      
  -                   ^^^                                                                                                                   
  -  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 2246, in wrapped_test                  
  +  File "<string>", line 6, in t                                                                                                            
  +  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 2267, in wrapped_test                
       raise the_error_hypothesis_found                                                                                                       
     File "/Users/tybug/Desktop/Liam/coding/hypothesis/sandbox.py", line 5, in t                                                              
       1/0                                                                                                                                    
       ~^~                                                                                                                                    
   ZeroDivisionError: division by zero                                                                                                        
   Falsifying example: t(                                                                                                                   
       _=None,                                                                                                                                
   )

There's a new File "<string>", line 6, in t line with no corresponding indented source code line. This feels like a traceback elision gone wrong?

Liam-DeVoe · 2026-04-13T01:49:06Z

                            report(msg)
-                        # The dance here is to avoid showing users long tracebacks
-                        # full of Hypothesis internals they don't care about.
-                        # We have to do this inline, to avoid adding another
-                        # internal stack frame just when we've removed the rest.
-                        #
-                        # Using a variable for our trimmed error ensures that the line
-                        # which will actually appear in tracebacks is as clear as
-                        # possible - "raise the_error_hypothesis_found".
+                        # Trim the traceback to remove hypothesis internals
                        the_error_hypothesis_found = e.with_traceback(


I disagree with removing this useful comment (did claude do this?)

Liam-DeVoe · 2026-04-13T01:56:52Z

+
+"""Snapshot tests investigating impersonation traceback issues.
+
+These tests capture the full pytest output for various scenarios where
+hypothesis's function impersonation (co_filename/co_firstlineno replacement)
+affects tracebacks. See https://github.com/HypothesisWorks/hypothesis/issues/4681
+"""
+


can we de-sloppify this file, by removing the overly-specific references to PRs and issues? (fine to leave an appropriate reference to the issue ofc, but we shouldn't have pr1582 in a test name)

Zac-HD · 2026-04-13T06:22:56Z

+                        if isinstance(e, BaseExceptionGroup):
+                            # Insert a frame here as otherwise all base frames are
+                            # trimmed which causes pytest problems
+                            _reraise_exception_group(the_error_hypothesis_found)


I'd rather fix this in Pytest than Hypothesis, optionally keeping a version-gated workaround here.

Zac-HD · 2026-04-13T06:24:36Z

+    # Strip any remaining hypothesis frames from the middle of the traceback
+    prev = tb
+    current = tb.tb_next
+    while current is not None:
+        if current.tb_next is not None and _is_hypothesis_frame(current.tb_frame):
+            prev.tb_next = current.tb_next
+        else:
+            prev = current
+        current = prev.tb_next


Unlike stripping leading frames, this modifies the traceback object in place and will thus affect anyone else who has a copy for whatever reason. Avoiding that is necessarily more expensive though, so maybe we just leave a comment?

Zac-HD · 2026-04-13T06:26:57Z

(man, I dislike the verbose pytest traceback format)

Can we also include some tests with --tb=native?

I also prefer --tb=native. Can we in fact make all but one of these tests use --tb=native, and only keep one for semantic coverage?

DRMacIver · 2026-04-16T10:47:50Z

There's a new File "<string>", line 6, in t line with no corresponding indented source code line. This feels like a traceback elision gone wrong?

Hmm. I thought I'd gotten rid of those. Will investigate.

DRMacIver and others added 13 commits April 6, 2026 11:00

Move file to right directory

8ef357d

Better handling of stack trace elision

3711e6e

Add release file

376ef5e

Remove spurious comment

30422cb

Clean up some more comments

f090b6d

DRMacIver marked this pull request as ready for review April 8, 2026 09:05

DRMacIver requested a review from Liam-DeVoe April 8, 2026 09:05

DRMacIver changed the title ~~Tentative fixes for #4681~~ Better handling of tracebacks Apr 8, 2026

Merge remote-tracking branch 'upstream/master' into DRMacIver/remove-…

d5a8f04

…line-number-impersonation

Liam-DeVoe reviewed Apr 13, 2026

View reviewed changes

remove duplicate test dir

acc81da

Zac-HD reviewed Apr 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Better handling of tracebacks#4696

Better handling of tracebacks#4696
DRMacIver wants to merge 15 commits intomasterfrom
DRMacIver/remove-line-number-impersonation

DRMacIver commented Apr 6, 2026 •

edited

Loading

Uh oh!

Liam-DeVoe left a comment •

edited

Loading

Uh oh!

Liam-DeVoe Apr 13, 2026

Uh oh!

Uh oh!

Liam-DeVoe Apr 13, 2026

Uh oh!

Zac-HD Apr 13, 2026

Uh oh!

Zac-HD Apr 13, 2026

Uh oh!

Zac-HD Apr 13, 2026

Uh oh!

Liam-DeVoe Apr 13, 2026

Uh oh!

DRMacIver commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

DRMacIver commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Liam-DeVoe left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Liam-DeVoe Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Liam-DeVoe Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Zac-HD Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Zac-HD Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Zac-HD Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Liam-DeVoe Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

DRMacIver commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

DRMacIver commented Apr 6, 2026 •

edited

Loading

Liam-DeVoe left a comment •

edited

Loading