feat: support for exception context propagation

[guibou]

This change is in order to have exception context propagation.

Everywhere we "re"-throw exception with throw, we instead use rethrow which does not remove the exception context. I've introduced rethrowIO' which is just a backward compatibility wrapper over rethrowIO and throwIO.

I will use this commit at work for a bit of time in order to gather some feedback and maybe comeback with a more robust solution.

Example of usage / changes:

The following code:

{-# LANGUAGE DeriveAnyClass #-}
import Control.Concurrent.Async
import Control.Exception
import Control.Exception.Context
import Control.Exception.Annotation
import Data.Typeable
import Data.Traversable
import GHC.Stack

data Ann = Ann String
  deriving (Show, ExceptionAnnotation)

asyncTask :: HasCallStack => IO ()
asyncTask = annotateIO (Ann "bonjour") $ do
  error "yoto"

asyncTask' :: HasCallStack => IO ()
asyncTask' = annotateIO (Ann "bonjour2") $ do
  error "yutu"

main = do
  -- withAsync asyncTask wait
  concurrently asyncTask asyncTask'
  -- race asyncTask asyncTask'

When run without this commit leads to:

ASyncException.hs: Uncaught exception ghc-internal:GHC.Internal.Exception.ErrorCall:

yoto

HasCallStack backtrace:
  throwIO, called at ./Control/Concurrent/Async/Internal.hs:630:24 in async-2.2.5-50rpfAJ7BEc1o5OswtTMUN:Control.Concurrent.Async.Internal

When run with this commit:

*** Exception: yoto

Ann "bonjour"
HasCallStack backtrace:
  error, called at /home/guillaume//ASyncException.hs:15:3 in async-2.2.5-inplace:Main
  asyncTask, called at /home/guillaume//ASyncException.hs:23:16 in async-2.2.5-inplace:Main

[mrkline]

Is there anything else that needs to be done while this is WIP? It would be excellent for exception annotations to propagate through.

[guibou]

@mrkline

Sorry, the only reason is that I had not took time to work on that.

I'm using this MR since month at work and did not had any surprising behavior.

Could be turned to "ready", if it builds with older GHC (I think so), I don't see any reason why not processing any further. Even if I missed some proper rethrow, it can be added in a future work.

[mrkline]

No worries, and thanks for the work!

I tried backporting this to base-4.20 so that it could also work with LTS-24. (mrkline@bb9d703) I hoped it would be simple since

rethrowIO e = throwIO (NoBacktrace e)

But sadly GHC 9.10 seems noisier - your example code gives

async-test: yoto
CallStack (from HasCallStack):
  error, called at Main.hs:15:3 in tacview-0.4.2.0-inplace-async-test:Main
  asyncTask, called at Main.hs:23:16 in tacview-0.4.2.0-inplace-async-test:Main
Ann "bonjour"
HasCallStack backtrace:
  collectBacktraces, called at libraries/ghc-internal/src/GHC/Internal/Exception.hs:169:13 in ghc-internal:GHC.Internal.Exception
  toExceptionWithBacktrace, called at libraries/ghc-internal/src/GHC/Internal/Exception.hs:204:5 in ghc-internal:GHC.Internal.Exception
  error, called at Main.hs:15:3 in tacview-0.4.2.0-inplace-async-test:Main
  asyncTask, called at Main.hs:23:16 in tacview-0.4.2.0-inplace-async-test:Main


HasCallStack backtrace:
  collectBacktraces, called at libraries/ghc-internal/src/GHC/Internal/Exception.hs:169:13 in ghc-internal:GHC.Internal.Exception
  toExceptionWithBacktrace, called at libraries/ghc-internal/src/GHC/Internal/IO.hs:260:11 in ghc-internal:GHC.Internal.IO
  throwIO, called at libraries/ghc-internal/src/GHC/Internal/Control/Exception/Base.hs:195:43 in ghc-internal:GHC.Internal.Control.Exception.Base

If you spot any obvious problem, let me know.

[guibou]

No worries, and thanks for the work!

I tried backporting this to base-4.20 so that it could also work with LTS-24. (mrkline@bb9d703) I hoped it would be simple since

rethrowIO e = throwIO (NoBacktrace e)

My understanding of NoBacktrace is that it is useful at catch site, to remove the backtrace, but not at throw site.

As always with exception, I need to experiment to really understand what is going on (I have an infinite respect for people who can reason about exception in their head without experimenting ;)

If you spot any obvious problem, let me know.

I need to get back into this MR actually.

It seems that your branch mrkline@bb9d703 is based on one commit that is not what's inside this MR. Your commit is bb9d703, which is really alike the commit on my local repo, 5cbf92365b4501d9c102b810e8f6159b778ae615, they all are on branch exception-context, however this MR is the branch novainsilico:no_discard_exception_context.

I'm lost in the history there, sorry, it may be my fault (not worked on this MR since a long time and I'm used to try a lot of thing locally, change branch, force push, ...). Do you remember how you ended with commit bb9d703 and branch exception-context?

[guibou]

@mrkline

Using GHC 9.12:

Indeed, throwIO with NoBacktrace does not include backtrace.

λ gecko ~ → cat Ex.hs 
import Control.Exception

main = do
  throwIO (NoBacktrace (ErrorCall "test"))
  -- throwIO (ErrorCall "test")
λ gecko ~ → runhaskell Ex.hs 
Ex.hs: Uncaught exception ghc-internal:GHC.Internal.Exception.ErrorCall:

test

With no NoBacktrace, there is a backtrace:

λ gecko ~ → cat Ex.hs       
import Control.Exception

main = do
  -- throwIO (NoBacktrace (ErrorCall "test"))
  throwIO (ErrorCall "test")
λ gecko ~ → runhaskell Ex.hs
Ex.hs: Uncaught exception ghc-internal:GHC.Internal.Exception.ErrorCall:

test

HasCallStack backtrace:
  throwIO, called at Ex.hs:5:3 in main:Main

(So I'm wrong, throwing with NoBacktrace does indeed not include the backtrace).

However, throwing an exception caught with ExceptionWithContext leads to a duplication of the backtrace, if NoBacktrace is not there:

λ gecko ~ → cat Ex.hs       
import Control.Exception

main = do
  -- throwIO (NoBacktrace (ErrorCall "test"))
  Left e <- try @(ExceptionWithContext SomeException) (throwIO (ErrorCall "test"))
  throwIO (NoBacktrace e)

λ gecko ~ → runhaskell Ex.hs
Ex.hs: Uncaught exception ghc-internal:GHC.Internal.Exception.ErrorCall:

test

HasCallStack backtrace:
  throwIO, called at Ex.hs:5:56 in main:Main

λ gecko ~ → cat Ex.hs       
import Control.Exception

main = do
  -- throwIO (NoBacktrace (ErrorCall "test"))
  Left e <- try @(ExceptionWithContext SomeException) (throwIO (ErrorCall "test"))
  throwIO e

λ gecko ~ → runhaskell Ex.hs
Ex.hs: Uncaught exception ghc-internal:GHC.Internal.Exception.ErrorCall:

test

HasCallStack backtrace:
  throwIO, called at Ex.hs:6:3 in main:Main

HasCallStack backtrace:
  throwIO, called at Ex.hs:5:56 in main:Main

So as a first approximation, it seems that your implementation should be correct.

I'll continue the experimentation, maybe that's different with ghc 9.10.

[guibou]

With GHC 9.10.2:

λ gecko ~ → cat Ex.hs
import Control.Exception
import GHC.Internal.Exception.Type

main = do
  -- throwIO (NoBacktrace (ErrorCall "test"))
  Left e <- try @(ExceptionWithContext SomeException) (throwIO (ErrorCall "test"))
  throwIO (NoBacktrace e)

λ gecko ~ → runhaskell Ex.hs
Ex.hs: test
HasCallStack backtrace:
  collectBacktraces, called at libraries/ghc-internal/src/GHC/Internal/Exception.hs:169:13 in ghc-internal:GHC.Internal.Exception
  toExceptionWithBacktrace, called at libraries/ghc-internal/src/GHC/Internal/IO.hs:260:11 in ghc-internal:GHC.Internal.IO
  throwIO, called at libraries/exceptions/src/Control/Monad/Catch.hs:308:12 in exceptions-0.10.9-f51c:Control.Monad.Catch
  throwM, called at libraries/exceptions/src/Control/Monad/Catch.hs:427:21 in exceptions-0.10.9-f51c:Control.Monad.Catch
  throwM, called at compiler/GHC/Driver/Monad.hs:167:54 in ghc-9.10.2-ef99:GHC.Driver.Monad
  a type signature in an instance, called at compiler/GHC/Driver/Monad.hs:167:54 in ghc-9.10.2-ef99:GHC.Driver.Monad
  throwM, called at libraries/exceptions/src/Control/Monad/Catch.hs:427:21 in exceptions-0.10.9-f51c:Control.Monad.Catch
  throwM, called at ghc/GHCi/UI/Monad.hs:288:15 in ghc-bin-9.10.2-4bb7:GHCi.UI.Monad
  a type signature in an instance, called at ghc/GHCi/UI/Monad.hs:288:15 in ghc-bin-9.10.2-4bb7:GHCi.UI.Monad
  throwM, called at libraries/exceptions/src/Control/Monad/Catch.hs:427:21 in exceptions-0.10.9-f51c:Control.Monad.Catch
  throwM, called at libraries/exceptions/src/Control/Monad/Catch.hs:427:21 in exceptions-0.10.9-f51c:Control.Monad.Catch
  throwM, called at libraries/exceptions/src/Control/Monad/Catch.hs:427:21 in exceptions-0.10.9-f51c:Control.Monad.Catch
  throwM, called at libraries/exceptions/src/Control/Monad/Catch.hs:427:21 in exceptions-0.10.9-f51c:Control.Monad.Catch
  throwM, called at libraries/exceptions/src/Control/Monad/Catch.hs:427:21 in exceptions-0.10.9-f51c:Control.Monad.Catch
  throwM, called at libraries/haskeline/System/Console/Haskeline/InputT.hs:53:39 in haskeline-0.8.2.1-0f10:System.Console.Haskeline.InputT
  a type signature in an instance, called at libraries/haskeline/System/Console/Haskeline/InputT.hs:53:39 in haskeline-0.8.2.1-0f10:System.Console.Haskeline.InputT
  throwM, called at ghc/GHCi/UI/Monad.hs:215:52 in ghc-bin-9.10.2-4bb7:GHCi.UI.Monad


λ gecko ~ → cat Ex.hs       
import Control.Exception
import GHC.Internal.Exception.Type

main = do
  -- throwIO (NoBacktrace (ErrorCall "test"))
  Left e <- try @(ExceptionWithContext SomeException) (throwIO (ErrorCall "test"))
  throwIO e

λ gecko ~ → runhaskell Ex.hs
Ex.hs: test
HasCallStack backtrace:
  collectBacktraces, called at libraries/ghc-internal/src/GHC/Internal/Exception.hs:169:13 in ghc-internal:GHC.Internal.Exception
  toExceptionWithBacktrace, called at libraries/ghc-internal/src/GHC/Internal/IO.hs:260:11 in ghc-internal:GHC.Internal.IO
  throwIO, called at libraries/exceptions/src/Control/Monad/Catch.hs:308:12 in exceptions-0.10.9-f51c:Control.Monad.Catch
  throwM, called at libraries/exceptions/src/Control/Monad/Catch.hs:427:21 in exceptions-0.10.9-f51c:Control.Monad.Catch
  throwM, called at compiler/GHC/Driver/Monad.hs:167:54 in ghc-9.10.2-ef99:GHC.Driver.Monad
  a type signature in an instance, called at compiler/GHC/Driver/Monad.hs:167:54 in ghc-9.10.2-ef99:GHC.Driver.Monad
  throwM, called at libraries/exceptions/src/Control/Monad/Catch.hs:427:21 in exceptions-0.10.9-f51c:Control.Monad.Catch
  throwM, called at ghc/GHCi/UI/Monad.hs:288:15 in ghc-bin-9.10.2-4bb7:GHCi.UI.Monad
  a type signature in an instance, called at ghc/GHCi/UI/Monad.hs:288:15 in ghc-bin-9.10.2-4bb7:GHCi.UI.Monad
  throwM, called at libraries/exceptions/src/Control/Monad/Catch.hs:427:21 in exceptions-0.10.9-f51c:Control.Monad.Catch
  throwM, called at libraries/exceptions/src/Control/Monad/Catch.hs:427:21 in exceptions-0.10.9-f51c:Control.Monad.Catch
  throwM, called at libraries/exceptions/src/Control/Monad/Catch.hs:427:21 in exceptions-0.10.9-f51c:Control.Monad.Catch
  throwM, called at libraries/exceptions/src/Control/Monad/Catch.hs:427:21 in exceptions-0.10.9-f51c:Control.Monad.Catch
  throwM, called at libraries/exceptions/src/Control/Monad/Catch.hs:427:21 in exceptions-0.10.9-f51c:Control.Monad.Catch
  throwM, called at libraries/haskeline/System/Console/Haskeline/InputT.hs:53:39 in haskeline-0.8.2.1-0f10:System.Console.Haskeline.InputT
  a type signature in an instance, called at libraries/haskeline/System/Console/Haskeline/InputT.hs:53:39 in haskeline-0.8.2.1-0f10:System.Console.Haskeline.InputT
  throwM, called at ghc/GHCi/UI/Monad.hs:215:52 in ghc-bin-9.10.2-4bb7:GHCi.UI.Monad


λ gecko ~ → 

There is no difference between NoBacktrace and no NoBacktrace.

This is surprising.

[guibou]

Haa, I think I got it. The problem is that in newest base, ErrorCall does not duplicate the callstack.

From base 4.12 changelog: https://hackage.haskell.org/package/base-4.21.0.0/changelog

Get rid of the HasCallStack mechanism manually propagated by ErrorCall in favour of the more general HasCallStack exception backtrace mechanism, to remove duplicate call stacks for uncaught exceptions.

So the reason for the duplicate call stack is that it is part of the context AND part of ErrorCall.

To be honest, I think we can keep that. This is painful, indeed, but that's part of a work in progress in the exception handling logic. Adding more complexity would, IMHO, just lead to problem in the future.

[mpickering]

@guibou It would be great to merge this, what's the latest? @simonmar would you have time soon to look at this PR and to potentially merge it?

[mpickering]

Also bear in mind that throwSTM doesn't attach a backtrace currently - https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13408#note_652255

[simonmar]

Seems reasonable. Could we have a test please?

[guibou]

Seems reasonable. Could we have a test please?

I'm doing that right now.

[guibou]

I do have a question for you, regarding the test.

The idea of test is to throw an exception in an async context (e.g. concurently thisThrow thisDoesNotThrow) and see if the backtrace pass through the concurrently call AND continue inside thisThrow or if it stops at the (re)throw inside concurrently.

Exceptions and backtrace are an opaque feature, we can get the backtrace, display it (see https://hackage-content.haskell.org/package/base-4.22.0.0/docs/Control-Exception-Backtrace.html#t:Backtraces), but nothing more without using a GHC.Internal module. And unfortunately, the displayed value contains volatile items (such as the line numbers).

My idea was to compare the exception caught from concurently to an exception caught directly by calling thisThrow. However part of the callstack will be the same, but the rest won't.

Is it OK if I just pick one line of the backtrace and compare (I've tested, it works), and we'll fix later if it break with a future GHC version, or should I go a bit deeper inside the GHC.Internal relevant module in order to extract more precise informations, at the risk of more breakage with next GHC version?

I'll continue with the first approach (e.g. working on the displayBacktraces bt :: String value and manual extraction) and submit an extended MR, but in the meantime if you think that the second solution is better, let me know.

[guibou]

Please take a look at the newly added tests.

Before merging, give me a bit of time. I would like to see if we can add context in the "non-local" operations so the rethrow location is not wasted. e.g. when running withAsync action $ \async -> .... wait async it would be great to have the callstack inside action as well as the callstack toward wait)

[mpickering]

Thanks @guibou, looks good to me. In theory you should be able to use ghc-experimental to inspect the Backtraces object but the patch for this didn't make it into 9.14.

[guibou]

I'm a bit annoyed.

Take this code sample:

data Ann = Ann String
  deriving (Show, ExceptionAnnotation)

data Exc' = Exc' String
  deriving (Show, Exception)

asyncTask :: HasCallStack => IO ()
asyncTask = annotateIO (Ann "bonjour") $ do
  throwIO (Exc' "yoto")

asyncTask' :: HasCallStack => IO ()
asyncTask' = annotateIO (Ann "bonjour2") $ do
  throwIO (Exc' "yutu")

t = do
  withAsync asyncTask $ \t ->
    wait t
  -- concurrently asyncTask asyncTask'

The current output is:

*** Exception: Exc' "yoto"

While handling Exc' "yoto"

Ann "bonjour"
HasCallStack backtrace:
  throwIO, called at test/test-async.hs:39:3 in async-2.2.5-inplace-test-async:Main
  asyncTask, called at test/test-async.hs:46:13 in async-2.2.5-inplace-test-async:Main

Here I'm annoyed by:

• The presence of While handling, which is added by the wait call, but that's still unclear why to me. edit: actually, there is another catch inside the withAsync block which recatch the exception thrown by wait and wraps with While handling. That's definitely a behavior which should be improved.
• The CallStack does NOT show anything related to the wait location, it only shows the callsite for asyncTask.

I think the correct solution would be to add some sort of While waiting at exception annotation on wait, adding another HasCallStack backtrace for the wait location ?

[guibou]

In the meantime, I've rebased the branch AND cleaned build with ghc 9.8 -> 9.14.

[guibou]

@simonmar I've fixed the While Handling annotation which was appearing with withAsync blocks.

Regarding the fact that the "non-local" primitive do not appear in the callstack (e.g. wait, waitBoth, waitAny). I've also added an exception annotation which shows the location of the wait call.

For example, using the previous example, an exception will now be displayed with a callstack locating the exception original throw, all of the annotations, AND a new annotation showing the location of the wait call:

*** Exception: Exc' "yoto"

AsyncWaitLocation CallStack (from HasCallStack):
  wait, called at test/test-async.hs:49:5 in async-2.2.6-inplace-test-async:Main
Ann "bonjour"
HasCallStack backtrace:
  throwIO, called at test/test-async.hs:41:3 in async-2.2.6-inplace-test-async:Main
  asyncTask, called at test/test-async.hs:48:13 in async-2.2.6-inplace-test-async:Main

I've tested that it builds and run tests with GHC 9.8 -> 9.14.

There are no tests for the new annotation for wait location feature. Give me a bit of time and I'll craft something.