Deadletter redrive fails on the -native graalvm docker image

[Spiral90210]

Running the -native redrive, and while it appears to succeed, it crashes immediately afterwards with the following logs:

12:56:41.212 [elasticmq-pekko.actor.default-dispatcher-11] INFO  o.a.pekko.actor.DeadLetterActorRef - Message [org.elasticmq.MessageData] to Actor[pekko://elasticmq/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
Uncaught error from thread [elasticmq-pekko.actor.default-dispatcher-13]: null, shutting down JVM since 'pekko.jvm-exit-on-fatal-error' is enabled for ActorSystem[elasticmq]
java.lang.ExceptionInInitializerError
	at org.elasticmq.rest.sqs.ListMessageMoveTasksDirectives.$anonfun$listMessageMoveTasks$1(ListMessageMoveTasksDirectives.scala:18)
	at org.apache.pekko.http.scaladsl.server.Directive$.$anonfun$addByNameNullaryApply$2(Directive.scala:184)
	at org.apache.pekko.http.scaladsl.server.Directive$.$anonfun$Empty$1(Directive.scala:170)
	at org.apache.pekko.http.scaladsl.server.Directive$$anon$1.tapply(Directive.scala:165)
	at org.apache.pekko.http.scaladsl.server.Directive$.$anonfun$addByNameNullaryApply$1(Directive.scala:184)
	at org.elasticmq.rest.sqs.ListMessageMoveTasksDirectives.listMessageMoveTasks(ListMessageMoveTasksDirectives.scala:17)
	at org.elasticmq.rest.sqs.ListMessageMoveTasksDirectives.listMessageMoveTasks$(ListMessageMoveTasksDirectives.scala:16)
	at org.elasticmq.rest.sqs.TheSQSRestServerBuilder$$anon$1.listMessageMoveTasks(SQSRestServerBuilder.scala:153)
	at org.elasticmq.rest.sqs.TheSQSRestServerBuilder.rawRoutes$1(SQSRestServerBuilder.scala:230)
	at org.elasticmq.rest.sqs.TheSQSRestServerBuilder.$anonfun$start$8(SQSRestServerBuilder.scala:255)
	at org.elasticmq.rest.sqs.directives.AnyParamDirectives.$anonfun$anyParamsMap$3(AnyParamDirectives.scala:76)
	at org.apache.pekko.http.scaladsl.server.util.ApplyConverterInstances$$anon$1.$anonfun$apply$1(ApplyConverterInstances.scala:23)
	at org.apache.pekko.http.scaladsl.server.ConjunctionMagnet$$anon$2.$anonfun$apply$3(Directive.scala:255)
	at org.apache.pekko.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:80)
	at org.apache.pekko.http.scaladsl.server.directives.FutureDirectives.$anonfun$onComplete$3(FutureDirectives.scala:47)
	at org.apache.pekko.http.scaladsl.util.FastFuture$.$anonfun$transformWith$1(FastFuture.scala:45)
	at org.apache.pekko.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:49)
	at org.apache.pekko.http.scaladsl.util.FastFuture$.$anonfun$transformWith$3(FastFuture.scala:59)
	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:517)
	at org.apache.pekko.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:72)
	at org.apache.pekko.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:109)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94)
	at org.apache.pekko.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:109)
	at org.apache.pekko.dispatch.TaskInvocation.run(AbstractDispatcher.scala:59)
	at org.apache.pekko.dispatch.ForkJoinExecutorConfigurator$PekkoForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:62)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
	at com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:704)
	at com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:202)
Caused by: scala.MatchError: [Ljava.lang.String;@37928abb (of class [Ljava.lang.String;)
	at spray.json.ProductFormatsInstances.jsonFormat2(ProductFormatsInstances.scala:46)
	at spray.json.ProductFormatsInstances.jsonFormat2$(ProductFormatsInstances.scala:45)
	at spray.json.DefaultJsonProtocol$.jsonFormat2(DefaultJsonProtocol.scala:30)
	at org.elasticmq.rest.sqs.ListMessageMoveTasksRequest$.<clinit>(ListMessageMoveTasksDirectives.scala:53)
	... 33 more
12:56:41.216 [elasticmq-pekko.actor.default-dispatcher-9] ERROR o.apache.pekko.actor.ActorSystemImpl - Uncaught error from thread [elasticmq-pekko.actor.default-dispatcher-13]: null, shutting down JVM since 'pekko.jvm-exit-on-fatal-error' is enabled for ActorSystem[elasticmq]
java.lang.ExceptionInInitializerError: null
	at org.elasticmq.rest.sqs.ListMessageMoveTasksDirectives.$anonfun$listMessageMoveTasks$1(ListMessageMoveTasksDirectives.scala:18)
	at org.apache.pekko.http.scaladsl.server.Directive$.$anonfun$addByNameNullaryApply$2(Directive.scala:184)
	at org.apache.pekko.http.scaladsl.server.Directive$.$anonfun$Empty$1(Directive.scala:170)
	at org.apache.pekko.http.scaladsl.server.Directive$$anon$1.tapply(Directive.scala:165)
	at org.apache.pekko.http.scaladsl.server.Directive$.$anonfun$addByNameNullaryApply$1(Directive.scala:184)
	at org.elasticmq.rest.sqs.ListMessageMoveTasksDirectives.listMessageMoveTasks(ListMessageMoveTasksDirectives.scala:17)
	at org.elasticmq.rest.sqs.ListMessageMoveTasksDirectives.listMessageMoveTasks$(ListMessageMoveTasksDirectives.scala:16)
	at org.elasticmq.rest.sqs.TheSQSRestServerBuilder$$anon$1.listMessageMoveTasks(SQSRestServerBuilder.scala:153)
	at org.elasticmq.rest.sqs.TheSQSRestServerBuilder.rawRoutes$1(SQSRestServerBuilder.scala:230)
	at org.elasticmq.rest.sqs.TheSQSRestServerBuilder.$anonfun$start$8(SQSRestServerBuilder.scala:255)

Claude is pretty certain this is a native graal vm problem - I've not worked with java in... 6(?) years now, so not much of an ability to dive straight into it anymore. I did have it produce this - hopefully it's useful to you. Apologies if the use of an LLM is inappropriate here - I didn't set it trying to find bugs, just as a diagnostic tool when I encountered this - here's its full output:

  **Symptom:** Any `StartMessageMoveTask` (DLQ redrive) request terminates the whole
  server. The move itself completes (messages are persisted to the destination queue),
  but the action handler throws during class initialization, and because
  `pekko.jvm-exit-on-fatal-error` is enabled the JVM exits — so the container dies and
  must be restarted. On the JVM image the same request works normally.

  **Stack trace (native, 1.7.1):**
  java.lang.ExceptionInInitializerError
    at org.elasticmq.rest.sqs.StartMessageMoveTaskDirectives.$anonfun$startMessageMoveTask$1(StartMessageMoveTaskDirectives.scala:21)
    ...
  Caused by: scala.MatchError: [Ljava.lang.String;@... (of class [Ljava.lang.String;)
    at spray.json.ProductFormatsInstances.jsonFormat3(ProductFormatsInstances.scala:68)
    at spray.json.DefaultJsonProtocol$.jsonFormat3(DefaultJsonProtocol.scala:30)
    at org.elasticmq.rest.sqs.StartMessageMoveTaskActionRequest$.<clinit>(StartMessageMoveTaskDirectives.scala:74)

  **Root cause:**
  `rest/rest-sqs/src/main/scala/org/elasticmq/rest/sqs/StartMessageMoveTaskDirectives.scala`
  derives the JSON format for the 3-field `StartMessageMoveTaskActionRequest` with the
  no-field-names `jsonFormat3(...)`. That overload calls spray-json's reflective
  `extractFieldNames`, then runs `val Array(p1, p2, p3) = <field names>`. In the GraalVM
  native image the case class is not registered for reflection, so `extractFieldNames`
  returns an `Array[String]` of the wrong length and the destructure throws
  `scala.MatchError: [Ljava.lang.String;`. This runs inside the companion object's
  `<clinit>`, so it surfaces as `ExceptionInInitializerError` the first time a
  `StartMessageMoveTask` request is handled.

  **Proposed fix (preferred, one line):** use the explicit-field-name overload, which
  does not reflect:
  ```scala
  implicit val requestJsonFormat: RootJsonFormat[StartMessageMoveTaskActionRequest] =
    jsonFormat(
      StartMessageMoveTaskActionRequest.apply,
      "SourceArn", "DestinationArn", "MaxNumberOfMessagesPerSecond"
    )

  Alternative fix: add StartMessageMoveTaskActionRequest (and any other case class
  whose format is derived via reflective jsonFormatN) to the native-image
  reflect-config.json / reachability metadata.

  Secondary hardening: a failing/unsupported action request should not take the whole
  broker down via pekko.jvm-exit-on-fatal-error; consider catching it in the route. But
  the format derivation above is the root cause.

  Note: the master source has this `jsonFormat3` line at ~56 and the case class at ~47-51; the `:74` in the trace is the line in the 1.7.1 build. Both point at the same
  `StartMessageMoveTaskActionRequest` companion. I couldn't fetch the exact 1.7.1-tagged file (gh isn't authenticated here and the raw URL for that tag 404'd), so verify the precise line numbers
  against the 1.7.1 tag before filing if you want them exact.

While I'm here, new UI is great, almost completely replaces some custom tooling I had written already (which stored recently sent messages in browser storage to be able to resend/template messages more quickly) - great job there!

[Spiral90210]

I've just tried switching to the jvm image. And while it doesn't crash, it doesn't seem to like the redrive either. Some sample logs:

15:25:26.643 [elasticmq-pekko.actor.default-dispatcher-6] INFO  o.a.pekko.actor.RepointableActorRef - Message [org.elasticmq.actor.queue.QueueEvent$Restore] to Actor[pekko://elasticmq/user/$a#1967717404] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/user/$a#1967717404] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
15:29:03.891 [elasticmq-pekko.actor.default-dispatcher-11] INFO  o.elasticmq.actor.QueueManagerActor - Creating queue CreateQueueData(admintest-orders-in-dev1-dlq,None,None,None,None,None,None,false,false,None,None,Map())
15:29:03.903 [elasticmq-pekko.actor.default-dispatcher-10] INFO  o.a.pekko.actor.RepointableActorRef - Message [org.elasticmq.actor.queue.QueueEvent$QueueCreated] to Actor[pekko://elasticmq/user/$a#1967717404] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/user/$a#1967717404] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
15:29:03.922 [elasticmq-pekko.actor.default-dispatcher-9] INFO  o.elasticmq.actor.QueueManagerActor - Creating queue CreateQueueData(admintest-orders-in-dev1,None,None,None,None,None,Some(DeadLettersQueueData(admintest-orders-in-dev1-dlq,1)),false,false,None,None,Map())
15:29:03.923 [elasticmq-pekko.actor.default-dispatcher-13] INFO  o.a.pekko.actor.RepointableActorRef - Message [org.elasticmq.actor.queue.QueueEvent$QueueCreated] to Actor[pekko://elasticmq/user/$a#1967717404] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/user/$a#1967717404] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
15:29:03.936 [elasticmq-pekko.actor.default-dispatcher-9] INFO  o.a.pekko.actor.RepointableActorRef - Message [org.elasticmq.actor.queue.QueueEvent$MessageAdded] to Actor[pekko://elasticmq/user/$a#1967717404] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/user/$a#1967717404] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
15:29:03.936 [elasticmq-pekko.actor.default-dispatcher-11] ERROR o.e.a.q.QueueActorStorage$ResultWithEvents - Failed to notify queue event listener. The state may be inconsistent.
org.apache.pekko.pattern.AskTimeoutException: Recipient [Actor[pekko://elasticmq/user/$a#1967717404]] had already been terminated. Message of type [org.elasticmq.actor.queue.QueueEvent$MessageAdded].
15:29:24.945 [elasticmq-pekko.actor.default-dispatcher-9] INFO  o.apache.pekko.actor.ActorSystemImpl - Request timeout encountered for request [POST / Strict(126 bytes)]
15:29:24.953 [elasticmq-pekko.actor.default-dispatcher-11] ERROR o.e.r.s.TheSQSRestServerBuilder$$anon$1 - Exception when running routes
org.apache.pekko.pattern.AskTimeoutException: Ask timed out on [Actor[pekko://elasticmq/user/$b/queue-admintest-orders-in-dev1-9a530cf4#-1319960575]] after [21000 ms]. Message of type [org.elasticmq.msg.SendMessage]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.
15:29:26.752 [elasticmq-pekko.actor.default-dispatcher-10] ERROR o.e.a.q.QueueActorStorage$ResultWithEvents - Failed to notify queue event listener. The state may be inconsistent.
org.apache.pekko.pattern.AskTimeoutException: Recipient [Actor[pekko://elasticmq/user/$a#1967717404]] had already been terminated. Message of type [org.elasticmq.actor.queue.QueueEvent$MessageAdded].
15:29:26.752 [elasticmq-pekko.actor.default-dispatcher-14] INFO  o.a.pekko.actor.RepointableActorRef - Message [org.elasticmq.actor.queue.QueueEvent$MessageAdded] to Actor[pekko://elasticmq/user/$a#1967717404] was not delivered. [5] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/user/$a#1967717404] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
15:29:33.823 [elasticmq-pekko.actor.default-dispatcher-14] INFO  o.elasticmq.actor.QueueManagerActor - Deleting queue admintest-orders-in-dev1
15:29:33.824 [elasticmq-pekko.actor.default-dispatcher-14] INFO  o.a.pekko.actor.RepointableActorRef - Message [org.elasticmq.actor.queue.QueueEvent$QueueDeleted] to Actor[pekko://elasticmq/user/$a#1967717404] was not delivered. [6] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/user/$a#1967717404] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
15:29:33.829 [elasticmq-pekko.actor.default-dispatcher-11] INFO  o.elasticmq.actor.QueueManagerActor - Deleting queue admintest-orders-in-dev1-dlq
15:29:33.830 [elasticmq-pekko.actor.default-dispatcher-11] INFO  o.a.pekko.actor.RepointableActorRef - Message [org.elasticmq.actor.queue.QueueEvent$QueueDeleted] to Actor[pekko://elasticmq/user/$a#1967717404] was not delivered. [7] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/user/$a#1967717404] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
15:29:47.763 [elasticmq-pekko.actor.default-dispatcher-11] INFO  o.apache.pekko.actor.ActorSystemImpl - Request timeout encountered for request [POST / Strict(126 bytes)]
15:29:47.772 [elasticmq-pekko.actor.default-dispatcher-11] ERROR o.e.r.s.TheSQSRestServerBuilder$$anon$1 - Exception when running routes
org.apache.pekko.pattern.AskTimeoutException: Ask timed out on [Actor[pekko://elasticmq/user/$b/queue-admintest-orders-in-dev1-9a530cf4#-1319960575]] after [21000 ms]. Message of type [org.elasticmq.msg.SendMessage]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.
15:31:38.781 [elasticmq-pekko.actor.default-dispatcher-9] INFO  o.elasticmq.actor.QueueManagerActor - Creating queue CreateQueueData(probe-send,None,None,None,None,None,None,false,false,None,None,Map())
15:31:38.783 [elasticmq-pekko.actor.default-dispatcher-11] INFO  o.a.pekko.actor.RepointableActorRef - Message [org.elasticmq.actor.queue.QueueEvent$QueueCreated] to Actor[pekko://elasticmq/user/$a#1967717404] was not delivered. [8] dead letters encountered. If this is not an expected behavior then Actor[pekko://elasticmq/user/$a#1967717404] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
15:31:38.817 [elasticmq-pekko.actor.default-dispatcher-11] ERROR o.e.a.q.QueueActorStorage$ResultWithEvents - Failed to notify queue event listener. The state may be inconsistent.
org.apache.pekko.pattern.AskTimeoutException: Recipient [Actor[pekko://elasticmq/user/$a#1967717404]] had already been terminated. Message of type [org.elasticmq.actor.queue.QueueEvent$MessageAdded].

It's not a happy bunny.

[Spiral90210]

Looking into it, there's some earlier storage problems - I think I have a different problem with the jvm image that I will continue to look into on my own. The -native image though, that's a definite problem.

[Spiral90210]

Ok - the jvm version was a permissions error on a mountable data volume mount (running as root cleared it). I confirmed that the DLQ redrive does work as expected on the jvm build, so this is an issue restricted graalvm.