Skip to content

stack-digest.md

Latest commit

 

History

History
210 lines (174 loc) · 14.6 KB

stack-digest.md

File metadata and controls

210 lines (174 loc) · 14.6 KB

Details about stack digest

This page gives details about the stack digest feature (goal and implementation).

Why generating stack digests?

A full Java stack trace is very helpful to analyse and debug a single error, but is not very handy to compare two errors.

The idea is to turn a full Java stack trace into a short and stable digest, that could help matching several distinct occurrences of the same type of error:

  • short for easing elasticsearch indexing, and take advantage of it (we use a MD5 hash),
  • stable is the tricky part, as the same type of error occurring twice may not generate exactly the same stack trace (see below).

This done, it becomes easy with elasticsearch or any other logs centralization and indexation system to:

  • count distinct type of errors that occur in your code over time,
  • count occurrences and frequency of a given type of error,
  • detect when a (new) type of error occurred for the first time (maybe linking this to a new version being deployed?).

The stack digest may also become a simple error id that you can link your bug tracker with...

Stack digest stability challenge by examples

Let's consider error stack 1

(the stack trace presented here has been cut by half from useless lines)

com.xyz.MyApp$MyClient$MyClientException: An error occurred while getting Alice's things(msg)
  at com.xyz.MyApp$MyClient.getTheThings(MyApp.java:26)
  at com.xyz.MyApp$MyService.displayThings(MyApp.java:16)
  at com.xyz.MyApp$MyService$$FastClassByCGLIB$$e7645040.invoke()(aop)
  at net.sf.cglib.proxy.MethodProxy.invoke()(aop)
  at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint()(fwk)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()(fwk)
  at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed()(fwk)
  at sun.reflect.NativeMethodAccessorImpl.invoke0()(aop)
  at sun.reflect.NativeMethodAccessorImpl.invoke()(aop)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke()(aop)
  at java.lang.reflect.Method.invoke()(aop)
  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs()(fwk)
  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod()(fwk)
  at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke()(fwk)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()(fwk)
  at org.springframework.aop.interceptor.AbstractTraceInterceptor.invoke()(fwk)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()(fwk)
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke()(fwk)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()(fwk)
  at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke()(fwk)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()(fwk)
  at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept()(fwk)
  at com.xyz.MyApp$MyService$$EnhancerBySpringCGLIB$$c673c675.displayThings(<generated>)(aop)
  at sun.reflect.GeneratedMethodAccessor647.invoke(Unknown Source)(aop)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)(aop)
  at java.lang.reflect.Method.invoke(Method.java:498)(aop)
  at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)(fwk)
  at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)(fwk)
  at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)(fwk)
  at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)(fwk)
  at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)(fwk)
  at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)(fwk)
  at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)(fwk)
  at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)(fwk)
  at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)(fwk)
  at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)(fwk)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)(jee)
  at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)(fwk)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)(jee)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)(jee)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)(jee)
  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)(jee)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)(jee)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)(jee)
  ...
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)(fwk)
  at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)(fwk)
  at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)(fwk)
  at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)(fwk)
  at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)(fwk)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)(jee)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)(jee)
  ...
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)(jee)
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)(jee)
  at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)(jee)
  at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)(jee)
  at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)(jee)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)(jee)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)(jee)
  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)(jee)
  at java.lang.Thread.run(Thread.java:748)(jee)
  ...
Caused by: com.xyz.MyApp$HttpStack$HttpError: I/O error on GET http://dummy/user/alice/things(msg)
  at com.xyz.MyApp$HttpStack.get(MyApp.java:40)
  at com.xyz.MyApp$MyClient.getTheThings(MyApp.java:24)
  ... 23 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out(msg)
  at com.xyz.MyApp$HttpStack.get(MyApp.java:38)
  ... 24 common frames omitted

Strike out elements may vary from one occurrence to the other:

  • error messages(msg) often contain stuff related to the very error occurrence context,
  • AOP generated classes(aop) may vary from one execution to another.

Italic elements are somewhat not stable, or at least useless (purely technical). Ex:

  • JEE container stuff(jee): may change when you upgrade your JEE container version or add/remove/reorganize your servlet filters chain for instance,
  • Spring Framework(fwk) underlying stacks (MVC, security) for pretty much the same reason,
  • AOP and dynamic invocation(aop): purely technical, and quite implementation-dependent.

Only bolded elements are supposed to be stable.

Now let's consider error stack 2

(shortened)

com.xyz.MyApp$MyClient$MyClientException: An error occurred while getting Bob's things(msg)
  at com.xyz.MyApp$MyClient.getTheThings(MyApp.java:26)
  at com.xyz.MyApp$MyService.displayThings(MyApp.java:16)
  at com.xyz.MyApp$MyService$$FastClassByCGLIB$$07e70d1e.invoke()(aop)
  at net.sf.cglib.proxy.MethodProxy.invoke()(aop)
  at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint()(fwk)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()(fwk)
  at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed()(fwk)
  at sun.reflect.NativeMethodAccessorImpl.invoke0()(aop)
  at sun.reflect.NativeMethodAccessorImpl.invoke()(aop)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke()(aop)
  at java.lang.reflect.Method.invoke()(aop)
  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs()(fwk)
  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod()(fwk)
  at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke()(fwk)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()(fwk)
  at org.springframework.aop.interceptor.AbstractTraceInterceptor.invoke()(fwk)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()(fwk)
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke()(fwk)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()(fwk)
  at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke()(fwk)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()(fwk)
  at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept()(fwk)
  at com.xyz.MyApp$MyService$$EnhancerBySpringCGLIB$$e3f570b1.displayThings(<generated>)(aop)
  at sun.reflect.GeneratedMethodAccessor737.invoke(Unknown Source)(aop)
  ...
Caused by: com.xyz.MyApp$HttpStack$HttpError: I/O error on GET http://dummy/user/bob/things(msg)
  at com.xyz.MyApp$HttpStack.get(MyApp.java:40)
  at com.xyz.MyApp$MyClient.getTheThings(MyApp.java:24)
  ... 23 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out(msg)
  at com.xyz.MyApp$HttpStack.get(MyApp.java:38)
  ... 24 common frames omitted

You may see in this example that most of the strike elements have slight differences from error stack 1 (messages and generated classes names).

Nevertheless it is the same exact error (despite the context is different as it applies to another user), and the goal here is to be able to count them as two occurrences of the same error.

Now let's consider error stack 3

(shortened)

com.xyz.MyApp$MyClient$MyClientException: An error occurred while getting Alice's things(msg)
  at com.xyz.MyApp$MyClient.getTheThings(MyApp.java:26)
  at com.xyz.MyApp$MyService.displayThings(MyApp.java:16)
  at com.xyz.MyApp$MyService$$FastClassByCGLIB$$e7645040.invoke()(aop)
  at net.sf.cglib.proxy.MethodProxy.invoke()(aop)
  at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint()(fwk)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()(fwk)
  at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed()(fwk)
  at sun.reflect.NativeMethodAccessorImpl.invoke0()(aop)
  at sun.reflect.NativeMethodAccessorImpl.invoke()(aop)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke()(aop)
  at java.lang.reflect.Method.invoke()(aop)
  ...
Caused by: com.xyz.MyApp$HttpStack$HttpError: I/O error on GET http://dummy/user/alice/things(msg)
  at com.xyz.MyApp$HttpStack.get(MyApp.java:40)
  at com.xyz.MyApp$MyClient.getTheThings(MyApp.java:24)
  ... 23 common frames omitted
Caused by: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown(msg)
  at com.sun.net.ssl.internal.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1172)
  ... 24 common frames omitted

Here, you can see that the first and second errors are the same as in error stack 1, but the root cause is different (SSLException instead of SocketTimeoutException).

So in that case we don't want the top error digest computed for error stack 3 to be the same as for error stack 1.

Stack digest computation rules

As a conclusion, stack digest computation applies the following rules:

  1. a stack digest shall not compute with the error message
  2. a stack digest shall compute with it's parent cause (recurses)
  3. in order to stabilize the stack digest (over time and space), it's recommended to exclude non-stable elements