Hot Threads API

edit

This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.

The hot threads API gets the current hot threads for Logstash. A hot thread is a Java thread that has high CPU usage and executes for a longer than normal period of time.

GET /_node/hot_threads

The output is a JSON document that contains a breakdown of the top hot threads for Logstash.

Example response:

{
    "time": "2017-01-12T12:09:45-08:00",
    "busiest_threads": 3,
    "threads": [
      {
        "name": "LogStash::Runner",
        "percent_of_cpu_time": 1.07,
        "state": "timed_waiting",
        "traces": [
          "java.lang.Object.wait(Native Method)",
          "java.lang.Thread.join(Thread.java:1253)",
          "org.jruby.internal.runtime.NativeThread.join(NativeThread.java:75)",
          "org.jruby.RubyThread.join(RubyThread.java:697)",
          "org.jruby.RubyThread$INVOKER$i$0$1$join.call(RubyThread$INVOKER$i$0$1$join.gen)",
          "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:663)",
          "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:198)",
          "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)",
          "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)",
          "org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)"
        ]
      },
      {
        "name": "[main]>worker7",
        "percent_of_cpu_time": 0.71,
        "state": "waiting",
        "traces": [
          "sun.misc.Unsafe.park(Native Method)",
          "java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)",
          "java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)",
          "java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:897)",
          "java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1222)",
          "java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)",
          "org.jruby.RubyThread.lockInterruptibly(RubyThread.java:1470)",
          "org.jruby.ext.thread.Mutex.lock(Mutex.java:91)",
          "org.jruby.ext.thread.Mutex.synchronize(Mutex.java:147)",
          "org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)"
        ]
      },
      {
        "name": "[main]>worker3",
        "percent_of_cpu_time": 0.71,
        "state": "waiting",
        "traces": [
          "sun.misc.Unsafe.park(Native Method)",
          "java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)",
          "java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)",
          "java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:897)",
          "java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1222)",
          "java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)",
          "org.jruby.RubyThread.lockInterruptibly(RubyThread.java:1470)",
          "org.jruby.ext.thread.Mutex.lock(Mutex.java:91)",
          "org.jruby.ext.thread.Mutex.synchronize(Mutex.java:147)",
          "org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)"
        ]
      }
    ]
  }
}

The parameters allowed are:

threads

The number of hot threads to return. The default is 3.

human

If true, returns plain text instead of JSON format. The default is false.

ignore_idle_threads

If true, does not return idle threads. The default is true.

See Common Options for a list of options that can be applied to all Logstash monitoring APIs.

You can use the ?human parameter to return the document in a human-readable format.

GET /_node/hot_threads?human=true

Example of a human-readable response:

 ::: {}
 Hot threads at 2017-01-12T12:10:15-08:00, busiestThreads=3:
 ================================================================================
 1.02 % of cpu usage, state: timed_waiting, thread name: 'LogStash::Runner'
	java.lang.Object.wait(Native Method)
	java.lang.Thread.join(Thread.java:1253)
	org.jruby.internal.runtime.NativeThread.join(NativeThread.java:75)
	org.jruby.RubyThread.join(RubyThread.java:697)
	org.jruby.RubyThread$INVOKER$i$0$1$join.call(RubyThread$INVOKER$i$0$1$join.gen)
	org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:663)
	org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:198)
	org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
	org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
	org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
 --------------------------------------------------------------------------------
 0.71 % of cpu usage, state: waiting, thread name: '[main]>worker7'
	sun.misc.Unsafe.park(Native Method)
	java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:897)
	java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1222)
	java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
	org.jruby.RubyThread.lockInterruptibly(RubyThread.java:1470)
	org.jruby.ext.thread.Mutex.lock(Mutex.java:91)
	org.jruby.ext.thread.Mutex.synchronize(Mutex.java:147)
	org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)
 --------------------------------------------------------------------------------
 0.71 % of cpu usage, state: timed_waiting, thread name: '[main]>worker3'
	sun.misc.Unsafe.park(Native Method)
	java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
	java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
	java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
	sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:497)
	org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:466)
	org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:324)