Task Manager troubleshooting

Theory: Kibana is configured to poll for tasks at a reduced rate.

Diagnosis: Evaluating the health stats, you can see the following output under stats.configuration.value:

{
  "request_capacity": 1000,
  "max_poll_inactivity_cycles": 10,
  "monitored_aggregated_stats_refresh_rate": 60000,
  "monitored_stats_running_average_window": 50,
  "monitored_task_execution_thresholds": {
    "default": {
      "error_threshold": 90,
      "warn_threshold": 80
    },
    "custom": {}
  },
  "poll_interval": 3000, 
  "max_workers": 10 
}

You can infer from this output that the Kibana instance polls for work every 3 seconds and can run 10 concurrent tasks.

Now suppose the output under stats.configuration.value is the following:

{
  "request_capacity": 1000,
  "max_poll_inactivity_cycles": 10,
  "monitored_aggregated_stats_refresh_rate": 60000,
  "monitored_stats_running_average_window": 50,
  "monitored_task_execution_thresholds": {
    "default": {
      "error_threshold": 90,
      "warn_threshold": 80
    },
    "custom": {}
  },
  "poll_interval": 60000, 
  "max_workers": 1 
}

You can infer from this output that the Kibana instance only polls for work once a minute and only picks up one task at a time. This throughput is unlikely to support mission critical services, such as Alerting or Reporting, and tasks will usually run late.

There are two possible reasons for such a configuration:

Theory: Kibana is not polling as frequently as it should

Diagnosis: Evaluating the health stats, you see the following output under stats.runtime.value.polling:

{
  "last_successful_poll": "2021-02-16T11:38:09.934Z", 
  "last_polling_delay": "2021-02-14T11:29:05.053Z",
  "duration": { 
    "p50": 13,
    "p90": 128,
    "p95": 143,
    "p99": 168
  },
  "claim_conflicts": { 
    "p50": 0,
    "p90": 0,
    "p95": 0,
    "p99": 2
  },
  "claim_mismatches": {
    "p50": 0,
    "p90": 0,
    "p95": 0,
    "p99": 0
  },
  "result_frequency_percent_as_number": { 
    "Failed": 0,
    "NoAvailableWorkers": 0,
    "NoTasksClaimed": 80,
    "RanOutOfCapacity": 0,
    "RunningAtCapacity": 0,
    "PoolFilled": 20
  }
}

You can infer from this output that the Kibana instance is polling regularly. This assessment is based on the following:

All of these stats are tracked as a running average, which means that they give a snapshot of a period of time (by default Kibana tracks up to 50 cycles), rather than giving a complete history.

Suppose the output under stats.runtime.value.polling.result_frequency_percent_as_number was the following:

{
  "Failed": 30, 
  "NoAvailableWorkers": 20, 
  "NoTasksClaimed": 10,
  "RanOutOfCapacity": 10, 
  "RunningAtCapacity": 10, 
  "PoolFilled": 20
}

You can infer from this output that Task Manager is not healthy, as the failure rate is high, and Task Manager is fetching tasks it has no capacity to run. Analyzing the Kibana Server Log should reveal the underlying issue causing the high error rate and capacity issues.

The high NoAvailableWorkers rate of 20% suggests that there are many tasks running for durations longer than the poll_interval. For details on analyzing long task execution durations, see the long running tasks theory.

Theory: Kibana is polling as frequently as it should, but that isn’t often enough to keep up with the workload

Diagnosis: Evaluating the health stats, you can see the following output of drift and load under stats.runtime.value:

{
  "drift": { 
    "p50": 99,
    "p90": 1245,
    "p95": 1845,
    "p99": 2878
  },
  "load": { 
    "p50": 0,
    "p90": 0,
    "p95": 10,
    "p99": 20
  },
}

You can infer from these stats that this Kibana has plenty of capacity, and any delays you might be experiencing are unlikely to be addressed by expanding the throughput.

Suppose the output of drift and load was the following:

{
  "drift": { 
    "p50": 2999,
    "p90": 3845,
    "p95": 3845.75,
    "p99": 4078
  },
  "load": { 
    "p50": 80,
    "p90": 100,
    "p95": 100,
    "p99": 100
  }
}

You can infer from these stats that this Kibana is using most of its capacity, but seems to keep up with the work most of the time. This assessment is based on the following:

For details on achieving higher throughput by adjusting your scaling strategy, see Scaling guidance.

Theory: Tasks run for too long, overrunning their schedule

Diagnosis: The Insufficient throughtput to handle the scheduled workload theory analyzed a hypothetical scenario where both drift and load were unusually high.

Suppose an alternate scenario, where drift is high, but load is not, such as the following:

{
    "drift": { 
        "p50": 9799,
        "p90": 83845,
        "p95": 90328,
        "p99": 123845
    },
    "load": { 
        "p50": 40,
        "p90": 75,
        "p95": 80,
        "p99": 100
    }
}

In the preceding scenario, the tasks are running far too late, but you have sufficient capacity to run more concurrent tasks. A high capacity allows Kibana to run multiple different tasks concurrently. If a task is already running when its next schedule run is due, Kibana will avoid running it a second time, and instead wait for the first execution to complete.

If a task takes longer to execute than the cadence of its schedule, then that task will always overrun and experience a high drift. For example, suppose a task is scheduled to execute every 3 seconds, but takes 6 seconds to complete. It will consistently suffer from a drift of, at least, 3 seconds.

Evaluating the health stats in this hypothetical scenario, you see the following output under stats.runtime.value.execution.duration:

{
  "alerting:.index-threshold": { 
    "p50": 95,
    "p90": 1725,
    "p95": 2761,
    "p99": 2761
  },
  "alerting:.es-query": { 
    "p50": 7149,
    "p90": 40071,
    "p95": 45282,
    "p99": 121845
  },
  "actions:.index": {
    "p50": 166,
    "p90": 166,
    "p95": 166,
    "p99": 166
  }
}

You can infer from these stats that the high drift the Task Manager is experiencing is most likely due to Elasticsearch query alerts that are running for a long time.

Resolving this issue is context dependent and changes from case to case. In the preceding example above, this would be resolved by modifying the queries in these alerts to make them faster, or improving the Elasticsearch throughput to speed up the exiting query.

Theory: Tasks take multiple attempts to succeed

Diagnosis: A high error rate could cause a task to appear to run late, when in fact it runs on time, but experiences a high failure rate.

Evaluating the preceding health stats, you see the following output under stats.runtime.value.execution.result_frequency_percent_as_number:

{
  "alerting:.index-threshold": { 
    "Success": 100,
    "RetryScheduled": 0,
    "Failed": 0,
    "status": "OK"
  },
  "alerting:xpack.uptime.alerts.monitorStatus": {
    "Success": 100,
    "RetryScheduled": 0,
    "Failed": 0,
    "status": "OK"
  },
  "actions:.index": { 
    "Success": 8,
    "RetryScheduled": 0,
    "Failed": 92,
    "status": "error" 
  }
}

You can infer from these stats that most actions:.index tasks, which back the ES Index Kibana action, fail. Resolving that would require deeper investigation into the Kibana Server Log, where the exact errors are logged, and addressing these specific errors.

Predicting the required throughput a deplyment might need to support Task Manager is difficult, as features can schedule an unpredictable number of tasks at a variety of scheduled cadences.

Health monitoring provides statistics that make it easier to monitor the adequacy of the existing throughput. By evaluating the workload, the required throughput can be estimated, which is used when following the Task Manager Scaling guidance.

Evaluating the preceding health stats above, you see the following output under stats.workload.value:

{
  "count": 26, 
  "task_types": {
    "alerting:.index-threshold": {
      "count": 2, 
      "status": {
        "idle": 2
      }
    },
    "actions:.index": {
      "count": 14,
      "status": {
        "idle": 2,
        "running": 2,
        "failed": 10 
      }
    },
    "alerting:xpack.uptime.alerts.monitorStatus": {
      "count": 10,
      "status": {
        "idle": 10
      }
    },
  },
  "schedule": [ 
    ["10s", 2],
    ["1m", 2],
    ["90s", 2],
    ["5m", 8]
  ],
  "overdue": 0, 
  "estimated_schedule_density": [  
    0, 1, 0, 0, 0, 1, 0, 1, 0, 1,
    0, 0, 0, 1, 0, 0, 1, 1, 1, 0,
    0, 3, 0, 0, 0, 1, 0, 1, 0, 1,
    0, 0, 0, 1, 0, 0, 1, 1, 1, 0
  ]
}

The workload section summarizes the work load across the cluster, listing the tasks in the system, their types, schedules, and current status.

You can infer from these stats that a default deployment should suffice. This assessment is based on the following:

Suppose the output of stats.workload.value looked something like this:

{
  "count": 2191, 
  "task_types": {
    "alerting:.index-threshold": {
      "count": 202,
      "status": {
        "idle": 183,
        "claiming": 2,
        "running": 19
      }
    },
    "alerting:.es-query": {
      "count": 225,
      "status": {
        "idle": 225,
      }
    },
    "actions:.index": {
      "count": 89,
      "status": {
        "idle": 24,
        "running": 2,
        "failed": 63
      }
    },
    "alerting:xpack.uptime.alerts.monitorStatus": {
      "count": 87,
      "status": {
        "idle": 74,
        "running": 13
      }
    },
  },
  "schedule": [ 
    ["10s", 38],
    ["1m", 101],
    ["90s", 55],
    ["5m", 89],
    ["20m", 62],
    ["60m", 106],
    ["1d", 61]
  ],
  "overdue": 0, 
  "estimated_schedule_density": [  
    10, 1, 0, 10, 0, 20, 0, 1, 0, 1,
    9, 0, 3, 10, 0, 0, 10, 10, 7, 0,
    0, 31, 0, 12, 16, 31, 0, 10, 0, 10,
    3, 22, 0, 10, 0, 2, 10, 10, 1, 0
  ]
}

You can infer several important attributes of your workload from this output:

These rough calculations give you a lower bound to the required throughput, which is at least 440 tasks per minute to ensure recurring tasks are executed, at their scheduled time. This throughput doesn’t account for nonrecurring tasks that might have been scheduled, nor does it account for tasks (recurring or otherwise) that might be scheduled in the future.

Given these inferred attributes, it would be safe to assume that a single Kibana instance with default settings would not provide the required throughput. It is possible that scaling horizontally by adding a couple more Kibana instances will.

For details on scaling Task Manager, see Scaling guidance.