Operate the Universal Profiling backend

This page outlines operating the backend when running Universal Profiling on a self-managed version of the Elastic Stack. Here you’ll find information on:

The resources needed to ingest and query Universal Profiling data vary based on the total number of CPU cores you’re profiling. The number of cores comes from the sum of all virtual cores as recorded in /proc/cpuinfo, adding up all the machines you’ll deploy the host-agent to.

Ingestion and query resource demand is almost directly proportional to the amount of data the host-agents generate. Calculate the data generated by the host-agents using the number of CPU samples collected, the number of executables processed, and the executables' debug metadata size. While the number of CPU samples collected is predictable, the number of executables processed and the executables' debug metadata size is not.

The following table provides recommended resources for ingesting and querying Universal Profiling data based on your number of CPU cores:

Because resource demand is nearly proportional to the amount of data the host-agents generate, you can calculate the necessary resources for use cases beyond those in the table by comparing your actual number of cores profiled with the number of cores in the table. When calculating, factor in the following:

Storage considerations: the Elasticsearch disks' bandwidth and latency will affect the latency of ingesting and querying the profiling data. Allocate data to hot nodes for best performance and user experience. If storage becomes a concern, tune the data retention by customizing the Universal Profiling index lifecycle management policy.

You can configure the collector and symbolizer using the YAML file and CLI flags, with the CLI flags taking precedence over the YAML file. The configuration files are created during the installation process, as seen in Create configuration files section. Comments in the configuration files explain the purpose of each configuration option.

Restart the backend binaries after modifying the configuration files for changes to take effect.

When building configuration options for each of the backend binaries, you can use CLI flags to override the values in the YAML configuration file. The overrides must contain the full path to the configuration option and must be in a key=value format. For example, -E application.field.key=value, where application is the name of the binary.

For example, to enable TLS in the HTTP server of the collector, you can pass the -E pf-elastic-collector.ssl.enabled=true flag. This will override the ssl.enabled option found in the YAML configuration file.

Monitor the collector and symbolizer through Logs and Metrics to ensure the services are running and healthy. Without both services running, profiling data will not be ingested and symbolized, and querying Kibana won’t return data.

The collector and symbolizer always log to standard output. You can turn on debug logs by setting the verbose configuration option to true in the YAML configuration file.

Avoid using debug logs in production, as they can be very verbose and impact backend performance. Only enable debug logs when troubleshooting a failed deployment or when instructed to do so by support.

Logs are formatted as "key=value" pairs, and Elasticsearch and Kibana can automatically parse them into fields.

A log collector, such as Filebeat, can collect and send logs to Elasticsearch for indexing and analysis. Depending on how it’s installed, a Filebeat input of type journald (for OS packages), log (for binaries), or container can be used to process the logs. Refer to the filebeat documentation for more information.

Metrics are not exposed by default. Enable metrics in the metrics section in the YAML configuration files. The collector and symbolizer can expose metrics in both JSON and Prometheus formats.

Metrics in JSON format can be exposed through an HTTP server or a Unix domain socket. Prometheus metrics can only be exposed through an HTTP server. Customize where the metrics are exposed using the metrics.prometheus_host and metrics.expvar_host configuration options.

You can use Metricbeat to scrape metrics. Consume the JSON directly through the http module. Consume the Prometheus endpoint using the prometheus module. When using an HTTP server for either format, the URI to scrape metrics from is /metrics.

For example, the following collector configuration would expose metrics in Prometheus format on port 9090 and in JSON format on port 9191. You can then scrape them by connecting to http://127.0.0.1:9090/metrics and http://127.0.0.1:9191/metrics respectively.

pf-elastic-collector:
  metrics:
    prometheus_host: ":9090"
    expvar_host: ":9191"

Optionally, you can also expose the expvar format over a Unix domain socket, by setting the expvar_socket configuration option to a valid path. For example, the following collector configuration would expose metrics in Prometheus format on port 9090 and in JSON format over a Unix domain socket at /tmp/collector.sock.

pf-elastic-collector:
  metrics:
    prometheus_host: ":9090"
    expvar_host: "/tmp/collector.sock"

The following sections show the most relevant metrics exposed by the backend binaries. Include these metrics in your monitoring dashboards to detect backend issues.

Common runtime metrics

Collector metrics

Symbolizer metrics

Health checks

The backend binaries expose two health check endpoints, /live and /ready, that you can use to monitor the health of the application. The endpoints return a 200 OK HTTP status code when the checks are successful.

The health check endpoints are hosted in the same HTTP server that accepts the incoming profiling data. This endpoint is configured through the application’s host configuration option.

For example, if the collector is configured with the default value host: 0.0.0.0:8260, you can check the health of the application by running curl -i localhost:8260/live and curl -i localhost:8260/ready.

In the resource guidance table, no options use more than one replica for the symbolizer. We do not recommend scaling the number of symbolizer replicas because of the technical limitations of the current implementation. We recommend scaling the symbolizer vertically, by increasing the memory and CPU cores it uses to process data.

You can increase the number of collector replicas at will, keeping their vertical sizing smaller, if this is more convenient for your deployment use case. The collector has a linear increase in memory usage and CPU threads with the number of host-agents that it serves. Keep in mind that since the host-agent/collector communication happens via gRPC, there may be long-lived TCP sessions that are bound to a single collector replica. When scaling out the number of replicas, depending on the load balancer that you have in place fronting the collector’s endpoint, you may want to shut down the older replicas after adding new replicas. This ensures that the load is evenly distributed across all replicas.

Upgrade the backend binaries whenever you upgrade the rest of the Elastic stack. While we try to keep backward compatibility between two consecutive minor version, we may introduce changes to the data format that require the applications to be at the same version of Elasticsearch and Kibana.

The upgrade process steps vary depending on the installation method used.

When using ECE, the upgrade process is managed by the platform itself. You don’t need to perform any action to upgrade the backend binaries.

Perform a helm upgrade using the helm upgrade command. You may reuse existing values or provide the full values YAML file on each upgrade.

Upgrade the package version using the OS package manager. Not all package managers will call into systemd to restart the service, so you may need to restart the service manually or through any other automation in place.

Download the corresponding binary version and replace the existing one, using the command seen in the Binary section of the setup guide. Replace the old binary and restart the services.

Pull the new container image, and replace the existing image with the new image.