Fleet and Elastic Agent 8.16.0

edit

Review important information about the Fleet and Elastic Agent 8.16.0 release.

Security updates

edit
Fleet Server
  • Update Fleet Server Go version to 1.23.1. #3924

Breaking changes

edit

Breaking changes can prevent your application from optimal operation and performance. Before you upgrade, review the breaking changes, then mitigate the impact to your application.

Elastic Agent
  • When using the System integration, uppercase characters in the host.hostname are being converted to lowercase in Elastic Agent output. This can possibly result in duplicated host entries appearing in Kibana. #3993

Known issues

edit
Fleet UI listing shows "No agent found"

Details

In the Fleet UI in Kibana, the listing Elastic Agents might show "No agent found" with a toast message "Error fetching agents" or "Agent policy …​ not found".

This error can happen if the Elastic Agents being searched and listed in the UI are using an Elastic Agent policy which doesn’t exist.

Impact

As a workaround for the issue, you can upgrade your Elastic Stack to verion 8.16.1. The issue has been resolved by Kibana #199325.

Elastic Agent throws exception when uninstalling on Windows

Details

Fleet-managed Elastic Agent sometimes throws an exception when uninstalling on Microsoft Windows systems.

For example:

C:\>"C:\Program Files\Elastic\Agent\elastic-agent.exe" uninstall
Elastic Agent will be uninstalled from your system at C:\Program Files\Elastic\Agent. Do you want to continue? [Y/n]:y
[====] Attempting to notify Fleet of uninstall  [37s] unexpected fault address 0x18000473ef1
fatal error: fault
[signal 0xc0000005 code=0x1 addr=0x18000473ef1 pc=0x9f3004]

goroutine 1 gp=0xc00007c000 m=5 mp=0xc000116008 [running]:
runtime.throw({0x207a4ba?, 0xa2d986?})
        runtime/panic.go:1023 +0x65 fp=0xc000067588 sp=0xc000067558 pc=0xcf8c5
runtime.sigpanic()
        runtime/signal_windows.go:414 +0xd0 fp=0xc0000675d0 sp=0xc000067588 pc=0xe6a10
(...)
        github.com/elastic/elastic-agent/internal/pkg/agent/errors/generators.go:23
github.com/elastic/elastic-agent/internal/pkg/fleetapi.(*AuditUnenrollCmd).Execute(0xc00073f998, {0x4, 0x23cf148}, 0x0)
        github.com/elastic/elastic-agent/internal/pkg/fleetapi/audit_unenroll_cmd.go:74 +0x324 fp=0xc000067738 sp=0xc0000675d0 pc=0x9f3004
runtime: g 1: unexpected return pc for github.com/elastic/elastic-agent/internal/pkg/fleetapi.(*AuditUnenrollCmd).Execute called from 0xc0006817a0
stack: frame={sp:0xc0000675d0, fp:0xc000067738} stack=[0xc000064000,0xc000068000)
0x000000c0000674d0:  0x000000c000067508  0x00000000000d14af <runtime.gwrite+0x00000000000000ef>
0x000000c0000674e0:  0x00000000023c9c90  0x0000000000000001
0x000000c0000674f0:  0x0000000000000001  0x000000c00006756b
(...)

For other examples, refer to Elastic Agent issue #5952.

This problem occurs when Elastic Agent notifies Fleet to audit the uninstall process.

Impact

As a workaround, we recommend trying again to uninstall the agent.

An Elastic Agent with the Defend integration may report an Orphaned status and will not be able to be issued an upgrade action through Fleet.

Details
A known issue in the Elastic Agent may prevent it from being targetted with an upgrade action for a future release. This may occur if the Defend integration is used and the agent is stopped on a running instance for too long. An agent may be stopped as part of an upgrade process.

Impact
A bug fix is present in the 8.16.3 and 8.17.1 releases of Fleet that will prevent this from occuring.

If you have agents that are affected, the workaround is as follows:

# Get a Token to issue an update_by_query request:
curl -XPOST --user elastic:${SUPERUSER_PASS} -H 'x-elastic-product-origin:fleet' -H'content-type:application/json' "https://${ELASTICSEARCH_HOST}/_security/service/elastic/fleet-server/credential/token/fix-unenrolled"

# Issue an update_by_query request that targets effected agents:
curl -XPOST -H 'Authorization: Bearer ${TOKEN}' -H 'x-elastic-product-origin:fleet' -H 'content-type:application/json' "https://${ELASTICSEARCH_HOST}/.fleet-agents/_update_by_query" -d '{"query": {"bool": {"must": [{ "exists": { "field": "unenrolled_at" } }],"must_not": [{ "term": { "active": "false" } }]}},"script": {"source": "ctx._source.unenrolled_at = null;","lang": "painless"}}'

New features

edit

The 8.16.0 release Added the following new and notable features.

Fleet
  • Add support for content-only packages in integrations UI. #195831
  • Add advanced agent monitoring options for HTTP endpoint and diagnostics. #193361
  • Add support for periodic unenrollment of inactive agents. Once an Elastic Agent transitions to an inactive state and after a configurable timeout has expired, the agent will be unenrolled. #189861
  • Add support for dynamic topics to the Kafka output. This allows the Kafka output to write to a topic which is dynamically set in an event field. #192720
  • Add support for GeoIP processor databases in Ingest Pipelines. #190830
  • Add support for reusable/shareable integration policies. This feature allows you to create integrations policies that can be shared with multiple Elastic Agent policies, thereby reducing the number of integrations policies that you need to actively manage. #187153
  • Add support for integration-level outputs. This feature enables you to send integration data to a specific output, overwriting the output defined in the Elastic Agent Policy. #189125
Fleet Server
  • Add /api/fleet/agents/:id/audit/unenroll API that an Elastic Agent or Endpoint process may use to report that an agent was uninstalled or unenrolled to Fleet. #3818 #484
  • Add a secret_paths attribute to the policy data sent to agents. This attribute is a list of keys that Fleet Server has replaced with a reference to a secret value. #3908 #3657
Elastic Agent
  • Uninstalling a Fleet-managed Elastic Agent instance will now do a best-effort attempt to notify Fleet Server of the agent removal so the agent status appears correctly in the Fleet UI (related to #3818 above). #5302 #484
  • Introduce a Helm Chart for deploying Elastic Agent in Kubernetes. #5331 #3847
  • Remove support for the experimental shippers feature. #5308 #4547
  • Add the GCP Asset Inventory input to Cloudbeat. #5422
  • Add support for passphrase protected mTLS client certificate key during install/enroll. #5494 #5489
  • Elastic Defend now accepts a passphrase protected client certificate key for mTLS. #5542 #5490
  • Add a Kustomize template to enable hints-based autodiscovery by default when deploying standalone Elastic Agent in a Kubernetes cluster. This also removes root privileges from the init container. #5643

Enhancements

edit
Fleet
  • Update maximum supported package version. #196551
  • Add additional columns to Elastic Agent Logs UI. #192262
  • Show +build versions for Elastic Agent upgrades. #192171
  • Add format parameter to agent_policies APIs. #191811
  • Add toggles for agent.monitoring.http.enabled and agent.monitoring.http.buffer.enabled to agent policy advanced settings. #190984
  • Support integration policies without agent policy references (aka orphaned integration policies). #190649
  • Allow traces to be added to the monitoring_enabled array in Agent policies. #189908
  • Add setup technology selector to the Add Integration page. #189612
Fleet Server
  • Alter the checkin API to remove attributes set by the audit or unenroll API (follow-up to #3818 above). #3827 #484
  • Enable warnings for configuration options that have been deprecated throughout the 8.x lifecycle. #3901
Elastic Agent
  • Re-enable support for Elastic Defend on Windows Server 2012 and 2012 R2. #5429
  • Include the correct Elastic License 2.0 file in build artifacts and packages. #5464
  • Add the pprofextension to the Elastic Agent OTel collector. #5556
  • Update the base container image from Ubuntu 20.04 to Ubuntu 24.04. #5644 #5501
  • Redact values from the elastic-agent inspect command output for any keys in the secret_paths array. #5621
  • Redact secret paths in files written in Elastic Agent diagnostics bundles. #5745
  • Update the versions of OpenTelemetry Collector components from v0.111.0/v1.17.0 to v0.112.0/v1.18.0. #5838

Bug fixes

edit
Fleet
  • Revert "Fix client-side validation for agent policy timeout fields". #194338
  • Add proxy arguments to install snippets. #193922
  • Rollover if dimension mappings changed in dynamic templates. #192098
Fleet Server
  • Fix the error handling when Fleet Server attempts to authenticate with Elasticsearch. #3935 #3929
  • Fix an issue that caused Fleet Server to report a 500 error on Elastic Agent check-in because the agent has upgrade details but the referenced action ID is not found. #3991
Elastic Agent
  • Fix Elastic Agent crashing when self unenrolling due to too many authentication failures against Fleet Server. #5438 #5434
  • Change the deprecated maintainer label in Dockerfile to use the org.opencontainers.image.authors label instead. #5527