Nginx

Collect logs and metrics from Nginx HTTP servers with Elastic Agent.

What is an Elastic integration?
Version
1.23.0 (View all)
Compatible Kibana version(s)
8.13.0 or higher
Supported Serverless project types

What's this?
Security
Observability
Subscription level
What's this?
Basic
Level of support
What's this?
Elastic

The Nginx integration allows you to monitor Nginx servers. Time series index mode enabled for metrics data stream.

Use the Nginx integration to collect metrics and logs from your server. Then visualize that data in Kibana, use the Machine Learning app to find unusual activity in HTTP access logs, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue.

For example, if you wanted to be notified if a certain number of client requests failed in a given time period, you could install the Nginx integration to send logs to Elastic. Then, you could view the logs stream into Elastic in real time in the Observability Logs app. You could also set up a new log threshold rule in the Logs app to alert you when there are more than a certain number of events with a failing status in a given time period.

Data streams

The Nginx integration collects two types of data: logs and metrics.

Logs help you keep a record of events that happen in your Nginx servers. This includes when a client request or error occurs.

Metrics give you insight into the state of your Nginx servers. This includes information like the total number of active client connections by status, the total number of client requests, and more.

Requirements

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.

Each data source was tested with a specific Nginx version. For more information see the Logs reference and Metrics reference.

Note: On Windows, the module was tested with Nginx installed from the Chocolatey repository.

Setup

For step-by-step instructions on how to set up an integration, see the Getting started guide.

Logs reference

Timezone support

This data source parses logs that don’t contain timezone information. For these logs, the Elastic Agent reads the local timezone and uses it when parsing to convert the timestamp to UTC. The timezone to be used for parsing is included in the event in the event.timezone field.

To disable this conversion, the event.timezone field can be removed using the drop_fields processor.

If logs originate from systems or applications with a timezone that is different than the local one, the event.timezone field can be overwritten with the original timezone using the add_fields processor.

Access Logs

Access logs collects the Nginx access logs.

Tested versions

The Nginx access logs stream was tested with Nginx 1.19.5.

An example event for access looks as following:

{
    "@timestamp": "2022-12-09T10:39:23.000Z",
    "_tmp": {},
    "agent": {
        "ephemeral_id": "34369a4a-4f24-4a39-9758-85fc2429d7e2",
        "id": "ef5e274d-4b53-45e6-943a-a5bcf1a6f523",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.5.0"
    },
    "data_stream": {
        "dataset": "nginx.access",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "ef5e274d-4b53-45e6-943a-a5bcf1a6f523",
        "snapshot": false,
        "version": "8.5.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "web"
        ],
        "created": "2022-12-09T10:39:38.896Z",
        "dataset": "nginx.access",
        "ingested": "2022-12-09T10:39:40Z",
        "kind": "event",
        "outcome": "success",
        "timezone": "+00:00",
        "type": [
            "access"
        ]
    },
    "host": {
        "architecture": "x86_64",
        "containerized": false,
        "hostname": "docker-fleet-agent",
        "id": "66392b0697b84641af8006d87aeb89f1",
        "ip": [
            "172.18.0.7"
        ],
        "mac": [
            "02-42-AC-12-00-07"
        ],
        "name": "docker-fleet-agent",
        "os": {
            "codename": "focal",
            "family": "debian",
            "kernel": "5.15.49-linuxkit",
            "name": "Ubuntu",
            "platform": "ubuntu",
            "type": "linux",
            "version": "20.04.5 LTS (Focal Fossa)"
        }
    },
    "http": {
        "request": {
            "method": "GET"
        },
        "response": {
            "body": {
                "bytes": 97
            },
            "status_code": 200
        },
        "version": "1.1"
    },
    "input": {
        "type": "log"
    },
    "log": {
        "file": {
            "path": "/tmp/service_logs/access.log"
        },
        "offset": 0
    },
    "nginx": {
        "access": {
            "remote_ip_list": [
                "127.0.0.1"
            ]
        }
    },
    "related": {
        "ip": [
            "127.0.0.1"
        ]
    },
    "source": {
        "address": "127.0.0.1",
        "ip": "127.0.0.1"
    },
    "tags": [
        "nginx-access"
    ],
    "url": {
        "original": "/server-status",
        "path": "/server-status"
    },
    "user_agent": {
        "device": {
            "name": "Other"
        },
        "name": "curl",
        "original": "curl/7.64.0",
        "version": "7.64.0"
    }
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields

FieldDescriptionType
@timestamp
Event timestamp.
date
cloud.image.id
Image ID for the cloud instance.
keyword
data_stream.dataset
Data stream dataset.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.dataset
Event dataset
constant_keyword
event.module
Event module
constant_keyword
host.containerized
If the host is a container.
boolean
host.os.build
OS build information.
keyword
host.os.codename
OS codename, if any.
keyword
input.type
Input type
keyword
log.offset
Log offset
long
nginx.access.remote_ip_list
An array of remote IP addresses. It is a list because it is common to include, besides the client IP address, IP addresses from headers like X-Forwarded-For. Real source IP is restored to source.ip.
keyword

Error Logs

Error logs collects the Nginx error logs.

Tested versions

The Nginx error logs stream was tested with Nginx 1.19.5.

An example event for error looks as following:

{
    "@timestamp": "2022-12-09T10:40:03.000Z",
    "agent": {
        "ephemeral_id": "34369a4a-4f24-4a39-9758-85fc2429d7e2",
        "id": "ef5e274d-4b53-45e6-943a-a5bcf1a6f523",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.5.0"
    },
    "data_stream": {
        "dataset": "nginx.error",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "ef5e274d-4b53-45e6-943a-a5bcf1a6f523",
        "snapshot": false,
        "version": "8.5.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "web"
        ],
        "created": "2022-12-09T10:40:18.973Z",
        "dataset": "nginx.error",
        "ingested": "2022-12-09T10:40:24Z",
        "kind": "event",
        "timezone": "+00:00",
        "type": [
            "error"
        ]
    },
    "host": {
        "architecture": "x86_64",
        "containerized": false,
        "hostname": "docker-fleet-agent",
        "id": "66392b0697b84641af8006d87aeb89f1",
        "ip": [
            "172.18.0.7"
        ],
        "mac": [
            "02-42-AC-12-00-07"
        ],
        "name": "docker-fleet-agent",
        "os": {
            "codename": "focal",
            "family": "debian",
            "kernel": "5.15.49-linuxkit",
            "name": "Ubuntu",
            "platform": "ubuntu",
            "type": "linux",
            "version": "20.04.5 LTS (Focal Fossa)"
        }
    },
    "input": {
        "type": "log"
    },
    "log": {
        "file": {
            "path": "/tmp/service_logs/error.log"
        },
        "level": "warn",
        "offset": 0
    },
    "message": "conflicting server name \"localhost\" on 0.0.0.0:80, ignored",
    "nginx": {
        "error": {}
    },
    "process": {
        "pid": 1,
        "thread": {
            "id": 1
        }
    },
    "tags": [
        "nginx-error"
    ]
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields

FieldDescriptionType
@timestamp
Event timestamp.
date
cloud.image.id
Image ID for the cloud instance.
keyword
data_stream.dataset
Data stream dataset.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.dataset
Event dataset
constant_keyword
event.module
Event module
constant_keyword
host.containerized
If the host is a container.
boolean
host.os.build
OS build information.
keyword
host.os.codename
OS codename, if any.
keyword
input.type
Input type
keyword
log.offset
Log offset
long
nginx.error.connection_id
Connection identifier.
long

Metrics reference

Stub Status Metrics

The Nginx stubstatus stream collects data from the Nginx ngx_http_stub_status module. It scrapes the server status data from the web page generated by ngx_http_stub_status. Please verify that your Nginx distribution comes with the mentioned module and it's enabled in the Nginx configuration file:

location /nginx_status {
    stub_status;
    allow 127.0.0.1; # only allow requests from localhost
    deny all;        # deny all other hosts
}

Replace 127.0.0.1 with your server’s IP address and make sure that this page accessible to only you.

Tested versions

The Nginx stubstatus stream was tested with Nginx 1.19.5 and is expected to work with all versions >= 1.19.

An example event for stubstatus looks as following:

{
    "@timestamp": "2024-02-08T08:12:10.668Z",
    "agent": {
        "ephemeral_id": "c3d516ba-b659-4190-a29d-d28200d74d48",
        "id": "2ea50bee-9250-43d1-8d70-949f242aa275",
        "name": "docker-fleet-agent",
        "type": "metricbeat",
        "version": "8.10.1"
    },
    "data_stream": {
        "dataset": "nginx.stubstatus",
        "namespace": "ep",
        "type": "metrics"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "2ea50bee-9250-43d1-8d70-949f242aa275",
        "snapshot": false,
        "version": "8.10.1"
    },
    "event": {
        "agent_id_status": "verified",
        "dataset": "nginx.stubstatus",
        "duration": 1156104,
        "ingested": "2024-02-08T08:12:13Z",
        "module": "nginx"
    },
    "host": {
        "architecture": "x86_64",
        "containerized": false,
        "hostname": "docker-fleet-agent",
        "id": "28da52b32df94b50aff67dfb8f1be3d6",
        "ip": [
            "172.24.0.7"
        ],
        "mac": [
            "02-42-AC-18-00-07"
        ],
        "name": "docker-fleet-agent",
        "os": {
            "codename": "focal",
            "family": "debian",
            "kernel": "5.15.0-89-generic",
            "name": "Ubuntu",
            "platform": "ubuntu",
            "type": "linux",
            "version": "20.04.6 LTS (Focal Fossa)"
        }
    },
    "metricset": {
        "name": "stubstatus",
        "period": 10000
    },
    "nginx": {
        "stubstatus": {
            "accepts": 33,
            "active": 1,
            "current": 33,
            "dropped": 0,
            "handled": 33,
            "hostname": "elastic-package-service-nginx-1:80",
            "reading": 0,
            "requests": 33,
            "waiting": 0,
            "writing": 1
        }
    },
    "service": {
        "address": "http://elastic-package-service-nginx-1:80/server-status",
        "type": "nginx"
    },
    "tags": [
        "nginx-stubstatus"
    ]
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields

FieldDescriptionTypeMetric Type
@timestamp
Event timestamp.
date
agent.id
keyword
cloud.account.id
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
keyword
cloud.availability_zone
Availability zone in which this host is running.
keyword
cloud.image.id
Image ID for the cloud instance.
keyword
cloud.instance.id
Instance ID of the host machine.
keyword
cloud.provider
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
keyword
cloud.region
Region in which this host is running.
keyword
container.id
Unique container id.
keyword
data_stream.dataset
Data stream dataset.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.dataset
Event dataset
constant_keyword
event.module
Event module
constant_keyword
host.containerized
If the host is a container.
boolean
host.name
Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.
keyword
host.os.build
OS build information.
keyword
host.os.codename
OS codename, if any.
keyword
nginx.stubstatus.accepts
The total number of accepted client connections.
long
counter
nginx.stubstatus.active
The current number of active client connections including Waiting connections.
long
gauge
nginx.stubstatus.current
The current number of client requests.
long
gauge
nginx.stubstatus.dropped
The total number of dropped client connections.
long
counter
nginx.stubstatus.handled
The total number of handled client connections.
long
counter
nginx.stubstatus.hostname
Nginx hostname.
keyword
nginx.stubstatus.reading
The current number of connections where Nginx is reading the request header.
long
gauge
nginx.stubstatus.requests
The total number of client requests.
long
counter
nginx.stubstatus.waiting
The current number of idle client connections waiting for a request.
long
gauge
nginx.stubstatus.writing
The current number of connections where Nginx is writing the response back to the client.
long
gauge
service.address
Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).
keyword

ML Modules

These anomaly detection jobs are available in the Machine Learning app in Kibana when you have data that matches the query specified in the manifest.

Nginx access logs

Find unusual activity in HTTP access logs.

JobDescription
visitor_rate_nginx
HTTP Access Logs: Detect unusual visitor rates
status_code_rate_nginx
HTTP Access Logs: Detect unusual status code rates
source_ip_url_count_nginx
HTTP Access Logs: Detect unusual source IPs - high distinct count of URLs
source_ip_request_rate_nginx
HTTP Access Logs: Detect unusual source IPs - high request rates
low_request_rate_nginx
HTTP Access Logs: Detect low request rates

Changelog

VersionDetailsKibana version(s)

1.23.0

Enhancement View pull request
Add processor support for stubstatus data stream.

8.13.0 or higher

1.22.0

Enhancement View pull request
ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

1.21.0

Enhancement View pull request
Add global filter on data_stream.dataset to improve performance.

8.12.0 or higher

1.20.0

Enhancement View pull request
Enable secrets for sensitive fields. For more details, refer https://www.elastic.co/guide/en/fleet/current/agent-policy.html#agent-policy-secret-values

8.12.0 or higher

1.19.1

Bug fix View pull request
Disable secrets for older stack versions due to errors.

8.8.0 or higher

1.19.0

Enhancement View pull request
Add support for tags in stub status metrics

8.8.0 or higher

1.18.0

Enhancement View pull request
Enable 'secret' for the sensitive fields, supported from 8.12.

8.8.0 or higher

1.17.0

Enhancement View pull request
Limit request tracer log count to five.

8.8.0 or higher

1.16.0

Enhancement View pull request
Update the package format_version to 3.0.0.

8.8.0 or higher

1.15.1

Bug fix View pull request
Add null check and ignore_missing check to the rename processor

8.8.0 or higher

1.15.0

Enhancement View pull request
Enable time series data streams for the metrics datasets. This improves storage usage and query performance. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html

8.8.0 or higher

1.14.1-beta

Bug fix View pull request
Add beta flag

—

1.14.0

Enhancement View pull request
Rename ownership from obs-service-integrations to obs-infraobs-integrations

8.8.0 or higher

1.13.0-beta

Enhancement View pull request
Add a new flag to enable request tracing

—

1.12.0-beta

Enhancement View pull request
Enable TSDB on Nginx for beta testing.

—

1.11.1

Bug fix View pull request
Modifed the dimension field mapping to support public cloud deployment.

8.5.0 or higher

1.11.0

Enhancement View pull request
Migrate "Access and error logs" dashboard visualizations to lens.

8.5.0 or higher

1.10.0

Enhancement View pull request
Migrate "Logs Overview" dashboard visualizations to lens.

8.5.0 or higher

1.9.0

Enhancement View pull request
Migrate "Metrics Overview" dashboard visualizations to lens.

8.5.0 or higher

1.8.0

Enhancement View pull request
Added changes to support TSDB Enablement.

8.0.0 or higher

1.7.1

Enhancement View pull request
Added categories and/or subcategories.

8.0.0 or higher

1.7.0

Enhancement View pull request
Update ECS version to 8.5.1.

8.0.0 or higher

1.6.0

Enhancement View pull request
Added infrastructure category.

8.0.0 or higher

1.5.1

Enhancement View pull request
Remove unused visualizations

8.0.0 or higher

1.5.0

Enhancement View pull request
Add ignore_older to log data streams

8.0.0 or higher

1.4.1

Enhancement View pull request
Update documentation with additional context for new users.

8.0.0 or higher

1.4.0

Enhancement View pull request
Migration of tile map to map object in dashboards

—

1.3.2

Enhancement View pull request
Add documentation for multi-fields

—

1.3.1

Enhancement View pull request
Add ML modules to readme

7.14.0 or higher
8.0.0 or higher

1.3.0

Enhancement View pull request
Update to ECS 8.0

—

1.2.3

Bug fix View pull request
Regenerate test files using the new GeoIP database

—

1.2.2

Bug fix View pull request
Change test public IPs to the supported subset

—

1.2.1

Bug fix View pull request
Fix ML module manifest query to ignore frozen and cold tiers

7.14.0 or higher
8.0.0 or higher

1.2.0

Enhancement View pull request
Release nginx package for v8.0.0

7.14.0 or higher
8.0.0 or higher

1.1.2

Enhancement View pull request
Uniform with guidelines

—

1.1.1

Bug fix View pull request
Fix logic that checks for the 'forwarded' tag

—

1.1.0

Enhancement View pull request
Update to ECS 1.12.0

7.14.0 or higher

1.0.0

Enhancement View pull request
Release Nginx as GA

—

0.8.2

Enhancement View pull request
Convert to generated ECS fields

—

0.8.1

Enhancement View pull request
update to ECS 1.11.0

—

0.8.0

Enhancement View pull request
Update integration description

—

0.7.0

Enhancement View pull request
Set "event.module" and "event.dataset"

—

0.6.2

Enhancement View pull request
Add support for Splunk authorization tokens

—

0.6.1

Bug fix View pull request
Fix bug in Third Party REST API ingest pipeline

—

0.6.0

Enhancement View pull request
update to ECS 1.10.0 and add event.original options

—

0.5.0

Enhancement View pull request
Adds ML jobs for finding unusual activity in HTTP access logs

—

0.4.1

Enhancement View pull request
update to ECS 1.9.0

—

0.3.11

Enhancement View pull request
Updating package owner

—

0.3.10

Enhancement View pull request
Fix compatibility with Kibana

—

0.1.0

Enhancement View pull request
initial release

—

On this page