« Nginx Nginx Ingress Controller Integration »
Elastic Docs ›Elastic integrations ›Nginx

Nginx Integration

edit

Nginx Integration

edit

Version

1.24.0 (View all)

Compatible Kibana version(s)

8.13.0 or higher

Supported Serverless project types
What’s this?

Security
Observability

Subscription level
What’s this?

Basic

Level of support
What’s this?

Elastic

The Nginx integration allows you to monitor Nginx servers. Time series index mode enabled for metrics data stream.

Use the Nginx integration to collect metrics and logs from your server. Then visualize that data in Kibana, use the Machine Learning app to find unusual activity in HTTP access logs, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue.

For example, if you wanted to be notified if a certain number of client requests failed in a given time period, you could install the Nginx integration to send logs to Elastic. Then, you could view the logs stream into Elastic in real time in the Observability Logs app. You could also set up a new log threshold rule in the Logs app to alert you when there are more than a certain number of events with a failing status in a given time period.

Data streams

edit

The Nginx integration collects two types of data: logs and metrics.

Logs help you keep a record of events that happen in your Nginx servers. This includes when a client request or error occurs.

Metrics give you insight into the state of your Nginx servers. This includes information like the total number of active client connections by status, the total number of client requests, and more.

Requirements

edit

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.

Each data source was tested with a specific Nginx version. For more information see the Logs reference and Metrics reference.

On Windows, the module was tested with Nginx installed from the Chocolatey repository.

Setup

edit

For step-by-step instructions on how to set up an integration, see the Getting started guide.

Logs reference

edit

Timezone support

This data source parses logs that don’t contain timezone information. For these logs, the Elastic Agent reads the local timezone and uses it when parsing to convert the timestamp to UTC. The timezone to be used for parsing is included in the event in the event.timezone field.

To disable this conversion, the event.timezone field can be removed using the drop_fields processor.

If logs originate from systems or applications with a timezone that is different than the local one, the event.timezone field can be overwritten with the original timezone using the add_fields processor.

Access Logs
edit

Access logs collects the Nginx access logs.

Tested versions
edit

The Nginx access logs stream was tested with Nginx 1.19.5.

Example

An example event for access looks as following:

{
    "@timestamp": "2022-12-09T10:39:23.000Z",
    "_tmp": {},
    "agent": {
        "ephemeral_id": "34369a4a-4f24-4a39-9758-85fc2429d7e2",
        "id": "ef5e274d-4b53-45e6-943a-a5bcf1a6f523",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.5.0"
    },
    "data_stream": {
        "dataset": "nginx.access",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "ef5e274d-4b53-45e6-943a-a5bcf1a6f523",
        "snapshot": false,
        "version": "8.5.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "web"
        ],
        "created": "2022-12-09T10:39:38.896Z",
        "dataset": "nginx.access",
        "ingested": "2022-12-09T10:39:40Z",
        "kind": "event",
        "outcome": "success",
        "timezone": "+00:00",
        "type": [
            "access"
        ]
    },
    "host": {
        "architecture": "x86_64",
        "containerized": false,
        "hostname": "docker-fleet-agent",
        "id": "66392b0697b84641af8006d87aeb89f1",
        "ip": [
            "172.18.0.7"
        ],
        "mac": [
            "02-42-AC-12-00-07"
        ],
        "name": "docker-fleet-agent",
        "os": {
            "codename": "focal",
            "family": "debian",
            "kernel": "5.15.49-linuxkit",
            "name": "Ubuntu",
            "platform": "ubuntu",
            "type": "linux",
            "version": "20.04.5 LTS (Focal Fossa)"
        }
    },
    "http": {
        "request": {
            "method": "GET"
        },
        "response": {
            "body": {
                "bytes": 97
            },
            "status_code": 200
        },
        "version": "1.1"
    },
    "input": {
        "type": "log"
    },
    "log": {
        "file": {
            "path": "/tmp/service_logs/access.log"
        },
        "offset": 0
    },
    "nginx": {
        "access": {
            "remote_ip_list": [
                "127.0.0.1"
            ]
        }
    },
    "related": {
        "ip": [
            "127.0.0.1"
        ]
    },
    "source": {
        "address": "127.0.0.1",
        "ip": "127.0.0.1"
    },
    "tags": [
        "nginx-access"
    ],
    "url": {
        "original": "/server-status",
        "path": "/server-status"
    },
    "user_agent": {
        "device": {
            "name": "Other"
        },
        "name": "curl",
        "original": "curl/7.64.0",
        "version": "7.64.0"
    }
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

input.type

Input type

keyword

log.offset

Log offset

long

nginx.access.remote_ip_list

An array of remote IP addresses. It is a list because it is common to include, besides the client IP address, IP addresses from headers like X-Forwarded-For. Real source IP is restored to source.ip.

keyword
Error Logs
edit

Error logs collects the Nginx error logs.

Tested versions
edit

The Nginx error logs stream was tested with Nginx 1.19.5.

Example

An example event for error looks as following:

{
    "@timestamp": "2022-12-09T10:40:03.000Z",
    "agent": {
        "ephemeral_id": "34369a4a-4f24-4a39-9758-85fc2429d7e2",
        "id": "ef5e274d-4b53-45e6-943a-a5bcf1a6f523",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.5.0"
    },
    "data_stream": {
        "dataset": "nginx.error",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "ef5e274d-4b53-45e6-943a-a5bcf1a6f523",
        "snapshot": false,
        "version": "8.5.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "web"
        ],
        "created": "2022-12-09T10:40:18.973Z",
        "dataset": "nginx.error",
        "ingested": "2022-12-09T10:40:24Z",
        "kind": "event",
        "timezone": "+00:00",
        "type": [
            "error"
        ]
    },
    "host": {
        "architecture": "x86_64",
        "containerized": false,
        "hostname": "docker-fleet-agent",
        "id": "66392b0697b84641af8006d87aeb89f1",
        "ip": [
            "172.18.0.7"
        ],
        "mac": [
            "02-42-AC-12-00-07"
        ],
        "name": "docker-fleet-agent",
        "os": {
            "codename": "focal",
            "family": "debian",
            "kernel": "5.15.49-linuxkit",
            "name": "Ubuntu",
            "platform": "ubuntu",
            "type": "linux",
            "version": "20.04.5 LTS (Focal Fossa)"
        }
    },
    "input": {
        "type": "log"
    },
    "log": {
        "file": {
            "path": "/tmp/service_logs/error.log"
        },
        "level": "warn",
        "offset": 0
    },
    "message": "conflicting server name \"localhost\" on 0.0.0.0:80, ignored",
    "nginx": {
        "error": {}
    },
    "process": {
        "pid": 1,
        "thread": {
            "id": 1
        }
    },
    "tags": [
        "nginx-error"
    ]
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

input.type

Input type

keyword

log.offset

Log offset

long

nginx.error.connection_id

Connection identifier.

long

Metrics reference

edit
Stub Status Metrics
edit

The Nginx stubstatus stream collects data from the Nginx ngx_http_stub_status module. It scrapes the server status data from the web page generated by ngx_http_stub_status. Please verify that your Nginx distribution comes with the mentioned module and it’s enabled in the Nginx configuration file:

location /nginx_status {
    stub_status;
    allow 127.0.0.1; # only allow requests from localhost
    deny all;        # deny all other hosts
}

Replace 127.0.0.1 with your server’s IP address and make sure that this page accessible to only you.

Tested versions
edit

The Nginx stubstatus stream was tested with Nginx 1.19.5 and is expected to work with all versions >= 1.19.

Example

An example event for stubstatus looks as following:

{
    "@timestamp": "2024-02-08T08:12:10.668Z",
    "agent": {
        "ephemeral_id": "c3d516ba-b659-4190-a29d-d28200d74d48",
        "id": "2ea50bee-9250-43d1-8d70-949f242aa275",
        "name": "docker-fleet-agent",
        "type": "metricbeat",
        "version": "8.10.1"
    },
    "data_stream": {
        "dataset": "nginx.stubstatus",
        "namespace": "ep",
        "type": "metrics"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "2ea50bee-9250-43d1-8d70-949f242aa275",
        "snapshot": false,
        "version": "8.10.1"
    },
    "event": {
        "agent_id_status": "verified",
        "dataset": "nginx.stubstatus",
        "duration": 1156104,
        "ingested": "2024-02-08T08:12:13Z",
        "module": "nginx"
    },
    "host": {
        "architecture": "x86_64",
        "containerized": false,
        "hostname": "docker-fleet-agent",
        "id": "28da52b32df94b50aff67dfb8f1be3d6",
        "ip": [
            "172.24.0.7"
        ],
        "mac": [
            "02-42-AC-18-00-07"
        ],
        "name": "docker-fleet-agent",
        "os": {
            "codename": "focal",
            "family": "debian",
            "kernel": "5.15.0-89-generic",
            "name": "Ubuntu",
            "platform": "ubuntu",
            "type": "linux",
            "version": "20.04.6 LTS (Focal Fossa)"
        }
    },
    "metricset": {
        "name": "stubstatus",
        "period": 10000
    },
    "nginx": {
        "stubstatus": {
            "accepts": 33,
            "active": 1,
            "current": 33,
            "dropped": 0,
            "handled": 33,
            "hostname": "elastic-package-service-nginx-1:80",
            "reading": 0,
            "requests": 33,
            "waiting": 0,
            "writing": 1
        }
    },
    "service": {
        "address": "http://elastic-package-service-nginx-1:80/server-status",
        "type": "nginx"
    },
    "tags": [
        "nginx-stubstatus"
    ]
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type Metric Type

@timestamp

Event timestamp.

date

agent.id

keyword

cloud.account.id

The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.

keyword

cloud.availability_zone

Availability zone in which this host is running.

keyword

cloud.image.id

Image ID for the cloud instance.

keyword

cloud.instance.id

Instance ID of the host machine.

keyword

cloud.provider

Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.

keyword

cloud.region

Region in which this host is running.

keyword

container.id

Unique container id.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

host.containerized

If the host is a container.

boolean

host.name

Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.

keyword

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

nginx.stubstatus.accepts

The total number of accepted client connections.

long

counter

nginx.stubstatus.active

The current number of active client connections including Waiting connections.

long

gauge

nginx.stubstatus.current

The current number of client requests.

long

gauge

nginx.stubstatus.dropped

The total number of dropped client connections.

long

counter

nginx.stubstatus.handled

The total number of handled client connections.

long

counter

nginx.stubstatus.hostname

Nginx hostname.

keyword

nginx.stubstatus.reading

The current number of connections where Nginx is reading the request header.

long

gauge

nginx.stubstatus.requests

The total number of client requests.

long

counter

nginx.stubstatus.waiting

The current number of idle client connections waiting for a request.

long

gauge

nginx.stubstatus.writing

The current number of connections where Nginx is writing the response back to the client.

long

gauge

service.address

Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).

keyword

Anomaly Detection Configurations

edit

These anomaly detection jobs are available in the Machine Learning app in Kibana when you have data that matches the query specified in the manifest.

Nginx access logs
edit

Find unusual activity in HTTP access logs.

Job Description

visitor_rate_nginx

HTTP Access Logs: Detect unusual visitor rates

status_code_rate_nginx

HTTP Access Logs: Detect unusual status code rates

source_ip_url_count_nginx

HTTP Access Logs: Detect unusual source IPs - high distinct count of URLs

source_ip_request_rate_nginx

HTTP Access Logs: Detect unusual source IPs - high request rates

low_request_rate_nginx

HTTP Access Logs: Detect low request rates

Changelog

edit
Changelog
Version Details Kibana version(s)

1.24.0

Enhancement (View pull request)
Deprecate third-party REST API import option.

8.13.0 or higher

1.23.0

Enhancement (View pull request)
Add processor support for stubstatus data stream.

8.13.0 or higher

1.22.0

Enhancement (View pull request)
ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

1.21.0

Enhancement (View pull request)
Add global filter on data_stream.dataset to improve performance.

8.12.0 or higher

1.20.0

Enhancement (View pull request)
Enable secrets for sensitive fields. For more details, refer https://www.elastic.co/guide/en/fleet/current/agent-policy.html#agent-policy-secret-values

8.12.0 or higher

1.19.1

Bug fix (View pull request)
Disable secrets for older stack versions due to errors.

8.8.0 or higher

1.19.0

Enhancement (View pull request)
Add support for tags in stub status metrics

8.8.0 or higher

1.18.0

Enhancement (View pull request)
Enable secret for the sensitive fields, supported from 8.12.

8.8.0 or higher

1.17.0

Enhancement (View pull request)
Limit request tracer log count to five.

8.8.0 or higher

1.16.0

Enhancement (View pull request)
Update the package format_version to 3.0.0.

8.8.0 or higher

1.15.1

Bug fix (View pull request)
Add null check and ignore_missing check to the rename processor

8.8.0 or higher

1.15.0

Enhancement (View pull request)
Enable time series data streams for the metrics datasets. This improves storage usage and query performance. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html

8.8.0 or higher

1.14.1-beta

Bug fix (View pull request)
Add beta flag

—

1.14.0

Enhancement (View pull request)
Rename ownership from obs-service-integrations to obs-infraobs-integrations

8.8.0 or higher

1.13.0-beta

Enhancement (View pull request)
Add a new flag to enable request tracing

—

1.12.0-beta

Enhancement (View pull request)
Enable TSDB on Nginx for beta testing.

—

1.11.1

Bug fix (View pull request)
Modifed the dimension field mapping to support public cloud deployment.

8.5.0 or higher

1.11.0

Enhancement (View pull request)
Migrate "Access and error logs" dashboard visualizations to lens.

8.5.0 or higher

1.10.0

Enhancement (View pull request)
Migrate "Logs Overview" dashboard visualizations to lens.

8.5.0 or higher

1.9.0

Enhancement (View pull request)
Migrate "Metrics Overview" dashboard visualizations to lens.

8.5.0 or higher

1.8.0

Enhancement (View pull request)
Added changes to support TSDB Enablement.

8.0.0 or higher

1.7.1

Enhancement (View pull request)
Added categories and/or subcategories.

8.0.0 or higher

1.7.0

Enhancement (View pull request)
Update ECS version to 8.5.1.

8.0.0 or higher

1.6.0

Enhancement (View pull request)
Added infrastructure category.

8.0.0 or higher

1.5.1

Enhancement (View pull request)
Remove unused visualizations

8.0.0 or higher

1.5.0

Enhancement (View pull request)
Add ignore_older to log data streams

8.0.0 or higher

1.4.1

Enhancement (View pull request)
Update documentation with additional context for new users.

8.0.0 or higher

1.4.0

Enhancement (View pull request)
Migration of tile map to map object in dashboards

—

1.3.2

Enhancement (View pull request)
Add documentation for multi-fields

—

1.3.1

Enhancement (View pull request)
Add ML modules to readme

7.14.0 or higher
8.0.0 or higher

1.3.0

Enhancement (View pull request)
Update to ECS 8.0

—

1.2.3

Bug fix (View pull request)
Regenerate test files using the new GeoIP database

—

1.2.2

Bug fix (View pull request)
Change test public IPs to the supported subset

—

1.2.1

Bug fix (View pull request)
Fix ML module manifest query to ignore frozen and cold tiers

7.14.0 or higher
8.0.0 or higher

1.2.0

Enhancement (View pull request)
Release nginx package for v8.0.0

7.14.0 or higher
8.0.0 or higher

1.1.2

Enhancement (View pull request)
Uniform with guidelines

—

1.1.1

Bug fix (View pull request)
Fix logic that checks for the forwarded tag

—

1.1.0

Enhancement (View pull request)
Update to ECS 1.12.0

7.14.0 or higher

1.0.0

Enhancement (View pull request)
Release Nginx as GA

—

0.8.2

Enhancement (View pull request)
Convert to generated ECS fields

—

0.8.1

Enhancement (View pull request)
update to ECS 1.11.0

—

0.8.0

Enhancement (View pull request)
Update integration description

—

0.7.0

Enhancement (View pull request)
Set "event.module" and "event.dataset"

—

0.6.2

Enhancement (View pull request)
Add support for Splunk authorization tokens

—

0.6.1

Bug fix (View pull request)
Fix bug in Third Party REST API ingest pipeline

—

0.6.0

Enhancement (View pull request)
update to ECS 1.10.0 and add event.original options

—

0.5.0

Enhancement (View pull request)
Adds ML jobs for finding unusual activity in HTTP access logs

—

0.4.1

Enhancement (View pull request)
update to ECS 1.9.0

—

0.3.11

Enhancement (View pull request)
Updating package owner

—

0.3.10

Enhancement (View pull request)
Fix compatibility with Kibana

—

0.1.0

Enhancement (View pull request)
initial release

—
« Nginx Nginx Ingress Controller Integration »