New

The executive guide to generative AI

Read more
About usPartnersSupport|Login
« Apache Apache Spark Integration »
Elastic Docs ›Elastic integrations ›Apache

Apache Integration

edit

Apache Integration

edit

Version

1.29.1 (View all)

Compatible Kibana version(s)

8.13.0 or higher

Supported Serverless project types
What’s this?

Security
Observability

Subscription level
What’s this?

Basic

Level of support
What’s this?

Elastic

This integration periodically fetches metrics from Apache servers. It can parse access and error logs created by the Apache server.

Compatibility

edit

The Apache datasets were tested with Apache 2.4.12 and 2.4.46 and are expected to work with all versions >= 2.2.31 and >= 2.4.16 (independent from operating system).

Logs

edit

Access Logs

edit

Access logs collects the Apache access logs.

Example

An example event for access looks as following:

{
    "@timestamp": "2024-06-21T13:03:30.000Z",
    "agent": {
        "ephemeral_id": "a8296a9f-087a-48ae-af44-8f064213f161",
        "id": "9326664e-5848-4401-a0fb-4494a1538c2e",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.13.0"
    },
    "apache": {
        "access": {
            "remote_addresses": [
                "127.0.0.1"
            ]
        }
    },
    "data_stream": {
        "dataset": "apache.access",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "9326664e-5848-4401-a0fb-4494a1538c2e",
        "snapshot": false,
        "version": "8.13.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "web"
        ],
        "created": "2024-06-21T13:03:44.637Z",
        "dataset": "apache.access",
        "ingested": "2024-06-21T13:03:56Z",
        "kind": "event",
        "outcome": "success"
    },
    "host": {
        "architecture": "x86_64",
        "containerized": true,
        "hostname": "docker-fleet-agent",
        "id": "8259e024976a406e8a54cdbffeb84fec",
        "ip": [
            "192.168.240.7"
        ],
        "mac": [
            "02-42-C0-A8-F0-07"
        ],
        "name": "docker-fleet-agent",
        "os": {
            "codename": "focal",
            "family": "debian",
            "kernel": "3.10.0-1160.118.1.el7.x86_64",
            "name": "Ubuntu",
            "platform": "ubuntu",
            "type": "linux",
            "version": "20.04.6 LTS (Focal Fossa)"
        }
    },
    "http": {
        "request": {
            "method": "GET"
        },
        "response": {
            "body": {
                "bytes": 45
            },
            "status_code": 200
        },
        "version": "1.1"
    },
    "input": {
        "type": "log"
    },
    "log": {
        "file": {
            "path": "/tmp/service_logs/access.log"
        },
        "offset": 0
    },
    "related": {
        "ip": [
            "127.0.0.1"
        ]
    },
    "source": {
        "address": "127.0.0.1",
        "ip": "127.0.0.1"
    },
    "tags": [
        "apache-access"
    ],
    "url": {
        "original": "/",
        "path": "/"
    },
    "user_agent": {
        "device": {
            "name": "Other"
        },
        "name": "curl",
        "original": "curl/7.64.0",
        "version": "7.64.0"
    }
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type Unit

@timestamp

Event timestamp.

date

apache.access.http.request_headers

Http request headers.

keyword

apache.access.identity

The client’s identity, as specified in RFC 1413, determined by the identd on the client’s machine.

keyword

apache.access.remote_addresses

An array of remote addresses. It is a list because it is common to include, besides the client IP address, IP addresses from headers like X-Forwarded-For.

keyword

apache.access.response_time

Time to serve the request in microseconds.

long

micros

apache.access.ssl.cipher

SSL cipher name. - name: nginx.access

keyword

apache.access.ssl.protocol

SSL protocol version.

keyword

apache.access.tls_handshake.error

TLS handshake error.

keyword

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

input.type

Input type

keyword

log.offset

Log offset

long

Supported format for the access logs are:

  • Common Log Format

    • The common LogFormat can be used as follows:

      %h %l %u %t \"%r\" %>s %b

    • Example:

      127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326

  • Combined Log Format

    • The combined LogFormat can be used as follows:

      1. %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"
      1. %A:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"
      1. %h:%p %l %u %t \"%{req}i %U %H\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"

    • Example:

      1. 127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://datawarehouse.us.oracle.com/datamining/contents.htm" "Mozilla/4.7 [en] (WinNT; I)"
      1. 127.0.0.1:80 127.0.0.1 - - [20/Jun/2024:16:23:43 +0530] "\x16\x03\x01" 400 226 "-" "-"
      1. 127.0.0.1:80 - - [20/Jun/2024:16:31:41 +0530] "<SCRIPT>NXSSTEST</SCRIPT> / HTTP/1.1" 403 4897 "-" "-"

  • Combined Log Format + X-Forwarded-For header

    • The combined LogFormat with x-forwarded-for header can be used as follows:

      %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" X-Forwarded-For=\"%{X-Forwarded-For}i\"

    • Example:

      127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://datawarehouse.us.oracle.com/datamining/contents.htm" "Mozilla/4.7 [en] (WinNT; I)" X-Forwarded-For="10.225.192.17, 10.2.2.121"

Error Logs

edit

Error logs collects the Apache error logs.

Example

An example event for error looks as following:

{
    "@timestamp": "2024-07-03T11:17:00.781Z",
    "agent": {
        "ephemeral_id": "7abcc15c-0d38-4f16-843e-622a20dcfe13",
        "id": "7417c67c-5b97-401f-b722-6becf94a2f17",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.13.0"
    },
    "apache": {
        "error": {
            "module": "mpm_event"
        }
    },
    "data_stream": {
        "dataset": "apache.error",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "7417c67c-5b97-401f-b722-6becf94a2f17",
        "snapshot": false,
        "version": "8.13.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "web"
        ],
        "dataset": "apache.error",
        "ingested": "2024-07-03T11:17:27Z",
        "kind": "event",
        "timezone": "+00:00",
        "type": [
            "info"
        ]
    },
    "host": {
        "architecture": "x86_64",
        "containerized": true,
        "hostname": "docker-fleet-agent",
        "id": "8259e024976a406e8a54cdbffeb84fec",
        "ip": [
            "192.168.246.7"
        ],
        "mac": [
            "02-42-C0-A8-F6-07"
        ],
        "name": "docker-fleet-agent",
        "os": {
            "codename": "focal",
            "family": "debian",
            "kernel": "3.10.0-1160.118.1.el7.x86_64",
            "name": "Ubuntu",
            "platform": "ubuntu",
            "type": "linux",
            "version": "20.04.6 LTS (Focal Fossa)"
        }
    },
    "input": {
        "type": "log"
    },
    "log": {
        "file": {
            "path": "/tmp/service_logs/error.log"
        },
        "level": "notice",
        "offset": 0
    },
    "message": "AH00489: Apache/2.4.46 (Unix) configured -- resuming normal operations",
    "process": {
        "pid": 1,
        "thread": {
            "id": 139928782480512
        }
    },
    "tags": [
        "apache-error"
    ]
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

apache.error.module

The module producing the logged message.

keyword

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

input.type

Input type

keyword

log.offset

Log offset

long

Metrics

edit

Status Metrics

edit

The server status stream collects data from the Apache Status module. It scrapes the status data from the web page generated by the mod_status module.

Example

An example event for status looks as following:

{
    "@timestamp": "2022-12-09T03:56:04.531Z",
    "agent": {
        "ephemeral_id": "de9a4641-fef3-4e54-b95a-cd2c722fb9d3",
        "id": "46343e0c-0d8c-464b-a216-cacf63027d6f",
        "name": "docker-fleet-agent",
        "type": "metricbeat",
        "version": "8.5.0"
    },
    "apache": {
        "status": {
            "bytes_per_request": 0,
            "bytes_per_sec": 0,
            "connections": {
                "async": {
                    "closing": 0,
                    "keep_alive": 0,
                    "writing": 0
                },
                "total": 0
            },
            "cpu": {
                "children_system": 0,
                "children_user": 0,
                "load": 0.133333,
                "system": 0.01,
                "user": 0.01
            },
            "load": {
                "1": 1.79,
                "15": 1.04,
                "5": 1.5
            },
            "requests_per_sec": 0.933333,
            "scoreboard": {
                "closing_connection": 0,
                "dns_lookup": 0,
                "gracefully_finishing": 0,
                "idle_cleanup": 0,
                "keepalive": 0,
                "logging": 0,
                "open_slot": 325,
                "reading_request": 0,
                "sending_reply": 1,
                "starting_up": 0,
                "total": 400,
                "waiting_for_connection": 74
            },
            "total_accesses": 14,
            "total_bytes": 0,
            "uptime": {
                "server_uptime": 15,
                "uptime": 15
            },
            "workers": {
                "busy": 1,
                "idle": 74
            }
        }
    },
    "data_stream": {
        "dataset": "apache.status",
        "namespace": "ep",
        "type": "metrics"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "46343e0c-0d8c-464b-a216-cacf63027d6f",
        "snapshot": false,
        "version": "8.5.0"
    },
    "event": {
        "agent_id_status": "verified",
        "dataset": "apache.status",
        "duration": 6186792,
        "ingested": "2022-12-09T03:56:04Z",
        "module": "apache"
    },
    "host": {
        "architecture": "x86_64",
        "containerized": false,
        "hostname": "docker-fleet-agent",
        "id": "66392b0697b84641af8006d87aeb89f1",
        "ip": [
            "172.18.0.7"
        ],
        "mac": [
            "02-42-AC-12-00-07"
        ],
        "name": "docker-fleet-agent",
        "os": {
            "codename": "focal",
            "family": "debian",
            "kernel": "5.15.49-linuxkit",
            "name": "Ubuntu",
            "platform": "ubuntu",
            "type": "linux",
            "version": "20.04.5 LTS (Focal Fossa)"
        }
    },
    "metricset": {
        "name": "status",
        "period": 30000
    },
    "service": {
        "address": "http://elastic-package-service-apache-1:80/server-status?auto=",
        "type": "apache"
    }
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type Unit Metric Type

@timestamp

Event timestamp.

date

agent.id

keyword

apache.status.bytes_per_request

Bytes per request.

scaled_float

gauge

apache.status.bytes_per_sec

Bytes per second.

scaled_float

gauge

apache.status.connections.async.closing

Async closed connections.

long

gauge

apache.status.connections.async.keep_alive

Async keeped alive connections.

long

gauge

apache.status.connections.async.writing

Async connection writing.

long

gauge

apache.status.connections.total

Total connections.

long

counter

apache.status.cpu.children_system

CPU of children system.

scaled_float

gauge

apache.status.cpu.children_user

CPU of children user.

scaled_float

gauge

apache.status.cpu.load

CPU Load.

scaled_float

gauge

apache.status.cpu.system

System cpu.

scaled_float

gauge

apache.status.cpu.user

CPU user load.

scaled_float

gauge

apache.status.load.1

Load average for the last minute.

scaled_float

gauge

apache.status.load.15

Load average for the last 15 minutes.

scaled_float

gauge

apache.status.load.5

Load average for the last 5 minutes.

scaled_float

gauge

apache.status.requests_per_sec

Requests per second.

scaled_float

gauge

apache.status.scoreboard.closing_connection

Closing connections.

long

gauge

apache.status.scoreboard.dns_lookup

Dns Lookups.

long

gauge

apache.status.scoreboard.gracefully_finishing

Gracefully finishing.

long

gauge

apache.status.scoreboard.idle_cleanup

Idle cleanups.

long

gauge

apache.status.scoreboard.keepalive

Keep alive.

long

gauge

apache.status.scoreboard.logging

Logging

long

gauge

apache.status.scoreboard.open_slot

Open slots.

long

gauge

apache.status.scoreboard.reading_request

Reading requests.

long

gauge

apache.status.scoreboard.sending_reply

Sending Reply.

long

gauge

apache.status.scoreboard.starting_up

Starting up.

long

gauge

apache.status.scoreboard.total

Total.

long

gauge

apache.status.scoreboard.waiting_for_connection

Waiting for connections.

long

gauge

apache.status.total_accesses

Total number of access requests.

long

counter

apache.status.total_bytes

Total number of bytes served.

long

byte

counter

apache.status.uptime.server_uptime

Server uptime in seconds.

long

counter

apache.status.uptime.uptime

Server uptime.

long

counter

apache.status.workers.busy

Number of busy workers.

long

gauge

apache.status.workers.idle

Number of idle workers.

long

gauge

cloud.account.id

The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.

keyword

cloud.availability_zone

Availability zone in which this host is running.

keyword

cloud.image.id

Image ID for the cloud instance.

keyword

cloud.instance.id

Instance ID of the host machine.

keyword

cloud.provider

Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.

keyword

cloud.region

Region in which this host is running.

keyword

container.id

Unique container id.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

host.containerized

If the host is a container.

boolean

host.name

Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.

keyword

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

service.address

Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).

keyword

Anomaly Detection Configurations

edit

These anomaly detection jobs are available in the Machine Learning app in Kibana when you have data that matches the query specified in the manifest.

Apache Access Logs

edit

Find unusual activity in HTTP access logs.

Job Description

visitor_rate_apache

HTTP Access Logs: Detect unusual visitor rates

status_code_rate_apache

HTTP Access Logs: Detect unusual status code rates

source_ip_url_count_apache

HTTP Access Logs: Detect unusual source IPs - high distinct count of URLs

source_ip_request_rate_apache

HTTP Access Logs: Detect unusual source IPs - high request rates

low_request_rate_apache

HTTP Access Logs: Detect low request rates

Changelog

edit
Changelog
Version Details Kibana version(s)

1.29.1

Bug fix (View pull request)
Added description to ssl nodes including links to documentation.

8.13.0 or higher

1.29.0

Enhancement (View pull request)
Support SSL/TLS for status data stream.

8.13.0 or higher

1.28.0

Enhancement (View pull request)
Allow the usage of deprecated log input

8.13.0 or higher

1.27.0

Enhancement (View pull request)
Support stack 9.0

8.13.0 or higher
9.0.0 or higher

1.26.0

Enhancement (View pull request)
Deprecate third-party REST API import option.

8.13.0 or higher

1.25.0

Enhancement (View pull request)
Allow configuration of ignoring older events in apache access log datastream.

8.13.0 or higher

1.24.0

Enhancement (View pull request)
Allow @custom pipeline access to event.original without setting preserve_original_event.

8.13.0 or higher

1.23.0

Enhancement (View pull request)
Update grok pattern to support access and error log format.

8.13.0 or higher

1.22.0

Enhancement (View pull request)
Add processor support for status data stream.

8.13.0 or higher

1.21.0

Enhancement (View pull request)
ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

1.20.0

Enhancement (View pull request)
Add optional response-time field in access logs.

8.10.2 or higher

1.19.0

Enhancement (View pull request)
Add global filter on data_stream.dataset to improve performance.

8.10.2 or higher

1.18.0

Enhancement (View pull request)
Prepare package for serverless.

Enhancement (View pull request)
Add secrets support.

Enhancement (View pull request)
Add filters to metrics dashboard.

Bug fix (View pull request)
Remove duplicated and ambiguous field definitions.

Bug fix (View pull request)
Fix normalization as arrays of event.category and event.type fields.

8.10.2 or higher

1.17.2

Bug fix (View pull request)
Rename field "apache2.error.integration" to "apache.error.module" in the Apache error visualizations.

8.10.2 or higher

1.17.1

Bug fix (View pull request)
Update grok for accepting user-identity.

8.10.2 or higher

1.17.0

Enhancement (View pull request)
Limit request tracer log count to five.

8.10.2 or higher

1.16.0

Enhancement (View pull request)
Migrate Uptime metric visualization to lens.

8.10.2 or higher

1.15.1

Bug fix (View pull request)
Add null check and ignore_missing check to the rename processor

8.8.0 or higher

1.15.0

Enhancement (View pull request)
Add ability to set condition for logs and metrics.

8.8.0 or higher

1.14.0

Enhancement (View pull request)
Enable time series data streams for the metrics datasets. This dramatically reduces storage for metrics and is expected to progressively improve query performance. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html.

8.8.0 or higher

1.13.0

Enhancement (View pull request)
Rename ownership from obs-service-integrations to obs-infraobs-integrations

8.7.1 or higher

1.12.0

Enhancement (View pull request)
Add a new flag to enable request tracing

8.7.1 or higher

1.11.0

Enhancement (View pull request)
Add dimension fields for status datastream for TSDB enablement.

8.3.0 or higher

1.10.0

Enhancement (View pull request)
Migrate Access and error logs dashboard visualizations to lens.

8.3.0 or higher

1.9.0

Enhancement (View pull request)
Migrate Overview dashboard visualizations to lens.

8.3.0 or higher

1.8.2

Bug fix (View pull request)
Fix a bug that may blank three visualizations

8.0.0 or higher

1.8.1

Enhancement (View pull request)
Added categories and/or subcategories.

8.0.0 or higher

1.8.0

Enhancement (View pull request)
Update ECS version to 8.5.1.

8.0.0 or higher

1.7.0

Enhancement (View pull request)
Added infrastructure category.

8.0.0 or higher

1.6.0

Enhancement (View pull request)
Add support for x_forwarded_for header

8.0.0 or higher

1.5.1

Enhancement (View pull request)
Remove unused visualizations

8.0.0 or higher

1.5.0

Enhancement (View pull request)
Use new labels for source license and subscription

8.0.0 or higher

1.4.1

Bug fix (View pull request)
Add correct field mapping for event.created

8.0.0 or higher

1.4.0

Enhancement (View pull request)
Migration of tile map to map in logs dashboard

—

1.3.6

Enhancement (View pull request)
Add documentation for multi-fields

—

1.3.5

Enhancement (View pull request)
Add ML modules to readme

7.14.0 or higher
8.0.0 or higher

1.3.4

Bug fix (View pull request)
Regenerate test files using the new GeoIP database

—

1.3.3

Bug fix (View pull request)
Change test public IPs to the supported subset

—

1.3.2

Bug fix (View pull request)
Fix ML module manifest query to ignore frozen and cold tiers

7.14.0 or higher
8.0.0 or higher

1.3.1

Bug fix (View pull request)
Fix parsing of trace log levels

—

1.3.0

Enhancement (View pull request)
Support Kibana 8.0

7.14.0 or higher
8.0.0 or higher

1.2.0

Enhancement (View pull request)
Uniform with guidelines

—

1.1.1

Bug fix (View pull request)
Fix logic that checks for the forwarded tag

—

1.1.0

Enhancement (View pull request)
Update to ECS 1.12.0

7.14.0 or higher

1.0.0

Enhancement (View pull request)
Release Apache as GA

—

0.9.2

Enhancement (View pull request)
Convert to generated ECS fields

—

0.9.1

Enhancement (View pull request)
update to ECS 1.11.0

—

0.9.0

Enhancement (View pull request)
Update integration description

—

0.8.1

Enhancement (View pull request)
Add support for Splunk authorization tokens

—

0.8.0

Bug fix (View pull request)
Set event.module and event.dataset

—

0.7.1

Bug fix (View pull request)
Fix bug in Third Party REST API ingest pipeline

—

0.7.0

Enhancement (View pull request)
Update to ECS 1.10.0 and adding items that all packages should have

—

0.6.0

Enhancement (View pull request)
Render units and metric types in exported fields table

—

0.5.1

Enhancement (View pull request)
Move ecs.version to the ingest pipeline and make event.original optional

—

0.5.0

Enhancement (View pull request)
Adds ML jobs for finding unusual activity in HTTP access logs

—

0.4.1

Enhancement (View pull request)
update to ECS 1.9.0

—

0.3.5

Enhancement (View pull request)
Updating package owner

—

0.3.4

Bug fix (View pull request)
Use correct types for source.port and source.ip

—

0.1.0

Enhancement (View pull request)
initial release

—
« Apache Apache Spark Integration »