Painless examples for transforms
editPainless examples for transforms
editThese examples demonstrate how to use Painless in transforms. You can learn more about the Painless scripting language in the Painless guide.
- Getting top hits by using scripted metric aggregation
- Getting time features by using aggregations
-
Using Painless in
group_by
- Getting duration by using bucket script
- Counting HTTP responses by using scripted metric aggregation
- Comparing indices by using scripted metric aggregations
- Getting web session details by using scripted metric aggregation
- While the context of the following examples is the transform use case, the Painless scripts in the snippets below can be used in other Elasticsearch search aggregations, too.
- All the following examples use scripts, transforms cannot deduce mappings of output fields when the fields are created by a script. Transforms don’t create any mappings in the destination index for these fields, which means they get dynamically mapped. Create the destination index prior to starting the transform in case you want explicit mappings.
Getting top hits by using scripted metric aggregation
editThis snippet shows how to find the latest document, in other words the document with the latest timestamp. From a technical perspective, it helps to achieve the function of a Top hits by using scripted metric aggregation in a transform, which provides a metric output.
"aggregations": { "latest_doc": { "scripted_metric": { "init_script": "state.timestamp_latest = 0L; state.last_doc = ''", "map_script": """ def current_date = doc['@timestamp'].getValue().toInstant().toEpochMilli(); if (current_date > state.timestamp_latest) {state.timestamp_latest = current_date; state.last_doc = new HashMap(params['_source']);} """, "combine_script": "return state", "reduce_script": """ def last_doc = ''; def timestamp_latest = 0L; for (s in states) {if (s.timestamp_latest > (timestamp_latest)) {timestamp_latest = s.timestamp_latest; last_doc = s.last_doc;}} return last_doc """ } } }
The |
|
The |
|
The |
|
The |
Check the scope of scripts for detailed explanation on the respective scripts.
You can retrieve the last value in a similar way:
"aggregations": { "latest_value": { "scripted_metric": { "init_script": "state.timestamp_latest = 0L; state.last_value = ''", "map_script": """ def current_date = doc['@timestamp'].getValue().toInstant().toEpochMilli(); if (current_date > state.timestamp_latest) {state.timestamp_latest = current_date; state.last_value = params['_source']['value'];} """, "combine_script": "return state", "reduce_script": """ def last_value = ''; def timestamp_latest = 0L; for (s in states) {if (s.timestamp_latest > (timestamp_latest)) {timestamp_latest = s.timestamp_latest; last_value = s.last_value;}} return last_value """ } } }
Getting time features by using aggregations
editThis snippet shows how to extract time based features by using Painless in a
transform. The snippet uses an index where @timestamp
is defined as a date
type field.
"aggregations": { "avg_hour_of_day": { "avg":{ "script": { "source": """ ZonedDateTime date = doc['@timestamp'].value; return date.getHour(); """ } } }, "avg_month_of_year": { "avg":{ "script": { "source": """ ZonedDateTime date = doc['@timestamp'].value; return date.getMonthValue(); """ } } }, ... }
Name of the aggregation. |
|
Contains the Painless script that returns the hour of the day. |
|
Sets |
|
Returns the hour value from |
|
Name of the aggregation. |
|
Contains the Painless script that returns the month of the year. |
|
Sets |
|
Returns the month value from |
Using Painless in group_by
editIt is possible to base the group_by
property of a transform on the output of
a script. The following example uses the Kibana sample web logs dataset. The goal
here is to make the transform output easier to understand through normalizing
the value of the fields that the data is grouped by.
POST _transform/_preview { "source": { "index": [ "kibana_sample_data_logs" ] }, "pivot": { "group_by": { "agent": { "terms": { "script": { "source": """String agent = doc['agent.keyword'].value; if (agent.contains("MSIE")) { return "internet explorer"; } else if (agent.contains("AppleWebKit")) { return "safari"; } else if (agent.contains('Firefox')) { return "firefox"; } else { return agent }""", "lang": "painless" } } } }, "aggregations": { "all": { "value_count": { "field": "response.keyword" } }, "200": { "filter": { "term": { "response": "200" } } }, "404": { "filter": { "term": { "response": "404" } } }, "503": { "filter": { "term": { "response": "503" } } } } }, "dest": { "index": "pivot_logs" } }
Specifies the source index or indices. |
|
The script defines an |
|
The aggregations object contains filters that narrow down the results to
documents that contains |
|
Specifies the destination index of the transform. |
The API returns the following result:
{ "preview" : [ { "all" : 4010, "agent" : "explorer", "200" : 3674, "404" : 210, "503" : 126 }, { "all" : 5362, "agent" : "firefox", "200" : 4931, "404" : 259, "503" : 172 }, { "all" : 4702, "agent" : "safari", "200" : 4227, "404" : 332, "503" : 143 } ], "generated_dest_index" : { "mappings" : { "_meta" : { "_transform" : { "transform" : "transform-preview", "version" : { "created" : "7.9.3" }, "creation_date_in_millis" : 1610454866455 }, "created_by" : "transform" }, "properties" : { "all" : { "type" : "long" }, "200" : { "type" : "long" }, "404" : { "type" : "long" }, "503" : { "type" : "long" } } }, "settings" : { "index" : { "number_of_shards" : "1", "auto_expand_replicas" : "0-1" } }, "aliases" : { } } }
You can see that the agent
values are simplified so it is easier to interpret
them. The table below shows how normalization modifies the output of the
transform in our example compared to the non-normalized values.
Non-normalized agent value |
Normalized agent value |
---|---|
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)" |
"internet explorer" |
"Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24" |
"safari" |
"Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1" |
"firefox" |
Getting duration by using bucket script
editThis example shows you how to get the duration of a session by client IP from a data log by using bucket script. The example uses the Kibana sample web logs dataset.
PUT _transform/data_log { "source": { "index": "kibana_sample_data_logs" }, "dest": { "index": "data-logs-by-client" }, "pivot": { "group_by": { "machine.os": {"terms": {"field": "machine.os.keyword"}}, "machine.ip": {"terms": {"field": "clientip"}} }, "aggregations": { "time_frame.lte": { "max": { "field": "timestamp" } }, "time_frame.gte": { "min": { "field": "timestamp" } }, "time_length": { "bucket_script": { "buckets_path": { "min": "time_frame.gte.value", "max": "time_frame.lte.value" }, "script": "params.max - params.min" } } } } }
To define the length of the sessions, we use a bucket script. |
|
The bucket path is a map of script variables and their associated path to
the buckets you want to use for the variable. In this particular case, |
|
Finally, the script substracts the start date of the session from the end date which results in the duration of the session. |
Counting HTTP responses by using scripted metric aggregation
editYou can count the different HTTP response types in a web log data set by using scripted metric aggregation as part of the transform. You can achieve a similar function with filter aggregations, check the Finding suspicious client IPs example for details.
The example below assumes that the HTTP response codes are stored as keywords in
the response
field of the documents.
"aggregations": { "responses.counts": { "scripted_metric": { "init_script": "state.responses = ['error':0L,'success':0L,'other':0L]", "map_script": """ def code = doc['response.keyword'].value; if (code.startsWith('5') || code.startsWith('4')) { state.responses.error += 1 ; } else if(code.startsWith('2')) { state.responses.success += 1; } else { state.responses.other += 1; } """, "combine_script": "state.responses", "reduce_script": """ def counts = ['error': 0L, 'success': 0L, 'other': 0L]; for (responses in states) { counts.error += responses['error']; counts.success += responses['success']; counts.other += responses['other']; } return counts; """ } }, ... }
The |
|
Object of the |
|
This |
|
The |
|
The |
|
The |
|
The |
Comparing indices by using scripted metric aggregations
editThis example shows how to compare the content of two indices by a transform that uses a scripted metric aggregation.
POST _transform/_preview { "id" : "index_compare", "source" : { "index" : [ "index1", "index2" ], "query" : { "match_all" : { } } }, "dest" : { "index" : "compare" }, "pivot" : { "group_by" : { "unique-id" : { "terms" : { "field" : "<unique-id-field>" } } }, "aggregations" : { "compare" : { "scripted_metric" : { "map_script" : "state.doc = new HashMap(params['_source'])", "combine_script" : "return state", "reduce_script" : """ if (states.size() != 2) { return "count_mismatch" } if (states.get(0).equals(states.get(1))) { return "match" } else { return "mismatch" } """ } } } } }
The indices referenced in the |
|
The |
|
The |
|
Object of the |
|
The |
|
The |
|
The |
Getting web session details by using scripted metric aggregation
editThis example shows how to derive multiple features from a single transaction. Let’s take a look on the example source document from the data:
Source document
{ "_index":"apache-sessions", "_type":"_doc", "_id":"KvzSeGoB4bgw0KGbE3wP", "_score":1.0, "_source":{ "@timestamp":1484053499256, "apache":{ "access":{ "sessionid":"571604f2b2b0c7b346dc685eeb0e2306774a63c2", "url":"http://www.leroymerlin.fr/v3/search/search.do?keyword=Carrelage%20salle%20de%20bain", "path":"/v3/search/search.do", "query":"keyword=Carrelage%20salle%20de%20bain", "referrer":"http://www.leroymerlin.fr/v3/p/produits/carrelage-parquet-sol-souple/carrelage-sol-et-mur/decor-listel-et-accessoires-carrelage-mural-l1308217717?resultOffset=0&resultLimit=51&resultListShape=MOSAIC&priceStyle=SALEUNIT_PRICE", "user_agent":{ "original":"Mobile Safari 10.0 Mac OS X (iPad) Apple Inc.", "os_name":"Mac OS X (iPad)" }, "remote_ip":"0337b1fa-5ed4-af81-9ef4-0ec53be0f45d", "geoip":{ "country_iso_code":"FR", "location":{ "lat":48.86, "lon":2.35 } }, "response_code":200, "method":"GET" } } } } ...
By using the sessionid
as a group-by field, you are able to enumerate events
through the session and get more details of the session by using scripted metric
aggregation.
POST _transform/_preview { "source": { "index": "apache-sessions" }, "pivot": { "group_by": { "sessionid": { "terms": { "field": "apache.access.sessionid" } } }, "aggregations": { "distinct_paths": { "cardinality": { "field": "apache.access.path" } }, "num_pages_viewed": { "value_count": { "field": "apache.access.url" } }, "session_details": { "scripted_metric": { "init_script": "state.docs = []", "map_script": """ Map span = [ '@timestamp':doc['@timestamp'].value, 'url':doc['apache.access.url'].value, 'referrer':doc['apache.access.referrer'].value ]; state.docs.add(span) """, "combine_script": "return state.docs;", "reduce_script": """ def all_docs = []; for (s in states) { for (span in s) { all_docs.add(span); } } all_docs.sort((HashMap o1, HashMap o2)->o1['@timestamp'].millis.compareTo(o2['@timestamp'].millis)); def size = all_docs.size(); def min_time = all_docs[0]['@timestamp']; def max_time = all_docs[size-1]['@timestamp']; def duration = max_time.millis - min_time.millis; def entry_page = all_docs[0]['url']; def exit_path = all_docs[size-1]['url']; def first_referrer = all_docs[0]['referrer']; def ret = new HashMap(); ret['first_time'] = min_time; ret['last_time'] = max_time; ret['duration'] = duration; ret['entry_page'] = entry_page; ret['exit_path'] = exit_path; ret['first_referrer'] = first_referrer; return ret; """ } } } } }
The data is grouped by |
|
The aggregations counts the number of paths and enumerate the viewed pages during the session. |
|
The |
|
The |
|
The |
|
The |
The API call results in a similar response:
{ "num_pages_viewed" : 2.0, "session_details" : { "duration" : 131374, "first_referrer" : "https://www.bing.com/", "entry_page" : "http://www.leroymerlin.fr/v3/p/produits/materiaux-menuiserie/porte-coulissante-porte-interieure-escalier-et-rambarde/barriere-de-securite-l1308218463", "first_time" : "2017-01-10T21:22:52.982Z", "last_time" : "2017-01-10T21:25:04.356Z", "exit_path" : "http://www.leroymerlin.fr/v3/p/produits/materiaux-menuiserie/porte-coulissante-porte-interieure-escalier-et-rambarde/barriere-de-securite-l1308218463?__result-wrapper?pageTemplate=Famille%2FMat%C3%A9riaux+et+menuiserie&resultOffset=0&resultLimit=50&resultListShape=PLAIN&nomenclatureId=17942&priceStyle=SALEUNIT_PRICE&fcr=1&*4294718806=4294718806&*14072=14072&*4294718593=4294718593&*17942=17942" }, "distinct_paths" : 1.0, "sessionid" : "000046f8154a80fd89849369c984b8cc9d795814" }, { "num_pages_viewed" : 10.0, "session_details" : { "duration" : 343112, "first_referrer" : "https://www.google.fr/", "entry_page" : "http://www.leroymerlin.fr/", "first_time" : "2017-01-10T16:57:39.937Z", "last_time" : "2017-01-10T17:03:23.049Z", "exit_path" : "http://www.leroymerlin.fr/v3/p/produits/porte-de-douche-coulissante-adena-e168578" }, "distinct_paths" : 8.0, "sessionid" : "000087e825da1d87a332b8f15fa76116c7467da6" } ...