Painless examples for transforms
editPainless examples for transforms
editThe examples that use the scripted_metric
aggregation are not supported on Elasticsearch Serverless.
These examples demonstrate how to use Painless in transforms. You can learn more about the Painless scripting language in the Painless guide.
- Getting top hits by using scripted metric aggregation
- Getting time features by using aggregations
- Getting duration by using bucket script
- Counting HTTP responses by using scripted metric aggregation
- Comparing indices by using scripted metric aggregations
- Getting web session details by using scripted metric aggregation
- While the context of the following examples is the transform use case, the Painless scripts in the snippets below can be used in other Elasticsearch search aggregations, too.
- All the following examples use scripts, transforms cannot deduce mappings of output fields when the fields are created by a script. Transforms don’t create any mappings in the destination index for these fields, which means they get dynamically mapped. Create the destination index prior to starting the transform in case you want explicit mappings.
Getting top hits by using scripted metric aggregation
editThis snippet shows how to find the latest document, in other words the document with the latest timestamp. From a technical perspective, it helps to achieve the function of a Top hits by using scripted metric aggregation in a transform, which provides a metric output.
This example uses a scripted_metric
aggregation which is not supported on Elasticsearch Serverless.
"aggregations": { "latest_doc": { "scripted_metric": { "init_script": "state.timestamp_latest = 0L; state.last_doc = ''", "map_script": """ def current_date = doc['@timestamp'].getValue().toInstant().toEpochMilli(); if (current_date > state.timestamp_latest) {state.timestamp_latest = current_date; state.last_doc = new HashMap(params['_source']);} """, "combine_script": "return state", "reduce_script": """ def last_doc = ''; def timestamp_latest = 0L; for (s in states) {if (s.timestamp_latest > (timestamp_latest)) {timestamp_latest = s.timestamp_latest; last_doc = s.last_doc;}} return last_doc """ } } }
The |
|
The |
|
The |
|
The |
Check the scope of scripts for detailed explanation on the respective scripts.
You can retrieve the last value in a similar way:
"aggregations": { "latest_value": { "scripted_metric": { "init_script": "state.timestamp_latest = 0L; state.last_value = ''", "map_script": """ def current_date = doc['@timestamp'].getValue().toInstant().toEpochMilli(); if (current_date > state.timestamp_latest) {state.timestamp_latest = current_date; state.last_value = params['_source']['value'];} """, "combine_script": "return state", "reduce_script": """ def last_value = ''; def timestamp_latest = 0L; for (s in states) {if (s.timestamp_latest > (timestamp_latest)) {timestamp_latest = s.timestamp_latest; last_value = s.last_value;}} return last_value """ } } }
Getting top hits by using stored scripts
editYou can also use the power of stored scripts to get the latest value. Stored scripts reduce compilation time, make searches faster, and are updatable.
-
Create the stored scripts:
POST _scripts/last-value-map-init { "script": { "lang": "painless", "source": """ state.timestamp_latest = 0L; state.last_value = '' """ } } POST _scripts/last-value-map { "script": { "lang": "painless", "source": """ def current_date = doc['@timestamp'].getValue().toInstant().toEpochMilli(); if (current_date > state.timestamp_latest) {state.timestamp_latest = current_date; state.last_value = doc[params['key']].value;} """ } } POST _scripts/last-value-combine { "script": { "lang": "painless", "source": """ return state """ } } POST _scripts/last-value-reduce { "script": { "lang": "painless", "source": """ def last_value = ''; def timestamp_latest = 0L; for (s in states) {if (s.timestamp_latest > (timestamp_latest)) {timestamp_latest = s.timestamp_latest; last_value = s.last_value;}} return last_value """ } }
-
Use the stored scripts in a scripted metric aggregation.
Getting time features by using aggregations
editThis snippet shows how to extract time based features by using Painless in a
transform. The snippet uses an index where @timestamp
is defined as a date
type field.
"aggregations": { "avg_hour_of_day": { "avg":{ "script": { "source": """ ZonedDateTime date = doc['@timestamp'].value; return date.getHour(); """ } } }, "avg_month_of_year": { "avg":{ "script": { "source": """ ZonedDateTime date = doc['@timestamp'].value; return date.getMonthValue(); """ } } }, ... }
Name of the aggregation. |
|
Contains the Painless script that returns the hour of the day. |
|
Sets |
|
Returns the hour value from |
|
Name of the aggregation. |
|
Contains the Painless script that returns the month of the year. |
|
Sets |
|
Returns the month value from |
Getting duration by using bucket script
editThis example shows you how to get the duration of a session by client IP from a data log by using bucket script. The example uses the Kibana sample web logs dataset.
resp = client.transform.put_transform( transform_id="data_log", source={ "index": "kibana_sample_data_logs" }, dest={ "index": "data-logs-by-client" }, pivot={ "group_by": { "machine.os": { "terms": { "field": "machine.os.keyword" } }, "machine.ip": { "terms": { "field": "clientip" } } }, "aggregations": { "time_frame.lte": { "max": { "field": "timestamp" } }, "time_frame.gte": { "min": { "field": "timestamp" } }, "time_length": { "bucket_script": { "buckets_path": { "min": "time_frame.gte.value", "max": "time_frame.lte.value" }, "script": "params.max - params.min" } } } }, ) print(resp)
const response = await client.transform.putTransform({ transform_id: "data_log", source: { index: "kibana_sample_data_logs", }, dest: { index: "data-logs-by-client", }, pivot: { group_by: { "machine.os": { terms: { field: "machine.os.keyword", }, }, "machine.ip": { terms: { field: "clientip", }, }, }, aggregations: { "time_frame.lte": { max: { field: "timestamp", }, }, "time_frame.gte": { min: { field: "timestamp", }, }, time_length: { bucket_script: { buckets_path: { min: "time_frame.gte.value", max: "time_frame.lte.value", }, script: "params.max - params.min", }, }, }, }, }); console.log(response);
PUT _transform/data_log { "source": { "index": "kibana_sample_data_logs" }, "dest": { "index": "data-logs-by-client" }, "pivot": { "group_by": { "machine.os": {"terms": {"field": "machine.os.keyword"}}, "machine.ip": {"terms": {"field": "clientip"}} }, "aggregations": { "time_frame.lte": { "max": { "field": "timestamp" } }, "time_frame.gte": { "min": { "field": "timestamp" } }, "time_length": { "bucket_script": { "buckets_path": { "min": "time_frame.gte.value", "max": "time_frame.lte.value" }, "script": "params.max - params.min" } } } } }
To define the length of the sessions, we use a bucket script. |
|
The bucket path is a map of script variables and their associated path to
the buckets you want to use for the variable. In this particular case, |
|
Finally, the script substracts the start date of the session from the end date which results in the duration of the session. |
Counting HTTP responses by using scripted metric aggregation
editYou can count the different HTTP response types in a web log data set by using scripted metric aggregation as part of the transform. You can achieve a similar function with filter aggregations, check the Finding suspicious client IPs example for details.
The example below assumes that the HTTP response codes are stored as keywords in
the response
field of the documents.
This example uses a scripted_metric
aggregation which is not supported on Elasticsearch Serverless.
"aggregations": { "responses.counts": { "scripted_metric": { "init_script": "state.responses = ['error':0L,'success':0L,'other':0L]", "map_script": """ def code = doc['response.keyword'].value; if (code.startsWith('5') || code.startsWith('4')) { state.responses.error += 1 ; } else if(code.startsWith('2')) { state.responses.success += 1; } else { state.responses.other += 1; } """, "combine_script": "state.responses", "reduce_script": """ def counts = ['error': 0L, 'success': 0L, 'other': 0L]; for (responses in states) { counts.error += responses['error']; counts.success += responses['success']; counts.other += responses['other']; } return counts; """ } }, ... }
The |
|
Object of the |
|
This |
|
The |
|
The |
|
The |
|
The |
Comparing indices by using scripted metric aggregations
editThis example shows how to compare the content of two indices by a transform that uses a scripted metric aggregation.
This example uses a scripted_metric
aggregation which is not supported on Elasticsearch Serverless.
resp = client.transform.preview_transform( id="index_compare", source={ "index": [ "index1", "index2" ], "query": { "match_all": {} } }, dest={ "index": "compare" }, pivot={ "group_by": { "unique-id": { "terms": { "field": "<unique-id-field>" } } }, "aggregations": { "compare": { "scripted_metric": { "map_script": "state.doc = new HashMap(params['_source'])", "combine_script": "return state", "reduce_script": " \n if (states.size() != 2) {\n return \"count_mismatch\"\n }\n if (states.get(0).equals(states.get(1))) {\n return \"match\"\n } else {\n return \"mismatch\"\n }\n " } } } }, ) print(resp)
const response = await client.transform.previewTransform({ id: "index_compare", source: { index: ["index1", "index2"], query: { match_all: {}, }, }, dest: { index: "compare", }, pivot: { group_by: { "unique-id": { terms: { field: "<unique-id-field>", }, }, }, aggregations: { compare: { scripted_metric: { map_script: "state.doc = new HashMap(params['_source'])", combine_script: "return state", reduce_script: ' \n if (states.size() != 2) {\n return "count_mismatch"\n }\n if (states.get(0).equals(states.get(1))) {\n return "match"\n } else {\n return "mismatch"\n }\n ', }, }, }, }, }); console.log(response);
POST _transform/_preview { "id" : "index_compare", "source" : { "index" : [ "index1", "index2" ], "query" : { "match_all" : { } } }, "dest" : { "index" : "compare" }, "pivot" : { "group_by" : { "unique-id" : { "terms" : { "field" : "<unique-id-field>" } } }, "aggregations" : { "compare" : { "scripted_metric" : { "map_script" : "state.doc = new HashMap(params['_source'])", "combine_script" : "return state", "reduce_script" : """ if (states.size() != 2) { return "count_mismatch" } if (states.get(0).equals(states.get(1))) { return "match" } else { return "mismatch" } """ } } } } }
The indices referenced in the |
|
The |
|
The |
|
Object of the |
|
The |
|
The |
|
The |
Getting web session details by using scripted metric aggregation
editThis example shows how to derive multiple features from a single transaction. Let’s take a look on the example source document from the data:
Source document
{ "_index":"apache-sessions", "_type":"_doc", "_id":"KvzSeGoB4bgw0KGbE3wP", "_score":1.0, "_source":{ "@timestamp":1484053499256, "apache":{ "access":{ "sessionid":"571604f2b2b0c7b346dc685eeb0e2306774a63c2", "url":"http://www.leroymerlin.fr/v3/search/search.do?keyword=Carrelage%20salle%20de%20bain", "path":"/v3/search/search.do", "query":"keyword=Carrelage%20salle%20de%20bain", "referrer":"http://www.leroymerlin.fr/v3/p/produits/carrelage-parquet-sol-souple/carrelage-sol-et-mur/decor-listel-et-accessoires-carrelage-mural-l1308217717?resultOffset=0&resultLimit=51&resultListShape=MOSAIC&priceStyle=SALEUNIT_PRICE", "user_agent":{ "original":"Mobile Safari 10.0 Mac OS X (iPad) Apple Inc.", "os_name":"Mac OS X (iPad)" }, "remote_ip":"0337b1fa-5ed4-af81-9ef4-0ec53be0f45d", "geoip":{ "country_iso_code":"FR", "location":{ "lat":48.86, "lon":2.35 } }, "response_code":200, "method":"GET" } } } } ...
By using the sessionid
as a group-by field, you are able to enumerate events
through the session and get more details of the session by using scripted metric
aggregation.
This example uses a scripted_metric
aggregation which is not supported on Elasticsearch Serverless.
POST _transform/_preview { "source": { "index": "apache-sessions" }, "pivot": { "group_by": { "sessionid": { "terms": { "field": "apache.access.sessionid" } } }, "aggregations": { "distinct_paths": { "cardinality": { "field": "apache.access.path" } }, "num_pages_viewed": { "value_count": { "field": "apache.access.url" } }, "session_details": { "scripted_metric": { "init_script": "state.docs = []", "map_script": """ Map span = [ '@timestamp':doc['@timestamp'].value, 'url':doc['apache.access.url'].value, 'referrer':doc['apache.access.referrer'].value ]; state.docs.add(span) """, "combine_script": "return state.docs;", "reduce_script": """ def all_docs = []; for (s in states) { for (span in s) { all_docs.add(span); } } all_docs.sort((HashMap o1, HashMap o2)->o1['@timestamp'].toEpochMilli().compareTo(o2['@timestamp'].toEpochMilli())); def size = all_docs.size(); def min_time = all_docs[0]['@timestamp']; def max_time = all_docs[size-1]['@timestamp']; def duration = max_time.toEpochMilli() - min_time.toEpochMilli(); def entry_page = all_docs[0]['url']; def exit_path = all_docs[size-1]['url']; def first_referrer = all_docs[0]['referrer']; def ret = new HashMap(); ret['first_time'] = min_time; ret['last_time'] = max_time; ret['duration'] = duration; ret['entry_page'] = entry_page; ret['exit_path'] = exit_path; ret['first_referrer'] = first_referrer; return ret; """ } } } } }
The data is grouped by |
|
The aggregations counts the number of paths and enumerate the viewed pages during the session. |
|
The |
|
The |
|
The |
|
The |
The API call results in a similar response:
{ "num_pages_viewed" : 2.0, "session_details" : { "duration" : 100300001, "first_referrer" : "https://www.bing.com/", "entry_page" : "http://www.leroymerlin.fr/v3/p/produits/materiaux-menuiserie/porte-coulissante-porte-interieure-escalier-et-rambarde/barriere-de-securite-l1308218463", "first_time" : "2017-01-10T21:22:52.982Z", "last_time" : "2017-01-10T21:25:04.356Z", "exit_path" : "http://www.leroymerlin.fr/v3/p/produits/materiaux-menuiserie/porte-coulissante-porte-interieure-escalier-et-rambarde/barriere-de-securite-l1308218463?__result-wrapper?pageTemplate=Famille%2FMat%C3%A9riaux+et+menuiserie&resultOffset=0&resultLimit=50&resultListShape=PLAIN&nomenclatureId=17942&priceStyle=SALEUNIT_PRICE&fcr=1&*4294718806=4294718806&*14072=14072&*4294718593=4294718593&*17942=17942" }, "distinct_paths" : 1.0, "sessionid" : "000046f8154a80fd89849369c984b8cc9d795814" }, { "num_pages_viewed" : 10.0, "session_details" : { "duration" : 343100405, "first_referrer" : "https://www.google.fr/", "entry_page" : "http://www.leroymerlin.fr/", "first_time" : "2017-01-10T16:57:39.937Z", "last_time" : "2017-01-10T17:03:23.049Z", "exit_path" : "http://www.leroymerlin.fr/v3/p/produits/porte-de-douche-coulissante-adena-e168578" }, "distinct_paths" : 8.0, "sessionid" : "000087e825da1d87a332b8f15fa76116c7467da6" } ...