Adjacency matrix aggregation
editAdjacency matrix aggregation
editA bucket aggregation returning a form of adjacency matrix.
The request provides a collection of named filter expressions, similar to the filters aggregation
request.
Each bucket in the response represents a non-empty cell in the matrix of intersecting filters.
Given filters named A, B and C the response would return buckets with the following names:
| A | B | C | |
|---|---|---|---|
A |
A |
A&B |
A&C |
B |
B |
B&C |
|
C |
C |
The intersecting buckets e.g A&C are labelled using a combination of the two filter names with a default separator
of &. Note that the response does not also include a C&A bucket as this would be the
same set of documents as A&C. The matrix is said to be symmetric so we only return half of it. To do this we sort
the filter name strings and always use the lowest of a pair as the value to the left of the separator.
Example
editThe following interactions aggregation uses adjacency_matrix to determine
which groups of individuals exchanged emails.
resp = client.bulk(
index="emails",
refresh=True,
operations=[
{
"index": {
"_id": 1
}
},
{
"accounts": [
"hillary",
"sidney"
]
},
{
"index": {
"_id": 2
}
},
{
"accounts": [
"hillary",
"donald"
]
},
{
"index": {
"_id": 3
}
},
{
"accounts": [
"vladimir",
"donald"
]
}
],
)
print(resp)
resp1 = client.search(
index="emails",
size=0,
aggs={
"interactions": {
"adjacency_matrix": {
"filters": {
"grpA": {
"terms": {
"accounts": [
"hillary",
"sidney"
]
}
},
"grpB": {
"terms": {
"accounts": [
"donald",
"mitt"
]
}
},
"grpC": {
"terms": {
"accounts": [
"vladimir",
"nigel"
]
}
}
}
}
}
},
)
print(resp1)
response = client.bulk(
index: 'emails',
refresh: true,
body: [
{
index: {
_id: 1
}
},
{
accounts: [
'hillary',
'sidney'
]
},
{
index: {
_id: 2
}
},
{
accounts: [
'hillary',
'donald'
]
},
{
index: {
_id: 3
}
},
{
accounts: [
'vladimir',
'donald'
]
}
]
)
puts response
response = client.search(
index: 'emails',
body: {
size: 0,
aggregations: {
interactions: {
adjacency_matrix: {
filters: {
"grpA": {
terms: {
accounts: [
'hillary',
'sidney'
]
}
},
"grpB": {
terms: {
accounts: [
'donald',
'mitt'
]
}
},
"grpC": {
terms: {
accounts: [
'vladimir',
'nigel'
]
}
}
}
}
}
}
}
)
puts response
const response = await client.bulk({
index: "emails",
refresh: "true",
operations: [
{
index: {
_id: 1,
},
},
{
accounts: ["hillary", "sidney"],
},
{
index: {
_id: 2,
},
},
{
accounts: ["hillary", "donald"],
},
{
index: {
_id: 3,
},
},
{
accounts: ["vladimir", "donald"],
},
],
});
console.log(response);
const response1 = await client.search({
index: "emails",
size: 0,
aggs: {
interactions: {
adjacency_matrix: {
filters: {
grpA: {
terms: {
accounts: ["hillary", "sidney"],
},
},
grpB: {
terms: {
accounts: ["donald", "mitt"],
},
},
grpC: {
terms: {
accounts: ["vladimir", "nigel"],
},
},
},
},
},
},
});
console.log(response1);
PUT emails/_bulk?refresh
{ "index" : { "_id" : 1 } }
{ "accounts" : ["hillary", "sidney"]}
{ "index" : { "_id" : 2 } }
{ "accounts" : ["hillary", "donald"]}
{ "index" : { "_id" : 3 } }
{ "accounts" : ["vladimir", "donald"]}
GET emails/_search
{
"size": 0,
"aggs" : {
"interactions" : {
"adjacency_matrix" : {
"filters" : {
"grpA" : { "terms" : { "accounts" : ["hillary", "sidney"] }},
"grpB" : { "terms" : { "accounts" : ["donald", "mitt"] }},
"grpC" : { "terms" : { "accounts" : ["vladimir", "nigel"] }}
}
}
}
}
}
The response contains buckets with document counts for each filter and combination of filters. Buckets with no matching documents are excluded from the response.
{
"took": 9,
"timed_out": false,
"_shards": ...,
"hits": ...,
"aggregations": {
"interactions": {
"buckets": [
{
"key":"grpA",
"doc_count": 2
},
{
"key":"grpA&grpB",
"doc_count": 1
},
{
"key":"grpB",
"doc_count": 2
},
{
"key":"grpB&grpC",
"doc_count": 1
},
{
"key":"grpC",
"doc_count": 1
}
]
}
}
}
Parameters
edit-
filters -
(Required, object) Filters used to create buckets.
Properties of
filters-
<filter> -
(Required, Query DSL object) Query used to filter documents. The key is the filter name.
At least one filter is required. The total number of filters cannot exceed the
indices.query.bool.max_clause_countsetting. See Filter limits.
-
-
separator -
(Optional, string)
Separator used to concatenate filter names. Defaults to
&.
Response body
edit-
key -
(string)
Filters for the bucket. If the bucket uses multiple filters, filter names are
concatenated using a
separator. -
doc_count - (integer) Number of documents matching the bucket’s filters.
Usage
editOn its own this aggregation can provide all of the data required to create an undirected weighted graph.
However, when used with child aggregations such as a date_histogram the results can provide the
additional levels of data required to perform dynamic network analysis
where examining interactions over time becomes important.
Filter limits
editFor N filters the matrix of buckets produced can be N²/2 which can be costly.
The circuit breaker settings prevent results producing too many buckets and to avoid excessive disk seeks
the indices.query.bool.max_clause_count setting is used to limit the number of filters.