Doc counts and overlapping jobs

There is an issue with doc counts, related to the above grouping limitation. Imagine you have two Rollup jobs saving to the same index, where one job is a "subset" of another job.

For example, you might have jobs with these two groupings:

PUT _xpack/rollup/job/sensor-all
{
    "groups" : {
      "date_histogram": {
        "field": "timestamp",
        "interval": "1h",
        "delay": "7d"
      },
      "terms": {
        "fields": ["node"]
      }
    },
    "metrics": [
        {
            "field": "price",
            "metrics": ["avg"]
        }
    ]
    ...
}

and

PUT _xpack/rollup/job/sensor-building
{
    "groups" : {
      "date_histogram": {
        "field": "timestamp",
        "interval": "1h",
        "delay": "7d"
      },
      "terms": {
        "fields": ["node", "building"]
      }
    }
    ...
}

The first job sensor-all contains the groupings and metrics that apply to all data in the index. The second job is rolling up a subset of data (in different buildings) which also include a building identifier. You did this because combining them would run into the limitation described in the previous section.

This mostly works, but can sometimes return incorrect doc_counts when you search. All metrics will be valid however.

The issue arises from the composite agg limitation described before, combined with search-time optimization. Imagine you try to run the following aggregation:

"aggs" : {
  "nodes": {
    "terms": {
      "field": "node"
    }
  }
}

This aggregation could be serviced by either sensor-all or sensor-building job, since they both group on the node field. So the RollupSearch API will search both of them and merge results. This will result in correct doc_counts and correct metrics. No problem here.

The issue arises from an aggregation that can only be serviced by sensor-building, like this one:

"aggs" : {
  "nodes": {
    "terms": {
      "field": "node"
    },
    "aggs": {
      "building": {
        "terms": {
          "field": "building"
        }
      }
    }
  }
}

Now we run into a problem. The RollupSearch API will correctly identify that only sensor-building job has all the required components to answer the aggregation, and will search it exclusively. Unfortunately, due to the composite aggregation limitation, that job only rolled up documents that have both a "node" and a "building" field. Meaning that the doc_counts for the "nodes" aggregation will not include counts for any document that doesn’t have [node, building] fields.

"aggs" : {
  "nodes": {
    "terms": {
      "field": "node"
    },
    "aggs": {
      "building": {
        "terms": {
          "field": "building"
        }
      },
      "avg_price": {
        "avg": { "field": "price" } 
      }
    }
  }
}