Elastic on Elastic: Configuring the Security app to use Cross Cluster Search

The Elastic Infosec Detections and Analytics team is responsible for building, tuning, and maintaining the security detections used to protect all Elastic systems. Within Elastic we call ourselves Customer Zero and we strive to always use the newest versions of our products. 

In the previous blog posts we gave an overview of our architecture and what data we send to our clusters. In this blog post we will provide instructions on how we use Cross Cluster Search (CCS) with the Security and Machine Learning (ML) applications

Configuring CCS

Getting the Security app to work with CCS only requires a few minor changes. The first step is to configure the Security app so that it knows to look for the CCS index patterns instead of local index patterns. To do this open the Kibana ‘Advanced Settings’ within the Stack Management menu, then located within the Security Solution section update the Elasticsearch indices. These should be set to match the CCS index patterns such as *:auditbeat-*, *:filebeat-*, *:logs-*, and any other index patterns you want to add to your Security app. 

Now when you return to the Security app you should see the Overview, Hosts, and Network pages displaying events from all of your remote clusters.

Configuring Built In Detection Rules to use CCS

Creating brand new custom Detection rules does not require any additional steps, the CCS index patterns will be available to select when you create a new detection rule. If you want to use the 500+ built in Detection rules with CCS it requires a little extra work. To do this you will need to import, duplicate, and bulk edit all of the rules to change the index patterns to use Cross Cluster Search. 

The first step is to load all of the pre-built Elastic rules into your Security App. Within the detections tab of the security app navigate to the ‘Manage Detections’ page and click the ‘Load Elastic prebuilt rules’ button.

The Prebuilt rules can not be directly modified but they can be duplicated and then the duplicates can be modified. To do this you will need to select all of the rules and use the ‘Bulk Actions’ menu to duplicate them all. Before starting this process I recommend changing the ‘refresh settings’ to disable automatically refreshing the table. If you don’t then the table may refresh automatically while you are trying to duplicate the rules causing you to deselect some rules.

To Select all rules on clusters older than 7.14 you will need to change the ‘Rows per page’ to the maximum amount.

When all of the rules are displayed select all rules and then in the bulk actions menu select ‘Duplicate Selected’. Duplicating all of the rules may take a couple minutes.

When this is complete you will have two copies of every rule. Using the filter on the right side of the rule table select only the ‘Custom rules’ that you just created

With only the Custom Rules displayed ‘select all’ again and select ‘Export Selected’ from the ‘bulk actions’ menu to download all of the rules as an ndjson file. 

Within this ndjson file you will need to find and replace all instances of the normal index patterns with the CCS index pattern. For example, to use the remote auditbeat index pattern you will need to find "auditbeat-*" and replace it with "*:auditbeat-*".  Repeat this for the other index patterns until all of the index patterns have been changed to the new CCS pattern.

I also recommend replacing the word ‘[Duplicate]’ that was appended to every rule name with the current stack version. This will help you manage your rules over time and track when each rule was installed or last updated.

After all of the changes have been saved to the custom rules ndjson file you can import the modified rules over the existing rules. Click the ‘Import rule’ button to open the rule import interface

Drag and drop the modified rule file to the pop up window and select the button to ‘Automatically overwrite saved objects with the same rule ID’. If you forget to do this it will create new rules instead of updating the existing rules and you will probably want to clean up all of the unnecessary rules.

After the rules have all been re-imported you can activate all of the rules that are applicable to your environment.

Note on Elastic Query Language (EQL)

EQL support for CCS is available as of Elastic release 7.14, but it requires that all of your remote clusters have also been upgraded to 7.14 or newer. Many of the built in Elastic detection rules use EQL so if you are using CCS with clusters older than 7.14 you will need to disable those rules.

Machine Learning with CCS

To use the Machine Learning with CCS you will need to update each of the datafeeds to use the CCS index patterns. When creating your own custom Machine Learning jobs this is easy to do, you simply select the CCS index pattern when creating the new job. To use the built in Security Machine Learning jobs requires modifying the datafeeds of each job to use the CCS index patterns. Because you cannot change a datafeed’s index pattern via the UI, doing this requires access to the Kibana Devtools or API access. My method for doing this is to load the built-in Machine Learning jobs, stop all of the jobs, and then use the Machine Learning API to Get the datafeed, make the change to the datafeed’s Index pattern, and then update the datafeed with the changes. After this is complete all of the ML jobs and datafeeds can be restarted. If you used the built in Security ML jobs you can now enable the built in Detection rules that use Machine Learning.

Conclusion

In this post we showed you how to configure your Security Application and Machine Learning jobs to work with Cross Cluster Search. Keep an eye out for future blog posts from the Elastic Infosec team on how we use Elastic to protect Elastic.