Want to jump right into the MITRE Engenuity ATT&CK® Eval Round 3 Kibana dashboards? Skip ahead to see the results.

Fresh out: the latest MITRE Engenuity Evaluation

Whether this is the third time you are looking at the MITRE Engenuity ATT&CK® evaluation results or your first, you may be asking yourself: what was unique about this year’s evaluation?

Well, let’s first start with: who is MITRE Engenuity? They are a tech foundation that collaborates with the private sector on many initiatives — most notably cybersecurity — and in recent years have become synonymous with cyber threat evaluations. Starting with APT3 in 2018 and then APT29 in 2019, MITRE Engenuity has continued to evaluate security vendors in their effectiveness at detecting adversary behaviors — especially those techniques as outlined by the MITRE ATT&CK framework.

Each year brings a new APT group and a new focus. This year’s Round 3 was based on Carbanak and Fin7, two threat groups known for “both sophisticated malware as well as legitimate administration tools capable of interacting with various platforms,“ per MITRE Engenuity. During the evaluations, we saw this in full effect, with the red team disguising routine operations by using many living-off-the-land techniques. This year also marked the first evaluation* where Linux systems were attacked, testing the visibility of security vendors on critical business infrastructure.

Interpreting the results yourself

Being open and transparent is part of our Elastic DNA. As we did last year, we’re extending that transparency to the analysis of this year’s evaluation results. We first converted all the vendor results into singular detection events — a single doc outlining all aspects of the detection outlined by MITRE Engenuity. From here, we used the power of Kibana to easily create visualizations of the data. More specifically, we used Kibana Lens, our easy-to-use visualization tool that allowed us to drag and drop fields of interest such as “Detection Type.”

Check out the dashboard with results of all participating vendors. After selecting vendor(s) of interest in the dropdown control list, your graphs will be generated. The visualizations will show a logical breakdown of detections by type per vendor.

Most notably is the first graph (shown below), which shows unique detections per substep. MITRE Engenuity defined this as “Visibility” in the vendor results data tables, which is defined as “the proportion of substeps with either an analytic (Technique, Tactic, or General) or telemetry detection.” This is important as many steps during this eval can create multiple detections per step. 

For instance, think about PowerShell being executed from a suspicious process. From a shell? With suspicious command line arguments? Did it generate network activity? All of these things can create alerts, so we uniqued the substeps to provide a clearer understanding of the results.

The evolution of Elastic Security, free and open

As you may recall, in 2019, Elastic joined forces with Endgame under the banner of Elastic Security — our free and open unified security solution for SIEM, security analytics, endpoint security, threat hunting, cloud monitoring, and more.

Our goal in late 2019 was to make endpoint security available to everyone using a single architecture. That goal was not only achieved in mid-2020, but also in time to undergo evaluation by MITRE for this third round! 

You will notice in this year’s results, we didn’t build a basic third-party integration of Endgame technology shoehorned in with the Elastic Stack. Rather, we went through the hard work of embedding endpoint security functionality directly into the new Elastic Agent. This means that with Elastic Security, security teams get the benefit of working off the same data store within a single architecture.

This embedded convergence is critical because protecting your environment is not a one-dimensional problem. Endpoint technology alone is not enough, and neither is traditional SIEM or security analytics as a standalone approach. Teams need to invest in holistic visibility of their environments — the deeper and more comprehensive the visibility, the better.

Our approach leverages holistic visibility across network logs, application performance monitoring, cloud-based authentication records, and more — all while providing much richer telemetry at the host to better address threats in the era of cloud with a highly distributed workforce. 

In addition to driving this convergence, Elastic Security provides free and open security features to equip SOC teams to prevent, detect, and respond to threats at scale — without the high degree of friction or prohibitive costs associated with starting down the path to operational maturity. Users can get started — or even build an entire solution — at no cost. Our code is housed in public repositories with a commitment to an open development process and transparent and direct engagement with our community. For example, in the /elastic/detection-rules GitHub repository, you can find rules written for Elastic Security, with coverage for many ATT&CK techniques. 

A free and open model ensures that our community and customers can innovate, evolve, and optimize the solution for their unique environment. Learn more about our corporate philosophy on the importance of an open approach to developing and delivering security capabilities. For this year’s MITRE Engenuity evaluation, testing was conducted using the capabilities available in the free and open tier (known as the Standard tier) of Elastic Security.

What’s new since the evaluation?

As this evaluation was kicking off back in October 2020, Elastic had submitted our first Beta release of the new architecture in Elastic Agent for the evaluation. Since then, we’ve launched three new releases! MITRE Engenuity tested Elastic v7.9, but v7.12 was released in March 2021. 

So what are the most notable differences between then and now?

• New endpoint security data source integrations (many of which were evaluated) including Crowdstrike, Symantec, Sophos, Microsoft, Cisco, Cylance, and many more.
• Out-of-the-box rules. Each release we ship more prebuilt detection logic in our public rules repository and there are 300+ more rules since 7.9 (now totaling 540+). Many are aligned to the ATT&CK Framework, which MITRE has referenced in their Cyber Analytics Repository (CAR). But also keep in mind, many rules are for other, non-ATT&CK-based use cases, like traditional SecOps.
• Machine learning. We also have emphasized our machine learning capabilities with the release of supervised learning models to support our existing unsupervised capabilities — most notably this DGA detection technique. 
• Correlation with Event Query Language (EQL). You can now search and create rules using EQL. As you may recall from previous evaluations, this technology is critical to our high fidelity detections, as we are able to correlate different events by using sequential detection logic. Dive into EQL.
• Elastic Agent. Ever feel the pain of juggling multiple products? Installations? Updates? We do too. So now with Elastic Agent, adding a new data type is as easy as clicking a button. For instance, you can select the Endpoint Security Integration with a single click. And if you are so inclined, it's a simple checkbox for PowerShell eventing too ;).
• Ransomware behavioral prevention and Master Boot Record (MBR) protection.
• ...and so much more. View our release blogs.

Thanks MITRE Engenuity!

As a vendor, it’s an exhilarating experience to have your software tested by MITRE Engenuity’s red team. It’s a fantastic way to assess our ATT&CK-based detections and our visibility capabilities. We were proud with the performance of Elastic Security this year and extend our thanks to MITRE Engenuity for another exciting evaluation. We look forward to the next one! 

Ready to give Elastic Security a spin? Try it free today!

*In round 3, MITRE also included protection testing for the first time. Elastic did not participate in the prevention portion of the tests during this round. Our free and open malware prevention would have also actively blocked many of these TTPs.