BigID tames massive data growth with Elastic for scalable security through Elastic Cloud, on AWS

BigID, a data security, compliance, privacy, and AI data management leader, helps organizations manage an ever-expanding sea of information scattered across disparate cloud, software as a service, and on-premise environments. Its clients gain a comprehensive understanding of their data across the entire data landscape. This allows them to mitigate risk by addressing potential security vulnerabilities and reduce compliance costs by simplifying adherence to relevant regulations.

Managing ever-growing data volumes is an ongoing challenge, especially in complex hybrid architectures. In these situations, data search can become a struggle. For BigID, frequent schema migrations and index additions became necessary to maintain performance, but can be challenging at scale. This also hampered the ability to support ad-hoc searches across various data fields.

Tomer Negbi, Director of Engineering, BigID turned to Elastic. "As we grew in scale and adoption, the need for a more scalable search solution became paramount," he says. "We saw the potential of Elasticsearch, running on Elastic Cloud and AWS, for real-time, indexed search and analysis of unstructured data that doesn't use native indexing."

Negbi gives the example of a large insurance company with billions of records. BigID empowers the organization to filter and explore data with ease. For instance, performing a complex query, such as identifying objects with certain attributes within a particular date range and data source, would traditionally be inefficient and require significant pre-indexing, potentially exceeding the data size itself. Elasticsearch's strength lies in handling these scenarios efficiently.

In addition, customers can easily configure the pipeline through BigID's user interface, specifying the data sources and connectors to be scanned, with scans prioritized based on their needs.

"We provide out-of-the-box dashboards highlighting key metrics of interest to users," says Negbi. Customers can also collaborate with BigID to define and build custom metrics tailored to their needs.

For example, a client might discover that the majority of business risk is concentrated in just 10% of the data. Meanwhile, other clients might want to understand specific geographic locations of data to ensure compliance with regional privacy regulations.

By migrating data-driven modules from its existing data storage platform to Elastic, BigID has achieved significant performance improvements. "Queries which previously took 20 minutes now run in a matter of seconds, 120 times faster - that's a massive improvement," says Negbi.

Additionally, query timeouts disappeared, replaced by faster responses and an improved user experience. Negbi says, "This newfound agility allows us to develop innovative features, reduce costs, and remain at the forefront of our industry."

Negbi and his team also benefit from Elastic's flexibility and speed. Development teams were previously forced to spend considerable time troubleshooting and building workarounds. Adding Elasticsearch can potentially resolve these issues and significantly improve customer satisfaction.

Working closely with the Elastic team enables BigID to give feedback on new releases. Negbi says, "Our data is not time-series based, where older data has less importance." Instead, all discovered data is critical and needs to be readily accessible regardless of scan time. "The Elastic team has been instrumental in collaborating with us to develop solutions that effectively address these unique requirements," he says.

Elastic has consistently provided excellent technical support. "They are proactive in offering guidance and anticipating our needs, ensuring we stay ahead of the curve. This is crucial for a fast-paced startup like BigID where rapid innovation is essential," says Negbi.

• Serverless is here! Find out how you can take advantage of Elastic, without the headache of hardware purchasing and maintenance
• Learn how other customers use Elastic to search, observe and protect their business data
• Start a free, no obligation trial today and see for yourself why so many organizations trust Elastic