Within the French Ministry of Agriculture and Food (the Ministry), our team of architects in the Methods, Support and Quality office (BMSQ) evaluate and supply software solutions to resolve issues encountered by project teams that affect various disciplines.

As data specialists, one area we’ve been involved in includes reconfiguring the traceability of activities for the commercial fishing industry. The aim is to improve the quality, speed and precision of how we collect and analyze large volumes of data connected to the industry — from declared fish hauls, harbor exit manifests, to GPS data tracking vessel locations.

The challenge we have is to provide up-to-date information that is verified, complete, dynamic and can be viewed in various formats depending on the target audience, and Elastic is the solution. Elastic handles ingesting all of our data and render visualizations to make it easy to share and use real-time information about commercial fishing activity. With this integrated data, we can take enforcement action, stop illegal fishing, and negotiate fishing rights with our neighboring countries.

Why we choose Elastic over Splunk and Graylog

The first stage of the project was to perform a proof of concept with a solution capable of storing, extracting, and presenting the data related to fishing activities in real time.

We benchmarked the Elastic Stack against other tools, such as Graylog and Splunk. We were ultimately won over by the Elastic Stack’s speed and ease of use, along with its power and scalability. The data presentation and visualization tools, Canvas, and Kibana, have also played crucial roles in this project, enabling us to efficiently provide information for our end users in the context of increasingly strict protective fishing regulations and measures. In addition, our Elastic subscription, reduces our development time and allows us to focus on our real work, thanks to the support team at Elastic.

Casting a wide regulatory fishing net with Elastic

With GPS systems required on fishing vessels larger than 12 meters (39 feet), we have been able to track boats while at the same time indexing that monitoring data into Elasticsearch, where it is visualized in either Kibana or Canvas. This enables us to help Ministry officials on several enforcement levels:

• Locating activity zones of boats and areas of intense fishing
• Monitoring fishing quotas in FAO (Food and Agriculture Organization) zones 
• Flagging infringements of the law

With Canvas, we can refine the granularity, quality, and format to ensure the fishery data is presented in the most suitable format for the audience, especially a non-technical audience.

Example of the infographics generated using Canvas to monitor fishing data of cod, sardine, and tuna

We could not render presentations like this with our legacy tools, which were conventional databases and a Java application. They were at their limits in terms of the required performance due to the number of filter fields, 300 and counting. Now, once the data has been processed in Logstash, stored and indexed in Elasticsearch, it can be filtered, cross-referenced and correlated in real time. 

Elastic gives us the ability to verify the precision and compliance of statements declared by boats compared with actual recorded events.

We are storing our raw data for 10 years. This amounts to 135 million records in Elasticsearch. In addition, each record contains more than 300 filter fields. We receive raw ERS (Electronic Reporting System) data in XML format, as issued by the boats using onboard software or GPS, and we model this data as it flows in so we can integrate it into our Elasticsearch cluster.

How Elastic fits in the architectural layout of the French Ministry of Agriculture

This sea of information allows us to pinpoint quantities and species fished, rejections of protected species, type of boat plus its flag, registration and equipment, fishing quotas per territorial area, satellite operator depending on the region of the globe, and much more. 

Real-time information detailed by region is the basis for consolidated analyses and discussion to facilitate immediate remedying of any infringements of the law, rapid reaction to media controversy relating to protected marine species, and even the renegotiation of quotas each year within the European Union.

Expanding to political, economical and environmental use cases 

The Ministry is continuing to closely monitor new releases of the Elastic Stack. We are anticipating the availability of a French version of Kibana, which would expand the solution’s user group. The most recent functionalities provided in versions 7.11 and 7.12 of Elastic are being tested with great interest — in particular the addition of the tracks layer in the Maps application. This feature takes an index of point locations, ordered by time, and displays them as a line, enabling us to track the route taken by boats, as shown below:

We have been grateful for the ease of use, flexibility, and creativity of Kibana and Canvas in enabling stakeholders to remain well informed and to react rapidly to increasingly stringent protective regulations and measures. Moreover, the fishing data provided in XML format is modeled by batch, which enables requests to be processed via Elasticsearch on a continuously growing volume of data, without having to wait for all 10 years of the current stored data to be processed. Elastic also enables the indexation process to be repeated each time new data is added to existing fields. 

In the short term, the Ministry’s aim is to conclude the roll-out of this solution and open it up to a growing group of users.

Beyond fishing data, we are going to need to store, process, and analyze increasing volumes of tracking data, particularly in regard to food — from farm to table. All of which means that Elastic could be of use for these new development projects with high political, economical and environmental stakes.

Sébastien Arnaud — Exchanges & Data architect at the French Ministry of Agriculture

Following initial training in the field of networks and IT security, he has worked on complex data exchange and transformation solutions. He likes to design and integrate innovative architectures for processing, storing and evaluating increasingly large volumes of data.