Capture Thrift traffic

Apache Thrift is a communication protocol and RPC framework initially created at Facebook. It is sometimes used in microservices architectures because it provides better performance when compared to the more obvious HTTP/RESTful API choice, while still supporting a wide range of programming languages and frameworks.

Packetbeat works based on a copy of the traffic, which means that you get performance management features without having to modify your services in any way and without any latency overhead. Packetbeat captures the transactions from the network and indexes them in Elasticsearch so that they can be analyzed and searched.

Packetbeat indexes the method, parameters, return value, and exceptions of each Thrift-RPC call. You can search by and create statistics based on any of these fields. Packetbeat automatically fills in the status column with either OK or Error, so it’s easy to find the problematic RPC calls. A transaction is put into the Error state if it returned an exception.

Packetbeat also indexes the event.duration field so you can get performance analytics and find the slow RPC calls.

Here is an example performance dashboard:

Thrift supports multiple transport and protocol types. Currently Packetbeat supports the default TSocket transport as well as the TFramed transport. From the protocol point of view, Packetbeat currently supports only the default TBinary protocol.

Packetbeat also has several configuration options that allow you to get the right balance between visibility, disk usage, and data protection. You can, for example, choose to obfuscate all strings or to store the requests but not the responses, while still capturing the response time for each of the RPC calls. You can also choose to limit the size of strings and lists to a given number of elements, so you can fine tune how much data you want to have stored in Elasticsearch.

The Thrift protocol has several specific configuration options. Here is an example configuration section for the Thrift protocol in the packetbeat.yml config file:

packetbeat.protocols:
- type: thrift
  transport_type: socket
  protocol_type: binary
  idl_files: ["tutorial.thrift", "shared.thrift"]
  string_max_size: 200
  collection_max_size: 20
  capture_reply: true
  obfuscate_strings: true
  drop_after_n_struct_fields: 100

Providing the Thrift IDL files to Packetbeat is optional. The binary Thrift messages include the called method name and enough structural information to decode the messages without needing the IDL files. However, if you provide the IDL files, Packetbeat can also resolve the service name, arguments, and exception names.