How to ingest data from Trello into Elastic Workplace Search

In our previous post, we introduced the concept of the Elastic Workplace Search Custom Source API as a means of adding data for which a prebuilt content source integration isn’t available. We used a simple example — a CSV file of contact information — to demonstrate the process along with the use of the associated REST API. In this post, we explore ingesting data from a more complex organizational source: Trello.

We use a bit of Python in this post to interact with the Trello and Workplace Search APIs. While you don’t need to be a developer to benefit from this post, more technical readers are welcome to follow along.

Why Trello?

Trello is a digital whiteboard, allowing users to track tasks through a Kanban-style board experience. Each Kanban board contains named lists such as “In Progress,” which in turn contain cards (to do items, effectively). Cards can move between lists and can also contain comments, attachments, due dates, and labels. The style of the board and available lists are configurable with a wide range of template offerings.

While commonly used by individuals to track their own “to dos”, Trello also offers the concepts of teams and board collaboration, thus potentially containing information relevant to the wider organization. By exposing this information within Workplace Search, your users are able to find tasks relevant to themselves, within a single unified experience. More importantly, making this information easily searchable — right alongside your other relevant sources of content — enhances the possibility of cross-team collaboration and reduction of duplicated work. For example, Trello data combined with quick search in Workplace Search can answer the common question “Is anyone else working or has worked on X?”

Getting Trello data into Workplace Search 

For the sake of simplicity this blog doesn’t aim to provide a comprehensive approach for ingesting all Trello data. Specifically, we focus on a subset for example purposes only:

• All team boards and their accompanying cards and comments. Any personal boards are ignored. Importantly we assume the user with whom you are accessing the Trello API has access to all of the boards you want to ingest.
• Useful metadata associated with each card. We ingest a subset of the fields including checklists, labels, comments, and links to attachments.
• Continued synchronization and security. The code samples provide an initial load capability only and assume all Workplace Search users have access to all Trello content. Possible approaches to security are suggested later.

Trello accounts can be created for free, if you don’t already have one. The code presented is configurable and can be easily adapted to ingest your Trello content to your Workplace Search instance. If you do create a Trello account, make sure to create a team and grant the team access to at least one board with content.

Creating a source

First up, you’ll create a new content source:

1. Navigate to the Sources page and click Add an organization content source.

2. Click Custom API Source.

3. Name your new source Trello and then click Create Custom API Source.

Make note of the access token and key, which you’ll need later. As a reminder the former provides authentication and is sensitive, while the latter uniquely identifies the custom source.

Setting up Trello authentication and authorization 

The Trello REST API also requires the user to generate an API key and token (see the Trello documentation). Assuming you’ve created an account, the former can be obtained by simply visiting this page and accepting the terms and conditions. Make a note of the developer API key at the top. This key on its own doesn’t provide access to Trello data and in itself isn’t sensitive. Under the API key, click the Token link to generate a token, noting the permissions you will be granting on the subsequent form. Again, record this token.

Using a script for ingest

Ingesting data into Workplace Search from Trello requires us to write at least a small script. In this case we’ve chosen to implement this in Python. In order to interact with Workplace Search we can use the official Python client.

The script assumes the user has access to all of the teams, and their associated boards, we wish to ingest. With this assumption, it simply iterates over each of the available teams and in turn their boards. Finally for each board, the cards and associated comments are fetched. In summary:

get_object(): 
for each team: 
 for each board in team: 
 yield board 
 for each card in board: 
 yield card 
 for each comment in card: 
 yield comment 
for each obj in get_objects(): 
workplace_client.index(obj)

Where yield produces a document for indexing. A few technical notes on the script for the more avid reader:

• The script requires the environment variables TRELLO_API_KEY, TRELLO_TOKEN, WORKPLACE_SEARCH_TOKEN, and WORKPLACE_SEARCH_URL to be set. These should correlate with the tokens and keys collected above.
• A WORKPLACE_SEARCH_URL can be specified if your instance is hosted at a different location than http://localhost:3002.
• The script makes a simple attempt at improving indexing performance by batching documents. The size of this batch defaults to 10 but can be increased by setting the environment variable BATCH_SIZE.
• Workplace Search prefers lowercase field names, with the underscore character also supported. Conversely, Trello prefers camel case for its field names. A short utility function converts Trello’s format to a Workplace Search snake case-compatible format.
• We standardize some field names across object types to provide a reliable and consistent title, description, and url.
• See technical details for assistance running the script.

The result of the above script is we index three types of documents to our Workplace Search source: boards, cards, and comments:

{ 
  "idOrganization": "5eda760b10ee4c37ab1c487c", 
  "name": "Everything about Nothing", 
  "desc": "A nice picture of some trees\n", 
  "dateLastActivity": null, 
  "id": "5edab0910c085b71f34ee7d7", 
  "url": "https://trello.com/b/v5weC7R3/everything-about-nothing", 
  "dateLastView": "2020-06-08T12:20:04.155Z", 
  "body": "A nice picture of some trees\n", 
  "title": "Everything about Nothing", 
  "type": "board" 
}

{ 
  "id": "5eda38905ca6ed13c6ecdda1", 
  "dateLastActivity": "2020-06-05T12:33:09.183Z", 
  "desc": "Tip: Use this card to create a handy link of resources around team structure, messaging docs, brand guidelines, etc. for easy access by teammates and colleagues. Consider using a checklist to link out to different resources or a bullet pointed list in the card description.", 
  "idBoard": "5eda38905ca6ed13c6ecdd8d", 
  "name": "Team Resources", 
  "idList": "5eda38905ca6ed13c6ecdd8e", 
  "closed": false, 
  "url": "https://trello.com/c/LmKTkCUl/1-team-resources", 
  "labels": [], 
  "dueComplete": false, 
  "type": "card", 
  "title": "Team Resources", 
  "body": "Tip: Use this card to create a handy link of resources around team structure, messaging docs, brand guidelines, etc. for easy access by teammates and colleagues. Consider using a checklist to link out to different resources or a bullet pointed list in the card description." 
}

{ 
  "id": "5edaaf0d42e2827eaf36814f", 
  "type": "comment", 
  "date": "2020-06-05T20:46:05.415Z", 
  "memberCreator": "dalemcdiarmid", 
  "body": "Curious if i can grab these comments", 
  "url": "https://trello.com/c/Tpoqtgru/7-blog-redesign#comment-5edaaf0d42e2827eaf36814f", 
  "title": "Comment by dalemcdiarmid at 2020-06-05T20:46:05.415Z" 
}

Setting the schema

All custom sources require a schema to be defined. This tells Workplace Search the types of each of the fields, allowing the appropriate search behavior to be exposed to the end user. Workplace Search intelligently identifies field types, but it also allows users to easily update the schema at any time. Simply navigate to the Sources tab and select the Schema tab. From here, we can modify the field types to be consistent with those shown below before clicking Save Schema:

After a short reindex process, the schema change will take effect. While this is occurring, incoming requests for searching and indexing will still be accepted; schema updates are seamless and do not incur any downtime or downgraded user experience. 

Customizing the result display

Now that our data is properly indexed and aligned with the correct types, we can tune the result display for this custom source. From the source’s configuration page, navigate to Display Settings. Our changes here will determine how Trello search results are displayed to the user. Given our limited set of fields, the required settings are simple. We also benefit from the time invested standardizing our field names — Workplace Search is able to automatically choose sensible fields for the title, url, and description as shown below.

We only modify the color here to align with the Trello branding. Finally, select the Result Detail tab to modify how our results will be displayed when clicked. We add the fields shown below:

Seeing the search results

From our search application we can now enjoy fully integrated Trello results. As shown below, results appear as configured. Clicking a result takes us through to the appropriate Trello object as specified by the url field. As well as the classic advantages of a unified information platform, we also immediately benefit from Workplace Search’s intelligent query parsing. For example, if we search for “cards about Blog redesign in Trello”, the query parser automatically identifies and highlights those keywords and delivers the appropriate content, already filtered.

Considering security

This simple example doesn’t consider the access permission of Trello users to specific boards. Trello restricts access to boards, and their associated content, through the concept of teams. While not implemented here, Workplace Search does offer the capability to assign permissions to documents ingested into custom sources. As well as adding a list of “_permissions” to each document, Workplace Search must also be instructed which users have which permissions. By doing both we are able to achieve document-level security.

For our limited dataset, we would need to create a permission for each team. For example it may be appropriate to use the team id, adding this to every board, card, and comment retrieved from this team. We would then ensure that users between Workplace Search and Trello were synchronized, with Workplace Search users being given the appropriate permissions based on the team. It is unlikely that Workplace Search and Trello will use the same usernames. A mapping of users between the systems, possibly based on their email address, would therefore need to be constructed. This introduces a level of complexity beyond the scope of this blog but will be addressed in a follow-up piece.

Next steps

If you’d like to try out Elastic Workplace Search and ingest custom sources of your own, you can spin up a free 14-day trial.