Introducing LangChain4j to simplify LLM integration into Java applications

The LangChain4j framework was created in 2023 with this target:

The goal of LangChain4j is to simplify integrating LLMs into Java applications.

LangChain4j is providing a standard way to:

• create embeddings (vectors) from a given content, let say a text for example
• store embeddings in an embedding store
• search for similar vectors in the embedding store
• discuss with LLMs
• use a chat memory to remember the context of a discussion with an LLM

This list is not exhaustive and the LangChain4j community is always implementing new features.

This post will cover the first main parts of the framework.

Adding LangChain4j OpenAI to our project

Like in all Java projects, it's just a matter of dependencies. Here we will be using Maven but the same could be achieved with any other dependency manager.

As a first step to the project we want to build here, we will be using OpenAI so we just need to add the langchain4j-open-ai artifact:

<properties>
  <langchain4j.version>0.34.0</langchain4j.version>
</properties>

<dependencies>
  <dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j-open-ai</artifactId>
    <version>${langchain4j.version}</version>
  </dependency>
</dependencies>

For the rest of the code we will be using either our own API key, which you can get by registering for an account with OpenAI, or the one provided by LangChain4j project for demo purposes only:

static String getOpenAiApiKey() {
  String apiKey = System.getenv(API_KEY_ENV_NAME);
  if (apiKey == null || apiKey.isEmpty()) {
    Logger.warn("Please provide your own key instead using [{}] env variable", API_KEY_ENV_NAME);
    return "demo";
  }
  return apiKey;
}

We can now create an instance of our ChatLanguageModel:

ChatLanguageModel model = OpenAiChatModel.withApiKey(getOpenAiApiKey());

And finally we can ask a simple question and get back the answer:

String answer = model.generate("Who is Thomas Pesquet?");
Logger.info("Answer is: {}", answer);

The given answer might be something like:

Thomas Pesquet is a French aerospace engineer, pilot, and European Space Agency astronaut.
He was selected as a member of the European Astronaut Corps in 2009 and has since completed 
two space missions to the International Space Station, including serving as a flight engineer 
for Expedition 50/51 in 2016-2017. Pesquet is known for his contributions to scientific 
research and outreach activities during his time in space.

If you'd like to run this code, please check out the Step1AiChatTest.java class.

Providing more context

Let's add the langchain4j artifact:

<dependency>
  <groupId>dev.langchain4j</groupId>
  <artifactId>langchain4j</artifactId>
  <version>${langchain4j.version}</version>
</dependency>

This one is providing a toolset which can help us build a more advanced LLM integration to build our assistant. Here we will just create an Assistant interface which provides the chat method which will be calling automagically the ChatLanguageModel we defined earlier:

interface Assistant {
  String chat(String userMessage);
}

We just have to ask LangChain4j AiServices class to build an instance for us:

Assistant assistant = AiServices.create(Assistant.class, model);

And then call the chat(String) method:

String answer = assistant.chat("Who is Thomas Pesquet?");
Logger.info("Answer is: {}", answer);

This is having the same behavior as before. So why did we change the code? In the first place, it's more elegant but more than that, you can now give some instructions to the LLM using simple annotations:

interface Assistant {
  @SystemMessage("Please answer in a funny way.")
  String chat(String userMessage);
}

This is now giving:

Ah, Thomas Pesquet is actually a super secret spy disguised as an astronaut! 
He's out there in space fighting aliens and saving the world one spacewalk at a time. 
Or maybe he's just a really cool French astronaut who has been to the International 
Space Station. But my spy theory is much more exciting, don't you think?

If you'd like to run this code, please check out the Step2AssistantTest.java class.

Switching to another LLM

We can use the great Ollama project. It helps to run a LLM locally on your machine.

Let's add the langchain4j-ollama artifact:

<dependency>
  <groupId>dev.langchain4j</groupId>
  <artifactId>langchain4j-ollama</artifactId>
  <version>${langchain4j.version}</version>
</dependency>

As we are running the sample code using tests, let's add Testcontainers to our project:

<dependency>
  <groupId>org.testcontainers</groupId>
  <artifactId>ollama</artifactId>
  <version>1.20.1</version>
  <scope>test</scope>
</dependency>

We can now start/stop Docker containers:

static String MODEL_NAME = "mistral";
static String DOCKER_IMAGE_NAME = "langchain4j/ollama-" + MODEL_NAME + ":latest";

static OllamaContainer ollama = new OllamaContainer(
  DockerImageName.parse(DOCKER_IMAGE_NAME).asCompatibleSubstituteFor("ollama/ollama"));

@BeforeAll
public static void setup() {
  ollama.start();
}

@AfterAll
public static void teardown() {
  ollama.stop();
}

We "just" have to change the model object to become an OllamaChatModel instead of the OpenAiChatModel we used previously:

OllamaChatModel model = OllamaChatModel.builder()
  .baseUrl(ollama.getEndpoint())
  .modelName(MODEL_NAME)
  .build();

Note that it could take some time to pull the image with its model, but after a while, you could get the answer:

Oh, Thomas Pesquet, the man who single-handedly keeps the French space program running 
while sipping on his crisp rosé and munching on a baguette! He's our beloved astronaut 
with an irresistible accent that makes us all want to learn French just so we can 
understand him better. When he's not floating in space, he's probably practicing his 
best "je ne sais quoi" face for the next family photo. Vive le Thomas Pesquet! 
🚀🌍🇫🇷 #FrenchSpaceHero

Better with memory

If we ask multiple questions, by default the system won't remember the previous questions and answers. So if we ask after the first question "When was he born?", our application will answer:

Oh, you're asking about this legendary figure from history, huh? Well, let me tell 
you a hilarious tale! He was actually born on Leap Year's Day, but only every 400 
years! So, do the math... if we count backwards from 2020 (which is also a leap year), 
then he was born in... *drumroll please* ...1600! Isn't that a hoot? But remember 
folks, this is just a joke, and historical records may vary.

Which is nonsense. Instead, we should use Chat Memory:

ChatMemory chatMemory = MessageWindowChatMemory.withMaxMessages(10);
Assistant assistant = AiServices.builder(Assistant.class)
  .chatLanguageModel(model)
  .chatMemory(chatMemory)
  .build();

Running the same questions now gives a meaningful answer:

Oh, Thomas Pesquet, the man who was probably born before sliced bread but after dinosaurs! 
You know, around the time when people started putting wheels on suitcases and calling it 
a revolution. So, roughly speaking, he came into this world somewhere in the late 70s or 
early 80s, give or take a year or two - just enough time for him to grow up, become an 
astronaut, and make us all laugh with his space-aged antics! Isn't that a hoot? 
*laughs maniacally*

Conclusion

In the next post, we will discover how we can ask questions to our private dataset using Elasticsearch as the embedding store. That will give us a way to transform our application search to the next level.