Connect to your own local LLM

This page provides instructions for setting up a connector to a large language model (LLM) of your choice using LM Studio. This allows you to use your chosen model within Elastic Security. You’ll first need to set up a reverse proxy to communicate with Elastic Security, then set up LM Studio on a server, and finally configure the connector in your Elastic Security project. Learn more about the benefits of using a local LLM.

This example uses a single server hosted in GCP to run the following components:

You need to set up a reverse proxy to enable communication between LM Studio and Elastic. For more complete instructions, refer to a guide such as this one.

The following is an example Nginx configuration file:

server {
    listen                          80;
    listen                          [::]:80;
    server_name                     <yourdomainname.com>;
    server_tokens off;
    add_header x-xss-protection "1; mode=block" always;
    add_header x-frame-options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    return 301                      https://$server_name$request_uri;
}

server {

    listen                          443 ssl http2;
    listen                          [::]:443 ssl http2;
    server_name                     <yourdomainname.com>;
    server_tokens off;
    ssl_certificate                 /etc/letsencrypt/live/<yourdomainname.com>/fullchain.pem;
    ssl_certificate_key             /etc/letsencrypt/live/<yourdomainname.com>/privkey.pem;
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets on;
    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256';
    ssl_protocols TLSv1.3 TLSv1.2;
    ssl_prefer_server_ciphers on;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    add_header x-xss-protection "1; mode=block" always;
    add_header x-frame-options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/letsencrypt/live/<yourdomainname.com>/fullchain.pem;
    resolver 1.1.1.1;
    location / {

    if ($http_authorization != "Bearer <secret token>") {
    return 401;
}

    proxy_pass http://localhost:1234/;
   }

}

You can use Elastic’s Nginx integration to monitor performance and populate monitoring dashboards in the Elastic Security app.

First, install LM Studio. LM Studio supports the OpenAI SDK, which makes it compatible with Elastic’s OpenAI connector, allowing you to connect to any model available in the LM Studio marketplace.

One current limitation of LM Studio is that when it is installed on a server, you must launch the application using its GUI before doing so using the CLI. For example, by using Chrome RDP with an X Window System. After you’ve opened the application the first time using the GUI, you can start it by using sudo lms server start in the CLI.

Once you’ve launched LM Studio:

In this example we used TheBloke/Mixtral-8x7B-Instruct-v0.1.Q3_K_M.gguf. It has 46.7B total parameters, a 32,000 token context window, and uses GGUF quanitization. For more information about model names and format information, refer to the following table.

After downloading a model, load it in LM Studio using the GUI or LM Studio’s CLI tool.

It is a best practice to download models from the marketplace using the GUI, and then load or unload them using the CLI. The GUI allows you to search for models, whereas the CLI only allows you to import specific paths, but the CLI provides a good interface for loading and unloading.

Use the following commands in your CLI:

After the model loads, you should see a Model loaded successfully message in the CLI.

To verify which model is loaded, use the lms ps command.

If your model uses NVIDIA drivers, you can check the GPU performance with the sudo nvidia-smi command.

Refer to the following video to see how to load a model using LM Studio’s GUI. You can change the port setting, which is referenced in the Nginx configuration file. Note that the GPU offload was set to Max.

You can monitor the performance of the host running LM Studio using Elastic’s Custom Logs integration. This can also help with troubleshooting. Note that the default path for LM Studio logs is /tmp/lmstudio-server-log.txt, as in the following screenshot:

Finally, configure the connector in your Security project:

Setup is now complete. You can use the model you’ve loaded in LM Studio to power Elastic’s generative AI features. You can test a variety of models as you interact with AI Assistant to see what works best without having to update your connector.