LocalAI

Overview

LocalAI is a drop-in replacement REST API that's compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families that are compatible with the ggml format. Does not require GPU.

Dify allows integration with LocalAI for local deployment of large language model inference and embedding capabilities.

Configure

1. First, clone the LocalAI code repository and navigate to the specified directory.

2. Download example LLM and Embedding models.

Here, we choose two smaller models that are compatible across all platforms. serves as the default LLM model, and serves as the default Embedding model, for quick local deployment.

3. Configure the .env file.

NOTE: Ensure that the THREADS variable value in doesn't exceed the number of CPU cores on your machine.

4. Start LocalAI.

The LocalAI request API endpoint will be available at http://127.0.0.1:8080.

And it provides two models, namely:

• LLM Model:
• External access name: (This name is customizable and can be configured in ).
• Embedding Model:
• External access name: (This name is customizable and can be configured in ).

If you use the Dify Docker deployment method, you need to pay attention to the network configuration to ensure that the Dify container can access the endpoint of LocalAI. The Dify container cannot access localhost inside, and you need to use the host IP address.

5. Integrate the models into Dify.
   Go to and fill in:
   Model 1:

• Model Type: Text Generation
• Model Name:
• Server URL: http://127.0.0.1:8080
• If Dify is deployed via docker, fill in the host domain: , which can be a LAN IP address, like: http://192.168.1.100:8080
  Click "Save" to use the model in the application.
  Model 2:
  • Model Type: Embeddings
  • Model Name:
  • Server URL: http://127.0.0.1:8080
    • If Dify is deployed via docker, fill in the host domain: , which can be a LAN IP address, like: http://192.168.1.100:8080

Click "Save" to use the model in the application.

For more information about LocalAI, please refer to: https://github.com/go-skynet/LocalAI