GEN-AI DIFY Dev platform with Ollama
DIFY GEN-AI Framework
Dify is an open-source LLM app development platform. Orchestrate LLM apps from agents to complex AI workflows, with an RAG engine.
More production-ready than LangChain.
You can find more information about
Ollama LLM API Server
Ollama is API Server compatible with OPENAI clients and allows you to run local LLM models
Run Llama 3.1, Phi 3, Mistral, Gemma 2, and other models. Customize and create your own
Read more about it at
For the sandbox we already downloaded and installed ollama in docker container and it's part of DIFY docker-compose.
Also we downloaded 2 models in it.
- text-embedding : all-minilm:l6-v2
- chat : gemma2:2b-instruct-q2_K (alias uv-gemma2)
As the name says text-embedding models support creating of embeddings from provided text, and chat or completion models is gemma2, 2 Billion parameter model with instruct trained dataset and quantized for low memory requirements.
Given the virtual machine memory limitations in the sandbox we will not load a larger memory.
Configuring DIFY to use OLLAMA models
Following are instructions to configure DIFY to use the local LLM models, typically the DIFY examples are configured to OpenAI which requires special API-Token and Cost is involved.
Open your browser and access the website on the VM IP address http://
It should redirect you to create a admin userid and password for the first time. Once you finish that login into the Server using the id and password.
On Login. Select the UserID Settings
-
From Models select > Ollama Provider
Select LLM Model Name : uv-gemma2 Base URL : http://ollama:11434
-
Models > Model Provider Now you should see the model is available on the main settings menu.
-
Let's add one more embedding model using ollama itself.
Click on Add model
Select : Text Embeddings Model Name : all-minilm:l6-v2 Base URL : http://ollama:11434
-
If all goes well you should see 2 models under ollama
-
Now make these as default one for System Model Settings
This should conclude setting up the models and your ready for workshop lab project. If you come across any issues report issue on Zulip support channel and we can figure out what went wrong.