Skip to content

GEN-AI DIFY Dev platform with Ollama

DIFY GEN-AI Framework

Dify is an open-source LLM app development platform. Orchestrate LLM apps from agents to complex AI workflows, with an RAG engine.

More production-ready than LangChain.

You can find more information about

DIFY at there website

DIFY Blogs

Ollama LLM API Server

Ollama is API Server compatible with OPENAI clients and allows you to run local LLM models

Run Llama 3.1, Phi 3, Mistral, Gemma 2, and other models. Customize and create your own

Read more about it at

Ollama Blog

Ollama Models

For the sandbox we already downloaded and installed ollama in docker container and it's part of DIFY docker-compose.

Also we downloaded 2 models in it.

  • text-embedding : all-minilm:l6-v2
  • chat : gemma2:2b-instruct-q2_K (alias uv-gemma2)

As the name says text-embedding models support creating of embeddings from provided text, and chat or completion models is gemma2, 2 Billion parameter model with instruct trained dataset and quantized for low memory requirements.

Given the virtual machine memory limitations in the sandbox we will not load a larger memory.

Configuring DIFY to use OLLAMA models

Following are instructions to configure DIFY to use the local LLM models, typically the DIFY examples are configured to OpenAI which requires special API-Token and Cost is involved.

Open your browser and access the website on the VM IP address http://

It should redirect you to create a admin userid and password for the first time. Once you finish that login into the Server using the id and password.

On Login. Select the UserID Settings

  • User > Settings
    Alt text

  • Settings > Model Provider Alt text

  • From Models select > Ollama Provider

    Select LLM Model Name : uv-gemma2 Base URL : http://ollama:11434

    Alt text

  • Models > Model Provider Now you should see the model is available on the main settings menu.

    Alt text

  • Let's add one more embedding model using ollama itself.

    Click on Add model

    Select : Text Embeddings Model Name : all-minilm:l6-v2 Base URL : http://ollama:11434

    Alt text

  • If all goes well you should see 2 models under ollama

    Alt text

  • Now make these as default one for System Model Settings

    Alt text

This should conclude setting up the models and your ready for workshop lab project. If you come across any issues report issue on Zulip support channel and we can figure out what went wrong.