No speedup. MacBook Pro 13, M1, 16GB, Ollama, orca-mini.
Local GPT assistance for maximum privacy and offline access.
The plugin allows you to open a context menu on selected text to pick an AI-assistant's action.
Also works with images
No speedup. MacBook Pro 13, M1, 16GB, Ollama, bakllava.
Default actions:
- Continue writing
- Summarize text
- Fix spelling and grammar
- Find action items in text
- General help (just use selected text as a prompt for any purpose)
You can also add yours, share the best actions or get one from the community.
Supported AI Providers:
- Ollama
- OpenAI compatible server (also OpenAI)
This plugin is available in the Obsidian community plugin store https://obsidian.md/plugins?id=local-gpt
You can also install this plugin via BRAT: pfrankov/obsidian-local-gpt
- Install Ollama.
- Install Qwen2 (default)
ollama pull qwen2
or any preferred model from the library.
Additional: if you want to enable streaming completion with Ollama you should set environment variable OLLAMA_ORIGINS
to *
:
- For MacOS run
launchctl setenv OLLAMA_ORIGINS "*"
. - For Linux and Windows check the docs.
There are several options to run local OpenAI-like server:
- llama.cpp
- llama-cpp-python
- LocalAI
- Obabooga Text generation web UI
- LM Studio
- ...maybe more
Here is an example for llama.cpp:
- Install llama.cpp and follow build instructions for your OS
- Download a model trained on the ChatML dialog format. For example, Mixtral 8X7B (Dolphin 2.5 version)
- Run the server by calling
./server -c 4096 --host 0.0.0.0 -t 16 --mlock -m models/dolphin-2.5-mixtral-8x7b.Q4_K_M.gguf
or as described in the documentation.
- Open Obsidian Settings
- Go to Hotkeys
- Filter "Local" and you should see "Local GPT: Show context menu"
- Click on
+
icon and press hotkey (e.g.⌘ + M
)
It is also possible to specify a fallback to handle requests — this allows you to use larger models when you are online and smaller ones when offline.
Example video
Kapture.2024-01-11.at.22.16.52.mp4
Since you can provide any OpenAI-like server, it is possible to use OpenAI servers themselves.
Despite the ease of configuration, I do not recommend this method, since the main purpose of the plugin is to work with private LLMs.
- Select
OpenAI compatible server
inSelected AI provider
- Set
OpenAI compatible server URL
tohttps://api.openai.com
- Retrieve and paste your
API key
from the API keys page - Click "refresh" button and select the model that suits your needs (e.g.
gpt-3.5-turbo
)
- Ability to select models from the list instead of typing their names
- Ability to share and apply presets (system prompt + prompt + model)
- Additional AI providers (OpenAI, etc...)
- Streaming completions
- Changing order of the actions
- Editing actions
- Fallback for action if first URL is unavailable (remote GPU)
- Support multimodal models like Llava
- Optional settings for prompts (top_p, top_k, temperature, repeat_penalty)
- Accounting your local documents in results as described here https://ollama.ai/blog/llms-in-obsidian
If you would like to use other providers, please let me know in the discussions.
- Colored Tags that colorizes tags in distinguishable colors.