AI Assistant Guide¶
Notolog includes an AI-powered assistant that supports multiple backends for text generation and assistance.
Table of Contents¶
- Overview
- Supported Backends
- OpenAI API Setup
- On-Device LLM (ONNX)
- Module llama.cpp (GGUF)
- Using the AI Assistant
- Best Practices
- Troubleshooting

Overview¶
The AI Assistant provides:
- Chat-style interface for natural conversations
- Context-aware responses with prompt history
- Multiple backends for flexibility
- Privacy options with local LLM support
Access via: Tools → AI Assistant
Supported Backends¶
| Backend | Model Format | Requirements | Privacy |
|---|---|---|---|
| OpenAI API | Cloud-based | API key, internet | Cloud processed |
| On-Device LLM | ONNX | Compatible GenAI model/runtime and sufficient memory | Local inference |
| Module llama.cpp | GGUF | Compatible model/runtime; compiler if building from source | Local inference |
OpenAI API Setup¶
Requirements¶
- OpenAI API account
- Valid API key
- Internet connection
Configuration¶
- Go to
Settings→OpenAI APItab - Enter your API URL (default: OpenAI endpoint)
- Enter your API key in the secure field
- Configure optional settings:
| Setting | Description | Default |
|---|---|---|
| API URL | API endpoint URL | https://api.openai.com/v1/chat/completions |
| API Key | Your OpenAI API key (stored encrypted) | (required) |
| Supported Models | GPT model to use | gpt-5.2 |
| System Prompt | Initial context/instructions | (empty) |
| Temperature | Response creativity (0-100 → 0.0-1.0) | 20 (0.2) |
| Maximum Response Tokens | Response limit; 0 leaves the limit to the API service | 0 |
| Prompt History Size | Number of turns to remember | 0 |
Security Notes¶
The API key, prompt, system prompt, and included conversation history are sent to the configured endpoint, which may be OpenAI or another compatible service. Verify the URL and use HTTPS for remote endpoints. Keys are encrypted in local settings with an application key, not stored in an OS credential vault. Anyone with access to both the settings and that key can recover them. The API key is configured in Settings; Notolog does not automatically read the OpenAI SDK's environment variables.
On-Device LLM (ONNX)¶
Run AI models locally using ONNX Runtime GenAI.
Requirements¶
- Python 3.11–3.14
- Enough RAM or VRAM for the selected model and context
- A model exported for the installed ONNX Runtime GenAI version
Supported Models¶
Choose an ONNX Runtime GenAI export containing genai_config.json and all referenced files. Use the model card to check runtime and execution-provider compatibility. The Microsoft model collection and Hugging Face ONNX catalog include model exports; check each export rather than relying on its ONNX label alone.
Configuration¶
- Go to
Settings→On Device LLMtab - Set ONNX Model Location to your ONNX model directory
- Configure generation parameters:
| Setting | Description | Default |
|---|---|---|
| ONNX Model Location | Path to ONNX model directory | (required) |
| Temperature | Response randomness (0-100 → 0.0-1.0) | 20 (0.2) |
| Maximum Response Tokens | GenAI sequence-length limit; 0 uses the module default of 4096 | 0 |
| Prompt History Size | Number of turns to remember | 0 |
First Run Notice¶
Model Loading Time
Loading time depends on the model, storage, and available memory. The application displays a loading message. Subsequent requests can reuse the loaded model.
Model Directory Structure¶
Your ONNX model directory should contain:
model_directory/
├── model.onnx # Main model file
├── model.onnx.data # Model weights (if separate)
├── tokenizer.json # Tokenizer configuration
└── genai_config.json # ONNX Runtime GenAI configuration
Filenames vary by export. Download the complete model directory referenced by genai_config.json and check compatibility with the installed GenAI version. See AI Modules.
Module llama.cpp (GGUF)¶
Run GGUF-format models using llama.cpp bindings.
Requirements¶
llama-cpp-pythonpackage- C++ compiler when building llama-cpp-python from source
- Enough RAM or VRAM for the selected model and context
- GGUF model file
Installation¶
# Option 1: Install with Notolog extras
pip install "notolog[llama]"
# Option 2: Install separately
pip install llama-cpp-python
# Linux users may need:
sudo apt-get install build-essential
Supported Models¶
GGUF models from: - Hugging Face GGUF Models - TheBloke's GGUF collection
Recommended quantizations: - Q4_K_M - Good balance of size/quality - Q5_K_M - Better quality, larger size - Q8_0 - Near-original quality
Configuration¶
- Go to
Settings→Module llama.cpptab - Set Model Location to your
.gguffile - Configure parameters:
| Setting | Description | Default |
|---|---|---|
| Model Location | Path to .gguf model file | (required) |
| Context Window Size | Token context size (1-65536) | 2048 |
| Chat Formats | Model's chat template | chatml |
| System Prompt | Initial instructions for the model | (empty) |
| Response Temperature | Response randomness (0-100 → 0.0-1.0) | 20 (0.2) |
| Max Tokens per Response | Response limit; 0 uses the available context window | 0 |
| Size of the Prompt History | Number of turns to remember | 0 |
Chat Formats¶
Select the template specified by your model card:
chatml- ChatML models (default)llama-2- Meta's Llama 2 modelsgemma- Google's Gemma models
The selector offers these three formats. Models requiring other templates are not supported through this setting; a GGUF file alone does not guarantee compatibility.
Using the AI Assistant¶
Basic Workflow¶
- Open AI Assistant
- Select your preferred backend in Settings
- Type your prompt in the input field
- Press
Enteror click Send - Wait for the response
Multi-line Prompts¶
For longer prompts: * Use Ctrl+Enter for new lines * Press Enter to send
Prompt Tips¶
Be Specific:
Provide Context:
"I'm writing a Markdown document about gardening.
Suggest 5 section headers for a beginner's guide."
Request Format:
Token Usage¶
The status area shows: - Input tokens: Your prompt size - Output tokens: Response size - Total tokens: Combined usage
Best Practices¶
For Privacy¶
- Use local models (ONNX or GGUF) for sensitive content
- Review system prompts before sending
- Clear chat history when done with sensitive topics
For Performance¶
- Start with a smaller model that fits the available RAM or VRAM
- Use quantized models (Q4_K_M) for speed
- Increase context window only if needed
- Close other applications when using large models
For Quality¶
- Use appropriate temperature:
- Low (0.1-0.3): Factual, consistent
- Medium (0.4-0.7): Balanced
- High (0.8-1.0): Creative, varied
- Provide clear, specific prompts
- Use system prompts to set context
Troubleshooting¶
"Model not found" Error¶
Cause: Invalid model path or corrupted model files.
Solution: 1. Verify the model path in Settings 2. Check that model files exist and are complete 3. Ensure file permissions allow reading
ONNX Model Takes Too Long to Load¶
Cause: First-time model initialization.
Solution: - Allow time for the model size and available hardware - Keep the model loaded to reuse it for subsequent requests - A loading message will be displayed
llama-cpp-python Installation Fails¶
Cause: Missing C++ compiler.
Solution:
# Ubuntu/Debian
sudo apt-get install build-essential
# Fedora
sudo dnf install gcc-c++
# macOS
xcode-select --install
Out of Memory Errors¶
Cause: Model too large for available RAM.
Solution: 1. Use smaller quantized models 2. Close other applications 3. Reduce context window size 4. Consider cloud API instead
API Reference¶
For developers extending AI functionality, see: - API Reference - Module Architecture - Creating Custom Modules
For general configuration, see Configuration Guide.