Model Gallery

12 models from 1 repositories

Filter by type:

Filter by tags:

deepseek-v4.1-flash-ista-iq1_m
DeepSeek-V4.1-Flash-1017, a 763B-parameter mixture-of-experts model (384 experts per layer, 6 used per token), in ISTA-DAS's 1.75 bits per weight GSQ-RCO quantization (IQ1_M), the smallest size of the three (about 239 GB to download). It writes a reasoning trace before the answer, and the Strata backend splits it from the reply and parses the model's DSML tool calls. Text only: images are not supported by this backend. Needs a GPU with 12 GB of VRAM or more and about 34 GB of RAM at the least (Strata's figure; its measurements use 46 to 64 GB: an RTX 5070 with 64 GB answered at about 5 tokens per second, an RTX 3060 with 46 GB at 8 to 10). Less RAM still runs, slowly. Slightly lower quality than 2.00 bpw, and noticeably weaker on plain English prose; ISTA-DASLab measured 98.7% of the base model on its task average (AIME25, GPQA-Diamond, LiveCodeBench v6). Needs about 240 GB of free disk, on a fast NVMe drive: the 111 GB n-gram table (Engram) stays on disk and is read for every token. The Engram table and the 8 GB DSpark draft model are shared by all DeepSeek-V4.1 entries and downloaded once into strata/models/deepseek-v4.1-flash-shared. Runs only on the experimental strata backend, on NVIDIA CUDA 13 (x86_64 and arm64), AMD Radeon RX 9070 and R9700 (ROCm) and Apple Silicon (Metal). Strata ships no DeepSeek engine for CUDA 12, Intel or other AMD cards. On the first load the strata backend builds the Strata pack from the GGUF files (under a minute, a few MB of extra disk); loading the model afterwards takes a few minutes. The default context is 8192 tokens, the size Strata checked; raise context_size if the GPU has room. Licensed under the MIT License (DeepSeek-V4.1-Flash-1017 and the quantized files). The GGUF files are the GSQ-RCO quantization by ISTA-DAS (ISTA-DASLab); the DSpark draft model is Unsloth's; Strata is by Niko1221.

Repository: localaiLicense: mit

deepseek-v4.1-flash-ista-iq2_xxs
DeepSeek-V4.1-Flash-1017, a 763B-parameter mixture-of-experts model (384 experts per layer, 6 used per token), in ISTA-DAS's 2.00 bits per weight GSQ-RCO quantization (IQ2_XXS), the middle size of the three (about 257 GB to download). It writes a reasoning trace before the answer, and the Strata backend splits it from the reply and parses the model's DSML tool calls. Text only: images are not supported by this backend. Needs a GPU with 12 GB of VRAM or more and about 40 GB of RAM at the least (Strata's figure; its measurements use 46 to 64 GB: an RTX 5070 with 64 GB answered at about 5 tokens per second, an RTX 3060 with 46 GB at 8 to 10). Less RAM still runs, slowly. Strata measured a perplexity 15.5% above the original model on 4,995 positions (code 8.6%, agent work 5.3%). Needs about 258 GB of free disk, on a fast NVMe drive: the 111 GB n-gram table (Engram) stays on disk and is read for every token. The Engram table and the 8 GB DSpark draft model are shared by all DeepSeek-V4.1 entries and downloaded once into strata/models/deepseek-v4.1-flash-shared. Runs only on the experimental strata backend, on NVIDIA CUDA 13 (x86_64 and arm64), AMD Radeon RX 9070 and R9700 (ROCm) and Apple Silicon (Metal). Strata ships no DeepSeek engine for CUDA 12, Intel or other AMD cards. On the first load the strata backend builds the Strata pack from the GGUF files (under a minute, a few MB of extra disk); loading the model afterwards takes a few minutes. The default context is 8192 tokens, the size Strata checked; raise context_size if the GPU has room. Licensed under the MIT License (DeepSeek-V4.1-Flash-1017 and the quantized files). The GGUF files are the GSQ-RCO quantization by ISTA-DAS (ISTA-DASLab); the DSpark draft model is Unsloth's; Strata is by Niko1221.

Repository: localaiLicense: mit

deepseek-v4.1-flash-ista-iq2_xs
DeepSeek-V4.1-Flash-1017, a 763B-parameter mixture-of-experts model (384 experts per layer, 6 used per token), in ISTA-DAS's 2.25 bits per weight GSQ-RCO quantization (IQ2_XS), the largest size of the three (about 274 GB to download). It writes a reasoning trace before the answer, and the Strata backend splits it from the reply and parses the model's DSML tool calls. Text only: images are not supported by this backend. Needs a GPU with 12 GB of VRAM or more and about 44 GB of RAM at the least (Strata's figure; its measurements use 46 to 64 GB: an RTX 5070 with 64 GB answered at about 5 tokens per second, an RTX 3060 with 46 GB at 8 to 10). Less RAM still runs, slowly. Strata measured a perplexity 7.3% above the original model on 4,995 positions (code 3.2%, agent work 2.2%). Needs about 275 GB of free disk, on a fast NVMe drive: the 111 GB n-gram table (Engram) stays on disk and is read for every token. The Engram table and the 8 GB DSpark draft model are shared by all DeepSeek-V4.1 entries and downloaded once into strata/models/deepseek-v4.1-flash-shared. Runs only on the experimental strata backend, on NVIDIA CUDA 13 (x86_64 and arm64), AMD Radeon RX 9070 and R9700 (ROCm) and Apple Silicon (Metal). Strata ships no DeepSeek engine for CUDA 12, Intel or other AMD cards. On the first load the strata backend builds the Strata pack from the GGUF files (under a minute, a few MB of extra disk); loading the model afterwards takes a few minutes. The default context is 8192 tokens, the size Strata checked; raise context_size if the GPU has room. Licensed under the MIT License (DeepSeek-V4.1-Flash-1017 and the quantized files). The GGUF files are the GSQ-RCO quantization by ISTA-DAS (ISTA-DASLab); the DSpark draft model is Unsloth's; Strata is by Niko1221.

Repository: localaiLicense: mit

deepseek-v4.1-flash-ista-coder-iq2_xxs
ISTA-DAS's coding version of DeepSeek-V4.1-Flash-1017: in every layer 192 of the 384 routed experts are kept and the other 192 removed, stored in the 2.00 bpw (IQ2_XXS) layout. It is the smallest DeepSeek-V4.1 set here (about 191 GB to download). It writes a reasoning trace before the answer, and the Strata backend splits it from the reply and parses the model's DSML tool calls. Text only: images are not supported by this backend. Needs a GPU with 12 GB of VRAM or more and about 24 GB of RAM at the least (Strata's figure; its measurements use 46 to 64 GB: an RTX 5070 with 64 GB answered at about 5 tokens per second, an RTX 3060 with 46 GB at 8 to 10). Less RAM still runs, slowly. Strata measured a perplexity 48% above the original model on its test set (KL 0.40), so it is weaker than the full sets on prose and general knowledge; it is meant for code and agent work. Needs about 192 GB of free disk, on a fast NVMe drive: the 111 GB n-gram table (Engram) stays on disk and is read for every token. The Engram table and the 8 GB DSpark draft model are shared by all DeepSeek-V4.1 entries and downloaded once into strata/models/deepseek-v4.1-flash-shared. Runs only on the experimental strata backend, on NVIDIA CUDA 13 (x86_64 and arm64), AMD Radeon RX 9070 and R9700 (ROCm) and Apple Silicon (Metal). Strata ships no DeepSeek engine for CUDA 12, Intel or other AMD cards. On the first load the strata backend builds the Strata pack from the GGUF files (under a minute, a few MB of extra disk); loading the model afterwards takes a few minutes. The default context is 8192 tokens, the size Strata checked; raise context_size if the GPU has room. Licensed under the MIT License (DeepSeek-V4.1-Flash-1017 and the quantized files). The GGUF files are the GSQ-RCO quantization by ISTA-DAS (ISTA-DASLab); the DSpark draft model is Unsloth's; Strata is by Niko1221.

Repository: localaiLicense: mit

qwen3-4b-vllm-cpp
Qwen3-4B on vllm.cpp, in bf16. The small end of the engine's gated dense family, which reaches parity with vLLM on every axis at concurrency 1. bf16 rather than NVFP4 on purpose: this is the entry that runs where the flagship NVFP4 checkpoints cannot, including Apple Silicon via Metal, Vulkan and plain CPU. Roughly 8 GB of weights, plus about 4.5 GB of KV cache at the context configured here. Tool calling and the thinking split are parsed inside the engine.

Repository: localaiLicense: apache-2.0

qwen3-tts-llamacpp
Qwen3-TTS 1.7B Base served by the llama.cpp backend, using upstream's own GGUF conversion. Runs on the full llama-cpp accelerator matrix (CUDA, ROCm, SYCL, Vulkan, Metal). Streaming output and zero-shot voice cloning: set `voice` to a reference clip or a saved Voice Library profile, which is required since the Base checkpoint has no built-in speaker. 24kHz mono, 10 languages. Q8_0 backbone (~1.8 GB) plus a Q8_0 projector.

Repository: localaiLicense: apache-2.0

deepseek-v4-flash-q2
DeepSeek V4 Flash (IQ2XXS GGUF, ~81 GB) - only loadable via the ds4 backend. Requires >=128 GB RAM. Metal (Darwin) or CUDA (Linux). See https://github.com/antirez/ds4 for details.

Repository: localai

deepseek-v4-flash-q2-q4
DeepSeek V4 Flash (mixed q2/q4 GGUF, ~91 GB) - only loadable via the ds4 backend. The last 6 expert layers are kept at Q4_K (the rest IQ2XXS), trading a little extra memory for higher quality than the pure-q2 build while still fitting in RAM on a 128 GB machine. imatrix-tuned. Metal (Darwin) or CUDA (Linux). See https://github.com/antirez/ds4 for details.

Repository: localai

deepseek-v4-flash-q4-ssd
DeepSeek V4 Flash (full 4-bit experts GGUF, ~153 GB) - only loadable via the ds4 backend, with SSD streaming enabled so it runs on a 128 GB machine even though the weights do not fit in RAM: routed MoE experts stream from the GGUF on SSD while the non-routed weights stay resident. SSD streaming is Metal (Darwin) only; generation speed depends on SSD speed and the expert cache. Tune the routed-expert cache with the 'ssd_streaming_cache_experts:NGB' option (default: automatic budget). See https://github.com/antirez/ds4.

Repository: localai

deepseek-v4-flash-q2-mtp
DeepSeek V4 Flash (IQ2XXS GGUF, ~81 GB) paired with the optional MTP speculative-decoding weights (~3.5 GB) for a slight speedup. Only loadable via the ds4 backend; requires >=128 GB RAM. MTP helps only with greedy decoding (temperature 0), so the override pins temperature to 0. Metal (Darwin) or CUDA (Linux). See https://github.com/antirez/ds4 for details.

Repository: localai

deepseek-v4-pro-q2-ssd
DeepSeek V4 Pro (IQ2XXS GGUF, ~433 GB, imatrix-tuned) - only loadable via the ds4 backend, with SSD streaming so the Pro-class model can be run on a 128 GB machine. This is experimental and slow: it needs ~433 GB of free SSD plus enough RAM for the resident weights, KV cache, and routed-expert cache, and is best used with thinking off for inspection or occasional work. SSD streaming is Metal (Darwin) only. See https://github.com/antirez/ds4.

Repository: localai

nimble-9b-vllm-cpp
Nimble is a decision model from Bespoke Labs (the model Ollama serves as nimble). For each question it runs one forward pass and reads the logits of the answer letters at the last prompt position. It answers typed choice, noul and score questions about a text state and does not generate text. This entry installs a converted redistribution of bespokelabs/Bespoke-Nimble-9B: the LoRA adapter is merged into its Qwen3.5-9B base in BF16 and config.json names the NimbleModel architecture. The checkpoint only works with vllm.cpp (the vllm-cpp backend); transformers and vLLM cannot load it. In LocalAI, serve it via POST /v1/systemone. The vllm.cpp project compared this converted directory on CPU against the Bespoke authors' own code over seven questions: 7 of 7 answers equal, largest probability difference 0.0054. This entry was installed and served through LocalAI on CPU and gave the answers shown in the model card example. There is no accuracy benchmark and no GPU (CUDA, ROCm, Metal) run. The engine refuses fields with more than 26 choices (upstream allows 255). The entry sets an 8192-token context (Nimble's own prompt limit) and a KV pool that holds four such sequences. BF16 weights, about 19.3 GB, pinned to a revision. On CPU the model needs about 20 GB of free RAM (measured peak 18.4 GB resident). On a 20-thread CPU a request with three questions (988 prompt tokens) took 30 seconds warm.

Repository: localaiLicense: apache-2.0