engine

engineLink to this section

engine is a local inference server - 2bit quants, kernels and runtime for consumer devices.

OverviewLink to this section

[![Docs][docs-shield]][docs-url] [![Website][website-shield]][website-url] [![GitHub][github-shield]][github-url] [![HuggingFace][hf-shield]][hf-url] [![Reddit][reddit-shield]][reddit-url] [![Blog][blog-shield]][blog-url]

A hybrid edge-cloud AI engine for mobile devices & wearables.

• LLM: Gemma-4-E2B-CQ4 (1k-context prefill / decode for 100 tokens) - VLM: Gemma-4-E2B-CQ4 (256px image encode time / decode) - Transcribe: Parakeet-TDT-0.6B-CQ4 (20s audio end-to-end transcribe time) - 1k-Context RAM: peak MB during the LLM benchmark - No speculative decode or MTP, pure decode

• Gemma-4-E2B-it accuracy across bit widths, averaged over 3 seeds. - CQ3.26 and CQ2.54 are mixed-precision, CQ2/CQ3/CQ4 are uniformly quantized. - Full results in docs/cactus_quants.md:

• Any HuggingFace model can be converted using 'cactus convert [HF-Name]', though experimental. - Liquid, Gemma. whisper. parakeet and Qwen model families are especially tested. - Some models have been pre-uploaded here, just run 'cactus download [HF-Name]'. - 'cactus run [HF-Name]' albeit first downloads or convert the model if not found.

N/B: Scroll all the way up and click the shields link for resources!

[docs-shield]: https://img.shields.io/badge/Docs-555?style=for-the-badge&logo=readthedocs&logoColor=white [docs-url]: https://cactus-compute.github.io/cactus/

[website-shield]: https://img.shields.io/badge/Website-555?style=for-the-badge&logo=safari&logoColor=white [website-url]: https://cactuscompute.com/

[github-shield]: https://img.shields.io/badge/GitHub-555?style=for-the-badge&logo=github&logoColor=white [github-url]: https://github.com/cactus-compute/cactus

[hf-shield]: https://img.shields.io/badge/HuggingFace-555?style=for-the-badge&logo=huggingface&logoColor=white [hf-url]: https://huggingface.co/Cactus-Compute

[reddit-shield]: https://img.shields.io/badge/Reddit-555?style=for-the-badge&logo=reddit&logoColor=white [reddit-url]: https://www.reddit.com/r/cactuscompute/

[blog-shield]: https://img.shields.io/badge/Blog-555?style=for-the-badge&logo=hashnode&logoColor=white [blog-url]: https://cactuscompute.com/blog

engine is open-source, written primarily in C++, with 6,184 GitHub stars under the Other license. The latest release is v2.2.2 (2026-09-26).

Key capabilitiesLink to this section

From the project's documentation:

• Step 1: brew install cactus-compute/cactus/cactus
• Step 2: cactus run
• LLM: Gemma-4-E2B-CQ4 (1k-context prefill / decode for 100 tokens)
• VLM: Gemma-4-E2B-CQ4 (256px image encode time / decode)
• Transcribe: Parakeet-TDT-0.6B-CQ4 (20s audio end-to-end transcribe time)
• 1k-Context RAM: peak MB during the LLM benchmark

InstallLink to this section

A quick way to get started (always check the official docs for the latest):

brew install cactus-compute/cactus/cactus

How it fits a local-AI stackLink to this section

engine runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related local inference servers in the directory:

• "airllm"
• "Atomic-Chat"
• "backburner"
• "BentoML"

SourcesLink to this section

• Source code & docs: cactus-compute/engine
• Official website: https://cactuscompute.com

Stats from GitHub, 2026-11-07.