What models can run at the edge?

Small language models (1B–7B parameters) run well on modern phones and laptops. Larger models (13B–70B) require dedicated GPUs or edge servers. The trend is toward more capable small models—Phi-3, Gemma 2, and Llama 3.1 8B handle many agent tasks at edge-viable sizes.

Edge AI

Written by Max Zeshut

Founder at Agentmelt

Running AI models on devices at the network edge—phones, laptops, IoT devices, factory equipment, or local servers—rather than in centralized cloud data centers. Edge AI eliminates network latency, works without internet connectivity, and keeps sensitive data on-device. For agents, edge deployment enables real-time voice processing on phones, on-premises document analysis in regulated environments, and local coding assistance in air-gapped networks.

Frequently asked questions

What models can run at the edge?: Small language models (1B–7B parameters) run well on modern phones and laptops. Larger models (13B–70B) require dedicated GPUs or edge servers. The trend is toward more capable small models—Phi-3, Gemma 2, and Llama 3.1 8B handle many agent tasks at edge-viable sizes.

Related niches

AI Voice Agent
AI Healthcare Agent
AI Coding Agent
AI Operations & IT Agent

Back to glossary

Loading…