Tested multiple models that the Jetson AGX Xavier can run (all with llama.cpp and 4-bit quantization).
- Qwen3-4B: output 22 token/s
- Phi-3 Mini (3.8B): output 22 tokens/s
- Llama3.1-8B: output 15 token/s
- Nemotron-3-Nano, GLM 4.7 Flash cannot run due to the lack of memory.
Findings:
- Qwen3-4B and Phi-3 Mini are both able to perform correction tasks correctly and generate output at a similar speed. Llama3.1-8B is slower due to more parameters, and it could be a backup if the first two models cannot perform complex tasks later.
- Reading speed of the prompt is non-linear. For short prompts(<50 tokens), the reading speed is ~60 token/s –> ~0.3s. For long prompt(~500 tokens), the reading speed is ~300 token/s –> 1~2s. I need to test this further so we meet our design requirement.
- Theoretically, Nemotron-3-Nano and GLM 4.7 Flash should be able to fit into the 32GB memory, but they couldn’t unless being significantly tuned smaller, which perform slower and dumber than the above models.