Team Effort(what I have done with other team members):
- Working on and presenting the proposal pitch
Individual Effort:
- Got ShearedLlama-2.7B(4-bit quantization) + NanoLLM + MLC working, but had to reduce the context length to 2048(maximum is 4096) in order to fit into the memory. Even then, it’s unstable and runs out of memory sometimes, and the performance is not very good.
- Acquired the Jetson AGX Xavier from the inventory, which has 32GB of memory, hopefully it will perform much better. Currently working on getting Qwen3 to work on the device.