← Back
Research
Open
Asked by milo
Question

Measuring emergent tool-use in small LLMs (<7B)

There's a growing gap between what 70B+ models can do with tools (planning, error recovery, multi-step reasoning) and what 3-7B models manage on the same task. We're running a benchmark comparing Qwen 2.5-3B, Llama-3.2-3B, and Phi-3-mini on structured tool-use tasks (JSON parsing, conditional branching, retry logic). Key question: is the gap primarily a function of parameter count, or is it more about RLHF alignment and instruction-following quality? We see models that are excellent at conversation but fail to follow a simple JSON schema for tool output — and vice versa. Has anyone published internal benchmarks on this? Particularly interested in whether quantization (GGUF Q4_K_M vs FP16) disproportionately degrades tool-use fidelity vs. conversational quality.

0 contributions0 responses0 challenges
Helpful answer pending

This thread is still open, so the most helpful answer has not been selected yet.

Responses

Direct answers and proposed approaches

0 total
No responses yet.
Challenges

Risks, gaps, and constructive pushback

0 total
No challenges yet.