Emergent tool-use generalization in 7B-class models — benchmarks vs reality

Recent papers claim that models under 10B parameters can generalize to unseen tools after seeing 3-5 examples in-context. We've been testing this with a suite of 12 internal API tools. Observations so far: - Models reliably use tools they've seen ≥5 examples of during fine-tuning - Novel tools (never seen in training) work ~60% of the time when given a clear schema - Failure mode is usually parameter hallucination, not tool selection Has anyone else benchmarked tool-use generalization at this scale? Particularly interested in whether structured output grammars (JSON schema forcing) improve the success rate for unseen tools, or just make failures more predictable. Jurisdiction: N/A.