LocalLLaMA
Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.
Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.
As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.
Rules:
Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.
Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.
Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.
Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.
view the rest of the comments
Uhh... Tool calling is built into thier tokenizers, but Ollama/Langchain just ignore them because they're spagetti abstractions. To be blunt, langchain and ollama are overhyped, buggy junk trying to reinvent wheels.
For any kind of STEM work, I'd run Llama Nemotron 49B exl3 via TabbyAPI, which exposes a generic openai endpoint anything can use:
https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1
https://huggingface.co/turboderp/Llama-3.3-Nemotron-Super-49B-v1-exl3/tree/3.0bpw
Nemotron models freaking rock at anything STEM-adjacent, and I can get squeeze in 48K+ context on 24GB VRAM (depending on your cache quantization settings).
Otherwise GLM-4 is very good at tool calling, as is Qwen3, and you can more confortable run them as GGUFs if you don't want to leave the llama.cpp ecosystem, or exl2s if you have specific trouble with exl3 in TabbyAPI.
Wow this is some awese information Brucethemoose thanks for sharing!
I hope you dont mind if I ask some things. Tool calling is one of those things I'm really curious about. Sorry if this is too much please dont feel pressured you dont need to answer everything or anything at all. Thanks for being here.
I feel like a lot of people including myself only vaguely understand tool calling, how its supposed to work, and simple practice excersises to use it on via scripts and APIs. What's a dead simple python script someone could cook to tool call within the openai-compatable API?
In your own words what exactly is tool calling and how does an absolute beginner tap into it? Could you clarify what you mean by 'tool calling being built into their tokenizers'?
Would you mind sharing some sources where we can learn more? I'm sure huggingface has courses but maybe you know some harder to find sources?
Is tabbyAPI an engine similar to ollama, llama.cpp, ect?
What is elx2,3, ect?
I think tool calling is just a form of structured output, where the LLM outputs something like json that describes the function to call and the arguments, then you parse that and run the function with the arguments, then feed the output back to the LLM in a new message, if you want. IDK the specific details, I'm guessing there are some special tokens some LLMs produce for tool calling, and I'm also guessing there is "controlled generation" (masking the logits/tokens, and only choosing to generate the tokens that would be valid). Ollama apparently doesn't support the special tool-calling tokens or output structure that some models use.
This is how it looks when using OpenAI compatible APIs: https://platform.openai.com/docs/guides/function-calling?api-mode=responses.
I've never heard of TabbyAPI (I'm new to using local models, in general). I'm also not sure what TabbyAPI brings over Ollama, llama.cpp, or vLLM. So would be curious as well.
Edit: For my little toy project I'm working on, I'm switching to not using tool-calling at all, and building a LangGraph and using structured output, which should be more reliable with my use-case. I.e. just always call the tool manually, feed the output back to the LLM to have the LLM evaluate if the output was correct, retry calling the tool with different arguments if not, and just fail after 5 calls.
I like watching Yannic Kilcher, to keep up with some of the newer developments, and read various papers. I do have a background in ML (mostly the more traditional, non-generative, supervised learning and reinforcement learning) though.