Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Their killer feature is the --grammar option which restricts the logits the LLM outputs which makes them great for bash scripts that do all manner of NLP classification work.

Otherwise I use ollama when I need a local LLM, vllm when I'm renting GPU servers, or OpenAI API when I just want the best model.



Interesting, you inspired me to https://github.com/jmorganca/ollama/issues/1507

I've also had good success instructing arbitrary grammars in the system prompt, though it doesn't work at the logits level, which can be helpful


There's an old PR for it: https://github.com/jmorganca/ollama/pull/565 (it just uses the underlying llama.cpp grammar feature which is what llamafile does)


thanks for the pointer, I'll update my issue to reference that PR




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: