Lars

@mkreutzfeldt@mastodon.social Glückwunsch! Bitte sag mir, dass er es nicht wieder als „eine Meinung“ oder „er hat bessere Experten gefragt“ abtut und sie ihn damit durchkommen lässt…

@mitsuhiko@hachyderm.io Okay, re-read it carefully and I think understand more clearly where you are coming from. One thought I immediately had when you mentioned all of the different decisions required to get an optimal local model was to slap a small benchmark on this and use something like optuna to optimize this awkward parameter space. Good luck with this endeavor; I’m unfortunately not in the 128GB Mac Studio Club 😅

@mitsuhiko Thanks for the write-up and thoughts! Are you already at a scale where DB performance comes into play? For example, how many steps per second this system can handle on a small-ish Postgres instance?

@mitsuhiko@hachyderm.io I’m sure you are aware of it but I did not see it mentioned in your post: There is llamafile by Mozilla which seems to me adresses the same issue. Do you think this is a good approach or do you have issues with it? https://mozilla-ai.github.io/llamafile/

@mitsuhiko@hachyderm.io Gotcha, you wish there were more „single-binary agents“ not just inference engines? In this regard I think you might be right, that there is not really a big selection of these around.

@mitsuhiko@hachyderm.io I guess I meant „single binary“ to mean easy to install. Don’t you feel we are quite close to this, at least on the inference side? I think llama.cpp is basically the basis for all local LLM projects, and also on the server side, we are basically at mostly vLLM with a side of SGlang and Triton. Given the development speed of newly released models and techniques like DDTree, I’m not sure it’s time to focus on a single model; who knows what will be released come monday.

@mitsuhiko@hachyderm.io I thought I did, but I guess my AI age attention span got the better of me. Will re read! Thanks for taking the time to answer 😊 And while I have your attention: Thanks for Flask and your „state of agentic coding“ series. Really loving it.