Selfhosted

48329 readers

1231 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago

MODERATORS

[email protected]

100

What is a self-hosted small LLM actually good for (<= 3B) (lemmy.world)

submitted 1 day ago* (last edited 1 day ago) by [email protected] to c/[email protected]

50 comments fedilink hide all child comments

I've tried coding and every one I've tried fails unless really, really basic small functions like what you learn as a newbie compared to say 4o mini that can spit out more sensible stuff that works.

I've tried explanations and they just regurgitate sentences that can be irrelevant, wrong, or get stuck in a loop.

So. what can I actually use a small LLM for? Which ones? I ask because I have an old laptop and the GPU can't really handle anything above 4B in a timely manner. 8B is about 1 t/s!

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 2 points 23 hours ago* (last edited 23 hours ago) (22 children)

Currently I've been using a local AI (a couple different kinds) to first - take the audio from a Twitch stream; so that I have context about the conversation, convert it to text, and then use a second AI; an LLM fed the first AIs translation + twitch chat and store 'facts' about specific users so that they can be referenced quickly for a streamer who has ADHD in order to be more personable.

That way, the guy can ask User X how their mothers surgery went. Or he can remember that User K has a birthday coming up. Or remember that User G's son just got a PS5 for Christmas, and wants a specific game.

It allows him to be more personable because he has issues remembering details about his users. It's still kind of a big alpha test at the moment, because we don't know the best way to display the 'data', but it functions as an aid.

[–] [email protected] 1 points 20 hours ago (1 children)

Surely none of that uses a small LLM <= 3B?

[–] [email protected] 1 points 16 hours ago* (last edited 16 hours ago) (1 children)

Yes. The small LLM isn't retrieving data, it's just understanding context of text enough to know what "Facts" need to be written to a file. I'm using the publicly released Deepseek models from a couple of months ago.

[–] [email protected] 1 points 1 hour ago

Some questions and because you don't actually understand, also, the answers.

what does the LLM understand the context of, (other user's data owned by Twitch)
How is the LLM fed that data? (You store it and feed it to the LLM)
Do you use Twitch's data and its users data through an AI without their consent? (Most likely, yes)
Do you have consent from the users to store 'facts' about them (You're pissy, so obviously not)
Are you then storing that processed data? (Yes, you are, written to a file)
Is the purpose this data processing commercial (Yes, it is, designed to increase viewer count for the user of this system - and before you retort "OMG it helps twitch too"... Uhm no, Twitch has the viewers if not watching him, watching someone else)

I mean yeah, it's a use case, but own up to the fact that you're wrong. Or be pissy. I don't care.

load more comments (20 replies)