this post was submitted on 12 Jun 2023
6 points (100.0% liked)

LocalLLaMA

3485 readers
143 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

Rules:

Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.

Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.

Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.

Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.

founded 2 years ago
MODERATORS
 

Let's talk about our experiences working with different models, either known or lesser-known.

Which locally run language models have you tried out? Share your insights, challenges, or anything you found interesting during your encounters with those models.

top 7 comments
sorted by: hot top controversial new old
[–] [email protected] 2 points 2 years ago

Guanaco, WizardLM (uncensored) and Camel-13b have been the best models I've tried that are 13b+.

Surprisinly, the LaMini-LM (Flan 3b) and OpenLlama (3b) have performed very well for smaller models.

[–] [email protected] 1 points 2 years ago (1 children)

I'd have to say I'm very impressed with WizardLM 30B (the newer one). I run it in GPT4ALL, and even though it is slow the results are quite impressive.

Looking forward to Orca 13b if it ever releases!

[–] [email protected] 1 points 2 years ago

Which one is the "newer" one? Looking at the quantised releases by TheBloke, I only see one version of 30B WizardLM (in multiple formats/quantisation sizes, plus the unofficial uncensored version).

[–] [email protected] 0 points 2 years ago (1 children)

guanaco-65B is my favorite. It's pretty hard to go back to 33B models after you've tried a 65B.

It's slow and requires a lot of resources to run though. Also, not like there are a lot of 65B model choices.

[–] [email protected] 1 points 2 years ago* (last edited 2 years ago) (1 children)

What do you even run a 65b model on?

[–] [email protected] 2 points 2 years ago (1 children)

With a quantized GGML version you can just run on it on CPU if you have 64GB RAM. It is fairly slow though, I get about 800ms/token on a 5900X. Basically you start it generating something and come back in 30minutes or so. Can't really carry on a conversation.

[–] [email protected] 1 points 2 years ago

Is it smart enough that it can get the thread of what you are looking for without as much rerolling or handholding, so this comes out better?