this post was submitted on 29 Jan 2025
150 points (96.9% liked)
Technology
71760 readers
4720 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
In one story they're using PTX on Nvidia H800s. In another they're on Huawei chips.
Which is it? Are we all just hypothesising?
Not the best on AI/LLM terms, but I assume that training the models was done on Nvidia, while inference (using the model/getting the data from the model) is done on Huawei chips
To add: Training the model is a huge single-cost expense, while inference is a continuous expense.
Wait, so after you train, you don't need all those fancy Nvidia chips?
They should make one place where there is an overabundance of geo thermal energy and train all models there...
Yes, so R&D and finalizing the model weight is done on NVIDIA GPUs (I guess you need an excessive amount of VRAM).
Inference is probably gonna be offloaded to consumers in the end where the NPU is taking care of the inference cost (See Apple, Qualcomm etc)