I already have stable diffusion on a local machine. I was trying to find motivation to install a LLM locally. You answered my question in a different response
use cases where customization helps while quality does matter much due to scale, i.e spam, then LLMs and related tools are amazing.
I want to work on my stuff in peace and in private without worrying about a company grabbing my stuff and using it for themselves and to give/sell it to other outfits, including the government. “If you have nothing to hide…” is bullshit and needs to die.
At the same time, the trouble with local LLMs is that they’re very resource heavy. Your average household computer isn’t going to be able to run one with much usability or speed.
Which, you know, is fine. Maybe if people had an idea of how much power is required to run them, they would think twice before using a gigawatt to output a poem about farts, and perhaps even wonder how OpenAI can offer that for free.
Btw, a 7b model should run ok on any PC with at least 16GB of RAM and a modern processor/GPU.
it’s a lot slower that chatgpt but on my integrated graphics i7 laptop it ran decent, def enough to be useable. Also there’s different models to play around with, some are faster but worse and some are smarter but slower
Okay but what problem does that solve? Is the solution setting up our own spambots to fill forums with arguments counter to their bullshit spambots? I don’t see how an LLM improves literally anything ever in any circumstance.
You seem unnecessarily hostile about this. If you don’t like LLM just move on.
This is exactly why this sub about technology is better off without business news. You’re just reacting to something you hate and directing that at others.
FWIW I did try a lot (LLMs, code, generative AI for images, 3D models) in a lot of ways (CLI, Web based, chat bot) both locally and using APIs.
I don’t use any on a daily basis. I find it exciting that we can theoretically do a lot “more” automatically but… so far the results have not been worth the efforts. Sadly some of the best use cases are exactly what you highlighted, i.e low effort engagement for spam. Overall I find that either working with a professional (script writer, 3D modeler, dev, designer, etc) is a lot more rewarding but also more efficient which itself makes it cheaper.
For use cases where customization helps while quality does matter much due to scale, i.e spam, then LLMs and related tools are amazing.
PS: I’d love to hear the opinion of a spammer actually, maybe they also think it’s not that efficient either.
I have personally found generative-text LLMs quite good for creating titles. As an example, I have a few hundred tweets that I’m trying to put into a file, and I’ll use an LLM to create a human-readable name for them. It’s much better than a lot of the other summarisation mechanisms (like BERT) I’ve tried with it, but it’s still not perfect, because the model tends to output the same thing in slightly different words each time, so repeat runs will often result in the same thing with a different title.
reminder, there are localy ran LLMs. Right now is a vital time for open source to fight against closed source in the AI arms race.
https://www.nomic.ai/gpt4all
Another good resource to help people find models https://llm.extractum.io
Or just straight up install https://ollama.com
I like Ollama, and recommend it to tinker, but I admit this “LLM Explorer” is quite neat thanks to sections like “LLMs Fit 16GB VRAM”
Ollama just works but it doesn’t help to pick which model best fits your needs.
What is the need I have to put the effort in to install all this locally. Websites win in terms of convenience.
I don’t think I understand your point, are you saying there is no benefit in running locally and that Websites or APIs are more convenient?
I already have stable diffusion on a local machine. I was trying to find motivation to install a LLM locally. You answered my question in a different response
I want to work on my stuff in peace and in private without worrying about a company grabbing my stuff and using it for themselves and to give/sell it to other outfits, including the government. “If you have nothing to hide…” is bullshit and needs to die.
Good point. Everything you feed into chatgpt is stored for future reference.
At the same time, the trouble with local LLMs is that they’re very resource heavy. Your average household computer isn’t going to be able to run one with much usability or speed.
Which, you know, is fine. Maybe if people had an idea of how much power is required to run them, they would think twice before using a gigawatt to output a poem about farts, and perhaps even wonder how OpenAI can offer that for free. Btw, a 7b model should run ok on any PC with at least 16GB of RAM and a modern processor/GPU.
Phi 3 can run on pretty low specs (requires 4gb RAM) and has relatively good output
it’s a lot slower that chatgpt but on my integrated graphics i7 laptop it ran decent, def enough to be useable. Also there’s different models to play around with, some are faster but worse and some are smarter but slower
Okay but what problem does that solve? Is the solution setting up our own spambots to fill forums with arguments counter to their bullshit spambots? I don’t see how an LLM improves literally anything ever in any circumstance.
You seem unnecessarily hostile about this. If you don’t like LLM just move on.
This is exactly why this sub about technology is better off without business news. You’re just reacting to something you hate and directing that at others.
But answer the question maybe
Also, my “hate” was very clearly directed towards LLMs and not a “person”.
FWIW I did try a lot (LLMs, code, generative AI for images, 3D models) in a lot of ways (CLI, Web based, chat bot) both locally and using APIs.
I don’t use any on a daily basis. I find it exciting that we can theoretically do a lot “more” automatically but… so far the results have not been worth the efforts. Sadly some of the best use cases are exactly what you highlighted, i.e low effort engagement for spam. Overall I find that either working with a professional (script writer, 3D modeler, dev, designer, etc) is a lot more rewarding but also more efficient which itself makes it cheaper.
For use cases where customization helps while quality does matter much due to scale, i.e spam, then LLMs and related tools are amazing.
PS: I’d love to hear the opinion of a spammer actually, maybe they also think it’s not that efficient either.
I have personally found generative-text LLMs quite good for creating titles. As an example, I have a few hundred tweets that I’m trying to put into a file, and I’ll use an LLM to create a human-readable name for them. It’s much better than a lot of the other summarisation mechanisms (like BERT) I’ve tried with it, but it’s still not perfect, because the model tends to output the same thing in slightly different words each time, so repeat runs will often result in the same thing with a different title.
But, that is also a fairly limited use case.