AI is getting more and more popular with the DeepSeek being online during the Chinese New Year and it suddenly becomes available to the general public to play with even there are no very strong GPU or high specs of computing resources. So more and more people share how they deploy and run LLM on their machine locally, maybe with some smaller models, like 7b, 14b or even 32b.
If you ask me what the value is for most of us general population to run the LLM locally, and very probably the very basic models which are not smart as the ones provided to the public from free from those vendors. I don’t know. I don’t think it’s worth for most of us to invest those resources to have a purely private LLM or AI agent for ourselves because firstly, it might not be as smart and convenient as the commercial ones provided by the LLM vendors for free and secondly, it’s a lot of cost to have one, even a small one, running and available from the internet, unless it’s just hosed within home network and can only be accessed inside home or lab network environment. The best value I can see from running the LLM locally is we can have a better understanding on how it works because if it’s going to be the normal trends for the public, having a better understanding of how it works will give us much advantages, especially if you are IT or IT related person. It’s like cloud several years ago when it just got some popularity. People were still debating whether Cloud will replace the on-premise infra, or only partially, but now it’s the normal situation for most of the companies. Most of the companies are either purely on cloud or with Hybrid infra and the cloud part will probably grow bigger and bigger. For now, knowing some cloud skills will definitely bring advantages to the job market.
Like many of the people who have tried to set up LLM running locally and have a AI agent or chatbox like functionality available, I have done the same in my lab environment and have played with it for quite some time. The next step I am planning now is to try to figure out how the Agent or the new setup can help me with my normal work or lab exploring, not just information searching but something more, like maybe helping me analyze the issues and then search for the solutions and then put the solution in deployment to fix the issues. While I know it’s a huge step forward and there must be more problems and questions to be fixed before this goal can be reached finally, for example, how should I censer the permissions and make sure the AI will help to fix the issue, instead of introducing more issues or even breaking down the whole infra. This is big topic and I think with the development of AI, it’s something that can be done, or maybe partially.
Will keep exploring on what new generations of AI can bring us and help us to work more efficiently.