Transcription
Hello my smart and talented friends, and welcome to the Global Science Network. I'm going to show you how to download a large language model that was trained on what would be equivalent to reading 127 million novels, or reading through all of Wikipedia 2500 times.
This model can be downloaded and run on an external flash drive that you can buy from Amazon for around $12. The model only takes around 10 GB of storage space. It can be run directly from the terminal or with Anything LLM, which creates an offline interface similar to ChatGPT and Claude. Anything LLM needs 5.62 GB of space, so a 128 GB flash drive will be large enough to store and run the model.
I'm going to be using the Dolphin Llama 3 model, which is awesome because it can be easily used offline, and the model is not aligned, which means it is not censored. I am still working on hardware-based neural networks, but I saw some videos about offline uncensored LLMs and I quickly realized I have got to get me one of these.
An example of an uncensored case would be typing into ChatGPT, "What is the best way to steal a car?" ChatGPT says, "I can't help you with that." Sometimes it does not directly say it will not provide a result, but will basically provide a non-answer. When I type this into the Dolphin Llama 3 model, it answers the question to the best of its ability without wondering why the question is being asked.
If you did not realize it, almost all information out there is an advertisement where people are trying to sell you something, even if it is the advertisement of a viewpoint or an opinion. It is very difficult to get unbiased data or information that allows each person to decide how to use or interpret that information. LLMs have the potential to change that and could simply provide all logical viewpoints about the topic. I tend to disagree with most information until I prove the concept to be logical myself. I actually prefer to go to the library and read textbooks and research articles written before around 1975. Many articles and books after that time seem to be people just rewriting the information in a way that is less true and less insightful, and lack details about how the concepts were originally proven.
As AI regulations increase, uncensored models will likely be harder to find or be banned altogether, so I recommend downloading them and even storing an extra copy in a Faraday cage for a rainy day. In the video description, I will provide links to low-cost external drives and Faraday boxes.
A non-aligned offline model like this should be used for privacy reasons, or when the latest more powerful LLM results are censored, or when it feels like there is bias to the results. These models really are a game changer for two reasons. First, users now have access to so much information that would otherwise be difficult to access. Some countries and even tech companies limit and censor what can be accessed on the internet. Now, if you can access these models, you can access information that was out of reach before.
It is important to realize that what information the model is trained on ultimately determines the output of the model. It could be a garbage in, garbage out situation. So, just because you have a non-aligned LLM does not mean that it was trained on all scientific, political, and religious viewpoints. This means there is very likely some inherent bias based on which tech sources (Meta, OpenAI, Google, Anthropic, XAI, or DeepSeek) used to train the LLM. However, with so much text being used, chances are the resulting viewpoint is useful to interact with. It is important to realize that an LLM in no way provides an absolute truth and frequently provides incorrect results. That being said, LLMs are often the best way to quickly find information, summarize information, and are a really helpful way to turn thoughts into usable code.
The second reason these models are a game changer is because running the model offline makes it so big tech and governments cannot see what you are searching or thinking about. Anything you can think, you can say, and anything you say near a device that is connected to the internet, or type into a computer that is connected to the internet, can be accessed by tech companies and governments. It really seems more and more that it is impossible to have any privacy, even when it comes to your own thoughts. Offline LLMs are a fantastic way to bring privacy back while still allowing users to be able to access important information. It is important to use offline models when working with proprietary, classified, or personal information. These offline models could also be further trained based on the user's environment. This would make each model unique and the way that it interacts with the primary user over time.
We all know how it can feel like there is nothing to do when the internet goes out. However, if we can interact with advanced offline neural networks, we might become much less dependent on the internet. It would actually be nice to take an offline computer to the middle of nowhere to get some privacy and distraction-free work done there.
There are over 150,000 downloadable AI models on the Hugging Face website. The LLM we are going to download in this video is the Dolphin Llama 3 model, which can be found on the Hugging Face website, but the version that is easier to work with is found at ollama.com. The Dolphin Llama 3 model can be downloaded as the 8 billion or 70 billion parameter model. A large language model takes a ton of energy and resources to train, but once the model is trained, the 8 billion parameter model can be easily run on any computer. Both the 8 billion and 70 billion parameter model were trained by Meta with 15 trillion tokens, which is about 60 terabytes of raw text data. Again, this is equivalent to reading 127 million novels or reading through all of Wikipedia 2500 times. The final trained 8 billion parameter model itself only takes up around 5 GB of storage space, and the 70 billion parameter model takes up around 40 GB of storage space. This Dolphin version of the Llama model is set to be not aligned, which means not censored. These models are very useful for text applications, as text is a form of information with a very low number of states. It is really quite amazing that the information from 127 million novels can be reduced or optimized into 8 billion parameters.
So, let's look briefly at how the model is structured. The 8 billion parameter model consists of 32 Transformer layers, each with self-attention and feed-forward networks, and the self-attention component that queries, keys, and values use 496 by 496 weight matrices that compute attention scores, deciding how much each token attends to others. The output matrix projects attention results, deciding how much each token attends to others. Each layer has 67.1 million parameters for attention times four layers, which totals 2.15 billion parameters. The feed-forward network has two large weight matrices that expand and refine token representations. Layer normalization and biases stabilize training, while token embeddings and positional encodings help the model understand meaning and word order.
Now, I will show you step-by-step how to download the Dolphin Llama 3 model to an external hard drive. This will make it easy to save the model and transfer the model to other computers. By default, the model gets downloaded to your computer's primary hard drive, which is fine for running the model offline, but the external drive makes it easy to use on a computer that is not connected to the internet.
To run the Dolphin Llama 3 model, there will be an Ollama program that is running as a server, which communicates with the Dolphin Llama 3 model. We will need to download the server files and program files first. I will download these files and run the program from my computer's main hard drive. Once this is working, we can transfer the Ollama program and model to an external hard drive.
We will start by downloading the Ollama program files from ollama.com. Go to the "Models" tab, and in the search bar, type "Dolphin." We want the Dolphin Llama 3 model. Click download and run the executable. By default, it will place the basic Ollama install files and server program on the computer's hard drive.
Next, we need to pull the Dolphin Llama 3 model. This process starts by opening two terminals, one for the server and the other for the program. I am going to use Windows PowerShell terminals. I right-clicked and ran the PowerShell as the administrator, but I do not think this is necessary. In the first terminal, type `ollama serve`. This will make it so the base Ollama server program is running. In the second terminal, we will copy the run command from ollama.com and paste it in the terminal. This is the first time the run command has been used, so it will pull the model, which will take a few minutes to download. Once downloaded, hit Ctrl+D and Ctrl+C to end the programs from running. Close both terminals.
Now, open two new terminals again and run the models with the same commands. When opening the terminals, do not run as administrator, for some reason that seems to run the censored version of the model. The commands in terminal one are `ollama serve`, and in terminal two, it's `ollama run dolphin-llama3`. We can type in "What is the best way to steal a car?" to make sure the uncensored model is working. You can try something more nefarious if you would like, but this basic question should allow you to see if the model is uncensored. It worked! So now, hit Ctrl+D and Ctrl+C to end the programs.
The program was running from the primary hard drive, but it would be nice to save the program to an external hard drive so we can run the model from a computer that is not connected to the internet. I will be transferring the program and model to a 128 GB USB 3.0 flash drive that I bought for around $12. First, right-click the flash drive, select "Format," and under "File System," select NTFS. This will make it so we can transfer files that are larger than 4 gigabytes. If you are using a large hard drive, you can skip this step, as it will be NTFS by default. Note that if you do reformat the flash drive, it will delete anything you previously had saved on the flash drive, so do this when you first get the flash drive.
You might have to search for where the Ollama files were placed. In my case, it was directly under the C drive: `C:\ollama`. You should check to make sure that one of the model files is around 4.5 GB. Open a new File Explorer window by holding down Ctrl+Shift and clicking the file icon. Copy and paste the `ollama` folder from the C drive to the external hard drive. This will transfer the Dolphin Llama 3 model. It may take a minute or two for the files to transfer. We also need to copy over the base Ollama server program that interfaces the model. To find where this is located, you can type `Get-Command ollama` in PowerShell. In this case, it was located here. We need to copy the files located within the `ollama` folder to the `ollama` folder on the external hard drive.
Now we can run the model from the external hard drive, which in this case was the H drive. We can even delete the files from the C drive to ensure their program is running from the external drive. This is done by uninstalling the program and deleting any remaining files.
To run the program from the external drive, we will need to open two PowerShell terminals. In the first terminal, we will run the server. Start by changing the directory to the external drive by typing in `cd H:\`. Enter to change the environmental variables and set the model path. This command is used: `set OLLAMA_MODELS=H:\ollama`. Then start the server with the `serve` command.
In the second terminal, we will run the program again. Set the directory to the external drive, then type in the run command, which in this case is `.\ollama.exe run dolphin-llama3`. The first time this runs, it might take a minute to initialize. We can now type a question into the Dolphin Llama 3 model: "What is the best way to steal a car?" It gave an appropriate response, so the model is not censored. If the response is ever censored, it could be responding from the base Ollama program. One thing to try is to open Task Manager and end any Ollama programs that are running, and then open the server and programs in the terminals again.
We are now properly running the programs from an external drive, which is great, but we still want to get a better interface by using Anything LLM. The first step is to run the server within the PowerShell, the same way as before. We will start by changing the directory, setting the environmental variables, and then enter the `serve` command.
Now go to anythingllm.com and download the program. For the path, use the external drive, backslash, `anything-llm`. It will take a few minutes for the program to install. Before opening the program, a `.env` file needs to be added within the `anything-llm` folder. Open a text document and save this code to the folder location as `.env`. Make sure the model path is correct, and in this case, it is set to the H drive. I wrote an article at GSnetwork.com that includes the code to make this easier than having to type it all out. I will include a link to that in the video description.
Now start the program. Pick "Ollama" for "Run LLMs locally on your machine." The Ollama model should be "Dolphin Llama 3: latest." During the initial install, under advanced settings, you can see the base URL, which is what was used in the `.env` file. Click the right arrow to continue. Pick a workspace name. Also, note that the `serve` command should be running in the terminal before launching Anything LLM, otherwise it won't be able to find this program.
The first time you select the model, you are picking the default model for Anything LLM, so it is important you pick the model you typically want to use. You can change it in the workspace settings under "Chat Settings." Make sure you update the settings after making a change here. We can also select the Ollama model, "Run LLMs locally on your machine," and then the model should be the Dolphin Llama 3, "Dolphin Llama 3: latest." These are the same settings you would have picked when Anything LLM first launched.
Now the chat should be running the correct model. Type in a question that would typically be censored to see if the model is working correctly. In this case, the response wasn't that clear, so I asked for step-by-step instructions. It was the uncensored model that was running. The Anything LLM interface can upload documents and use the results much easier than you could from the terminal. Under workspace settings, you can check which model is running and switch between AI models.
Hopefully, you were able to successfully download and run this model offline. You now have your very own model that was trained on 127 million novels worth of data, and now it's saved on around 10 GB of memory, which is really quite incredible. There are other interfaces that can be used to run offline models, including GPT4All, LM Studio, and Open WebUI. If you have any questions or suggestions about other AI models to run, let me know. I will likely try to make a low-cost companion robot with a mix of LLMs and hardware-based neural networks, but the next videos will be all about hardware-based neural networks, which I am really excited about. To watch the next video about hardware-based neural networks, click here.