Listen to the audio version of the article
We know that the most famous large language models (Gpt4, Gemini Ultra, Microsoft Copilot,) are only accessible online through their respective companies’ cloud. We know that I can also read data and words on the Web and thus formulate answers to our questions by integrating this information with the datasets they have trained on. However, there are also Llm (Laege Language Model) on devices which can be consulted offline and which use the device’s own computing power. Chat With RTX by Nvidia is a demo app that allows you to have your own personalized chatbot directly on your PC. Presented in January, we tested it on a very powerful Legion Pro 7i equipped with an Nvidia GeForce RTX 4090 graphics card. As Nvidia was keen to underline, this is a tech demo which at present only serves to help us understand the potential of the technology. Technically it combines RAG (augmented recovery generation), the TensorRT-LLM model and the acceleration of the latest Nvidia cards (from the 30 series onwards and with at least 8GB of VRAM). Rag in particular is an AI technique we’ll hear a lot about because it allows large language models (LLMs) to take advantage of additional data resources without the need for retraining. It means that the chatbot can be “fed” with information outside the dataset on which it was trained. Hence the possibility of customizing it by giving it access to files in pdf, doc/docx and xml format present on your computer. The limit should be 500 MB of text files. Additionally, you can specify the URL of a YouTube playlist and the app will load transcripts of the videos in the playlist, allowing you to query the content covered.
What is it for? in theory you can give him technical manuals to read, hundreds of page guides that you can then query. During my test I gave him the documents for a review guide for a giant video game and he answered correctly. However, it must be said that to date it is not a conversational chatbot like Gpt4 or Gemini. Makes inaccuracies. Furthermore, for now it is only in English. If you give him a YouTube video in Italian he struggles, but he is more precise with English. As mentioned, it is a tech demo that demonstrates how having a local LLM translates into greater security and privacy, because the large language model (LLM) runs locally on a PC and can be combined with a customized dataset, chosen by the user. It is therefore a very safe solution compared to the use of data that remains on the machine and does not have to be processed on third-party servers. However imperfect it may be, it is already a concrete piece of the future of generative intelligence, certainly for professionals, artists and content creators. To date it exploits the capacity of two open source Llms such as the French Mistral Ai and Meta’s Llama2. But Google’s two Gemma chatbots will also arrive soon.