Llama.cpp
≈ 2 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
llama.cpp is an open-source software library that performs inference on various large language models such as Meta's Llama model. It is co-developed alongside the project, a general-purpose tensor library. Command-line tools are included with the library, alongside a server with a simple web interface. llama.cpp has been considered as the de facto standard as the core of almost all local inference tools, including Ollama and LM Studio.
llama.cpp is an open-source software library that performs inference on various large language models such as Meta's Llama model. It is co-developed alongside the project, a general-purpose tensor library. Command-line tools are included with the library, alongside a server with a simple web interface. llama.cpp has been considered as the de facto standard as the core of almost all local inference tools, including Ollama and LM Studio. // example metadata general.architecture: 'llama', general.name: 'LLaMA v2', llama.context_length: 4096, ... , general.file_type: 10, // (typically indicates quantization level, here "MOSTLY_Q2_K") tokenizer.ggml.model: 'llama', tokenizer.ggml.tokens: [ ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ... ], ... // n-th tensor name: GGUF string, // ex: "blk.0.ffn_gate.weight" n_dimensions: UINT32, // ex: 2 dimensions: UINT64[], // ex: [ 4096, 32000 ] type: UINT32, // ex: 10 (typically indicates quantization level, here "GGML_TYPE_Q2_K") offset: UINT64 // starting position within the tensor_data block, relative to the start of the block // (n+1)-th tensor ...
Text derived from https://en.wikipedia.org/w/index.php?oldid=1378655897 under CC BY-SA 4.0; modifications have been made.