From 37681dd4e8a37cba86fd7bd75830f3f5fc09ed27 Mon Sep 17 00:00:00 2001 From: Albarqawi Date: Sat, 16 Sep 2023 07:43:46 +0100 Subject: [PATCH 001/117] Update the tools in the README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 3007c00..90e09d2 100644 --- a/README.md +++ b/README.md @@ -307,6 +307,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Langroid](https://github.com/langroid/langroid) - Harness LLMs with Multi-Agent Programming - [Embedchain](https://github.com/embedchain/embedchain) - Framework to create ChatGPT like bots over your dataset. - [CometLLM](https://github.com/comet-ml/comet-llm) - A 100% opensource LLMOps platform to log, manage, and visualize your LLM prompts and chains. Track prompt templates, prompt variables, prompt duration, token usage, and other metadata. Score prompt outputs and visualize chat history all within a single UI. +- [IntelliServer](https://github.com/intelligentnode/IntelliServer) - simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models. ## Tutorials about LLM From 1ec53270abdae204bd7ebdfd482486102a67d9f6 Mon Sep 17 00:00:00 2001 From: "Xin(Leo) Jing" Date: Mon, 18 Sep 2023 23:18:11 -0700 Subject: [PATCH 002/117] Update README.md --- README.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/README.md b/README.md index 8cfa91a..7dedf98 100644 --- a/README.md +++ b/README.md @@ -310,6 +310,28 @@ The above tables coule be better summarized by this wonderful visualization from - [IntelliServer](https://github.com/intelligentnode/IntelliServer) - simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models. +## Prompting libraries & tools + +- [YiVal](https://github.com/YiVal/YiVal): Evaluate and Evolve — YiVal is an open-source GenAI-Ops tool for tuning and evaluating prompts, configurations, and model parameters using customizable datasets, evaluation methods, and improvement strategies. +- [Guidance](https://github.com/microsoft/guidance): A handy looking Python library from Microsoft that uses Handlebars templating to interleave generation, prompting, and logical control. +- [LangChain](https://github.com/hwchase17/langchain): A popular Python/JavaScript library for chaining sequences of language model prompts. +- [FLAML (A Fast Library for Automated Machine Learning & Tuning)](https://microsoft.github.io/FLAML/docs/Getting-Started/): A Python library for automating selection of models, hyperparameters, and other tunable choices. +- [Chainlit](https://docs.chainlit.io/overview): A Python library for making chatbot interfaces. +- [Guardrails.ai](https://shreyar.github.io/guardrails/): A Python library for validating outputs and retrying failures. Still in alpha, so expect sharp edges and bugs. +- [Semantic Kernel](https://github.com/microsoft/semantic-kernel): A Python/C#/Java library from Microsoft that supports prompt templating, function chaining, vectorized memory, and intelligent planning. +- [Prompttools](https://github.com/hegelai/prompttools): Open-source Python tools for testing and evaluating models, vector DBs, and prompts. +- [Outlines](https://github.com/normal-computing/outlines): A Python library that provides a domain-specific language to simplify prompting and constrain generation. +- [Promptify](https://github.com/promptslab/Promptify): A small Python library for using language models to perform NLP tasks. +- [Scale Spellbook](https://scale.com/spellbook): A paid product for building, comparing, and shipping language model apps. +- [PromptPerfect](https://promptperfect.jina.ai/prompts): A paid product for testing and improving prompts. +- [Weights & Biases](https://wandb.ai/site/solutions/llmops): A paid product for tracking model training and prompt engineering experiments. +- [OpenAI Evals](https://github.com/openai/evals): An open-source library for evaluating task performance of language models and prompts. +- [LlamaIndex](https://github.com/jerryjliu/llama_index): A Python library for augmenting LLM apps with data. +- [Arthur Shield](https://www.arthur.ai/get-started): A paid product for detecting toxicity, hallucination, prompt injection, etc. +- [LMQL](https://lmql.ai): A programming language for LLM interaction with support for typed prompting, control flow, constraints, and tools. + + + ## Tutorials about LLM - [Andrej Karpathy] State of GPT [video](https://build.microsoft.com/en-US/sessions/db3f4859-cd30-4445-a0cd-553c3304f8e2) - [Hyung Won Chung] Instruction finetuning and RLHF lecture [Youtube](https://www.youtube.com/watch?v=zjrM-MW-0y0) From 580e1a9391080d8847749c2df607f1f7bec638c4 Mon Sep 17 00:00:00 2001 From: "Xin(Leo) Jing" Date: Mon, 18 Sep 2023 23:19:40 -0700 Subject: [PATCH 003/117] Update README.md --- README.md | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 7dedf98..af02ec5 100644 --- a/README.md +++ b/README.md @@ -312,23 +312,23 @@ The above tables coule be better summarized by this wonderful visualization from ## Prompting libraries & tools -- [YiVal](https://github.com/YiVal/YiVal): Evaluate and Evolve — YiVal is an open-source GenAI-Ops tool for tuning and evaluating prompts, configurations, and model parameters using customizable datasets, evaluation methods, and improvement strategies. -- [Guidance](https://github.com/microsoft/guidance): A handy looking Python library from Microsoft that uses Handlebars templating to interleave generation, prompting, and logical control. -- [LangChain](https://github.com/hwchase17/langchain): A popular Python/JavaScript library for chaining sequences of language model prompts. +- [YiVal](https://github.com/YiVal/YiVal) — Evaluate and Evolve: YiVal is an open-source GenAI-Ops tool for tuning and evaluating prompts, configurations, and model parameters using customizable datasets, evaluation methods, and improvement strategies. +- [Guidance](https://github.com/microsoft/guidance) — A handy looking Python library from Microsoft that uses Handlebars templating to interleave generation, prompting, and logical control. +- [LangChain](https://github.com/hwchase17/langchain) — A popular Python/JavaScript library for chaining sequences of language model prompts. - [FLAML (A Fast Library for Automated Machine Learning & Tuning)](https://microsoft.github.io/FLAML/docs/Getting-Started/): A Python library for automating selection of models, hyperparameters, and other tunable choices. -- [Chainlit](https://docs.chainlit.io/overview): A Python library for making chatbot interfaces. -- [Guardrails.ai](https://shreyar.github.io/guardrails/): A Python library for validating outputs and retrying failures. Still in alpha, so expect sharp edges and bugs. -- [Semantic Kernel](https://github.com/microsoft/semantic-kernel): A Python/C#/Java library from Microsoft that supports prompt templating, function chaining, vectorized memory, and intelligent planning. -- [Prompttools](https://github.com/hegelai/prompttools): Open-source Python tools for testing and evaluating models, vector DBs, and prompts. -- [Outlines](https://github.com/normal-computing/outlines): A Python library that provides a domain-specific language to simplify prompting and constrain generation. -- [Promptify](https://github.com/promptslab/Promptify): A small Python library for using language models to perform NLP tasks. -- [Scale Spellbook](https://scale.com/spellbook): A paid product for building, comparing, and shipping language model apps. -- [PromptPerfect](https://promptperfect.jina.ai/prompts): A paid product for testing and improving prompts. -- [Weights & Biases](https://wandb.ai/site/solutions/llmops): A paid product for tracking model training and prompt engineering experiments. -- [OpenAI Evals](https://github.com/openai/evals): An open-source library for evaluating task performance of language models and prompts. -- [LlamaIndex](https://github.com/jerryjliu/llama_index): A Python library for augmenting LLM apps with data. -- [Arthur Shield](https://www.arthur.ai/get-started): A paid product for detecting toxicity, hallucination, prompt injection, etc. -- [LMQL](https://lmql.ai): A programming language for LLM interaction with support for typed prompting, control flow, constraints, and tools. +- [Chainlit](https://docs.chainlit.io/overview) — A Python library for making chatbot interfaces. +- [Guardrails.ai](https://shreyar.github.io/guardrails/) — A Python library for validating outputs and retrying failures. Still in alpha, so expect sharp edges and bugs. +- [Semantic Kernel](https://github.com/microsoft/semantic-kernel) — A Python/C#/Java library from Microsoft that supports prompt templating, function chaining, vectorized memory, and intelligent planning. +- [Prompttools](https://github.com/hegelai/prompttools) — Open-source Python tools for testing and evaluating models, vector DBs, and prompts. +- [Outlines](https://github.com/normal-computing/outlines) — A Python library that provides a domain-specific language to simplify prompting and constrain generation. +- [Promptify](https://github.com/promptslab/Promptify) — A small Python library for using language models to perform NLP tasks. +- [Scale Spellbook](https://scale.com/spellbook) — A paid product for building, comparing, and shipping language model apps. +- [PromptPerfect](https://promptperfect.jina.ai/prompts) — A paid product for testing and improving prompts. +- [Weights & Biases](https://wandb.ai/site/solutions/llmops) — A paid product for tracking model training and prompt engineering experiments. +- [OpenAI Evals](https://github.com/openai/evals) — An open-source library for evaluating task performance of language models and prompts. +- [LlamaIndex](https://github.com/jerryjliu/llama_index) — A Python library for augmenting LLM apps with data. +- [Arthur Shield](https://www.arthur.ai/get-started) — A paid product for detecting toxicity, hallucination, prompt injection, etc. +- [LMQL](https://lmql.ai) — A programming language for LLM interaction with support for typed prompting, control flow, constraints, and tools. From 748bbb80a2decfbf25577f69a8e41d3450b5e8ba Mon Sep 17 00:00:00 2001 From: Su-Nadendla <110320693+Sumedhn97@users.noreply.github.com> Date: Thu, 28 Sep 2023 15:36:08 -0400 Subject: [PATCH 004/117] Update README.md --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index af02ec5..cb774db 100644 --- a/README.md +++ b/README.md @@ -137,6 +137,10 @@ If you're interested in the field of LLM, you may find the above list of milesto > Finetune a language model on a collection of tasks described via instructions +- [Retrieval-Augmented Generation](paper_list/instruction-tuning.md) + + > Retrieval-Augmented Generation (RAG) combines retrieval from a corpus with generative text models to enhance response accuracy using external knowledge. + ## LLM Leaderboard
From 530df7672bcf05fe72871e32af021afe2b2e07a8 Mon Sep 17 00:00:00 2001 From: Su-Nadendla <110320693+Sumedhn97@users.noreply.github.com> Date: Thu, 28 Sep 2023 15:41:48 -0400 Subject: [PATCH 005/117] Create Retrieval-Augmented Generation.md --- paper_list/Retrieval-Augmented Generation.md | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 paper_list/Retrieval-Augmented Generation.md diff --git a/paper_list/Retrieval-Augmented Generation.md b/paper_list/Retrieval-Augmented Generation.md new file mode 100644 index 0000000..9f6b197 --- /dev/null +++ b/paper_list/Retrieval-Augmented Generation.md @@ -0,0 +1,4 @@ +# Retrieval-Augmented Generation +> Large language models (LLMs) demonstrate an in-context learning (ICL) ability, that is, learning from a few examples in the context. +## Useful Resource +- [Retrieval-Augmented Generation_Paper](https://arxiv.org/abs/2005.11401v4) - The Original Paper on RAG published by Meta in 2020. From 28d44c3f9ce4596e5af760c058cdce901415d868 Mon Sep 17 00:00:00 2001 From: Su-Nadendla <110320693+Sumedhn97@users.noreply.github.com> Date: Thu, 28 Sep 2023 15:42:55 -0400 Subject: [PATCH 006/117] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index cb774db..d6798fb 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,7 @@ If you're interested in the field of LLM, you may find the above list of milesto > Finetune a language model on a collection of tasks described via instructions -- [Retrieval-Augmented Generation](paper_list/instruction-tuning.md) +- [Retrieval-Augmented Generation](paper_list/Retrieval-Augmented Generation.md) > Retrieval-Augmented Generation (RAG) combines retrieval from a corpus with generative text models to enhance response accuracy using external knowledge. From fd7ff1b034d4f73bd25598e238a2cefee840b000 Mon Sep 17 00:00:00 2001 From: Su-Nadendla <110320693+Sumedhn97@users.noreply.github.com> Date: Thu, 28 Sep 2023 15:43:51 -0400 Subject: [PATCH 007/117] Rename Retrieval-Augmented Generation.md to Retrieval_Augmented_Generation.md --- ...-Augmented Generation.md => Retrieval_Augmented_Generation.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename paper_list/{Retrieval-Augmented Generation.md => Retrieval_Augmented_Generation.md} (100%) diff --git a/paper_list/Retrieval-Augmented Generation.md b/paper_list/Retrieval_Augmented_Generation.md similarity index 100% rename from paper_list/Retrieval-Augmented Generation.md rename to paper_list/Retrieval_Augmented_Generation.md From 816d4df7f2b9fdcda3074909bef765ad6436cdfb Mon Sep 17 00:00:00 2001 From: Su-Nadendla <110320693+Sumedhn97@users.noreply.github.com> Date: Thu, 28 Sep 2023 15:44:04 -0400 Subject: [PATCH 008/117] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d6798fb..c32d843 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,7 @@ If you're interested in the field of LLM, you may find the above list of milesto > Finetune a language model on a collection of tasks described via instructions -- [Retrieval-Augmented Generation](paper_list/Retrieval-Augmented Generation.md) +- [Retrieval-Augmented Generation](paper_list/Retrieval_Augmented_Generation.md) > Retrieval-Augmented Generation (RAG) combines retrieval from a corpus with generative text models to enhance response accuracy using external knowledge. From 9332678ed1466feed7da8e01410bc206666f3678 Mon Sep 17 00:00:00 2001 From: Su-Nadendla <110320693+Sumedhn97@users.noreply.github.com> Date: Thu, 28 Sep 2023 15:44:33 -0400 Subject: [PATCH 009/117] Update Retrieval_Augmented_Generation.md --- paper_list/Retrieval_Augmented_Generation.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/paper_list/Retrieval_Augmented_Generation.md b/paper_list/Retrieval_Augmented_Generation.md index 9f6b197..cfe0487 100644 --- a/paper_list/Retrieval_Augmented_Generation.md +++ b/paper_list/Retrieval_Augmented_Generation.md @@ -1,4 +1,5 @@ # Retrieval-Augmented Generation -> Large language models (LLMs) demonstrate an in-context learning (ICL) ability, that is, learning from a few examples in the context. +> Retrieval-Augmented Generation (RAG) combines a retriever model to fetch relevant documents from a corpus and a generator model to produce responses based on both the retrieved documents and the original input, enhancing the generation with external knowledge. + ## Useful Resource - [Retrieval-Augmented Generation_Paper](https://arxiv.org/abs/2005.11401v4) - The Original Paper on RAG published by Meta in 2020. From 467f7923f2731f083c6acbbf9387b07b94f0353d Mon Sep 17 00:00:00 2001 From: Hannibal046 <38466901+Hannibal046@users.noreply.github.com> Date: Sat, 30 Sep 2023 14:20:37 +0800 Subject: [PATCH 010/117] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index c32d843..6935435 100644 --- a/README.md +++ b/README.md @@ -424,6 +424,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Awesome-Code-LLM](https://github.com/huybery/Awesome-Code-LLM) - An awesome and curated list of best code-LLM for research. - [Awesome-LLM-Compression](https://github.com/HuangOwen/Awesome-LLM-Compression) - Awesome LLM compression research papers and tools. - [Awesome-LLM-Systems](https://github.com/AmberLJC/LLMSys-PaperList) - Awesome LLM systems research papers. +- [awesome-llm-webapps](https://github.com/snowfort-ai/awesome-llm-webapps) - A collection of open source, actively maintained web apps for LLM applications. ## Other Useful Resources From 68d4b522091072e0109ceedb00228ecaaff41639 Mon Sep 17 00:00:00 2001 From: Chanjun Park Date: Mon, 2 Oct 2023 10:48:59 +0900 Subject: [PATCH 011/117] add add --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index 6935435..6718697 100644 --- a/README.md +++ b/README.md @@ -156,6 +156,9 @@ The following list makes sure that all LLMs are compared **apples to apples**. > - [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) - aims to track, rank and evaluate LLMs and chatbots as they are released. > - [Chatbot Arena Leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard) - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner. > - [AlpacaEval Leaderboard](https://tatsu-lab.github.io/alpaca_eval/) - An Automatic Evaluator for Instruction-following Language Models + > - [Open Ko-LLM Leaderboard](https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard) - The Open Ko-LLM Leaderboard objectively evaluates the performance of Korean Large Language Model (LLM). + + ### Base LLM From 3d006dd204c10a9c647e39d3a9eed83b16d69046 Mon Sep 17 00:00:00 2001 From: Lars Grammel Date: Mon, 2 Oct 2023 13:35:31 +0200 Subject: [PATCH 012/117] Add: ModelFusion library --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6718697..ce63c89 100644 --- a/README.md +++ b/README.md @@ -336,7 +336,7 @@ The above tables coule be better summarized by this wonderful visualization from - [LlamaIndex](https://github.com/jerryjliu/llama_index) — A Python library for augmenting LLM apps with data. - [Arthur Shield](https://www.arthur.ai/get-started) — A paid product for detecting toxicity, hallucination, prompt injection, etc. - [LMQL](https://lmql.ai) — A programming language for LLM interaction with support for typed prompting, control flow, constraints, and tools. - +- [ModelFusion](https://github.com/lgrammel/modelfusion) - A TypeScript library for building apps with LLMs and other ML models (speech-to-text, text-to-speech, image generation). ## Tutorials about LLM From 267c567f2c727a7a6c5f6a184b812784415cf06c Mon Sep 17 00:00:00 2001 From: Aaron Pham <29749331+aarnphm@users.noreply.github.com> Date: Tue, 3 Oct 2023 10:04:41 -0400 Subject: [PATCH 013/117] docs: Add OpenLLM to list of tooling --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index ce63c89..384ae0f 100644 --- a/README.md +++ b/README.md @@ -315,6 +315,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Embedchain](https://github.com/embedchain/embedchain) - Framework to create ChatGPT like bots over your dataset. - [CometLLM](https://github.com/comet-ml/comet-llm) - A 100% opensource LLMOps platform to log, manage, and visualize your LLM prompts and chains. Track prompt templates, prompt variables, prompt duration, token usage, and other metadata. Score prompt outputs and visualize chat history all within a single UI. - [IntelliServer](https://github.com/intelligentnode/IntelliServer) - simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models. +- [OpenLLM](https://github.com/bentoml/OpenLLM) - Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at [BentoML](https://bentoml.com/) for LLMs-based applications. ## Prompting libraries & tools From 74ca66be646d5b7ffe40875af22a9d474f446487 Mon Sep 17 00:00:00 2001 From: Li Ding <3971036+0xDing@users.noreply.github.com> Date: Thu, 12 Oct 2023 04:29:15 -0500 Subject: [PATCH 014/117] docs: Add Flappy to list of tooling --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 384ae0f..9885df0 100644 --- a/README.md +++ b/README.md @@ -338,6 +338,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Arthur Shield](https://www.arthur.ai/get-started) — A paid product for detecting toxicity, hallucination, prompt injection, etc. - [LMQL](https://lmql.ai) — A programming language for LLM interaction with support for typed prompting, control flow, constraints, and tools. - [ModelFusion](https://github.com/lgrammel/modelfusion) - A TypeScript library for building apps with LLMs and other ML models (speech-to-text, text-to-speech, image generation). +- [Flappy](https://github.com/pleisto/flappy) — Production-Ready LLM Agent SDK for Every Developer. ## Tutorials about LLM From fa3981d55627474260bbf6c19cffa53f0ea27b8f Mon Sep 17 00:00:00 2001 From: CHEN Liang Date: Sat, 14 Oct 2023 21:55:40 +0800 Subject: [PATCH 015/117] Update evaluation.md --- paper_list/evaluation.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/paper_list/evaluation.md b/paper_list/evaluation.md index f6b6d09..281b3bd 100644 --- a/paper_list/evaluation.md +++ b/paper_list/evaluation.md @@ -58,5 +58,7 @@ - (2023-04) **Are Emergent Abilities of Large Language Models a Mirage?** [paper](https://arxiv.org/abs/2304.15004) +- (2023-10) **Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators** [paper](https://arxiv.org/abs/2310.07289) | [code](https://github.com/ChanLiang/CONNER) + ## Useful Resources From d734ed445ff69fbeb1a590640b00cf85ae4d1993 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E8=BF=87=E6=8B=9F=E5=90=88?= Date: Mon, 16 Oct 2023 09:37:09 -0500 Subject: [PATCH 016/117] =?UTF-8?q?add=20blog=20"llm=E5=A4=A7=E6=A8=A1?= =?UTF-8?q?=E5=9E=8B=E8=AE=AD=E7=BB=83=E7=9F=A5=E4=B9=8E=E4=B8=93=E6=A0=8F?= =?UTF-8?q?"=20on=20Tutorials?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 9885df0..4b0b78d 100644 --- a/README.md +++ b/README.md @@ -366,6 +366,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Jingfeng Yang] Why did all of the public reproduction of GPT-3 fail? [Link](https://jingfengyang.github.io/gpt) - [Hung-yi Lee] ChatGPT (可能)是怎麼煉成的 - GPT 社會化的過程 [Video](https://www.youtube.com/watch?v=e0aKI2GGZNg) - [Keyvan Kambakhsh] Pure Rust implementation of a minimal Generative Pretrained Transformer [code](https://github.com/keyvank/femtoGPT) +- [过拟合] llm大模型训练知乎专栏 [Link](https://www.zhihu.com/column/c_1252604770952642560) ## Courses about LLM From 958340aba0fee718ac4951ae8667d73631ad1af4 Mon Sep 17 00:00:00 2001 From: Kaito Sugimoto Date: Mon, 23 Oct 2023 19:06:50 +0900 Subject: [PATCH 017/117] add awesome-japanese-llm --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 4b0b78d..e861929 100644 --- a/README.md +++ b/README.md @@ -431,6 +431,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Awesome-LLM-Compression](https://github.com/HuangOwen/Awesome-LLM-Compression) - Awesome LLM compression research papers and tools. - [Awesome-LLM-Systems](https://github.com/AmberLJC/LLMSys-PaperList) - Awesome LLM systems research papers. - [awesome-llm-webapps](https://github.com/snowfort-ai/awesome-llm-webapps) - A collection of open source, actively maintained web apps for LLM applications. +- [awesome-japanese-llm](https://github.com/llm-jp/awesome-japanese-llm) - 日本語LLMまとめ - Overview of Japanese LLMs ## Other Useful Resources From 55144cfe22899c63bbadd7236b904676ba8eda2a Mon Sep 17 00:00:00 2001 From: Sartaj Bhuvaji Date: Tue, 31 Oct 2023 19:09:34 -0700 Subject: [PATCH 018/117] Added StatQuest videos URLs Added 3 videos from Youtube/StatQuest . These videos would help people looking to understand the basics of LLM. --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index e861929..1d95b43 100644 --- a/README.md +++ b/README.md @@ -367,6 +367,9 @@ The above tables coule be better summarized by this wonderful visualization from - [Hung-yi Lee] ChatGPT (可能)是怎麼煉成的 - GPT 社會化的過程 [Video](https://www.youtube.com/watch?v=e0aKI2GGZNg) - [Keyvan Kambakhsh] Pure Rust implementation of a minimal Generative Pretrained Transformer [code](https://github.com/keyvank/femtoGPT) - [过拟合] llm大模型训练知乎专栏 [Link](https://www.zhihu.com/column/c_1252604770952642560) +- [StatQuest] Sequence-to-Sequence (seq2seq) Encoder-Decoder Neural Networks [Link](https://www.youtube.com/watch?v=L8HKweZIOmg) +- [StatQuest] Transformer Neural Networks, ChatGPT's foundation [Link](https://www.youtube.com/watch?v=zxQyTK8quyY) +- [StatQuest] Decoder-Only Transformers, ChatGPTs specific Transformer [Link](https://www.youtube.com/watch?v=bQ5BoolX9Ag) ## Courses about LLM From c961bc0b9ad105f59867835a4e3e943c7006f8b9 Mon Sep 17 00:00:00 2001 From: Jason Cox Date: Thu, 2 Nov 2023 23:52:50 -0400 Subject: [PATCH 019/117] Add Mistral.ai LLM --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 1d95b43..848bf20 100644 --- a/README.md +++ b/README.md @@ -279,6 +279,8 @@ The above tables coule be better summarized by this wonderful visualization from - [XGen](https://github.com/salesforce/xgen) - Salesforce open-source LLMs with 8k sequence length. - [baichuan-7B](https://github.com/baichuan-inc/baichuan-7B) - baichuan-7B 是由百川智能开发的一个开源可商用的大规模预训练语言模型. - [Aquila](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/Aquila) - 悟道·天鹰语言大模型是首个具备中英双语知识、支持商用许可协议、国内数据合规需求的开源语言大模型。 +- [Mistral](https://mistral.ai/) - Mistral-7B-v0.1 is a small, yet powerful model adaptable to many use-cases including code and 8k sequence length. Apache 2.0 licence. + ## LLM Training Frameworks - [DeepSpeed](https://github.com/microsoft/DeepSpeed) - DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. From 1253433215fd3c905b28901cdd47ff7dd7af72a4 Mon Sep 17 00:00:00 2001 From: Hannibal046 <38466901+Hannibal046@users.noreply.github.com> Date: Wed, 8 Nov 2023 13:32:22 +0800 Subject: [PATCH 020/117] Update README.md --- README.md | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 848bf20..103c459 100644 --- a/README.md +++ b/README.md @@ -5,18 +5,10 @@ 🔥 Large Language Models(LLM) have taken the ~~NLP community~~ ~~AI community~~ **the Whole World** by storm. Here is a curated list of papers about large language models, especially relating to ChatGPT. It also contains frameworks for LLM training, tools to deploy LLM, courses and tutorials about LLM and all publicly available LLM checkpoints and APIs. -## Updates - -- [2023-07-01] Add some open-source models: Aquila, Chatglm2, Ultra-LM -- [2023-07-01] Add some deploying tools: vLLM, Text Generation Inference -- [2023-07-01] Add some great post about LLM from Yao Fu, Lilian and Andrej - ### ToDos - Add LLM data (Pretraining data/Instruction Tuning data/Chat data/RLHF data) :sparkles:**Contributions Wanted** -> Also check out the project that I am currently working on: [nanoRWKV](https://github.com/Hannibal046/nanoRWKV) - The nanoGPT-style implementation of RWKV Language Model (an RNN with GPT-level LLM performance.) - ## Table of Content @@ -436,7 +428,8 @@ The above tables coule be better summarized by this wonderful visualization from - [Awesome-LLM-Compression](https://github.com/HuangOwen/Awesome-LLM-Compression) - Awesome LLM compression research papers and tools. - [Awesome-LLM-Systems](https://github.com/AmberLJC/LLMSys-PaperList) - Awesome LLM systems research papers. - [awesome-llm-webapps](https://github.com/snowfort-ai/awesome-llm-webapps) - A collection of open source, actively maintained web apps for LLM applications. -- [awesome-japanese-llm](https://github.com/llm-jp/awesome-japanese-llm) - 日本語LLMまとめ - Overview of Japanese LLMs +- [awesome-japanese-llm](https://github.com/llm-jp/awesome-japanese-llm) - 日本語LLMまとめ - Overview of Japanese LLMs. +- [Awesome-LLM-Healthcare](https://github.com/mingze-yuan/Awesome-LLM-Healthcare) - The paper list of the review on LLMs in medicine. ## Other Useful Resources From bc1e6bc59ac53a7691213a228cc3ddf10650d642 Mon Sep 17 00:00:00 2001 From: Michael Feil <63565275+michaelfeil@users.noreply.github.com> Date: Sun, 12 Nov 2023 23:15:13 +0100 Subject: [PATCH 021/117] Update README.md --- README.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 103c459..a717219 100644 --- a/README.md +++ b/README.md @@ -289,19 +289,15 @@ The above tables coule be better summarized by this wonderful visualization from ## Tools for deploying LLM - [FastChat](https://github.com/lm-sys/FastChat) - A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs. - - [SkyPilot](https://github.com/skypilot-org/skypilot) - Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface. - - [vLLM](https://github.com/vllm-project/vllm) - A high-throughput and memory-efficient inference and serving engine for LLMs - -- [Text Generation Inference](https://github.com/huggingface/text-generation-inference) - A Rust, Python and gRPC server for text generation inference. Used in production at [HuggingFace](https://huggingface.co/) to power LLMs api-inference widgets. - +- [Text Generation Inference](https://github.com/huggingface/text-generation-inference) - A Rust, Python and gRPC server for text generation inference. Used in production at [HuggingFace](https://huggingface.co/) to power LLMs api-inference widgets, HFOIL Licence. - [Haystack](https://haystack.deepset.ai/) - an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data. -- [Sidekick](https://github.com/ai-sidekick/sidekick) - Data integration platform for LLMs. +- [Sidekick](https://github.com/ai-sidekick/sidekick) - Data integration platform for LLMs. - [LangChain](https://github.com/hwchase17/langchain) - Building applications with LLMs through composability - [LiteChain](https://github.com/rogeriochaves/litechain) - Lightweight alternative to LangChain for composing LLMs - [magentic](https://github.com/jackmpcollins/magentic) - Seamlessly integrate LLMs as Python functions -- [wechat-chatgpt](https://github.com/fuergaosi233/wechat-chatgpt) - Use ChatGPT On Wechat via wechaty +- [wechat-chatgpt](https://github.com/fuergaosi233/wechat-chatgpt) - Use ChatGPT On Wechat via wechaty - [promptfoo](https://github.com/typpo/promptfoo) - Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality. - [Agenta](https://github.com/agenta-ai/agenta) - Easily build, version, evaluate and deploy your LLM-powered apps. - [Serge](https://github.com/serge-chat/serge) - a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted! @@ -310,7 +306,9 @@ The above tables coule be better summarized by this wonderful visualization from - [CometLLM](https://github.com/comet-ml/comet-llm) - A 100% opensource LLMOps platform to log, manage, and visualize your LLM prompts and chains. Track prompt templates, prompt variables, prompt duration, token usage, and other metadata. Score prompt outputs and visualize chat history all within a single UI. - [IntelliServer](https://github.com/intelligentnode/IntelliServer) - simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models. - [OpenLLM](https://github.com/bentoml/OpenLLM) - Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at [BentoML](https://bentoml.com/) for LLMs-based applications. - +- [DeepSpeed-Mii] - MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed. +- [Text-Embeddings-Inference](https://github.com/huggingface/text-embeddings-inference) - Inference for text-embeddings in Rust, HFOIL Licence. +- [Infinity](https://github.com/michaelfeil/infinity) - Inference for text-embeddings in Python ## Prompting libraries & tools @@ -436,6 +434,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Arize-Phoenix](https://phoenix.arize.com/) - Open-source tool for ML observability that runs in your notebook environment. Monitor and fine tune LLM, CV and Tabular Models. - [Emergent Mind](https://www.emergentmind.com) - The latest AI news, curated & explained by GPT-4. - [ShareGPT](https://sharegpt.com) - Share your wildest ChatGPT conversations with one click. +- [Gradient](https://gradient.ai) - Option to fast fine-tune a QLora Adapter for Llama - [Major LLMs + Data Availability](https://docs.google.com/spreadsheets/d/1bmpDdLZxvTCleLGVPgzoMTQ0iDP2-7v7QziPrzPdHyM/edit#gid=0) - [500+ Best AI Tools](https://vaulted-polonium-23c.notion.site/500-Best-AI-Tools-e954b36bf688404ababf74a13f98d126) - [Cohere Summarize Beta](https://txt.cohere.ai/summarize-beta/) - Introducing Cohere Summarize Beta: A New Endpoint for Text Summarization From 3f108a75267f80229f8191ae7815d54c49557a77 Mon Sep 17 00:00:00 2001 From: Michael Feil <63565275+michaelfeil@users.noreply.github.com> Date: Sun, 12 Nov 2023 23:17:12 +0100 Subject: [PATCH 022/117] Update README.md --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index a717219..0c122e7 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ | 2022-01 | COT | Google | [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/pdf/2201.11903.pdf) | NeurIPS
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F1b6e810ce0afd0dd093f789d2b2742d047e316d5%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2022-01 | LaMDA | Google | [LaMDA: Language Models for Dialog Applications](https://arxiv.org/pdf/2201.08239.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fb3848d32f7294ec708627897833c4097eb4d8778%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2022-01 | Minerva | Google | [Solving Quantitative Reasoning Problems with Language Models](https://arxiv.org/abs/2206.14858) | NeurIPS
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fab0e3d3e4d42369de5933a3b4c237780b41c0d77%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | -| 2022-01 | Megatron-Turing NLG | Microsoft&NVIDIA | [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://arxiv.org/pdf/2201.11990.pdf) | ![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F7cbc2a7843411a1768ab762930707af0a3c33a19%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| +| 2022-01 | Megatron-Turing NLG | Microsoft&NVIDIA | [Using Deep and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://arxiv.org/pdf/2201.11990.pdf) | ![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F7cbc2a7843411a1768ab762930707af0a3c33a19%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2022-03 | InstructGPT | OpenAI | [Training language models to follow instructions with human feedback](https://arxiv.org/pdf/2203.02155.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fd766bffc357127e0dc86dd69561d5aeb520d6f4c%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2022-04 | PaLM | Google | [PaLM: Scaling Language Modeling with Pathways](https://arxiv.org/pdf/2204.02311.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F094ff971d6a8b8ff870946c9b3ce5aa173617bfb%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2022-04 | Chinchilla | DeepMind | [An empirical analysis of compute-optimal large language model training](https://www.deepmind.com/publications/an-empirical-analysis-of-compute-optimal-large-language-model-training) | NeurIPS
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fbb0656031cb17adf6bac5fd0fe8d53dd9c291508%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | @@ -306,10 +306,10 @@ The above tables coule be better summarized by this wonderful visualization from - [CometLLM](https://github.com/comet-ml/comet-llm) - A 100% opensource LLMOps platform to log, manage, and visualize your LLM prompts and chains. Track prompt templates, prompt variables, prompt duration, token usage, and other metadata. Score prompt outputs and visualize chat history all within a single UI. - [IntelliServer](https://github.com/intelligentnode/IntelliServer) - simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models. - [OpenLLM](https://github.com/bentoml/OpenLLM) - Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at [BentoML](https://bentoml.com/) for LLMs-based applications. -- [DeepSpeed-Mii] - MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed. +- [DeepSpeed-Mii](https://github.com/microsoft/DeepSpeed-MII) - MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed. - [Text-Embeddings-Inference](https://github.com/huggingface/text-embeddings-inference) - Inference for text-embeddings in Rust, HFOIL Licence. - [Infinity](https://github.com/michaelfeil/infinity) - Inference for text-embeddings in Python - +- [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) - Nvidia Framework for LLM Inference ## Prompting libraries & tools - [YiVal](https://github.com/YiVal/YiVal) — Evaluate and Evolve: YiVal is an open-source GenAI-Ops tool for tuning and evaluating prompts, configurations, and model parameters using customizable datasets, evaluation methods, and improvement strategies. From c5154b29df1b96f890216d22564cda8ac3da8d25 Mon Sep 17 00:00:00 2001 From: Jeff Emanuel <35050222+Dicklesworthstone@users.noreply.github.com> Date: Sun, 12 Nov 2023 17:37:02 -0500 Subject: [PATCH 023/117] Update README.md What is this Python project? https://github.com/Dicklesworthstone/swiss_army_llama The Swiss Army Llama is a comprehensive toolkit for working with local Large Language Models (LLMs). It uses FastAPI to provide REST endpoints for a variety of tasks, including text embeddings, completions, semantic similarity measurements, and more. It supports a wide range of document types, including those requiring OCR, and audio files for transcription. Embeddings are cached in SQLite for efficiency, and RAM Disks are used for quick loading of multiple LLMs. The toolkit is accessible via a Swagger UI and integrates with various technologies like FAISS for semantic search and the Fast Vector Similarity library for advanced similarity measures. What's the difference between this Python project and similar ones? 1. FastAPI Integration: Swiss Army Llama is fully integrated with FastAPI, providing a Swagger page for easy access and interaction with its REST API, a feature not commonly found in similar projects. 2. Comprehensive Caching: The project implements automatic caching for all processes, significantly enhancing efficiency and reducing redundant computations. This level of automatic caching is not standard in similar tools. 3. RAM Disk Utilization: It uniquely uses RAM Disk to accelerate the loading of models, providing a substantial speed advantage in accessing and using various LLMs. 4. Broad File Format Support with Textract: The integration with Textract allows the Swiss Army Llama to handle a wide array of file formats, far exceeding the capabilities of similar projects which often have limited file format support. 5. Integration with Whisper: The project is integrated with OpenAI's Whisper model for advanced audio transcription, a feature that is not typically included in similar LLM toolkits. 6. BNF Grammar Tools: It includes specialized tools for working with Backus-Naur Form (BNF) grammars, offering unique capabilities in generating structured LLM outputs based on specific grammar rules. 7. Support for Token-Level Embeddings: In addition to standard embeddings, Swiss Army Llama supports token-level embeddings, providing more nuanced data representation and analysis. This level of detail in embeddings is not commonly available in similar projects. Features: Versatile File Processing: Supports an extensive array of file types including PDFs, Word documents, images, and audio files, with advanced text preprocessing and OCR capabilities. Comprehensive Embedding Features: Offers both fixed-sized and token-level embeddings, with the unique introduction of combined feature vectors for comparing strings of unequal length. Advanced Semantic Search: Combines FAISS vector searching with sophisticated similarity measures like spearman_rho, kendall_tau, and jensen_shannon_similarity, enabling nuanced text comparisons. Efficient Caching and RAM Disk Usage: Implements efficient caching in SQLite and optional RAM Disk usage for faster model loading and execution. Comprehensive Logging and Real-Time Monitoring: Features a real-time log file viewer in the browser, and uses Redis for efficient request handling and logging. Interactive and User-Friendly Interface: Integrates with Swagger UI for easy access and management of large result sets, making it user-friendly and easily integrable into applications. Customizable and Scalable: Built on FastAPI, it is highly scalable and customizable with configurable settings for response formats and parallel inference. Multiple Model and Measure Support: Accommodates a variety of models and measures, providing flexibility and customization according to user needs. Specialized Grammar-Enforced Text Completions: Offers the ability to generate multiple text completions with specified grammar, enhancing structured LLM output. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 103c459..64e389d 100644 --- a/README.md +++ b/README.md @@ -299,6 +299,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Haystack](https://haystack.deepset.ai/) - an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data. - [Sidekick](https://github.com/ai-sidekick/sidekick) - Data integration platform for LLMs. - [LangChain](https://github.com/hwchase17/langchain) - Building applications with LLMs through composability +- [Swiss Army Llama](https://github.com/Dicklesworthstone/swiss_army_llama) - Comprehensive set of tools for working with local LLMs for various tasks. - [LiteChain](https://github.com/rogeriochaves/litechain) - Lightweight alternative to LangChain for composing LLMs - [magentic](https://github.com/jackmpcollins/magentic) - Seamlessly integrate LLMs as Python functions - [wechat-chatgpt](https://github.com/fuergaosi233/wechat-chatgpt) - Use ChatGPT On Wechat via wechaty From a0bb3e17f5f107bcaa4b0009b73b92e850715131 Mon Sep 17 00:00:00 2001 From: Michael Feil <63565275+michaelfeil@users.noreply.github.com> Date: Mon, 13 Nov 2023 09:49:09 +0100 Subject: [PATCH 024/117] Update README.md --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index 0c122e7..1fa3dbb 100644 --- a/README.md +++ b/README.md @@ -434,7 +434,6 @@ The above tables coule be better summarized by this wonderful visualization from - [Arize-Phoenix](https://phoenix.arize.com/) - Open-source tool for ML observability that runs in your notebook environment. Monitor and fine tune LLM, CV and Tabular Models. - [Emergent Mind](https://www.emergentmind.com) - The latest AI news, curated & explained by GPT-4. - [ShareGPT](https://sharegpt.com) - Share your wildest ChatGPT conversations with one click. -- [Gradient](https://gradient.ai) - Option to fast fine-tune a QLora Adapter for Llama - [Major LLMs + Data Availability](https://docs.google.com/spreadsheets/d/1bmpDdLZxvTCleLGVPgzoMTQ0iDP2-7v7QziPrzPdHyM/edit#gid=0) - [500+ Best AI Tools](https://vaulted-polonium-23c.notion.site/500-Best-AI-Tools-e954b36bf688404ababf74a13f98d126) - [Cohere Summarize Beta](https://txt.cohere.ai/summarize-beta/) - Introducing Cohere Summarize Beta: A New Endpoint for Text Summarization From 702dad8c6d033f6421c8db3ff91a04858dc67a6c Mon Sep 17 00:00:00 2001 From: Yuanhe Tian Date: Sun, 19 Nov 2023 20:04:04 -0800 Subject: [PATCH 025/117] Add ChiMed-GPT, a Chinese medical large language model --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 787fbf3..d3ad4cc 100644 --- a/README.md +++ b/README.md @@ -235,6 +235,7 @@ The above tables coule be better summarized by this wonderful visualization from - [BayLing](https://github.com/ictnlp/BayLing) - an English/Chinese LLM equipped with advanced language alignment, showing superior capability in English/Chinese generation, instruction following and multi-turn interaction. - [UltraLM](https://github.com/thunlp/UltraChat) - Large-scale, Informative, and Diverse Multi-round Chat Models. - [Guanaco](https://github.com/artidoro/qlora) - QLoRA tuned LLaMA + - [ChiMed-GPT](https://github.com/synlp/ChiMed-GPT) - A Chinese medical large language model. - [BLOOM](https://huggingface.co/bigscience/bloom) - BigScience Large Open-science Open-access Multilingual Language Model [BLOOM-LoRA](https://github.com/linhduongtuan/BLOOM-LORA) - [BLOOMZ&mT0](https://huggingface.co/bigscience/bloomz) - a family of models capable of following human instructions in dozens of languages zero-shot. - [Phoenix](https://github.com/FreedomIntelligence/LLMZoo) From aac285b431f5da8cfa3fdd98e2d154b2c28311d5 Mon Sep 17 00:00:00 2001 From: JaejinCho Date: Tue, 5 Dec 2023 08:34:27 -0500 Subject: [PATCH 026/117] Add a repo link for LLM dataset list --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index d3ad4cc..59c1379 100644 --- a/README.md +++ b/README.md @@ -430,6 +430,7 @@ The above tables coule be better summarized by this wonderful visualization from - [awesome-llm-webapps](https://github.com/snowfort-ai/awesome-llm-webapps) - A collection of open source, actively maintained web apps for LLM applications. - [awesome-japanese-llm](https://github.com/llm-jp/awesome-japanese-llm) - 日本語LLMまとめ - Overview of Japanese LLMs. - [Awesome-LLM-Healthcare](https://github.com/mingze-yuan/Awesome-LLM-Healthcare) - The paper list of the review on LLMs in medicine. +- [LLMDatahub](https://github.com/Zjh-819/LLMDataHub) - a curated collection of datasets specifically designed for chatbot training, including links, size, language, usage, and a brief description of each dataset ## Other Useful Resources From 443a289f81bda7e9c398464322f17f0d0ced9037 Mon Sep 17 00:00:00 2001 From: DefTruth <31974251+DefTruth@users.noreply.github.com> Date: Mon, 11 Dec 2023 15:25:16 +0800 Subject: [PATCH 027/117] Add Awesome-LLM-Inference --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index d3ad4cc..7f6f095 100644 --- a/README.md +++ b/README.md @@ -430,6 +430,7 @@ The above tables coule be better summarized by this wonderful visualization from - [awesome-llm-webapps](https://github.com/snowfort-ai/awesome-llm-webapps) - A collection of open source, actively maintained web apps for LLM applications. - [awesome-japanese-llm](https://github.com/llm-jp/awesome-japanese-llm) - 日本語LLMまとめ - Overview of Japanese LLMs. - [Awesome-LLM-Healthcare](https://github.com/mingze-yuan/Awesome-LLM-Healthcare) - The paper list of the review on LLMs in medicine. +- [Awesome-LLM-Inference](https://github.com/DefTruth/Awesome-LLM-Inference) - A curated list of Awesome LLM Inference Paper with codes. ## Other Useful Resources From 37253df880010c0f18b85c091367a744812a0ee4 Mon Sep 17 00:00:00 2001 From: Himanshu Date: Fri, 15 Dec 2023 14:34:35 +0530 Subject: [PATCH 028/117] added GPTRouter --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 4e2ac26..2049e47 100644 --- a/README.md +++ b/README.md @@ -432,6 +432,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Awesome-LLM-Healthcare](https://github.com/mingze-yuan/Awesome-LLM-Healthcare) - The paper list of the review on LLMs in medicine. - [Awesome-LLM-Inference](https://github.com/DefTruth/Awesome-LLM-Inference) - A curated list of Awesome LLM Inference Paper with codes. - [LLMDatahub](https://github.com/Zjh-819/LLMDataHub) - a curated collection of datasets specifically designed for chatbot training, including links, size, language, usage, and a brief description of each dataset +- [GPTRouter](https://gpt-router.writesonic.com/) - GPTRouter is an open source LLM API Gateway that offers a universal API for 30+ LLMs, vision, and image models, with smart fallbacks based on uptime and latency, automatic retries, and streaming. Stay operational even when OpenAI is down ## Other Useful Resources From 9e53089d2f4adc5359a4442895f6d980144b5dd9 Mon Sep 17 00:00:00 2001 From: wsl Date: Sat, 16 Dec 2023 18:12:53 +0800 Subject: [PATCH 029/117] reorg --- README.md | 120 ++++++++++++++++++++++++++---------------------- contributing.md | 6 +++ 2 files changed, 70 insertions(+), 56 deletions(-) diff --git a/README.md b/README.md index 4e2ac26..6193019 100644 --- a/README.md +++ b/README.md @@ -5,26 +5,28 @@ 🔥 Large Language Models(LLM) have taken the ~~NLP community~~ ~~AI community~~ **the Whole World** by storm. Here is a curated list of papers about large language models, especially relating to ChatGPT. It also contains frameworks for LLM training, tools to deploy LLM, courses and tutorials about LLM and all publicly available LLM checkpoints and APIs. -### ToDos + -## Table of Content +## Trending LLM Projects +- [Mixtral 8x7B](https://mistral.ai/news/mixtral-of-experts/) - a high-quality sparse mixture of experts model (SMoE) with open weights. +- [promptbase](https://github.com/microsoft/promptbase) - All things prompt engineering. +- [ollama](https://github.com/jmorganca/ollama) - Get up and running with Llama 2 and other large language models locally. +- [anything-llm](https://github.com/Mintplex-Labs/anything-llm) - A private ChatGPT to chat with anything! +- [phi-2](https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/) - a 2.7 billion-parameter language model that demonstrates outstanding reasoning and language understanding capabilities, showcasing state-of-the-art performance among base language models with less than 13 billion parameters. +## Table of Content - [Awesome-LLM ](#awesome-llm-) - - [Updates](#updates) - - [Table of Content](#table-of-content) - [Milestone Papers](#milestone-papers) - [Other Papers](#other-papers) - - [LLM Leaderboard](#llm-leaderboard) - [Open LLM](#open-llm) - [LLM Training Frameworks](#llm-training-frameworks) - - [Tools for deploying LLM](#tools-for-deploying-llm) - - [Tutorials about LLM](#tutorials-about-llm) - - [Courses about LLM](#courses-about-llm) - - [Opinions about LLM](#opinions-about-llm) - - [Other Awesome Lists](#other-awesome-lists) + - [Tools for deploying LLM](#deploying-tools) + - [Tutorials about LLM](#tutorials) + - [Courses about LLM](#courses) + - [Opinions about LLM](#opinions) - [Other Useful Resources](#other-useful-resources) - [Contributing](#contributing) @@ -77,17 +79,45 @@ | 2023-04 | Pythia | EleutherAI et al. | [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://arxiv.org/abs/2304.01373)|ICML
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fbe55e8ec4213868db08f2c3168ae666001bea4b8%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-05 | Dromedary | CMU et al. | [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://arxiv.org/abs/2305.03047)|![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fe01515c6138bc525f7aec30fc85f2adf028d4156%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-05 | PaLM 2 | Google | [PaLM 2 Technical Report](https://ai.google/static/documents/palm2techreport.pdf)|![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Feccee350691708972370b7a12c2a78ad3bddd159%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-05 | RWKV | Bo Peng | [RWKV: Reinventing RNNs for the Transformer Era](https://arxiv.org/abs/2305.13048) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F026b3396a63ed5772329708b7580d633bb86bec9%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-05 | DPO | Stanford | [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/pdf/2305.18290.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F0d1c76d45afa012ded7ab741194baf142117c495%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| +| 2023-05 | RWKV | Bo Peng | [RWKV: Reinventing RNNs for the Transformer Era](https://arxiv.org/abs/2305.13048) |EMNLP
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F026b3396a63ed5772329708b7580d633bb86bec9%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| +| 2023-05 | DPO | Stanford | [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/pdf/2305.18290.pdf) |Neurips
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F0d1c76d45afa012ded7ab741194baf142117c495%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-07 | LLaMA 2 | Meta | [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/pdf/2307.09288.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F104b0bb1da562d53cbda87aec79ef6a2827d191a%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| +| 2023-12 | Mamba | CMU&Princeton | [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/ftp/arxiv/papers/2312/2312.00752.pdf) |ICLR
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F432bef8e34014d726c674bc458008ac895297b51%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| ## Other Papers -If you're interested in the field of LLM, you may find the above list of milestone papers helpful to explore its history and state-of-the-art. However, each direction of LLM offers a unique set of insights and contributions, which are essential to understanding the field as a whole. For a detailed list of papers in various subfields, please refer to the following link (it is possible that there are overlaps between different subfields): +If you're interested in the field of LLM, you may find the above list of milestone papers helpful to explore its history and state-of-the-art. However, each direction of LLM offers a unique set of insights and contributions, which are essential to understanding the field as a whole. For a detailed list of papers in various subfields, please refer to the following link: -(:exclamation: **We would greatly appreciate and welcome your contribution to the following list. :exclamation:**) +- [LLMsPracticalGuide](https://github.com/Mooler0410/LLMsPracticalGuide) - A curated (still actively updated) list of practical guide resources of LLMs +- [Awesome ChatGPT Prompts](https://github.com/f/awesome-chatgpt-prompts) - A collection of prompt examples to be used with the ChatGPT model. +- [awesome-chatgpt-prompts-zh](https://github.com/PlexPt/awesome-chatgpt-prompts-zh) - A Chinese collection of prompt examples to be used with the ChatGPT model. +- [Awesome ChatGPT](https://github.com/humanloop/awesome-chatgpt) - Curated list of resources for ChatGPT and GPT-3 from OpenAI. +- [Chain-of-Thoughts Papers](https://github.com/Timothyxxx/Chain-of-ThoughtsPapers) - A trend starts from "Chain of Thought Prompting Elicits Reasoning in Large Language Models. +- [Instruction-Tuning-Papers](https://github.com/SinclairCoder/Instruction-Tuning-Papers) - A trend starts from `Natrural-Instruction` (ACL 2022), `FLAN` (ICLR 2022) and `T0` (ICLR 2022). +- [LLM Reading List](https://github.com/crazyofapple/Reading_groups/) - A paper & resource list of large language models. +- [Reasoning using Language Models](https://github.com/atfortes/LM-Reasoning-Papers) - Collection of papers and resources on Reasoning using Language Models. +- [Chain-of-Thought Hub](https://github.com/FranxYao/chain-of-thought-hub) - Measuring LLMs' Reasoning Performance +- [Awesome GPT](https://github.com/formulahendry/awesome-gpt) - A curated list of awesome projects and resources related to GPT, ChatGPT, OpenAI, LLM, and more. +- [Awesome GPT-3](https://github.com/elyase/awesome-gpt3) - a collection of demos and articles about the [OpenAI GPT-3 API](https://openai.com/blog/openai-api/). +- [Awesome LLM Human Preference Datasets](https://github.com/PolisAI/awesome-llm-human-preference-datasets) - a collection of human preference datasets for LLM instruction tuning, RLHF and evaluation. +- [RWKV-howto](https://github.com/Hannibal046/RWKV-howto) - possibly useful materials and tutorial for learning RWKV. +- [ModelEditingPapers](https://github.com/zjunlp/ModelEditingPapers) - A paper & resource list on model editing for large language models. +- [Awesome LLM Security](https://github.com/corca-ai/awesome-llm-security) - A curation of awesome tools, documents and projects about LLM Security. +- [Awesome-Align-LLM-Human](https://github.com/GaryYufei/AlignLLMHumanSurvey) - A collection of papers and resources about aligning large language models (LLMs) with human. +- [Awesome-Code-LLM](https://github.com/huybery/Awesome-Code-LLM) - An awesome and curated list of best code-LLM for research. +- [Awesome-LLM-Compression](https://github.com/HuangOwen/Awesome-LLM-Compression) - Awesome LLM compression research papers and tools. +- [Awesome-LLM-Systems](https://github.com/AmberLJC/LLMSys-PaperList) - Awesome LLM systems research papers. +- [awesome-llm-webapps](https://github.com/snowfort-ai/awesome-llm-webapps) - A collection of open source, actively maintained web apps for LLM applications. +- [awesome-japanese-llm](https://github.com/llm-jp/awesome-japanese-llm) - 日本語LLMまとめ - Overview of Japanese LLMs. +- [Awesome-LLM-Healthcare](https://github.com/mingze-yuan/Awesome-LLM-Healthcare) - The paper list of the review on LLMs in medicine. +- [Awesome-LLM-Inference](https://github.com/DefTruth/Awesome-LLM-Inference) - A curated list of Awesome LLM Inference Paper with codes. +- [LLMDatahub](https://github.com/Zjh-819/LLMDataHub) - a curated collection of datasets specifically designed for chatbot training, including links, size, language, usage, and a brief description of each dataset +- [Awesome-Chinese-LLM](https://github.com/HqWu-HITCS/Awesome-Chinese-LLM) - 整理开源的中文大语言模型，以规模较小、可私有化部署、训练成本较低的模型为主，包括底座模型，垂直领域微调及应用，数据集与教程等。 +- [llm-course](https://github.com/mlabonne/llm-course) - Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks. + + -## LLM Leaderboard +## Open LLM
There are three important steps for a ChatGPT-like LLM: -1. **Pre-training** -2. **Instruction Tuning** -3. **Alignment** +- **Pre-training** +- **Instruction Tuning** +- **Alignment** -The following list makes sure that all LLMs are compared **apples to apples**. + > You may also find these leaderboards helpful: > - [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) - aims to track, rank and evaluate LLMs and chatbots as they are released. > - [Chatbot Arena Leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard) - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner. @@ -152,7 +182,7 @@ The following list makes sure that all LLMs are compared **apples to apples**. -### Base LLM + -- [LLaMA](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) - A foundational, 65-billion-parameter large language model. [LLaMA.cpp](https://github.com/ggerganov/llama.cpp) [Lit-LLaMA](https://github.com/Lightning-AI/lit-llama) +- [Mistral](https://mistral.ai/) - Mistral-7B-v0.1 is a small, yet powerful model adaptable to many use-cases including code and 8k sequence length. Apache 2.0 licence. +- [Mixtral 8x7B](https://mistral.ai/news/mixtral-of-experts/) - a high-quality sparse mixture of experts model (SMoE) with open weights. +- [LLaMA](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) & [LLaMA-2](https://ai.meta.com/llama/) - A foundational large language model. [LLaMA.cpp](https://github.com/ggerganov/llama.cpp) [Lit-LLaMA](https://github.com/Lightning-AI/lit-llama) - [Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html) - A model fine-tuned from the LLaMA 7B model on 52K instruction-following demonstrations. [Alpaca.cpp](https://github.com/antimatter15/alpaca.cpp) [Alpaca-LoRA](https://github.com/tloen/alpaca-lora) - [Flan-Alpaca](https://github.com/declare-lab/flan-alpaca) - Instruction Tuning from Humans and Machines. - [Baize](https://github.com/project-baize/baize-chatbot) - Baize is an open-source chat model trained with [LoRA](https://github.com/microsoft/LoRA). It uses 100k dialogs generated by letting ChatGPT chat with itself. @@ -272,7 +304,9 @@ The above tables coule be better summarized by this wonderful visualization from - [XGen](https://github.com/salesforce/xgen) - Salesforce open-source LLMs with 8k sequence length. - [baichuan-7B](https://github.com/baichuan-inc/baichuan-7B) - baichuan-7B 是由百川智能开发的一个开源可商用的大规模预训练语言模型. - [Aquila](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/Aquila) - 悟道·天鹰语言大模型是首个具备中英双语知识、支持商用许可协议、国内数据合规需求的开源语言大模型。 -- [Mistral](https://mistral.ai/) - Mistral-7B-v0.1 is a small, yet powerful model adaptable to many use-cases including code and 8k sequence length. Apache 2.0 licence. +- [phi-1](https://arxiv.org/abs/2306.11644) - a new large language model for code, with significantly smaller size than competing models. +- [phi-1.5](https://arxiv.org/abs/2309.05463) - a 1.3 billion parameter model trained on a dataset of 30 billion tokens, which achieves common sense reasoning benchmark results comparable to models ten times its size that were trained on datasets more than ten times larger. +- [phi-2](https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/) - a 2.7 billion-parameter language model that demonstrates outstanding reasoning and language understanding capabilities, showcasing state-of-the-art performance among base language models with less than 13 billion parameters. ## LLM Training Frameworks @@ -287,7 +321,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Alpa](https://alpa.ai/index.html) - Alpa is a system for training and serving large-scale neural networks. - [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) - An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library. -## Tools for deploying LLM +## Deploying Tools - [FastChat](https://github.com/lm-sys/FastChat) - A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs. - [SkyPilot](https://github.com/skypilot-org/skypilot) - Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface. @@ -312,6 +346,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Text-Embeddings-Inference](https://github.com/huggingface/text-embeddings-inference) - Inference for text-embeddings in Rust, HFOIL Licence. - [Infinity](https://github.com/michaelfeil/infinity) - Inference for text-embeddings in Python - [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) - Nvidia Framework for LLM Inference + ## Prompting libraries & tools - [YiVal](https://github.com/YiVal/YiVal) — Evaluate and Evolve: YiVal is an open-source GenAI-Ops tool for tuning and evaluating prompts, configurations, and model parameters using customizable datasets, evaluation methods, and improvement strategies. @@ -335,7 +370,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Flappy](https://github.com/pleisto/flappy) — Production-Ready LLM Agent SDK for Every Developer. -## Tutorials about LLM +## Tutorials - [Andrej Karpathy] State of GPT [video](https://build.microsoft.com/en-US/sessions/db3f4859-cd30-4445-a0cd-553c3304f8e2) - [Hyung Won Chung] Instruction finetuning and RLHF lecture [Youtube](https://www.youtube.com/watch?v=zjrM-MW-0y0) - [Jason Wei] Scaling, emergence, and reasoning in large language models [Slides](https://docs.google.com/presentation/d/1EUV7W7X_w0BDrscDhPg7lMGzJCkeaPkGCJ3bN8dluXc/edit?pli=1&resourcekey=0-7Nz5A7y8JozyVrnDtcEKJA#slide=id.g16197112905_0_0) @@ -365,7 +400,7 @@ The above tables coule be better summarized by this wonderful visualization from - [StatQuest] Transformer Neural Networks, ChatGPT's foundation [Link](https://www.youtube.com/watch?v=zxQyTK8quyY) - [StatQuest] Decoder-Only Transformers, ChatGPTs specific Transformer [Link](https://www.youtube.com/watch?v=bQ5BoolX9Ag) -## Courses about LLM +## Courses - [DeepLearning.AI] ChatGPT Prompt Engineering for Developers [Homepage](https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/) - [Princeton] Understanding Large Language Models [Homepage](https://www.cs.princeton.edu/courses/archive/fall22/cos597G/) @@ -381,13 +416,11 @@ The above tables coule be better summarized by this wonderful visualization from - [Aston Zhang] Chain of Thought论文 [Bilibili](https://www.bilibili.com/video/BV1t8411e7Ug/?spm_id_from=333.788&vd_source=1e55c5426b48b37e901ff0f78992e33f) [Youtube](https://www.youtube.com/watch?v=H4J59iG3t5o&list=PLFXJ6jwg0qW-7UM8iUTj3qKqdhbQULP5I&index=29) - [MIT] Introduction to Data-Centric AI [Homepage](https://dcai.csail.mit.edu) -## Opinions about LLM +## Opinions - [A Stage Review of Instruction Tuning](https://yaofu.notion.site/June-2023-A-Stage-Review-of-Instruction-Tuning-f59dbfc36e2d4e12a33443bd6b2012c2) [2023-06-29] [Yao Fu] - - [LLM Powered Autonomous Agents](https://lilianweng.github.io/posts/2023-06-23-agent/) [2023-06-23] [Lilian] - [Why you should work on AI AGENTS!](https://www.youtube.com/watch?v=fqVLjtvWgq8) [2023-06-22] [Andrej Karpathy] - - [Google "We Have No Moat, And Neither Does OpenAI"](https://www.semianalysis.com/p/google-we-have-no-moat-and-neither) [2023-05-05] - [AI competition statement](https://petergabriel.com/news/ai-competition-statement/) [2023-04-20] [petergabriel] - [我的大模型世界观](https://mp.weixin.qq.com/s/_ZvyxRpgIA4L4pqfcQtPTQ) [2023-04-23] [陆奇] @@ -406,32 +439,7 @@ The above tables coule be better summarized by this wonderful visualization from - [What Are Large Language Models Used For? ](https://www.notion.so/Awesome-LLM-40c8aa3f2b444ecc82b79ae8bbd2696b) \[2023-01-26][NVIDIA] - [Large Language Models: A New Moore's Law ](https://huggingface.co/blog/large-language-models) \[2021-10-26\]\[Huggingface\] -## Other Awesome Lists -- [LLMsPracticalGuide](https://github.com/Mooler0410/LLMsPracticalGuide) - A curated (still actively updated) list of practical guide resources of LLMs -- [Awesome ChatGPT Prompts](https://github.com/f/awesome-chatgpt-prompts) - A collection of prompt examples to be used with the ChatGPT model. -- [awesome-chatgpt-prompts-zh](https://github.com/PlexPt/awesome-chatgpt-prompts-zh) - A Chinese collection of prompt examples to be used with the ChatGPT model. -- [Awesome ChatGPT](https://github.com/humanloop/awesome-chatgpt) - Curated list of resources for ChatGPT and GPT-3 from OpenAI. -- [Chain-of-Thoughts Papers](https://github.com/Timothyxxx/Chain-of-ThoughtsPapers) - A trend starts from "Chain of Thought Prompting Elicits Reasoning in Large Language Models. -- [Instruction-Tuning-Papers](https://github.com/SinclairCoder/Instruction-Tuning-Papers) - A trend starts from `Natrural-Instruction` (ACL 2022), `FLAN` (ICLR 2022) and `T0` (ICLR 2022). -- [LLM Reading List](https://github.com/crazyofapple/Reading_groups/) - A paper & resource list of large language models. -- [Reasoning using Language Models](https://github.com/atfortes/LM-Reasoning-Papers) - Collection of papers and resources on Reasoning using Language Models. -- [Chain-of-Thought Hub](https://github.com/FranxYao/chain-of-thought-hub) - Measuring LLMs' Reasoning Performance -- [Awesome GPT](https://github.com/formulahendry/awesome-gpt) - A curated list of awesome projects and resources related to GPT, ChatGPT, OpenAI, LLM, and more. -- [Awesome GPT-3](https://github.com/elyase/awesome-gpt3) - a collection of demos and articles about the [OpenAI GPT-3 API](https://openai.com/blog/openai-api/). -- [Awesome LLM Human Preference Datasets](https://github.com/PolisAI/awesome-llm-human-preference-datasets) - a collection of human preference datasets for LLM instruction tuning, RLHF and evaluation. -- [RWKV-howto](https://github.com/Hannibal046/RWKV-howto) - possibly useful materials and tutorial for learning RWKV. -- [ModelEditingPapers](https://github.com/zjunlp/ModelEditingPapers) - A paper & resource list on model editing for large language models. -- [Awesome LLM Security](https://github.com/corca-ai/awesome-llm-security) - A curation of awesome tools, documents and projects about LLM Security. -- [Awesome-Align-LLM-Human](https://github.com/GaryYufei/AlignLLMHumanSurvey) - A collection of papers and resources about aligning large language models (LLMs) with human. -- [Awesome-Code-LLM](https://github.com/huybery/Awesome-Code-LLM) - An awesome and curated list of best code-LLM for research. -- [Awesome-LLM-Compression](https://github.com/HuangOwen/Awesome-LLM-Compression) - Awesome LLM compression research papers and tools. -- [Awesome-LLM-Systems](https://github.com/AmberLJC/LLMSys-PaperList) - Awesome LLM systems research papers. -- [awesome-llm-webapps](https://github.com/snowfort-ai/awesome-llm-webapps) - A collection of open source, actively maintained web apps for LLM applications. -- [awesome-japanese-llm](https://github.com/llm-jp/awesome-japanese-llm) - 日本語LLMまとめ - Overview of Japanese LLMs. -- [Awesome-LLM-Healthcare](https://github.com/mingze-yuan/Awesome-LLM-Healthcare) - The paper list of the review on LLMs in medicine. -- [Awesome-LLM-Inference](https://github.com/DefTruth/Awesome-LLM-Inference) - A curated list of Awesome LLM Inference Paper with codes. -- [LLMDatahub](https://github.com/Zjh-819/LLMDataHub) - a curated collection of datasets specifically designed for chatbot training, including links, size, language, usage, and a brief description of each dataset ## Other Useful Resources diff --git a/contributing.md b/contributing.md index 68e995f..3376c24 100644 --- a/contributing.md +++ b/contributing.md @@ -16,3 +16,9 @@ edit your PR before we merge it. There's no need to open a new PR, just edit the existing one. If you're not sure how to do that, [here is a guide](https://github.com/RichardLitt/knowledge/blob/master/github/amending-a-commit-guide.md) on the different ways you can update your PR so that we can merge it. + +## how to add dynamic citation badge + +1. get paper id from semantic scholar paper page +2. create dynamic badge at [this site](https://shields.io/badges/dynamic-json-badge) with this link: https://api.semanticscholar.org/graph/v1/paper/{paper_id}?fields=citationCount +3. ![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F432bef8e34014d726c674bc458008ac895297b51%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) \ No newline at end of file From a2f59c06ec66a1ab1811c90c781b903112448131 Mon Sep 17 00:00:00 2001 From: Xianzheng Ma Date: Sat, 16 Dec 2023 22:35:37 +0000 Subject: [PATCH 030/117] add Awesome-LLM-3D repo A curated list of Multi-modal Large Language Model in 3D world, including 3D understanding, reasoning, generation, and embodied agents. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 6193019..25c52b9 100644 --- a/README.md +++ b/README.md @@ -113,6 +113,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [awesome-japanese-llm](https://github.com/llm-jp/awesome-japanese-llm) - 日本語LLMまとめ - Overview of Japanese LLMs. - [Awesome-LLM-Healthcare](https://github.com/mingze-yuan/Awesome-LLM-Healthcare) - The paper list of the review on LLMs in medicine. - [Awesome-LLM-Inference](https://github.com/DefTruth/Awesome-LLM-Inference) - A curated list of Awesome LLM Inference Paper with codes. +- [Awesome-LLM-3D](https://github.com/ActiveVisionLab/Awesome-LLM-3D) - A curated list of Multi-modal Large Language Model in 3D world, including 3D understanding, reasoning, generation, and embodied agents. - [LLMDatahub](https://github.com/Zjh-819/LLMDataHub) - a curated collection of datasets specifically designed for chatbot training, including links, size, language, usage, and a brief description of each dataset - [Awesome-Chinese-LLM](https://github.com/HqWu-HITCS/Awesome-Chinese-LLM) - 整理开源的中文大语言模型，以规模较小、可私有化部署、训练成本较低的模型为主，包括底座模型，垂直领域微调及应用，数据集与教程等。 - [llm-course](https://github.com/mlabonne/llm-course) - Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks. From 337e0a4be83f9dead256b2f5c96e33075ccd2dc0 Mon Sep 17 00:00:00 2001 From: wsl Date: Sun, 17 Dec 2023 14:40:38 +0800 Subject: [PATCH 031/117] add deepseek, baichuan, Yi --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 25c52b9..d3cb687 100644 --- a/README.md +++ b/README.md @@ -272,6 +272,9 @@ The above tables coule be better summarized by this wonderful visualization from - [BLOOM](https://huggingface.co/bigscience/bloom) - BigScience Large Open-science Open-access Multilingual Language Model [BLOOM-LoRA](https://github.com/linhduongtuan/BLOOM-LORA) - [BLOOMZ&mT0](https://huggingface.co/bigscience/bloomz) - a family of models capable of following human instructions in dozens of languages zero-shot. - [Phoenix](https://github.com/FreedomIntelligence/LLMZoo) +- [Deepseek Coder](https://github.com/deepseek-ai/DeepSeek-Coder) - Let the Code Write Itself. +- [Deepseek LLM](https://github.com/deepseek-ai/DeepSeek-LLM) - Let there be answers. +- [Yi](https://github.com/01-ai/Yi) - A series of large language models trained from scratch by developers @01-ai. - [T5](https://arxiv.org/abs/1910.10683) - Text-to-Text Transfer Transformer - [T0](https://arxiv.org/abs/2110.08207) - Multitask Prompted Training Enables Zero-Shot Task Generalization - [OPT](https://arxiv.org/abs/2205.01068) - Open Pre-trained Transformer Language Models. @@ -303,7 +306,7 @@ The above tables coule be better summarized by this wonderful visualization from - [MPT-7B](https://www.mosaicml.com/blog/mpt-7b) - Open LLM for commercial use by MosaicML - [Falcon](https://falconllm.tii.ae) - Falcon LLM is a foundational large language model (LLM) with 40 billion parameters trained on one trillion tokens. TII has now released Falcon LLM – a 40B model. - [XGen](https://github.com/salesforce/xgen) - Salesforce open-source LLMs with 8k sequence length. -- [baichuan-7B](https://github.com/baichuan-inc/baichuan-7B) - baichuan-7B 是由百川智能开发的一个开源可商用的大规模预训练语言模型. +- [Baichuan](https://github.com/baichuan-inc) - A series of large language models developed by Baichuan Intelligent Technology. - [Aquila](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/Aquila) - 悟道·天鹰语言大模型是首个具备中英双语知识、支持商用许可协议、国内数据合规需求的开源语言大模型。 - [phi-1](https://arxiv.org/abs/2306.11644) - a new large language model for code, with significantly smaller size than competing models. - [phi-1.5](https://arxiv.org/abs/2309.05463) - a 1.3 billion parameter model trained on a dataset of 30 billion tokens, which achieves common sense reasoning benchmark results comparable to models ten times its size that were trained on datasets more than ten times larger. From 7bdaa1c3152011d6b7eefcdd2119bfe3ff5fd54d Mon Sep 17 00:00:00 2001 From: Himanshu Date: Mon, 18 Dec 2023 10:28:36 +0530 Subject: [PATCH 032/117] Update README.md --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 2049e47..9f48621 100644 --- a/README.md +++ b/README.md @@ -333,7 +333,7 @@ The above tables coule be better summarized by this wonderful visualization from - [LMQL](https://lmql.ai) — A programming language for LLM interaction with support for typed prompting, control flow, constraints, and tools. - [ModelFusion](https://github.com/lgrammel/modelfusion) - A TypeScript library for building apps with LLMs and other ML models (speech-to-text, text-to-speech, image generation). - [Flappy](https://github.com/pleisto/flappy) — Production-Ready LLM Agent SDK for Every Developer. - +- [GPTRouter](https://gpt-router.writesonic.com/) - GPTRouter is an open source LLM API Gateway that offers a universal API for 30+ LLMs, vision, and image models, with smart fallbacks based on uptime and latency, automatic retries, and streaming. Stay operational even when OpenAI is down ## Tutorials about LLM - [Andrej Karpathy] State of GPT [video](https://build.microsoft.com/en-US/sessions/db3f4859-cd30-4445-a0cd-553c3304f8e2) @@ -432,7 +432,6 @@ The above tables coule be better summarized by this wonderful visualization from - [Awesome-LLM-Healthcare](https://github.com/mingze-yuan/Awesome-LLM-Healthcare) - The paper list of the review on LLMs in medicine. - [Awesome-LLM-Inference](https://github.com/DefTruth/Awesome-LLM-Inference) - A curated list of Awesome LLM Inference Paper with codes. - [LLMDatahub](https://github.com/Zjh-819/LLMDataHub) - a curated collection of datasets specifically designed for chatbot training, including links, size, language, usage, and a brief description of each dataset -- [GPTRouter](https://gpt-router.writesonic.com/) - GPTRouter is an open source LLM API Gateway that offers a universal API for 30+ LLMs, vision, and image models, with smart fallbacks based on uptime and latency, automatic retries, and streaming. Stay operational even when OpenAI is down ## Other Useful Resources From ffa8438d2e100b4b6c8e473720b53a61facbff0a Mon Sep 17 00:00:00 2001 From: Ben Auffarth <10786684+benman1@users.noreply.github.com> Date: Mon, 25 Dec 2023 13:38:14 +0100 Subject: [PATCH 033/117] Generative AI with LangChain: Build large language model (LLM) apps with Python, ChatGPT, and other LLMs --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index d0e29ad..aef8c45 100644 --- a/README.md +++ b/README.md @@ -420,6 +420,10 @@ The above tables coule be better summarized by this wonderful visualization from - [Aston Zhang] Chain of Thought论文 [Bilibili](https://www.bilibili.com/video/BV1t8411e7Ug/?spm_id_from=333.788&vd_source=1e55c5426b48b37e901ff0f78992e33f) [Youtube](https://www.youtube.com/watch?v=H4J59iG3t5o&list=PLFXJ6jwg0qW-7UM8iUTj3qKqdhbQULP5I&index=29) - [MIT] Introduction to Data-Centric AI [Homepage](https://dcai.csail.mit.edu) +## Books +- [Generative AI with LangChain: Build large language model (LLM) apps with Python, ChatGPT, and other LLMs](https://amzn.to/3GUlRng) + + ## Opinions - [A Stage Review of Instruction Tuning](https://yaofu.notion.site/June-2023-A-Stage-Review-of-Instruction-Tuning-f59dbfc36e2d4e12a33443bd6b2012c2) [2023-06-29] [Yao Fu] From c6b6e1a8071ef6d52dd42645dc543883f34ad57b Mon Sep 17 00:00:00 2001 From: Ben Auffarth <10786684+benman1@users.noreply.github.com> Date: Mon, 25 Dec 2023 15:39:11 +0100 Subject: [PATCH 034/117] link to repository --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index aef8c45..46cf9f3 100644 --- a/README.md +++ b/README.md @@ -421,7 +421,7 @@ The above tables coule be better summarized by this wonderful visualization from - [MIT] Introduction to Data-Centric AI [Homepage](https://dcai.csail.mit.edu) ## Books -- [Generative AI with LangChain: Build large language model (LLM) apps with Python, ChatGPT, and other LLMs](https://amzn.to/3GUlRng) +- [Generative AI with LangChain: Build large language model (LLM) apps with Python, ChatGPT, and other LLMs](https://amzn.to/3GUlRng) - it comes with a [GitHub repository](https://github.com/benman1/generative_ai_with_langchain) that showcases a lot of the functionality ## Opinions From 2872a8aa5738e615ceed49c48869c6c8e8b03fd5 Mon Sep 17 00:00:00 2001 From: Mark Needham Date: Sun, 31 Dec 2023 22:13:09 +0000 Subject: [PATCH 035/117] Update README.md Guardrails moved --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d0e29ad..67531c0 100644 --- a/README.md +++ b/README.md @@ -358,7 +358,7 @@ The above tables coule be better summarized by this wonderful visualization from - [LangChain](https://github.com/hwchase17/langchain) — A popular Python/JavaScript library for chaining sequences of language model prompts. - [FLAML (A Fast Library for Automated Machine Learning & Tuning)](https://microsoft.github.io/FLAML/docs/Getting-Started/): A Python library for automating selection of models, hyperparameters, and other tunable choices. - [Chainlit](https://docs.chainlit.io/overview) — A Python library for making chatbot interfaces. -- [Guardrails.ai](https://shreyar.github.io/guardrails/) — A Python library for validating outputs and retrying failures. Still in alpha, so expect sharp edges and bugs. +- [Guardrails.ai](https://www.guardrailsai.com/docs/) — A Python library for validating outputs and retrying failures. Still in alpha, so expect sharp edges and bugs. - [Semantic Kernel](https://github.com/microsoft/semantic-kernel) — A Python/C#/Java library from Microsoft that supports prompt templating, function chaining, vectorized memory, and intelligent planning. - [Prompttools](https://github.com/hegelai/prompttools) — Open-source Python tools for testing and evaluating models, vector DBs, and prompts. - [Outlines](https://github.com/normal-computing/outlines) — A Python library that provides a domain-specific language to simplify prompting and constrain generation. From b2983bbeefe1df8186d295276b57efeddaf93c8d Mon Sep 17 00:00:00 2001 From: Hannibal046 <38466901+Hannibal046@users.noreply.github.com> Date: Thu, 4 Jan 2024 19:56:07 +0800 Subject: [PATCH 036/117] Update README.md add llm_course --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 9fffe46..7808184 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,7 @@ - Add LLM data (Pretraining data/Instruction Tuning data/Chat data/RLHF data) :sparkles:**Contributions Wanted** --> ## Trending LLM Projects +- [llm-course](https://github.com/mlabonne/llm-course) - Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks. - [Mixtral 8x7B](https://mistral.ai/news/mixtral-of-experts/) - a high-quality sparse mixture of experts model (SMoE) with open weights. - [promptbase](https://github.com/microsoft/promptbase) - All things prompt engineering. - [ollama](https://github.com/jmorganca/ollama) - Get up and running with Llama 2 and other large language models locally. From 006410de8cf8c1bfac002696033fb831d71dd68a Mon Sep 17 00:00:00 2001 From: ewired <37567272+ewired@users.noreply.github.com> Date: Fri, 12 Jan 2024 19:27:51 -0600 Subject: [PATCH 037/117] Add YALL It's yet another leaderboard, YALL! --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 7808184..470e181 100644 --- a/README.md +++ b/README.md @@ -181,6 +181,7 @@ There are three important steps for a ChatGPT-like LLM: > - [Chatbot Arena Leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard) - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner. > - [AlpacaEval Leaderboard](https://tatsu-lab.github.io/alpaca_eval/) - An Automatic Evaluator for Instruction-following Language Models > - [Open Ko-LLM Leaderboard](https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard) - The Open Ko-LLM Leaderboard objectively evaluates the performance of Korean Large Language Model (LLM). + > - [Yet Another LLM Leaderboard](https://huggingface.co/spaces/mlabonne/Yet_Another_LLM_Leaderboard) - Leaderboard made with LLM AutoEval using Nous benchmark suite. From fe9b60ba86e378437aafa69484197e54ae43c3be Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Stjepan=20Jurekovi=C4=87?= Date: Wed, 17 Jan 2024 14:08:14 +0100 Subject: [PATCH 038/117] Added Build a Large Language Model (From Scratch) Hi, Stjepan here from Manning. I thought this title might be a good match for your list. Thank you for considering it. Best, --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 470e181..96220d6 100644 --- a/README.md +++ b/README.md @@ -424,6 +424,7 @@ The above tables coule be better summarized by this wonderful visualization from ## Books - [Generative AI with LangChain: Build large language model (LLM) apps with Python, ChatGPT, and other LLMs](https://amzn.to/3GUlRng) - it comes with a [GitHub repository](https://github.com/benman1/generative_ai_with_langchain) that showcases a lot of the functionality +- [Build a Large Language Model (From Scratch)](https://www.manning.com/books/build-a-large-language-model-from-scratch) - A guide to building your own working LLM. ## Opinions From c6eb3a4d9a2726bc1d14759548c8706cf0c6bdbd Mon Sep 17 00:00:00 2001 From: Hannibal046 <38466901+Hannibal046@users.noreply.github.com> Date: Thu, 1 Feb 2024 12:33:26 +0800 Subject: [PATCH 039/117] Update README.md Add awesome-hallucination-detection --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 96220d6..6775632 100644 --- a/README.md +++ b/README.md @@ -91,6 +91,7 @@ ## Other Papers If you're interested in the field of LLM, you may find the above list of milestone papers helpful to explore its history and state-of-the-art. However, each direction of LLM offers a unique set of insights and contributions, which are essential to understanding the field as a whole. For a detailed list of papers in various subfields, please refer to the following link: +- [awesome-hallucination-detection](https://github.com/EdinburghNLP/awesome-hallucination-detection) - List of papers on hallucination detection in LLMs. - [LLMsPracticalGuide](https://github.com/Mooler0410/LLMsPracticalGuide) - A curated (still actively updated) list of practical guide resources of LLMs - [Awesome ChatGPT Prompts](https://github.com/f/awesome-chatgpt-prompts) - A collection of prompt examples to be used with the ChatGPT model. - [awesome-chatgpt-prompts-zh](https://github.com/PlexPt/awesome-chatgpt-prompts-zh) - A Chinese collection of prompt examples to be used with the ChatGPT model. From b5d360a92b679cfe69fd90ee1fdccadf9a4b6663 Mon Sep 17 00:00:00 2001 From: XavierSpycy <115918068+XavierSpycy@users.noreply.github.com> Date: Fri, 2 Feb 2024 01:20:09 +0800 Subject: [PATCH 040/117] Update the latest info --- README.md | 36 ++++++++++++++++++-- paper_list/Retrieval_Augmented_Generation.md | 1 + 2 files changed, 35 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 6775632..dbde756 100644 --- a/README.md +++ b/README.md @@ -83,6 +83,7 @@ | 2023-05 | RWKV | Bo Peng | [RWKV: Reinventing RNNs for the Transformer Era](https://arxiv.org/abs/2305.13048) |EMNLP
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F026b3396a63ed5772329708b7580d633bb86bec9%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-05 | DPO | Stanford | [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/pdf/2305.18290.pdf) |Neurips
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F0d1c76d45afa012ded7ab741194baf142117c495%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-07 | LLaMA 2 | Meta | [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/pdf/2307.09288.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F104b0bb1da562d53cbda87aec79ef6a2827d191a%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| +|2023-10| Mistral 7B| Mistral |[Mistral 7B](https://arxiv.org/pdf/2310.06825.pdf%5D%5D%3E)|
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fdb633c6b1c286c0386f0078d8a2e6224e03a6227%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-12 | Mamba | CMU&Princeton | [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/ftp/arxiv/papers/2312/2312.00752.pdf) |ICLR
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F432bef8e34014d726c674bc458008ac895297b51%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| @@ -183,6 +184,7 @@ There are three important steps for a ChatGPT-like LLM: > - [AlpacaEval Leaderboard](https://tatsu-lab.github.io/alpaca_eval/) - An Automatic Evaluator for Instruction-following Language Models > - [Open Ko-LLM Leaderboard](https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard) - The Open Ko-LLM Leaderboard objectively evaluates the performance of Korean Large Language Model (LLM). > - [Yet Another LLM Leaderboard](https://huggingface.co/spaces/mlabonne/Yet_Another_LLM_Leaderboard) - Leaderboard made with LLM AutoEval using Nous benchmark suite. + > - [OpenCompass 2.0 LLM Leaderboard](https://rank.opencompass.org.cn/leaderboard-llm-v2) - OpenCompass is an LLM evaluation platform, supporting a wide range of models (InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets. @@ -275,8 +277,10 @@ The above tables coule be better summarized by this wonderful visualization from - [BLOOM](https://huggingface.co/bigscience/bloom) - BigScience Large Open-science Open-access Multilingual Language Model [BLOOM-LoRA](https://github.com/linhduongtuan/BLOOM-LORA) - [BLOOMZ&mT0](https://huggingface.co/bigscience/bloomz) - a family of models capable of following human instructions in dozens of languages zero-shot. - [Phoenix](https://github.com/FreedomIntelligence/LLMZoo) -- [Deepseek Coder](https://github.com/deepseek-ai/DeepSeek-Coder) - Let the Code Write Itself. -- [Deepseek LLM](https://github.com/deepseek-ai/DeepSeek-LLM) - Let there be answers. +- [Deepseek](https://github.com/deepseek-ai/) + - [Coder](https://github.com/deepseek-ai/DeepSeek-Coder) - Let the Code Write Itself. + - [LLM](https://github.com/deepseek-ai/DeepSeek-LLM) - Let there be answers. + - 知名私募巨头幻方量化旗下的人工智能公司深度求索（DeepSeek）自主研发的大语言模型开发的智能助手。包括 [7B-base](https://modelscope.cn/models/deepseek-ai/deepseek-llm-7b-base/summary), [67B-base](https://modelscope.cn/models/deepseek-ai/deepseek-llm-67b-base/summary), - [Yi](https://github.com/01-ai/Yi) - A series of large language models trained from scratch by developers @01-ai. - [T5](https://arxiv.org/abs/1910.10683) - Text-to-Text Transfer Transformer - [T0](https://arxiv.org/abs/2110.08207) - Multitask Prompted Training Enables Zero-Shot Task Generalization @@ -285,8 +289,10 @@ The above tables coule be better summarized by this wonderful visualization from - [GLM](https://github.com/THUDM/GLM)- GLM is a General Language Model pretrained with an autoregressive blank-filling objective and can be finetuned on various natural language understanding and generation tasks. - [ChatGLM-6B](https://github.com/THUDM/ChatGLM-6B) - ChatGLM-6B 是一个开源的、支持中英双语的对话语言模型，基于 [General Language Model (GLM)](https://github.com/THUDM/GLM) 架构，具有 62 亿参数. - [ChatGLM2-6B](https://github.com/THUDM/ChatGLM2-6B) - An Open Bilingual Chat LLM | 开源双语对话语言模型 + - [ChatGLM3-6B](https://github.com/THUDM/ChatGLM3) - An Open Bilingual Chat LLMs | 开源双语对话语言模型 ; Including [ChatGLM3-6B-32k](https://huggingface.co/THUDM/chatglm3-6b-32k), [ChatGLM3-6B-128k](https://huggingface.co/THUDM/chatglm3-6b-128k). - [RWKV](https://github.com/BlinkDL/RWKV-LM) - Parallelizable RNN with Transformer-level LLM Performance. - [ChatRWKV](https://github.com/BlinkDL/ChatRWKV) - ChatRWKV is like ChatGPT but powered by my RWKV (100% RNN) language model. + - [Trending Demo](https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-2) - RWKV-5 trained on 100+ world languages (70% English, 15% multilang, 15% code). - [StableLM](https://stability.ai/blog/stability-ai-launches-the-first-of-its-stablelm-suite-of-language-models) - Stability AI Language Models. - [YaLM](https://medium.com/yandex/yandex-publishes-yalm-100b-its-the-largest-gpt-like-neural-network-in-open-source-d1df53d0e9a6) - a GPT-like neural network for generating and processing text. It can be used freely by developers and researchers from all over the world. - [GPT-Neo](https://github.com/EleutherAI/gpt-neo) - An implementation of model & data parallel [GPT3](https://arxiv.org/abs/2005.14165)-like models using the [mesh-tensorflow](https://github.com/tensorflow/mesh) library. @@ -314,6 +320,12 @@ The above tables coule be better summarized by this wonderful visualization from - [phi-1](https://arxiv.org/abs/2306.11644) - a new large language model for code, with significantly smaller size than competing models. - [phi-1.5](https://arxiv.org/abs/2309.05463) - a 1.3 billion parameter model trained on a dataset of 30 billion tokens, which achieves common sense reasoning benchmark results comparable to models ten times its size that were trained on datasets more than ten times larger. - [phi-2](https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/) - a 2.7 billion-parameter language model that demonstrates outstanding reasoning and language understanding capabilities, showcasing state-of-the-art performance among base language models with less than 13 billion parameters. +- [InternLM / 书生·浦语](https://github.com/InternLM/InternLM) - Official release of InternLM2 7B and 20B base and chat models. 200K context support. [Homepage](https://internlm.intern-ai.org.cn/) | [ModelScope](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm-7b/summary) +- [BlueLM-7B](https://github.com/vivo-ai-lab/BlueLM) - BlueLM(蓝心大模型): Open large language models developed by vivo AI Lab. [Homepage](https://developers.vivo.com/product/ai/bluelm) | [ModelScope](https://modelscope.cn/models/vivo-ai/BlueLM-7B-Base/summary) +[MoE-16B-base](https://modelscope.cn/models/deepseek-ai/deepseek-moe-16b-base), 等. | [Chat with DeepSeek (Beta)](https://chat.deepseek.com/sign_in) +- [Qwen series](https://huggingface.co/Qwen) - The large language model series proposed by Alibaba Cloud. ｜ 阿里云研发的通义千问大模型系列. 包括 [7B](https://huggingface.co/Qwen/Qwen-7B), [72B](https://huggingface.co/Qwen/Qwen-72B), 及各种量化和Chat版本. [Chat Demo](https://huggingface.co/spaces/Qwen/Qwen-72B-Chat-Demo) +- [XVERSE series](https://github.com/xverse-ai) - Multilingual large language model developed by XVERSE Technology Inc | 由深圳元象科技自主研发的支持多语言的大语言模型. 包括[7B](https://github.com/xverse-ai/XVERSE-7B), [13B](https://github.com/xverse-ai/XVERSE-13B), [65B](https://github.com/xverse-ai/XVERSE-65B)等. +- [Skywork series](https://github.com/SkyworkAI/Skywork) - A series of large models developed by the Kunlun Group · Skywork team | 昆仑万维集团·天工团队开发的一系列大型模型. ## LLM Training Frameworks @@ -353,6 +365,10 @@ The above tables coule be better summarized by this wonderful visualization from - [Text-Embeddings-Inference](https://github.com/huggingface/text-embeddings-inference) - Inference for text-embeddings in Rust, HFOIL Licence. - [Infinity](https://github.com/michaelfeil/infinity) - Inference for text-embeddings in Python - [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) - Nvidia Framework for LLM Inference +- [FasterTransformer](https://github.com/NVIDIA/FasterTransformer) - NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM) +- [Flash-Attention](https://github.com/Dao-AILab/flash-attention) - A method designed to enhance the efficiency of Transformer models +- [Langchain-Chatchat](https://github.com/chatchat-space/Langchain-Chatchat) - Formerly langchain-ChatGLM, local knowledge based LLM (like ChatGLM) QA app with langchain. +- [Search with Lepton](https://github.com/leptonai/search_with_lepton) - Build your own conversational search engine using less than 500 lines of code by [LeptonAI](https://github.com/leptonai). ## Prompting libraries & tools @@ -376,6 +392,8 @@ The above tables coule be better summarized by this wonderful visualization from - [ModelFusion](https://github.com/lgrammel/modelfusion) - A TypeScript library for building apps with LLMs and other ML models (speech-to-text, text-to-speech, image generation). - [Flappy](https://github.com/pleisto/flappy) — Production-Ready LLM Agent SDK for Every Developer. - [GPTRouter](https://gpt-router.writesonic.com/) - GPTRouter is an open source LLM API Gateway that offers a universal API for 30+ LLMs, vision, and image models, with smart fallbacks based on uptime and latency, automatic retries, and streaming. Stay operational even when OpenAI is down +- [QAnything](https://github.com/netease-youdao/QAnything) - A local knowledge base question-answering system designed to support a wide range of file formats and databases. + - Core modules: [BCEmbedding](https://github.com/netease-youdao/BCEmbedding) - Bilingual and Crosslingual Embedding for RAG ## Tutorials - [Andrej Karpathy] State of GPT [video](https://build.microsoft.com/en-US/sessions/db3f4859-cd30-4445-a0cd-553c3304f8e2) @@ -422,6 +440,18 @@ The above tables coule be better summarized by this wonderful visualization from - [李沐] GPT，GPT-2，GPT-3 论文精读 [Bilibili](https://www.bilibili.com/video/BV1AF411b7xQ/?spm_id_from=333.788&vd_source=1e55c5426b48b37e901ff0f78992e33f) [Youtube](https://www.youtube.com/watch?v=t70Bl3w7bxY&list=PLFXJ6jwg0qW-7UM8iUTj3qKqdhbQULP5I&index=18) - [Aston Zhang] Chain of Thought论文 [Bilibili](https://www.bilibili.com/video/BV1t8411e7Ug/?spm_id_from=333.788&vd_source=1e55c5426b48b37e901ff0f78992e33f) [Youtube](https://www.youtube.com/watch?v=H4J59iG3t5o&list=PLFXJ6jwg0qW-7UM8iUTj3qKqdhbQULP5I&index=29) - [MIT] Introduction to Data-Centric AI [Homepage](https://dcai.csail.mit.edu) +- [DeepLearning.AI] Building Applications with Vector Databases [Homepage](https://www.deeplearning.ai/short-courses/building-applications-vector-databases/) +- [DeepLearning.AI] Building Systems with the ChatGPT API [Homepage](https://www.deeplearning.ai/short-courses/building-systems-with-chatgpt/) +- [DeepLearning.AI] LangChain for LLM Application Development [Homepage](https://www.deeplearning.ai/short-courses/langchain-for-llm-application-development/) +- [DeepLearning.AI] LangChain: Chat with Your Data [Homepage](https://www.deeplearning.ai/short-courses/langchain-chat-with-your-data/) +- [DeepLearning.AI] Finetuning Large Language Models [Homepage](https://www.deeplearning.ai/short-courses/finetuning-large-language-models/) +- [DeepLearning.AI] Build LLM Apps with LangChain.js [Homepage](https://www.deeplearning.ai/short-courses/build-llm-apps-with-langchain-js/) +- [DeepLearning.AI] Large Language Models with Semantic Search [Homepage](https://www.deeplearning.ai/short-courses/large-language-models-semantic-search/) +- [DeepLearning.AI] LLMOps [Homepage](https://www.deeplearning.ai/short-courses/llmops/) +- [DeepLearning.AI] Building and Evaluating Advanced RAG Applications [Homepage](https://www.deeplearning.ai/short-courses/building-evaluating-advanced-rag/) +- [DeepLearning.AI] Quality and Safety for LLM Applications [Homepage](https://www.deeplearning.ai/short-courses/quality-safety-llm-applications/) +- [DeepLearning.AI] Vector Databases: from Embeddings to Applications [Homepage](https://www.deeplearning.ai/short-courses/vector-databases-embeddings-applications/) +- [DeepLearning.AI] Functions, Tools and Agents with LangChain [Homepage](https://www.deeplearning.ai/short-courses/functions-tools-agents-langchain/) ## Books - [Generative AI with LangChain: Build large language model (LLM) apps with Python, ChatGPT, and other LLMs](https://amzn.to/3GUlRng) - it comes with a [GitHub repository](https://github.com/benman1/generative_ai_with_langchain) that showcases a lot of the functionality @@ -469,6 +499,8 @@ The above tables coule be better summarized by this wonderful visualization from - [HuggingGPT](https://github.com/microsoft/JARVIS) - Solving AI Tasks with ChatGPT and its Friends in HuggingFace. - [EasyEdit](https://github.com/zjunlp/EasyEdit) - An easy-to-use framework to edit large language models. - [chatgpt-shroud](https://github.com/guyShilo/chatgpt-shroud) - A Chrome extension for OpenAI's ChatGPT, enhancing user privacy by enabling easy hiding and unhiding of chat history. Ideal for privacy during screen shares. +- [MTEB](https://huggingface.co/spaces/mteb/leaderboard) - Massive Text Embedding Benchmark Leaderboard +- [xFormer](https://github.com/facebookresearch/xformers) - A PyTorch based library which hosts flexible Transformers parts ## Contributing This is an active repository and your contributions are always welcome! diff --git a/paper_list/Retrieval_Augmented_Generation.md b/paper_list/Retrieval_Augmented_Generation.md index cfe0487..5a1d092 100644 --- a/paper_list/Retrieval_Augmented_Generation.md +++ b/paper_list/Retrieval_Augmented_Generation.md @@ -3,3 +3,4 @@ ## Useful Resource - [Retrieval-Augmented Generation_Paper](https://arxiv.org/abs/2005.11401v4) - The Original Paper on RAG published by Meta in 2020. +- [Retrieval-Augmented Geneartion Survey](https://arxiv.org/pdf/2312.10997.pdf) - A Comprehensive and High-quality Survey Conducted by Tongji University and Fudan University on RAG in 2023. From ed8773524f57b0f284122ec3f3fa0d3785733f99 Mon Sep 17 00:00:00 2001 From: Hannibal046 <38466901+Hannibal046@users.noreply.github.com> Date: Mon, 5 Feb 2024 17:06:04 +0800 Subject: [PATCH 041/117] Update README.md --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index dbde756..56a9055 100644 --- a/README.md +++ b/README.md @@ -10,12 +10,13 @@ - Add LLM data (Pretraining data/Instruction Tuning data/Chat data/RLHF data) :sparkles:**Contributions Wanted** --> ## Trending LLM Projects +- [OLMo](https://github.com/allenai/OLMo) - Open Language Model. +- [miqu-1-70b](https://huggingface.co/miqudev/miqu-1-70b) - A leaked 70B model from Mistral AI. - [llm-course](https://github.com/mlabonne/llm-course) - Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks. - [Mixtral 8x7B](https://mistral.ai/news/mixtral-of-experts/) - a high-quality sparse mixture of experts model (SMoE) with open weights. - [promptbase](https://github.com/microsoft/promptbase) - All things prompt engineering. - [ollama](https://github.com/jmorganca/ollama) - Get up and running with Llama 2 and other large language models locally. - [anything-llm](https://github.com/Mintplex-Labs/anything-llm) - A private ChatGPT to chat with anything! -- [phi-2](https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/) - a 2.7 billion-parameter language model that demonstrates outstanding reasoning and language understanding capabilities, showcasing state-of-the-art performance among base language models with less than 13 billion parameters. ## Table of Content From e14d9947fad4557cd0c8290ed8f92fe6145e3330 Mon Sep 17 00:00:00 2001 From: Leon Knauer Date: Tue, 20 Feb 2024 15:05:59 +0100 Subject: [PATCH 042/117] Added Awesome Deliberative Prompting --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 56a9055..5bf522f 100644 --- a/README.md +++ b/README.md @@ -99,6 +99,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [awesome-chatgpt-prompts-zh](https://github.com/PlexPt/awesome-chatgpt-prompts-zh) - A Chinese collection of prompt examples to be used with the ChatGPT model. - [Awesome ChatGPT](https://github.com/humanloop/awesome-chatgpt) - Curated list of resources for ChatGPT and GPT-3 from OpenAI. - [Chain-of-Thoughts Papers](https://github.com/Timothyxxx/Chain-of-ThoughtsPapers) - A trend starts from "Chain of Thought Prompting Elicits Reasoning in Large Language Models. +- [Awesome Deliberative Prompting](https://github.com/logikon-ai/awesome-deliberative-prompting) - How to ask LLMs to produce reliable reasoning and make reason-responsive decisions. - [Instruction-Tuning-Papers](https://github.com/SinclairCoder/Instruction-Tuning-Papers) - A trend starts from `Natrural-Instruction` (ACL 2022), `FLAN` (ICLR 2022) and `T0` (ICLR 2022). - [LLM Reading List](https://github.com/crazyofapple/Reading_groups/) - A paper & resource list of large language models. - [Reasoning using Language Models](https://github.com/atfortes/LM-Reasoning-Papers) - Collection of papers and resources on Reasoning using Language Models. From c0539e680fc41c218e9856d886d0f4b79064d767 Mon Sep 17 00:00:00 2001 From: David <119470903+PubliusAu@users.noreply.github.com> Date: Fri, 23 Feb 2024 09:28:25 -0600 Subject: [PATCH 043/117] Update README.md Adding two free certification courses on LLM observability. --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 5bf522f..636ed68 100644 --- a/README.md +++ b/README.md @@ -454,6 +454,8 @@ The above tables coule be better summarized by this wonderful visualization from - [DeepLearning.AI] Quality and Safety for LLM Applications [Homepage](https://www.deeplearning.ai/short-courses/quality-safety-llm-applications/) - [DeepLearning.AI] Vector Databases: from Embeddings to Applications [Homepage](https://www.deeplearning.ai/short-courses/vector-databases-embeddings-applications/) - [DeepLearning.AI] Functions, Tools and Agents with LangChain [Homepage](https://www.deeplearning.ai/short-courses/functions-tools-agents-langchain/) +- [Arize] LLM Observability: Evaluations [Homepage](https://courses.arize.com/p/llm-evaluations/) +- [Arize] LLM Observability: Traces and Spans [Homepage](https://courses.arize.com/p/llm-observability-traces-spans/) ## Books - [Generative AI with LangChain: Build large language model (LLM) apps with Python, ChatGPT, and other LLMs](https://amzn.to/3GUlRng) - it comes with a [GitHub repository](https://github.com/benman1/generative_ai_with_langchain) that showcases a lot of the functionality From 98ba0328f1071682c2549dae2cd56ea33c0fd192 Mon Sep 17 00:00:00 2001 From: wsl Date: Sat, 24 Feb 2024 15:23:38 +0800 Subject: [PATCH 044/117] add llm-eval, gemma, minbpe, mamba tutorial --- README.md | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 636ed68..3e03c97 100644 --- a/README.md +++ b/README.md @@ -10,13 +10,9 @@ - Add LLM data (Pretraining data/Instruction Tuning data/Chat data/RLHF data) :sparkles:**Contributions Wanted** --> ## Trending LLM Projects -- [OLMo](https://github.com/allenai/OLMo) - Open Language Model. -- [miqu-1-70b](https://huggingface.co/miqudev/miqu-1-70b) - A leaked 70B model from Mistral AI. -- [llm-course](https://github.com/mlabonne/llm-course) - Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks. -- [Mixtral 8x7B](https://mistral.ai/news/mixtral-of-experts/) - a high-quality sparse mixture of experts model (SMoE) with open weights. -- [promptbase](https://github.com/microsoft/promptbase) - All things prompt engineering. -- [ollama](https://github.com/jmorganca/ollama) - Get up and running with Llama 2 and other large language models locally. -- [anything-llm](https://github.com/Mintplex-Labs/anything-llm) - A private ChatGPT to chat with anything! +- [Sora](https://openai.com/sora) - Sora is an AI model that can create realistic and imaginative scenes from text instructions. +- [Gemma](https://blog.google/technology/developers/gemma-open-models/) - Gemma is built for responsible AI development from the same research and technology used to create Gemini models. +- [minbpe](https://github.com/karpathy/minbpe) - Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. ## Table of Content @@ -25,6 +21,7 @@ - [Other Papers](#other-papers) - [Open LLM](#open-llm) - [LLM Training Frameworks](#llm-training-frameworks) + - [LLM Evaluation Frameworks](#llm-evaluation-frameworks) - [Tools for deploying LLM](#deploying-tools) - [Tutorials about LLM](#tutorials) - [Courses about LLM](#courses) @@ -93,6 +90,7 @@ ## Other Papers If you're interested in the field of LLM, you may find the above list of milestone papers helpful to explore its history and state-of-the-art. However, each direction of LLM offers a unique set of insights and contributions, which are essential to understanding the field as a whole. For a detailed list of papers in various subfields, please refer to the following link: +- [Awesome-LLM-hallucination](https://github.com/LuckyyySTA/Awesome-LLM-hallucination) - LLM hallucination paper list. - [awesome-hallucination-detection](https://github.com/EdinburghNLP/awesome-hallucination-detection) - List of papers on hallucination detection in LLMs. - [LLMsPracticalGuide](https://github.com/Mooler0410/LLMsPracticalGuide) - A curated (still actively updated) list of practical guide resources of LLMs - [Awesome ChatGPT Prompts](https://github.com/f/awesome-chatgpt-prompts) - A collection of prompt examples to be used with the ChatGPT model. @@ -251,7 +249,7 @@ The above tables coule be better summarized by this wonderful visualization from --- ## Open LLM --> - +- [Gemma](https://blog.google/technology/developers/gemma-open-models/) - Gemma is built for responsible AI development from the same research and technology used to create Gemini models. - [Mistral](https://mistral.ai/) - Mistral-7B-v0.1 is a small, yet powerful model adaptable to many use-cases including code and 8k sequence length. Apache 2.0 licence. - [Mixtral 8x7B](https://mistral.ai/news/mixtral-of-experts/) - a high-quality sparse mixture of experts model (SMoE) with open weights. - [LLaMA](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) & [LLaMA-2](https://ai.meta.com/llama/) - A foundational large language model. [LLaMA.cpp](https://github.com/ggerganov/llama.cpp) [Lit-LLaMA](https://github.com/Lightning-AI/lit-llama) @@ -342,6 +340,12 @@ The above tables coule be better summarized by this wonderful visualization from - [Alpa](https://alpa.ai/index.html) - Alpa is a system for training and serving large-scale neural networks. - [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) - An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library. +## LLM Evaluation Frameworks: +- [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) - A framework for few-shot evaluation of language models. +- [lighteval](https://github.com/huggingface/lighteval) - a lightweight LLM evaluation suite that Hugging Face has been using internally. +- [OLMO-eval](https://github.com/allenai/OLMo-Eval) - a repository for evaluating open language models. +- [instruct-eval](https://github.com/declare-lab/instruct-eval) - This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks. + ## Deploying Tools - [FastChat](https://github.com/lm-sys/FastChat) - A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs. @@ -398,6 +402,9 @@ The above tables coule be better summarized by this wonderful visualization from - Core modules: [BCEmbedding](https://github.com/netease-youdao/BCEmbedding) - Bilingual and Crosslingual Embedding for RAG ## Tutorials +- [Maarten Grootendorst] A Visual Guide to Mamba and State Space Models [blog](https://maartengrootendorst.substack.com/p/a-visual-guide-to-mamba-and-state?utm_source=multiple-personal-recommendations-email&utm_medium=email&open=false) +- [Jack Cook] (Mamba: The Easy Way)[https://jackcook.com/2024/02/23/mamba.html] +- [Andrej Karpathy] minbpe [video](https://www.youtube.com/watch?v=zduSFxRajkE&t=1157s) - [Andrej Karpathy] State of GPT [video](https://build.microsoft.com/en-US/sessions/db3f4859-cd30-4445-a0cd-553c3304f8e2) - [Hyung Won Chung] Instruction finetuning and RLHF lecture [Youtube](https://www.youtube.com/watch?v=zjrM-MW-0y0) - [Jason Wei] Scaling, emergence, and reasoning in large language models [Slides](https://docs.google.com/presentation/d/1EUV7W7X_w0BDrscDhPg7lMGzJCkeaPkGCJ3bN8dluXc/edit?pli=1&resourcekey=0-7Nz5A7y8JozyVrnDtcEKJA#slide=id.g16197112905_0_0) @@ -429,6 +436,7 @@ The above tables coule be better summarized by this wonderful visualization from ## Courses +- [UWaterloo] CS 886: Recent Advances on Foundation Models [Homepage](https://cs.uwaterloo.ca/~wenhuche/teaching/cs886/) - [DeepLearning.AI] ChatGPT Prompt Engineering for Developers [Homepage](https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/) - [Princeton] Understanding Large Language Models [Homepage](https://www.cs.princeton.edu/courses/archive/fall22/cos597G/) - [OpenBMB] 大模型公开课 [主页](https://www.openbmb.org/community/course) @@ -486,7 +494,6 @@ The above tables coule be better summarized by this wonderful visualization from - [Large Language Models: A New Moore's Law ](https://huggingface.co/blog/large-language-models) \[2021-10-26\]\[Huggingface\] - ## Other Useful Resources - [Arize-Phoenix](https://phoenix.arize.com/) - Open-source tool for ML observability that runs in your notebook environment. Monitor and fine tune LLM, CV and Tabular Models. From 990a124c4dff507599773b63e67117ac57dca048 Mon Sep 17 00:00:00 2001 From: Hannibal046 <38466901+Hannibal046@users.noreply.github.com> Date: Sat, 24 Feb 2024 18:52:50 +0800 Subject: [PATCH 045/117] Update README.md fix mamba blog link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 3e03c97..dfc7e8f 100644 --- a/README.md +++ b/README.md @@ -403,7 +403,7 @@ The above tables coule be better summarized by this wonderful visualization from ## Tutorials - [Maarten Grootendorst] A Visual Guide to Mamba and State Space Models [blog](https://maartengrootendorst.substack.com/p/a-visual-guide-to-mamba-and-state?utm_source=multiple-personal-recommendations-email&utm_medium=email&open=false) -- [Jack Cook] (Mamba: The Easy Way)[https://jackcook.com/2024/02/23/mamba.html] +- [Jack Cook] [Mamba: The Easy Way](https://jackcook.com/2024/02/23/mamba.html) - [Andrej Karpathy] minbpe [video](https://www.youtube.com/watch?v=zduSFxRajkE&t=1157s) - [Andrej Karpathy] State of GPT [video](https://build.microsoft.com/en-US/sessions/db3f4859-cd30-4445-a0cd-553c3304f8e2) - [Hyung Won Chung] Instruction finetuning and RLHF lecture [Youtube](https://www.youtube.com/watch?v=zjrM-MW-0y0) From c05d45fe0af48908ffc70ee3f1d898bec801f401 Mon Sep 17 00:00:00 2001 From: Hannibal046 <38466901+Hannibal046@users.noreply.github.com> Date: Sun, 25 Feb 2024 11:15:54 +0800 Subject: [PATCH 046/117] Update README.md Add LWM --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index dfc7e8f..dc47435 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,7 @@ - Add LLM data (Pretraining data/Instruction Tuning data/Chat data/RLHF data) :sparkles:**Contributions Wanted** --> ## Trending LLM Projects +- [LWM](https://github.com/LargeWorldModel/LWM) - Large World Model (LWM) is a general-purpose large-context multimodal autoregressive model. - [Sora](https://openai.com/sora) - Sora is an AI model that can create realistic and imaginative scenes from text instructions. - [Gemma](https://blog.google/technology/developers/gemma-open-models/) - Gemma is built for responsible AI development from the same research and technology used to create Gemini models. - [minbpe](https://github.com/karpathy/minbpe) - Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. From 436dde35ac7eb79d0c71cb58c6599d7cbe8c2cfa Mon Sep 17 00:00:00 2001 From: Bogdan Condurache Date: Thu, 29 Feb 2024 01:28:02 +0200 Subject: [PATCH 047/117] Add Robocorp Action Server --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index dc47435..7ba1f13 100644 --- a/README.md +++ b/README.md @@ -376,6 +376,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Flash-Attention](https://github.com/Dao-AILab/flash-attention) - A method designed to enhance the efficiency of Transformer models - [Langchain-Chatchat](https://github.com/chatchat-space/Langchain-Chatchat) - Formerly langchain-ChatGLM, local knowledge based LLM (like ChatGLM) QA app with langchain. - [Search with Lepton](https://github.com/leptonai/search_with_lepton) - Build your own conversational search engine using less than 500 lines of code by [LeptonAI](https://github.com/leptonai). +- [Robocorp](https://github.com/robocorp/robocorp) - Create, deploy and operate Actions using Python anywhere to enhance your AI agents and assistants. Batteries included with an extensive set of libraries, helpers and logging. ## Prompting libraries & tools From 5f3a30ee19de8dd415681378b21fd9d906348a4b Mon Sep 17 00:00:00 2001 From: lvhan028 Date: Thu, 29 Feb 2024 21:01:04 +0800 Subject: [PATCH 048/117] add introduction to lmdeploy --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 7ba1f13..a652744 100644 --- a/README.md +++ b/README.md @@ -377,6 +377,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Langchain-Chatchat](https://github.com/chatchat-space/Langchain-Chatchat) - Formerly langchain-ChatGLM, local knowledge based LLM (like ChatGLM) QA app with langchain. - [Search with Lepton](https://github.com/leptonai/search_with_lepton) - Build your own conversational search engine using less than 500 lines of code by [LeptonAI](https://github.com/leptonai). - [Robocorp](https://github.com/robocorp/robocorp) - Create, deploy and operate Actions using Python anywhere to enhance your AI agents and assistants. Batteries included with an extensive set of libraries, helpers and logging. +- [LMDeploy](https://github.com/InternLM/lmdeploy) - A high-throughput and low-latency inference and serving framework for LLMs and VLs ## Prompting libraries & tools From 905cf7482fc92e3ac57fada23211a61c57c7eab9 Mon Sep 17 00:00:00 2001 From: Samar Patel <77489054+Sammindinventory@users.noreply.github.com> Date: Thu, 7 Mar 2024 09:39:24 +0530 Subject: [PATCH 049/117] Update README.md Added Text to SQL Package --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index a652744..542a723 100644 --- a/README.md +++ b/README.md @@ -350,6 +350,7 @@ The above tables coule be better summarized by this wonderful visualization from ## Deploying Tools - [FastChat](https://github.com/lm-sys/FastChat) - A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs. +- [MindSQL](https://github.com/Mindinventory/MindSQL) - A python package for Txt-to-SQL with self hosting functionalities and RESTful APIs compatible with proprietary as well as open source LLM. - [SkyPilot](https://github.com/skypilot-org/skypilot) - Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface. - [vLLM](https://github.com/vllm-project/vllm) - A high-throughput and memory-efficient inference and serving engine for LLMs - [Text Generation Inference](https://github.com/huggingface/text-generation-inference) - A Rust, Python and gRPC server for text generation inference. Used in production at [HuggingFace](https://huggingface.co/) to power LLMs api-inference widgets, HFOIL Licence. From 77c78940f99636586e7a1e70ef668f3a68a58f8a Mon Sep 17 00:00:00 2001 From: Hao-Ting Li Date: Tue, 12 Mar 2024 21:24:47 +0800 Subject: [PATCH 050/117] Update README.md Mamba did not be accepted by ICLR. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 542a723..5474372 100644 --- a/README.md +++ b/README.md @@ -83,7 +83,7 @@ | 2023-05 | DPO | Stanford | [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/pdf/2305.18290.pdf) |Neurips
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F0d1c76d45afa012ded7ab741194baf142117c495%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-07 | LLaMA 2 | Meta | [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/pdf/2307.09288.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F104b0bb1da562d53cbda87aec79ef6a2827d191a%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| |2023-10| Mistral 7B| Mistral |[Mistral 7B](https://arxiv.org/pdf/2310.06825.pdf%5D%5D%3E)|
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fdb633c6b1c286c0386f0078d8a2e6224e03a6227%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-12 | Mamba | CMU&Princeton | [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/ftp/arxiv/papers/2312/2312.00752.pdf) |ICLR
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F432bef8e34014d726c674bc458008ac895297b51%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| +| 2023-12 | Mamba | CMU&Princeton | [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/ftp/arxiv/papers/2312/2312.00752.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F432bef8e34014d726c674bc458008ac895297b51%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| From 8c11a159325150168cbe3a51d1308bf43dd070eb Mon Sep 17 00:00:00 2001 From: Nathan Chen <120630832+Nathancgy@users.noreply.github.com> Date: Fri, 15 Mar 2024 17:04:24 +0800 Subject: [PATCH 051/117] Added Embedding Overview to Tutorials --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 5474372..d853cc7 100644 --- a/README.md +++ b/README.md @@ -437,6 +437,7 @@ The above tables coule be better summarized by this wonderful visualization from - [StatQuest] Sequence-to-Sequence (seq2seq) Encoder-Decoder Neural Networks [Link](https://www.youtube.com/watch?v=L8HKweZIOmg) - [StatQuest] Transformer Neural Networks, ChatGPT's foundation [Link](https://www.youtube.com/watch?v=zxQyTK8quyY) - [StatQuest] Decoder-Only Transformers, ChatGPTs specific Transformer [Link](https://www.youtube.com/watch?v=bQ5BoolX9Ag) +- [康斯坦丁] Understanding Language Processing Through Embedding [Link](https://zhuanlan.zhihu.com/p/643560252) ## Courses From b88848d13f3b1579ca43e5f405825fcb2f33896e Mon Sep 17 00:00:00 2001 From: Nathan Chen <120630832+Nathancgy@users.noreply.github.com> Date: Fri, 15 Mar 2024 18:21:30 +0800 Subject: [PATCH 052/117] Added conference for milestone papers --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index d853cc7..94f4b58 100644 --- a/README.md +++ b/README.md @@ -60,7 +60,7 @@ | 2022-04 | PaLM | Google | [PaLM: Scaling Language Modeling with Pathways](https://arxiv.org/pdf/2204.02311.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F094ff971d6a8b8ff870946c9b3ce5aa173617bfb%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2022-04 | Chinchilla | DeepMind | [An empirical analysis of compute-optimal large language model training](https://www.deepmind.com/publications/an-empirical-analysis-of-compute-optimal-large-language-model-training) | NeurIPS
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fbb0656031cb17adf6bac5fd0fe8d53dd9c291508%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | | 2022-05 | OPT | Meta | [OPT: Open Pre-trained Transformer Language Models](https://arxiv.org/pdf/2205.01068.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F13a0d8bb38f739990c8cd65a44061c6534f17221%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-05 | UL2 | Google | [Unifying Language Learning Paradigms](https://arxiv.org/abs/2205.05131v1) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Ff40aeae3e522ada1f6a9f326841b01ef5c8657b6%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| +| 2022-05 | UL2 | Google | [Unifying Language Learning Paradigms](https://arxiv.org/abs/2205.05131v1) | ICLR
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Ff40aeae3e522ada1f6a9f326841b01ef5c8657b6%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2022-06 | Emergent Abilities | Google | [Emergent Abilities of Large Language Models](https://openreview.net/pdf?id=yzkSU5zdwD) | TMLR
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fdac3a172b504f4e33c029655e9befb3386e5f63a%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2022-06 | BIG-bench | Google | [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://github.com/google/BIG-bench) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F34503c0b6a615124eaf82cb0e4a1dab2866e8980%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2022-06 | METALM | Microsoft | [Language Models are General-Purpose Interfaces](https://arxiv.org/pdf/2206.06336.pdf) | ![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fa8fd9c1625011741f74401ff9bdc1c584e25c86d%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | @@ -71,13 +71,13 @@ | 2022-11 | BLOOM | BigScience | [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://arxiv.org/pdf/2211.05100.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F964bd39b546f0f6625ff3b9ef1083f797807ef2e%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2022-11 | Galactica | Meta | [Galactica: A Large Language Model for Science](https://arxiv.org/pdf/2211.09085.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F7d645a3fd276918374fd9483fd675c28e46506d1%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2022-12 | OPT-IML | Meta | [OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization](https://arxiv.org/pdf/2212.12017) | ![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fe965e93e76a9e6c4e4863d145b5c007b540d575d%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-01 | Flan 2022 Collection | Google | [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://arxiv.org/pdf/2301.13688.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Ff2b0017ddd77fa38760a18145e63553105a1a236%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| +| 2023-01 | Flan 2022 Collection | Google | [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://arxiv.org/pdf/2301.13688.pdf) | ICML
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Ff2b0017ddd77fa38760a18145e63553105a1a236%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-02 | LLaMA|Meta|[LLaMA: Open and Efficient Foundation Language Models](https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/)|![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F57e849d0de13ed5f91d086936296721d4ff75a75%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-02 | Kosmos-1|Microsoft|[Language Is Not All You Need: Aligning Perception with Language Models](https://arxiv.org/abs/2302.14045)|![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Ffbfef4723d8c8467d7bd523e1d0b703cce0e0f9c%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-03 | PaLM-E | Google | [PaLM-E: An Embodied Multimodal Language Model](https://palm-e.github.io)|![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F38fe8f324d2162e63a967a9ac6648974fc4c66f3%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| +| 2023-03 | PaLM-E | Google | [PaLM-E: An Embodied Multimodal Language Model](https://palm-e.github.io)| ICML
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F38fe8f324d2162e63a967a9ac6648974fc4c66f3%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-03 | GPT 4 | OpenAI | [GPT-4 Technical Report](https://openai.com/research/gpt-4)|![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F8ca62fdf4c276ea3052dc96dcfd8ee96ca425a48%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-04 | Pythia | EleutherAI et al. | [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://arxiv.org/abs/2304.01373)|ICML
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fbe55e8ec4213868db08f2c3168ae666001bea4b8%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-05 | Dromedary | CMU et al. | [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://arxiv.org/abs/2305.03047)|![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fe01515c6138bc525f7aec30fc85f2adf028d4156%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| +| 2023-05 | Dromedary | CMU et al. | [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://arxiv.org/abs/2305.03047)| NeurIPS
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fe01515c6138bc525f7aec30fc85f2adf028d4156%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-05 | PaLM 2 | Google | [PaLM 2 Technical Report](https://ai.google/static/documents/palm2techreport.pdf)|![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Feccee350691708972370b7a12c2a78ad3bddd159%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-05 | RWKV | Bo Peng | [RWKV: Reinventing RNNs for the Transformer Era](https://arxiv.org/abs/2305.13048) |EMNLP
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F026b3396a63ed5772329708b7580d633bb86bec9%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-05 | DPO | Stanford | [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/pdf/2305.18290.pdf) |Neurips
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F0d1c76d45afa012ded7ab741194baf142117c495%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| From 259e37e72d1344167b6c6a1b9c2a431073e5967f Mon Sep 17 00:00:00 2001 From: Nathan Chen <120630832+Nathancgy@users.noreply.github.com> Date: Sat, 16 Mar 2024 00:54:00 +0800 Subject: [PATCH 053/117] Added ToT (Tree-of_Thought) to milestone papers section --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 94f4b58..3e16f89 100644 --- a/README.md +++ b/README.md @@ -81,13 +81,12 @@ | 2023-05 | PaLM 2 | Google | [PaLM 2 Technical Report](https://ai.google/static/documents/palm2techreport.pdf)|![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Feccee350691708972370b7a12c2a78ad3bddd159%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-05 | RWKV | Bo Peng | [RWKV: Reinventing RNNs for the Transformer Era](https://arxiv.org/abs/2305.13048) |EMNLP
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F026b3396a63ed5772329708b7580d633bb86bec9%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-05 | DPO | Stanford | [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/pdf/2305.18290.pdf) |Neurips
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F0d1c76d45afa012ded7ab741194baf142117c495%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| +| 2023-05 | ToT | Google&Princeton | [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://arxiv.org/pdf/2305.10601.pdf) | NeurIPS
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F2f3822eb380b5e753a6d579f31dfc3ec4c4a0820%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-07 | LLaMA 2 | Meta | [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/pdf/2307.09288.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F104b0bb1da562d53cbda87aec79ef6a2827d191a%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| |2023-10| Mistral 7B| Mistral |[Mistral 7B](https://arxiv.org/pdf/2310.06825.pdf%5D%5D%3E)|
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fdb633c6b1c286c0386f0078d8a2e6224e03a6227%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-12 | Mamba | CMU&Princeton | [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/ftp/arxiv/papers/2312/2312.00752.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F432bef8e34014d726c674bc458008ac895297b51%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| - - ## Other Papers If you're interested in the field of LLM, you may find the above list of milestone papers helpful to explore its history and state-of-the-art. However, each direction of LLM offers a unique set of insights and contributions, which are essential to understanding the field as a whole. For a detailed list of papers in various subfields, please refer to the following link: From 490b06369cc96c83a29b7eb359f1a96a8eee272c Mon Sep 17 00:00:00 2001 From: Max Brin Date: Mon, 18 Mar 2024 08:53:14 +0200 Subject: [PATCH 054/117] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index d853cc7..8e2069c 100644 --- a/README.md +++ b/README.md @@ -357,6 +357,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Haystack](https://haystack.deepset.ai/) - an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data. - [Sidekick](https://github.com/ai-sidekick/sidekick) - Data integration platform for LLMs. - [LangChain](https://github.com/hwchase17/langchain) - Building applications with LLMs through composability +- [Floom](https://github.com/FloomAI/Floom) AI gateway and marketplace for developers, enables streamlined integration of AI features into products - [Swiss Army Llama](https://github.com/Dicklesworthstone/swiss_army_llama) - Comprehensive set of tools for working with local LLMs for various tasks. - [LiteChain](https://github.com/rogeriochaves/litechain) - Lightweight alternative to LangChain for composing LLMs - [magentic](https://github.com/jackmpcollins/magentic) - Seamlessly integrate LLMs as Python functions From 33f8d84446d6fe6e5974f7f606c8d701932b496c Mon Sep 17 00:00:00 2001 From: Clemo <121163007+clemra@users.noreply.github.com> Date: Mon, 18 Mar 2024 14:28:18 +0100 Subject: [PATCH 055/117] docs: add langfuse docs: add langfuse (oss platform featuring traces, evals, prompt management and metrics to debug and improve your LLM application, 2,3k stars) --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 7167736..03fdbf9 100644 --- a/README.md +++ b/README.md @@ -348,6 +348,7 @@ The above tables coule be better summarized by this wonderful visualization from ## Deploying Tools +- [Langfuse](https://github.com/langfuse/langfuse) - Open Source LLM Engineering Platform 🪢 Tracing, Evaluations, Prompt Management, Evaluations and Playground. - [FastChat](https://github.com/lm-sys/FastChat) - A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs. - [MindSQL](https://github.com/Mindinventory/MindSQL) - A python package for Txt-to-SQL with self hosting functionalities and RESTful APIs compatible with proprietary as well as open source LLM. - [SkyPilot](https://github.com/skypilot-org/skypilot) - Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface. From 5151dd1a9f21be34ef5eb00b12ecf9b22f3f0699 Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Wed, 27 Mar 2024 15:12:42 -0700 Subject: [PATCH 056/117] LLaVA: Large Language and Vision Assistant LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 03fdbf9..f6a965d 100644 --- a/README.md +++ b/README.md @@ -274,6 +274,7 @@ The above tables coule be better summarized by this wonderful visualization from - [UltraLM](https://github.com/thunlp/UltraChat) - Large-scale, Informative, and Diverse Multi-round Chat Models. - [Guanaco](https://github.com/artidoro/qlora) - QLoRA tuned LLaMA - [ChiMed-GPT](https://github.com/synlp/ChiMed-GPT) - A Chinese medical large language model. + - [LLaVa](https://github.com/haotian-liu/LLaVA) - LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding. - [BLOOM](https://huggingface.co/bigscience/bloom) - BigScience Large Open-science Open-access Multilingual Language Model [BLOOM-LoRA](https://github.com/linhduongtuan/BLOOM-LORA) - [BLOOMZ&mT0](https://huggingface.co/bigscience/bloomz) - a family of models capable of following human instructions in dozens of languages zero-shot. - [Phoenix](https://github.com/FreedomIntelligence/LLMZoo) From c825b0fec4bb6fcf968028c25489ec11bf82ae22 Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Wed, 27 Mar 2024 15:14:43 -0700 Subject: [PATCH 057/117] Gorilla: Large Language Model Connected with Massive APIs --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 03fdbf9..ef409c1 100644 --- a/README.md +++ b/README.md @@ -274,6 +274,7 @@ The above tables coule be better summarized by this wonderful visualization from - [UltraLM](https://github.com/thunlp/UltraChat) - Large-scale, Informative, and Diverse Multi-round Chat Models. - [Guanaco](https://github.com/artidoro/qlora) - QLoRA tuned LLaMA - [ChiMed-GPT](https://github.com/synlp/ChiMed-GPT) - A Chinese medical large language model. + - [Gorilla LLM](https://github.com/ShishirPatil/gorilla) - Gorilla: Large Language Model Connected with Massive APIs - [BLOOM](https://huggingface.co/bigscience/bloom) - BigScience Large Open-science Open-access Multilingual Language Model [BLOOM-LoRA](https://github.com/linhduongtuan/BLOOM-LORA) - [BLOOMZ&mT0](https://huggingface.co/bigscience/bloomz) - a family of models capable of following human instructions in dozens of languages zero-shot. - [Phoenix](https://github.com/FreedomIntelligence/LLMZoo) From 8410f3f7eadcc9ce7d5f3dc4a378833ebfae7c5f Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Wed, 27 Mar 2024 15:21:05 -0700 Subject: [PATCH 058/117] RAFT: A new way to teach LLMs to be better at RAG --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 03fdbf9..5cb1847 100644 --- a/README.md +++ b/README.md @@ -274,6 +274,7 @@ The above tables coule be better summarized by this wonderful visualization from - [UltraLM](https://github.com/thunlp/UltraChat) - Large-scale, Informative, and Diverse Multi-round Chat Models. - [Guanaco](https://github.com/artidoro/qlora) - QLoRA tuned LLaMA - [ChiMed-GPT](https://github.com/synlp/ChiMed-GPT) - A Chinese medical large language model. + - [RAFT](https://aka.ms/raft-blog) - RAFT: A new way to teach LLMs to be better at RAG ([paper](https://arxiv.org/abs/2403.10131)). - [BLOOM](https://huggingface.co/bigscience/bloom) - BigScience Large Open-science Open-access Multilingual Language Model [BLOOM-LoRA](https://github.com/linhduongtuan/BLOOM-LORA) - [BLOOMZ&mT0](https://huggingface.co/bigscience/bloomz) - a family of models capable of following human instructions in dozens of languages zero-shot. - [Phoenix](https://github.com/FreedomIntelligence/LLMZoo) From 32310ef163ae68f4ff336381c1d579337c358365 Mon Sep 17 00:00:00 2001 From: Krish Katyal Date: Fri, 5 Apr 2024 01:22:12 +0530 Subject: [PATCH 059/117] added new deployment tool added Tune Studio --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 61e9eda..de340e8 100644 --- a/README.md +++ b/README.md @@ -383,6 +383,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Search with Lepton](https://github.com/leptonai/search_with_lepton) - Build your own conversational search engine using less than 500 lines of code by [LeptonAI](https://github.com/leptonai). - [Robocorp](https://github.com/robocorp/robocorp) - Create, deploy and operate Actions using Python anywhere to enhance your AI agents and assistants. Batteries included with an extensive set of libraries, helpers and logging. - [LMDeploy](https://github.com/InternLM/lmdeploy) - A high-throughput and low-latency inference and serving framework for LLMs and VLs +- [Tune Studio](https://studio.tune.app/) - Playground for devs to finetune & deploy LLMs ## Prompting libraries & tools From 259bec30667821898837a986766f6fe0b3c4ac2b Mon Sep 17 00:00:00 2001 From: Sean <118865326+seanxuu@users.noreply.github.com> Date: Wed, 10 Apr 2024 10:21:15 +0800 Subject: [PATCH 060/117] Update README.md The last update of LLMsPracticalGuide is 8 month ago --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index de340e8..c4f1d4d 100644 --- a/README.md +++ b/README.md @@ -92,7 +92,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [Awesome-LLM-hallucination](https://github.com/LuckyyySTA/Awesome-LLM-hallucination) - LLM hallucination paper list. - [awesome-hallucination-detection](https://github.com/EdinburghNLP/awesome-hallucination-detection) - List of papers on hallucination detection in LLMs. -- [LLMsPracticalGuide](https://github.com/Mooler0410/LLMsPracticalGuide) - A curated (still actively updated) list of practical guide resources of LLMs +- [LLMsPracticalGuide](https://github.com/Mooler0410/LLMsPracticalGuide) - A curated list of practical guide resources of LLMs - [Awesome ChatGPT Prompts](https://github.com/f/awesome-chatgpt-prompts) - A collection of prompt examples to be used with the ChatGPT model. - [awesome-chatgpt-prompts-zh](https://github.com/PlexPt/awesome-chatgpt-prompts-zh) - A Chinese collection of prompt examples to be used with the ChatGPT model. - [Awesome ChatGPT](https://github.com/humanloop/awesome-chatgpt) - Curated list of resources for ChatGPT and GPT-3 from OpenAI. From 43176ccb26f0d8524c9f2b9dc6cdc8bb15bb76a5 Mon Sep 17 00:00:00 2001 From: Nils Herzig <72463901+nilsherzig@users.noreply.github.com> Date: Thu, 18 Apr 2024 11:47:40 +0200 Subject: [PATCH 061/117] readme: added llocalsearch --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index c4f1d4d..c4f58f5 100644 --- a/README.md +++ b/README.md @@ -384,6 +384,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Robocorp](https://github.com/robocorp/robocorp) - Create, deploy and operate Actions using Python anywhere to enhance your AI agents and assistants. Batteries included with an extensive set of libraries, helpers and logging. - [LMDeploy](https://github.com/InternLM/lmdeploy) - A high-throughput and low-latency inference and serving framework for LLMs and VLs - [Tune Studio](https://studio.tune.app/) - Playground for devs to finetune & deploy LLMs +- [LLocalSearch](https://github.com/nilsherzig/LLocalSearch) - Locally running websearch using LLM chains ## Prompting libraries & tools From 8165cec30f2df311f8fb7859651042aa171cd57d Mon Sep 17 00:00:00 2001 From: Guangsi SHI Date: Fri, 19 Apr 2024 14:43:27 +0800 Subject: [PATCH 062/117] Update README.md add OneKE --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index c4f58f5..a23596d 100644 --- a/README.md +++ b/README.md @@ -410,6 +410,7 @@ The above tables coule be better summarized by this wonderful visualization from - [GPTRouter](https://gpt-router.writesonic.com/) - GPTRouter is an open source LLM API Gateway that offers a universal API for 30+ LLMs, vision, and image models, with smart fallbacks based on uptime and latency, automatic retries, and streaming. Stay operational even when OpenAI is down - [QAnything](https://github.com/netease-youdao/QAnything) - A local knowledge base question-answering system designed to support a wide range of file formats and databases. - Core modules: [BCEmbedding](https://github.com/netease-youdao/BCEmbedding) - Bilingual and Crosslingual Embedding for RAG +- [OneKE](https://openspg.yuque.com/ndx6g9/ps5q6b/vfoi61ks3mqwygvy) — A bilingual Chinese-English knowledge extraction model with knowledge graphs and natural language processing technologies. ## Tutorials - [Maarten Grootendorst] A Visual Guide to Mamba and State Space Models [blog](https://maartengrootendorst.substack.com/p/a-visual-guide-to-mamba-and-state?utm_source=multiple-personal-recommendations-email&utm_medium=email&open=false) From b86d77c30243fa59244f6de899e6e7e9aae74224 Mon Sep 17 00:00:00 2001 From: pproulx4 <145417947+pproulx4@users.noreply.github.com> Date: Fri, 19 Apr 2024 13:31:24 -0300 Subject: [PATCH 063/117] Update README.md - Add Command-R/R+ Added Cohere's Command-R and Command-R+ models to list of open weight LLMs at line 332 --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index c4f58f5..a774333 100644 --- a/README.md +++ b/README.md @@ -329,6 +329,7 @@ The above tables coule be better summarized by this wonderful visualization from - [Qwen series](https://huggingface.co/Qwen) - The large language model series proposed by Alibaba Cloud. ｜ 阿里云研发的通义千问大模型系列. 包括 [7B](https://huggingface.co/Qwen/Qwen-7B), [72B](https://huggingface.co/Qwen/Qwen-72B), 及各种量化和Chat版本. [Chat Demo](https://huggingface.co/spaces/Qwen/Qwen-72B-Chat-Demo) - [XVERSE series](https://github.com/xverse-ai) - Multilingual large language model developed by XVERSE Technology Inc | 由深圳元象科技自主研发的支持多语言的大语言模型. 包括[7B](https://github.com/xverse-ai/XVERSE-7B), [13B](https://github.com/xverse-ai/XVERSE-13B), [65B](https://github.com/xverse-ai/XVERSE-65B)等. - [Skywork series](https://github.com/SkyworkAI/Skywork) - A series of large models developed by the Kunlun Group · Skywork team | 昆仑万维集团·天工团队开发的一系列大型模型. +- [Command-R series](https://huggingface.co/CohereForAI) - Two multilingual large language models intended for retrieval augmented generation (RAG) and conversational use, at [35](https://huggingface.co/CohereForAI/c4ai-command-r-v01) and [104](https://huggingface.co/CohereForAI/c4ai-command-r-plus) billion parameters. 128k context support. ## LLM Training Frameworks From 0c32736952f8e601b169f015638f5d1919bca5db Mon Sep 17 00:00:00 2001 From: Richard Gill Date: Fri, 3 May 2024 13:25:59 +0100 Subject: [PATCH 064/117] - [llm-ui](https://github.com/llm-ui-kit/llm-ui) - A React library for building LLM UIs. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index efdfcb3..c8e000e 100644 --- a/README.md +++ b/README.md @@ -412,6 +412,7 @@ The above tables coule be better summarized by this wonderful visualization from - [QAnything](https://github.com/netease-youdao/QAnything) - A local knowledge base question-answering system designed to support a wide range of file formats and databases. - Core modules: [BCEmbedding](https://github.com/netease-youdao/BCEmbedding) - Bilingual and Crosslingual Embedding for RAG - [OneKE](https://openspg.yuque.com/ndx6g9/ps5q6b/vfoi61ks3mqwygvy) — A bilingual Chinese-English knowledge extraction model with knowledge graphs and natural language processing technologies. +- [llm-ui](https://github.com/llm-ui-kit/llm-ui) - A React library for building LLM UIs. ## Tutorials - [Maarten Grootendorst] A Visual Guide to Mamba and State Space Models [blog](https://maartengrootendorst.substack.com/p/a-visual-guide-to-mamba-and-state?utm_source=multiple-personal-recommendations-email&utm_medium=email&open=false) From 4c1cdf8004c07e85696809fbf266a25d7b69a9ab Mon Sep 17 00:00:00 2001 From: Dilip Parasu Date: Sun, 5 May 2024 11:27:47 +0530 Subject: [PATCH 065/117] Add Jamba Jamba - A major milestone showing how good really hybrid mamba models are, showing its scalability --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index efdfcb3..7cd9351 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,7 @@ | 2023-07 | LLaMA 2 | Meta | [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/pdf/2307.09288.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F104b0bb1da562d53cbda87aec79ef6a2827d191a%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| |2023-10| Mistral 7B| Mistral |[Mistral 7B](https://arxiv.org/pdf/2310.06825.pdf%5D%5D%3E)|
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fdb633c6b1c286c0386f0078d8a2e6224e03a6227%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-12 | Mamba | CMU&Princeton | [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/ftp/arxiv/papers/2312/2312.00752.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F432bef8e34014d726c674bc458008ac895297b51%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| +| 2024-03 | Jamba | AI21 Labs | [Jamba: A Hybrid Transformer-Mamba Language Model](https://arxiv.org/pdf/2403.19887) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fcbaf689fd9ea9bc939510019d90535d6249b3367%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | ## Other Papers @@ -330,6 +331,7 @@ The above tables coule be better summarized by this wonderful visualization from - [XVERSE series](https://github.com/xverse-ai) - Multilingual large language model developed by XVERSE Technology Inc | 由深圳元象科技自主研发的支持多语言的大语言模型. 包括[7B](https://github.com/xverse-ai/XVERSE-7B), [13B](https://github.com/xverse-ai/XVERSE-13B), [65B](https://github.com/xverse-ai/XVERSE-65B)等. - [Skywork series](https://github.com/SkyworkAI/Skywork) - A series of large models developed by the Kunlun Group · Skywork team | 昆仑万维集团·天工团队开发的一系列大型模型. - [Command-R series](https://huggingface.co/CohereForAI) - Two multilingual large language models intended for retrieval augmented generation (RAG) and conversational use, at [35](https://huggingface.co/CohereForAI/c4ai-command-r-v01) and [104](https://huggingface.co/CohereForAI/c4ai-command-r-plus) billion parameters. 128k context support. +- - [Jamba](https://huggingface.co/ai21labs/Jamba-v0.1) - A Hybrid Transformer-Mamba MoE model, with 52B params, first production grade mamba based LLM, 256K context support. ## LLM Training Frameworks From 67e4faef126b4d895423a73a22f33ecabe3ddee6 Mon Sep 17 00:00:00 2001 From: Hannibal046 <38466901+Hannibal046@users.noreply.github.com> Date: Tue, 7 May 2024 02:47:32 +0800 Subject: [PATCH 066/117] Update README.md add book: BUILD GPT: HOW AI WORKS --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 26df6c0..8b6bf54 100644 --- a/README.md +++ b/README.md @@ -484,6 +484,7 @@ The above tables coule be better summarized by this wonderful visualization from ## Books - [Generative AI with LangChain: Build large language model (LLM) apps with Python, ChatGPT, and other LLMs](https://amzn.to/3GUlRng) - it comes with a [GitHub repository](https://github.com/benman1/generative_ai_with_langchain) that showcases a lot of the functionality - [Build a Large Language Model (From Scratch)](https://www.manning.com/books/build-a-large-language-model-from-scratch) - A guide to building your own working LLM. +- [BUILD GPT: HOW AI WORKS](https://www.amazon.com/dp/9152799727?ref_=cm_sw_r_cp_ud_dp_W3ZHCD6QWM3DPPC0ARTT_1) - explains how to code a Generative Pre-trained Transformer, or GPT, from scratch. ## Opinions From b129cf453daa5d5d0957b526b70f94b9b0347bdb Mon Sep 17 00:00:00 2001 From: Feiliu <40708416+FeiLiu36@users.noreply.github.com> Date: Tue, 7 May 2024 23:55:05 +0800 Subject: [PATCH 067/117] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 8b6bf54..a0f3b10 100644 --- a/README.md +++ b/README.md @@ -121,6 +121,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [LLMDatahub](https://github.com/Zjh-819/LLMDataHub) - a curated collection of datasets specifically designed for chatbot training, including links, size, language, usage, and a brief description of each dataset - [Awesome-Chinese-LLM](https://github.com/HqWu-HITCS/Awesome-Chinese-LLM) - 整理开源的中文大语言模型，以规模较小、可私有化部署、训练成本较低的模型为主，包括底座模型，垂直领域微调及应用，数据集与教程等。 - [llm-course](https://github.com/mlabonne/llm-course) - Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks. +- [LLM4Opt]([https://github.com/mlabonne/llm-course](https://github.com/FeiLiu36/LLM4Opt)) - Applying Large language models (LLMs) for diverse optimization tasks (Opt) is an emerging research area. This is a collection of references and papers of LLM4Opt. - ## Trending LLM Projects -- [LWM](https://github.com/LargeWorldModel/LWM) - Large World Model (LWM) is a general-purpose large-context multimodal autoregressive model. -- [Sora](https://openai.com/sora) - Sora is an AI model that can create realistic and imaginative scenes from text instructions. -- [Gemma](https://blog.google/technology/developers/gemma-open-models/) - Gemma is built for responsible AI development from the same research and technology used to create Gemini models. -- [minbpe](https://github.com/karpathy/minbpe) - Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. +- [GPT-4o](https://openai.com/index/hello-gpt-4o/) - OpenAI's new flagship model that can reason across audio, vision, and text in real time. +- [MiniCPM-V](https://github.com/OpenBMB/MiniCPM-V) - A GPT-4V Level Multimodal LLM on Your Phone. +- [llama3-from-scratch](https://github.com/naklecha/llama3-from-scratch) - llama3 implementation one matrix multiplication at a time. +- [ChatGPT Desktop Application](https://github.com/lencx/ChatGPT) - ChatGPT Desktop Application (Mac, Windows and Linux). +- [llm.c](https://github.com/karpathy/llm.c) - LLM training in simple, raw C/CUDA. ## Table of Content - [Awesome-LLM ](#awesome-llm-) - [Milestone Papers](#milestone-papers) - [Other Papers](#other-papers) + - [LLM Leaderboard](#llm-leaderboard) - [Open LLM](#open-llm) - - [LLM Training Frameworks](#llm-training-frameworks) - - [LLM Evaluation Frameworks](#llm-evaluation-frameworks) - - [Tools for deploying LLM](#deploying-tools) - - [Tutorials about LLM](#tutorials) - - [Courses about LLM](#courses) - - [Opinions about LLM](#opinions) - - [Other Useful Resources](#other-useful-resources) - - [Contributing](#contributing) + - [LLM Data](#llm-data) + - [LLM Evaluation](#llm-evaluation) + - [LLM Training Framework](#llm-training-frameworks) + - [LLM Deployment](#llm-deployment) + - [LLM Applications](#llm-applications) + - [LLM Books](#llm-books) + - [Great thoughts about LLM](#great-thoughts-about-llm) + - [Miscellaneous](#miscellaneous) ## Milestone Papers @@ -83,7 +81,7 @@ | 2023-05 | DPO | Stanford | [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/pdf/2305.18290.pdf) |Neurips
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F0d1c76d45afa012ded7ab741194baf142117c495%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-05 | ToT | Google&Princeton | [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://arxiv.org/pdf/2305.10601.pdf) | NeurIPS
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F2f3822eb380b5e753a6d579f31dfc3ec4c4a0820%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-07 | LLaMA 2 | Meta | [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/pdf/2307.09288.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F104b0bb1da562d53cbda87aec79ef6a2827d191a%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -|2023-10| Mistral 7B| Mistral |[Mistral 7B](https://arxiv.org/pdf/2310.06825.pdf%5D%5D%3E)|
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fdb633c6b1c286c0386f0078d8a2e6224e03a6227%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| +|2023-10| Mistral 7B| Mistral |[Mistral 7B](https://arxiv.org/pdf/2310.06825.pdf)|
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fdb633c6b1c286c0386f0078d8a2e6224e03a6227%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-12 | Mamba | CMU&Princeton | [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/ftp/arxiv/papers/2312/2312.00752.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F432bef8e34014d726c674bc458008ac895297b51%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2024-03 | Jamba | AI21 Labs | [Jamba: A Hybrid Transformer-Mamba Language Model](https://arxiv.org/pdf/2403.19887) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fcbaf689fd9ea9bc939510019d90535d6249b3367%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | @@ -120,225 +118,105 @@ If you're interested in the field of LLM, you may find the above list of milesto - [Awesome-LLM-3D](https://github.com/ActiveVisionLab/Awesome-LLM-3D) - A curated list of Multi-modal Large Language Model in 3D world, including 3D understanding, reasoning, generation, and embodied agents. - [LLMDatahub](https://github.com/Zjh-819/LLMDataHub) - a curated collection of datasets specifically designed for chatbot training, including links, size, language, usage, and a brief description of each dataset - [Awesome-Chinese-LLM](https://github.com/HqWu-HITCS/Awesome-Chinese-LLM) - 整理开源的中文大语言模型，以规模较小、可私有化部署、训练成本较低的模型为主，包括底座模型，垂直领域微调及应用，数据集与教程等。 -- [llm-course](https://github.com/mlabonne/llm-course) - Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks. -- [LLM4Opt](https://github.com/FeiLiu36/LLM4Opt) - Applying Large language models (LLMs) for diverse optimization tasks (Opt) is an emerging research area. This is a collection of references and papers of LLM4Opt. - - +## LLM Leaderboard +- [Chatbot Arena Leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard) - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner. +- [AlpacaEval Leaderboard](https://tatsu-lab.github.io/alpaca_eval/) - An Automatic Evaluator for Instruction-following Language Models + using Nous benchmark suite. +- [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) - aims to track, rank and evaluate LLMs and chatbots as they are released. +- [OpenCompass 2.0 LLM Leaderboard](https://rank.opencompass.org.cn/leaderboard-llm-v2) - OpenCompass is an LLM evaluation platform, supporting a wide range of models (InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets. ## Open LLM -
- -
- -There are three important steps for a ChatGPT-like LLM: -- **Pre-training** -- **Instruction Tuning** -- **Alignment** - - - > You may also find these leaderboards helpful: - > - [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) - aims to track, rank and evaluate LLMs and chatbots as they are released. - > - [Chatbot Arena Leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard) - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner. - > - [AlpacaEval Leaderboard](https://tatsu-lab.github.io/alpaca_eval/) - An Automatic Evaluator for Instruction-following Language Models - > - [Open Ko-LLM Leaderboard](https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard) - The Open Ko-LLM Leaderboard objectively evaluates the performance of Korean Large Language Model (LLM). - > - [Yet Another LLM Leaderboard](https://huggingface.co/spaces/mlabonne/Yet_Another_LLM_Leaderboard) - Leaderboard made with LLM AutoEval using Nous benchmark suite. - > - [OpenCompass 2.0 LLM Leaderboard](https://rank.opencompass.org.cn/leaderboard-llm-v2) - OpenCompass is an LLM evaluation platform, supporting a wide range of models (InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets. - - - - -- [Gemma](https://blog.google/technology/developers/gemma-open-models/) - Gemma is built for responsible AI development from the same research and technology used to create Gemini models. -- [Mistral](https://mistral.ai/) - Mistral-7B-v0.1 is a small, yet powerful model adaptable to many use-cases including code and 8k sequence length. Apache 2.0 licence. -- [Mixtral 8x7B](https://mistral.ai/news/mixtral-of-experts/) - a high-quality sparse mixture of experts model (SMoE) with open weights. -- [LLaMA](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) & [LLaMA-2](https://ai.meta.com/llama/) - A foundational large language model. [LLaMA.cpp](https://github.com/ggerganov/llama.cpp) [Lit-LLaMA](https://github.com/Lightning-AI/lit-llama) - - [Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html) - A model fine-tuned from the LLaMA 7B model on 52K instruction-following demonstrations. [Alpaca.cpp](https://github.com/antimatter15/alpaca.cpp) [Alpaca-LoRA](https://github.com/tloen/alpaca-lora) - - [Flan-Alpaca](https://github.com/declare-lab/flan-alpaca) - Instruction Tuning from Humans and Machines. - - [Baize](https://github.com/project-baize/baize-chatbot) - Baize is an open-source chat model trained with [LoRA](https://github.com/microsoft/LoRA). It uses 100k dialogs generated by letting ChatGPT chat with itself. - - [Cabrita](https://github.com/22-hours/cabrita) - A portuguese finetuned instruction LLaMA. - - [Vicuna](https://lmsys.org/blog/2023-03-30-vicuna/) - An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality. - - [Llama-X](https://github.com/AetherCortex/Llama-X) - Open Academic Research on Improving LLaMA to SOTA LLM. - - [Chinese-Vicuna](https://github.com/Facico/Chinese-Vicuna) - A Chinese Instruction-following LLaMA-based Model. - - [GPTQ-for-LLaMA](https://github.com/qwopqwop200/GPTQ-for-LLaMa) - 4 bits quantization of [LLaMA](https://arxiv.org/abs/2302.13971) using [GPTQ](https://arxiv.org/abs/2210.17323). - - [GPT4All](https://github.com/nomic-ai/gpt4all) - Demo, data, and code to train open-source assistant-style large language model based on GPT-J and LLaMa. - - [Koala](https://bair.berkeley.edu/blog/2023/04/03/koala/) - A Dialogue Model for Academic Research - - [BELLE](https://github.com/LianjiaTech/BELLE) - Be Everyone's Large Language model Engine - - [StackLLaMA](https://huggingface.co/blog/stackllama) - A hands-on guide to train LLaMA with RLHF. - - [RedPajama](https://github.com/togethercomputer/RedPajama-Data) - An Open Source Recipe to Reproduce LLaMA training dataset. - - [Chimera](https://github.com/FreedomIntelligence/LLMZoo) - Latin Phoenix. - - [WizardLM|WizardCoder](https://github.com/nlpxucan/WizardLM) - Family of instruction-following LLMs powered by Evol-Instruct: WizardLM, WizardCoder. - - [CaMA](https://github.com/zjunlp/CaMA) - a Chinese-English Bilingual LLaMA Model. - - [Orca](https://aka.ms/orca-lm) - Microsoft's finetuned LLaMA model that reportedly matches GPT3.5, finetuned against 5M of data, ChatGPT, and GPT4 - - [BayLing](https://github.com/ictnlp/BayLing) - an English/Chinese LLM equipped with advanced language alignment, showing superior capability in English/Chinese generation, instruction following and multi-turn interaction. - - [UltraLM](https://github.com/thunlp/UltraChat) - Large-scale, Informative, and Diverse Multi-round Chat Models. - - [Guanaco](https://github.com/artidoro/qlora) - QLoRA tuned LLaMA - - [ChiMed-GPT](https://github.com/synlp/ChiMed-GPT) - A Chinese medical large language model. - - [RAFT](https://aka.ms/raft-blog) - RAFT: A new way to teach LLMs to be better at RAG ([paper](https://arxiv.org/abs/2403.10131)). - - [Gorilla LLM](https://github.com/ShishirPatil/gorilla) - Gorilla: Large Language Model Connected with Massive APIs - - [LLaVa](https://github.com/haotian-liu/LLaVA) - LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding. -- [BLOOM](https://huggingface.co/bigscience/bloom) - BigScience Large Open-science Open-access Multilingual Language Model [BLOOM-LoRA](https://github.com/linhduongtuan/BLOOM-LORA) - - [BLOOMZ&mT0](https://huggingface.co/bigscience/bloomz) - a family of models capable of following human instructions in dozens of languages zero-shot. - - [Phoenix](https://github.com/FreedomIntelligence/LLMZoo) -- [Deepseek](https://github.com/deepseek-ai/) - - [Coder](https://github.com/deepseek-ai/DeepSeek-Coder) - Let the Code Write Itself. - - [LLM](https://github.com/deepseek-ai/DeepSeek-LLM) - Let there be answers. - - 知名私募巨头幻方量化旗下的人工智能公司深度求索（DeepSeek）自主研发的大语言模型开发的智能助手。包括 [7B-base](https://modelscope.cn/models/deepseek-ai/deepseek-llm-7b-base/summary), [67B-base](https://modelscope.cn/models/deepseek-ai/deepseek-llm-67b-base/summary), -- [Yi](https://github.com/01-ai/Yi) - A series of large language models trained from scratch by developers @01-ai. -- [T5](https://arxiv.org/abs/1910.10683) - Text-to-Text Transfer Transformer - - [T0](https://arxiv.org/abs/2110.08207) - Multitask Prompted Training Enables Zero-Shot Task Generalization -- [OPT](https://arxiv.org/abs/2205.01068) - Open Pre-trained Transformer Language Models. -- [UL2](https://arxiv.org/abs/2205.05131v1) - a unified framework for pretraining models that are universally effective across datasets and setups. -- [GLM](https://github.com/THUDM/GLM)- GLM is a General Language Model pretrained with an autoregressive blank-filling objective and can be finetuned on various natural language understanding and generation tasks. - - [ChatGLM-6B](https://github.com/THUDM/ChatGLM-6B) - ChatGLM-6B 是一个开源的、支持中英双语的对话语言模型，基于 [General Language Model (GLM)](https://github.com/THUDM/GLM) 架构，具有 62 亿参数. - - [ChatGLM2-6B](https://github.com/THUDM/ChatGLM2-6B) - An Open Bilingual Chat LLM | 开源双语对话语言模型 - - [ChatGLM3-6B](https://github.com/THUDM/ChatGLM3) - An Open Bilingual Chat LLMs | 开源双语对话语言模型 ; Including [ChatGLM3-6B-32k](https://huggingface.co/THUDM/chatglm3-6b-32k), [ChatGLM3-6B-128k](https://huggingface.co/THUDM/chatglm3-6b-128k). -- [RWKV](https://github.com/BlinkDL/RWKV-LM) - Parallelizable RNN with Transformer-level LLM Performance. - - [ChatRWKV](https://github.com/BlinkDL/ChatRWKV) - ChatRWKV is like ChatGPT but powered by my RWKV (100% RNN) language model. - - [Trending Demo](https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-2) - RWKV-5 trained on 100+ world languages (70% English, 15% multilang, 15% code). -- [StableLM](https://stability.ai/blog/stability-ai-launches-the-first-of-its-stablelm-suite-of-language-models) - Stability AI Language Models. -- [YaLM](https://medium.com/yandex/yandex-publishes-yalm-100b-its-the-largest-gpt-like-neural-network-in-open-source-d1df53d0e9a6) - a GPT-like neural network for generating and processing text. It can be used freely by developers and researchers from all over the world. -- [GPT-Neo](https://github.com/EleutherAI/gpt-neo) - An implementation of model & data parallel [GPT3](https://arxiv.org/abs/2005.14165)-like models using the [mesh-tensorflow](https://github.com/tensorflow/mesh) library. -- [GPT-J](https://github.com/kingoflolz/mesh-transformer-jax/#gpt-j-6b) - A 6 billion parameter, autoregressive text generation model trained on [The Pile](https://pile.eleuther.ai/). - - [Dolly](https://www.databricks.com/blog/2023/03/24/hello-dolly-democratizing-magic-chatgpt-open-models.html) - a cheap-to-build LLM that exhibits a surprising degree of the instruction following capabilities exhibited by ChatGPT. -- [Pythia](https://github.com/EleutherAI/pythia) - Interpreting Autoregressive Transformers Across Time and Scale - - [Dolly 2.0](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm) - the first open source, instruction-following LLM, fine-tuned on a human-generated instruction dataset licensed for research and commercial use. -- [OpenFlamingo](https://github.com/mlfoundations/open_flamingo) - an open-source reproduction of DeepMind's Flamingo model. -- [Cerebras-GPT](https://www.cerebras.net/blog/cerebras-gpt-a-family-of-open-compute-efficient-large-language-models/) - A Family of Open, Compute-efficient, Large Language Models. -- [GALACTICA](https://github.com/paperswithcode/galai/blob/main/docs/model_card.md) - The GALACTICA models are trained on a large-scale scientific corpus. - - [GALPACA](https://huggingface.co/GeorgiaTechResearchInstitute/galpaca-30b) - GALACTICA 30B fine-tuned on the Alpaca dataset. -- [Palmyra](https://huggingface.co/Writer/palmyra-base) - Palmyra Base was primarily pre-trained with English text. -- [Camel](https://huggingface.co/Writer/camel-5b-hf) - a state-of-the-art instruction-following large language model designed to deliver exceptional performance and versatility. -- [h2oGPT](https://github.com/h2oai/h2ogpt) -- [PanGu-α](https://openi.org.cn/pangu/) - PanGu-α is a 200B parameter autoregressive pretrained Chinese language model develped by Huawei Noah's Ark Lab, MindSpore Team and Peng Cheng Laboratory. -- [MOSS](https://github.com/OpenLMLab/MOSS) - MOSS是一个支持中英双语和多种插件的开源对话语言模型. -- [Open-Assistant](https://github.com/LAION-AI/Open-Assistant) - a project meant to give everyone access to a great chat based large language model. - - [HuggingChat](https://huggingface.co/chat/) - Powered by Open Assistant's latest model – the best open source chat model right now and @huggingface Inference API. -- [StarCoder](https://huggingface.co/blog/starcoder) - Hugging Face LLM for Code -- [MPT-7B](https://www.mosaicml.com/blog/mpt-7b) - Open LLM for commercial use by MosaicML -- [Falcon](https://falconllm.tii.ae) - Falcon LLM is a foundational large language model (LLM) with 40 billion parameters trained on one trillion tokens. TII has now released Falcon LLM – a 40B model. -- [XGen](https://github.com/salesforce/xgen) - Salesforce open-source LLMs with 8k sequence length. -- [Baichuan](https://github.com/baichuan-inc) - A series of large language models developed by Baichuan Intelligent Technology. -- [Aquila](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/Aquila) - 悟道·天鹰语言大模型是首个具备中英双语知识、支持商用许可协议、国内数据合规需求的开源语言大模型。 -- [phi-1](https://arxiv.org/abs/2306.11644) - a new large language model for code, with significantly smaller size than competing models. -- [phi-1.5](https://arxiv.org/abs/2309.05463) - a 1.3 billion parameter model trained on a dataset of 30 billion tokens, which achieves common sense reasoning benchmark results comparable to models ten times its size that were trained on datasets more than ten times larger. -- [phi-2](https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/) - a 2.7 billion-parameter language model that demonstrates outstanding reasoning and language understanding capabilities, showcasing state-of-the-art performance among base language models with less than 13 billion parameters. -- [InternLM / 书生·浦语](https://github.com/InternLM/InternLM) - Official release of InternLM2 7B and 20B base and chat models. 200K context support. [Homepage](https://internlm.intern-ai.org.cn/) | [ModelScope](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm-7b/summary) -- [BlueLM-7B](https://github.com/vivo-ai-lab/BlueLM) - BlueLM(蓝心大模型): Open large language models developed by vivo AI Lab. [Homepage](https://developers.vivo.com/product/ai/bluelm) | [ModelScope](https://modelscope.cn/models/vivo-ai/BlueLM-7B-Base/summary) -[MoE-16B-base](https://modelscope.cn/models/deepseek-ai/deepseek-moe-16b-base), 等. | [Chat with DeepSeek (Beta)](https://chat.deepseek.com/sign_in) -- [Qwen series](https://huggingface.co/Qwen) - The large language model series proposed by Alibaba Cloud. ｜ 阿里云研发的通义千问大模型系列. 包括 [7B](https://huggingface.co/Qwen/Qwen-7B), [72B](https://huggingface.co/Qwen/Qwen-72B), 及各种量化和Chat版本. [Chat Demo](https://huggingface.co/spaces/Qwen/Qwen-72B-Chat-Demo) -- [XVERSE series](https://github.com/xverse-ai) - Multilingual large language model developed by XVERSE Technology Inc | 由深圳元象科技自主研发的支持多语言的大语言模型. 包括[7B](https://github.com/xverse-ai/XVERSE-7B), [13B](https://github.com/xverse-ai/XVERSE-13B), [65B](https://github.com/xverse-ai/XVERSE-65B)等. -- [Skywork series](https://github.com/SkyworkAI/Skywork) - A series of large models developed by the Kunlun Group · Skywork team | 昆仑万维集团·天工团队开发的一系列大型模型. -- [Command-R series](https://huggingface.co/CohereForAI) - Two multilingual large language models intended for retrieval augmented generation (RAG) and conversational use, at [35](https://huggingface.co/CohereForAI/c4ai-command-r-v01) and [104](https://huggingface.co/CohereForAI/c4ai-command-r-plus) billion parameters. 128k context support. -- - [Jamba](https://huggingface.co/ai21labs/Jamba-v0.1) - A Hybrid Transformer-Mamba MoE model, with 52B params, first production grade mamba based LLM, 256K context support. +- Meta + - [Llama 3-8|70B](https://llama.meta.com/llama3/) + - [Llama 2-7|13|70B](https://llama.meta.com/llama2/) + - [Llama 1-7|13|33|65B](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) + - [OPT-1.3|6.7|13|30|66B](https://arxiv.org/abs/2205.01068) +- Mistral AI + - [Mistral-7B](https://mistral.ai/news/announcing-mistral-7b/) + - [Mixtral-8x7B](https://mistral.ai/news/mixtral-of-experts/) + - [Mixtral-8x22B](https://mistral.ai/news/mixtral-8x22b/) +- Google + - [Gemma-2|7B](https://blog.google/technology/developers/gemma-open-models/) + - [RecurrentGemma-2B](https://github.com/google-deepmind/recurrentgemma) + - [T5](https://arxiv.org/abs/1910.10683) +- Apple + - [OpenELM-1.1|3B](https://huggingface.co/apple/OpenELM) +- Microsoft + - [Phi1-1.3B](https://huggingface.co/microsoft/phi-1) + - [Phi2-2.7B](https://huggingface.co/microsoft/phi-2) + - [Phi3-3.8|7|14B](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) +- AllenAI + - [OLMo-7B](https://huggingface.co/collections/allenai/olmo-suite-65aeaae8fe5b6b2122b46778) +- xAI + - [Grok-1-314B-MoE](https://x.ai/blog/grok-os) +- Cohere + - [Command R-35B](https://huggingface.co/CohereForAI/c4ai-command-r-v01) +- DeepSeek + - [DeepSeek-Math-7B](https://huggingface.co/collections/deepseek-ai/deepseek-math-65f2962739da11599e441681) + - [DeepSeek-Coder-1.3|6.7|7|33B](https://huggingface.co/collections/deepseek-ai/deepseek-coder-65f295d7d8a0a29fe39b4ec4) + - [DeepSeek-VL-1.3B|7B](https://huggingface.co/collections/deepseek-ai/deepseek-vl-65f295948133d9cf92b706d3) + - [DeepSeek-MoE-16B](https://huggingface.co/collections/deepseek-ai/deepseek-moe-65f29679f5cf26fe063686bf) + - [DeepSeek-v2-236B-MoE](https://arxiv.org/abs/2405.04434) +- Alibaba + - [Qwen-1.8|7|14|72B](https://huggingface.co/collections/Qwen/qwen-65c0e50c3f1ab89cb8704144) + - [Qwen1.5-1.8|4|7|14|32|72|110B](https://huggingface.co/collections/Qwen/qwen15-65c0a2f577b1ecb76d786524) + - [CodeQwen-7B](https://huggingface.co/Qwen/CodeQwen1.5-7B) + - [Qwen-VL-7B](https://huggingface.co/Qwen/Qwen-VL) +- 01-ai + - [Yi-34B](https://huggingface.co/collections/01-ai/yi-2023-11-663f3f19119ff712e176720f) + - [Yi1.5-6|9|34B](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) + - [Yi-VL-6B|34B](https://huggingface.co/collections/01-ai/yi-vl-663f557228538eae745769f3) +- Baichuan + - [Baichuan-7|13B](https://huggingface.co/baichuan-inc) + - [Baichuan2-7|13B](https://huggingface.co/baichuan-inc) +- BLOOM + - [BLOOMZ&mT0](https://huggingface.co/bigscience/bloomz) +- Zhipu AI + - [GLM-2|6|10|13|70B](https://huggingface.co/THUDM) + - [CogVLM2-19B](https://huggingface.co/collections/THUDM/cogvlm2-6645f36a29948b67dc4eef75) +- OpenBMB + - [MiniCPM-2B](https://huggingface.co/collections/openbmb/minicpm-2b-65d48bf958302b9fd25b698f) + - [OmniLLM-12B](https://huggingface.co/openbmb/OmniLMM-12B) + - [VisCPM-10B](https://huggingface.co/openbmb/VisCPM-Chat) + - [CPM-Bee-1|2|5|10B](https://huggingface.co/collections/openbmb/cpm-bee-65d491cc84fc93350d789361) +- RWKV Foundation + - [RWKV-v4|5|6](https://huggingface.co/RWKV) +- ElutherAI + - [Pythia-1|1.4|2.8|6.9|12B](https://github.com/EleutherAI/pythia) +- Stability AI + - [StableLM-3B](https://huggingface.co/collections/stabilityai/stable-lm-650852cfd55dd4e15cdcb30a) + - [StableLM-v2-1.6|12B](https://huggingface.co/collections/stabilityai/stable-lm-650852cfd55dd4e15cdcb30a) + - [StableCode-3B](https://huggingface.co/collections/stabilityai/stable-code-64f9dfb4ebc8a1be0a3f7650) +- BigCode + - [StarCoder-1|3|7B](https://huggingface.co/collections/bigcode/%E2%AD%90-starcoder-64f9bd5740eb5daaeb81dbec) + - [StarCoder2-3|7|15B](https://huggingface.co/collections/bigcode/starcoder2-65de6da6e87db3383572be1a) +- DataBricks + - [MPT-7B](https://www.databricks.com/blog/mpt-7b) +- Shanghai AI Laboratory + - [InternLM2-1.8|7|20B](https://huggingface.co/collections/internlm/internlm2-65b0ce04970888799707893c) + - [InternLM-Math-7B|20B](https://huggingface.co/collections/internlm/internlm2-math-65b0ce88bf7d3327d0a5ad9f) + - [InternLM-XComposer2-1.8|7B](https://huggingface.co/collections/internlm/internlm-xcomposer2-65b3706bf5d76208998e7477) + - [InternVL-2|6|14|26](https://huggingface.co/collections/OpenGVLab/internvl-65b92d6be81c86166ca0dde4) + +## LLM Data +- [LLMDataHub](https://github.com/Zjh-819/LLMDataHub) + +## LLM Evaluation: +- [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) - A framework for few-shot evaluation of language models. +- [lighteval](https://github.com/huggingface/lighteval) - a lightweight LLM evaluation suite that Hugging Face has been using internally. +- [OLMO-eval](https://github.com/allenai/OLMo-Eval) - a repository for evaluating open language models. +- [instruct-eval](https://github.com/declare-lab/instruct-eval) - This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks. ## LLM Training Frameworks - [DeepSpeed](https://github.com/microsoft/DeepSpeed) - DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - [Megatron-DeepSpeed](https://github.com/microsoft/Megatron-DeepSpeed) - DeepSpeed version of NVIDIA's Megatron-LM that adds additional support for several features such as MoE model training, Curriculum Learning, 3D Parallelism, and others. -- [FairScale](https://fairscale.readthedocs.io/en/latest/what_is_fairscale.html) - FairScale is a PyTorch extension library for high performance and large scale training. +- [torchtune](https://github.com/pytorch/torchtune) - A Native-PyTorch Library for LLM Fine-tuning. +- [torchtitan](https://github.com/pytorch/torchtitan) - A native PyTorch Library for large model training. - [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) - Ongoing research training transformer models at scale. - [Colossal-AI](https://github.com/hpcaitech/ColossalAI) - Making large AI models cheaper, faster, and more accessible. - [BMTrain](https://github.com/OpenBMB/BMTrain) - Efficient Training for Big Models. @@ -347,22 +225,23 @@ The above tables coule be better summarized by this wonderful visualization from - [Alpa](https://alpa.ai/index.html) - Alpa is a system for training and serving large-scale neural networks. - [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) - An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library. -## LLM Evaluation Frameworks: -- [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) - A framework for few-shot evaluation of language models. -- [lighteval](https://github.com/huggingface/lighteval) - a lightweight LLM evaluation suite that Hugging Face has been using internally. -- [OLMO-eval](https://github.com/allenai/OLMo-Eval) - a repository for evaluating open language models. -- [instruct-eval](https://github.com/declare-lab/instruct-eval) - This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks. -## Deploying Tools +## LLM Deployment +> Reference: [llm-inference-solutions](https://github.com/mani-kantap/llm-inference-solutions) +- [vLLM](https://github.com/vllm-project/vllm) - A high-throughput and memory-efficient inference and serving engine for LLMs. +- [TGI](https://huggingface.co/docs/text-generation-inference/en/index) - a toolkit for deploying and serving Large Language Models (LLMs). +- [exllama](https://github.com/turboderp/exllama) - A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights. +- [llama.cpp](https://github.com/ggerganov/llama.cpp) - LLM inference in C/C++. +- [ollama](https://github.com/ollama/ollama) - Get up and running with Llama 3, Mistral, Gemma, and other large language models. - [Langfuse](https://github.com/langfuse/langfuse) - Open Source LLM Engineering Platform 🪢 Tracing, Evaluations, Prompt Management, Evaluations and Playground. - [FastChat](https://github.com/lm-sys/FastChat) - A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs. - [MindSQL](https://github.com/Mindinventory/MindSQL) - A python package for Txt-to-SQL with self hosting functionalities and RESTful APIs compatible with proprietary as well as open source LLM. - [SkyPilot](https://github.com/skypilot-org/skypilot) - Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface. -- [vLLM](https://github.com/vllm-project/vllm) - A high-throughput and memory-efficient inference and serving engine for LLMs -- [Text Generation Inference](https://github.com/huggingface/text-generation-inference) - A Rust, Python and gRPC server for text generation inference. Used in production at [HuggingFace](https://huggingface.co/) to power LLMs api-inference widgets, HFOIL Licence. - [Haystack](https://haystack.deepset.ai/) - an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data. -- [Sidekick](https://github.com/ai-sidekick/sidekick) - Data integration platform for LLMs. +- [Sidekick](https://github.com/ai-sidekick/sidekick) - Data integration platform for LLMs. +- [QA-Pilot](https://github.com/reid41/QA-Pilot) - An interactive chat project that leverages Ollama/OpenAI/MistralAI LLMs for rapid understanding and navigation of GitHub code repository or compressed file resources. +- [Shell-Pilot](https://github.com/reid41/shell-pilot) - Interact with LLM using Ollama models(or openAI, mistralAI)via pure shell scripts on your Linux(or MacOS) system, enhancing intelligent system management without any dependencies. - [LangChain](https://github.com/hwchase17/langchain) - Building applications with LLMs through composability - [Floom](https://github.com/FloomAI/Floom) AI gateway and marketplace for developers, enables streamlined integration of AI features into products - [Swiss Army Llama](https://github.com/Dicklesworthstone/swiss_army_llama) - Comprehensive set of tools for working with local LLMs for various tasks. @@ -391,7 +270,7 @@ The above tables coule be better summarized by this wonderful visualization from - [LLocalSearch](https://github.com/nilsherzig/LLocalSearch) - Locally running websearch using LLM chains - [AI Gateway](https://github.com/Portkey-AI/gateway) — Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency. -## Prompting libraries & tools +## LLM Applications - [YiVal](https://github.com/YiVal/YiVal) — Evaluate and Evolve: YiVal is an open-source GenAI-Ops tool for tuning and evaluating prompts, configurations, and model parameters using customizable datasets, evaluation methods, and improvement strategies. - [Guidance](https://github.com/microsoft/guidance) — A handy looking Python library from Microsoft that uses Handlebars templating to interleave generation, prompting, and logical control. @@ -414,107 +293,49 @@ The above tables coule be better summarized by this wonderful visualization from - [Flappy](https://github.com/pleisto/flappy) — Production-Ready LLM Agent SDK for Every Developer. - [GPTRouter](https://gpt-router.writesonic.com/) - GPTRouter is an open source LLM API Gateway that offers a universal API for 30+ LLMs, vision, and image models, with smart fallbacks based on uptime and latency, automatic retries, and streaming. Stay operational even when OpenAI is down - [QAnything](https://github.com/netease-youdao/QAnything) - A local knowledge base question-answering system designed to support a wide range of file formats and databases. - - Core modules: [BCEmbedding](https://github.com/netease-youdao/BCEmbedding) - Bilingual and Crosslingual Embedding for RAG - [OneKE](https://openspg.yuque.com/ndx6g9/ps5q6b/vfoi61ks3mqwygvy) — A bilingual Chinese-English knowledge extraction model with knowledge graphs and natural language processing technologies. - [llm-ui](https://github.com/llm-ui-kit/llm-ui) - A React library for building LLM UIs. - [Wordware](https://www.wordware.ai) - A web-hosted IDE where non-technical domain experts work with AI Engineers to build task-specific AI agents. We approach prompting as a new programming language rather than low/no-code blocks. +- [Wallaroo.AI](https://github.com/WallarooLabs) - Deploy, manage, optimize any model at scale across any environment from cloud to edge. Let's you go from python notebook to inferencing in minutes. -## Tutorials -- [Maarten Grootendorst] A Visual Guide to Mamba and State Space Models [blog](https://maartengrootendorst.substack.com/p/a-visual-guide-to-mamba-and-state?utm_source=multiple-personal-recommendations-email&utm_medium=email&open=false) -- [Jack Cook] [Mamba: The Easy Way](https://jackcook.com/2024/02/23/mamba.html) -- [Andrej Karpathy] minbpe [video](https://www.youtube.com/watch?v=zduSFxRajkE&t=1157s) -- [Andrej Karpathy] State of GPT [video](https://build.microsoft.com/en-US/sessions/db3f4859-cd30-4445-a0cd-553c3304f8e2) -- [Hyung Won Chung] Instruction finetuning and RLHF lecture [Youtube](https://www.youtube.com/watch?v=zjrM-MW-0y0) -- [Jason Wei] Scaling, emergence, and reasoning in large language models [Slides](https://docs.google.com/presentation/d/1EUV7W7X_w0BDrscDhPg7lMGzJCkeaPkGCJ3bN8dluXc/edit?pli=1&resourcekey=0-7Nz5A7y8JozyVrnDtcEKJA#slide=id.g16197112905_0_0) -- [Susan Zhang] Open Pretrained Transformers [Youtube](https://www.youtube.com/watch?v=p9IxoSkvZ-M&t=4s) -- [Ameet Deshpande] How Does ChatGPT Work? [Slides](https://docs.google.com/presentation/d/1TTyePrw-p_xxUbi3rbmBI3QQpSsTI1btaQuAUvvNc8w/edit#slide=id.g206fa25c94c_0_24) -- [Yao Fu] 预训练，指令微调，对齐，专业化：论大语言模型能力的来源 [Bilibili](https://www.bilibili.com/video/BV1Qs4y1h7pn/?spm_id_from=333.337.search-card.all.click&vd_source=1e55c5426b48b37e901ff0f78992e33f) -- [Hung-yi Lee] ChatGPT 原理剖析 [Youtube](https://www.youtube.com/watch?v=yiY4nPOzJEg&list=RDCMUC2ggjtuuWvxrHHHiaDH1dlQ&index=2) -- [Jay Mody] GPT in 60 Lines of NumPy [Link](https://jaykmody.com/blog/gpt-from-scratch/) -- [ICML 2022] Welcome to the "Big Model" Era: Techniques and Systems to Train and Serve Bigger Models [Link](https://icml.cc/virtual/2022/tutorial/18440) -- [NeurIPS 2022] Foundational Robustness of Foundation Models [Link](https://nips.cc/virtual/2022/tutorial/55796) -- [Andrej Karpathy] Let's build GPT: from scratch, in code, spelled out. [Video](https://www.youtube.com/watch?v=kCc8FmEb1nY)|[Code](https://github.com/karpathy/ng-video-lecture) -- [DAIR.AI] Prompt Engineering Guide [Link](https://github.com/dair-ai/Prompt-Engineering-Guide) -- [邱锡鹏] 大型语言模型的能力分析与应用 [Slides](resources/大型语言模型的能力分析与应用%20-%2030min.pdf) | [Video](https://www.bilibili.com/video/BV1Xb411X7c3/?buvid=XY2DA82257CC34DECD40B00CAE8AFB7F3B43C&is_story_h5=false&mid=dM1oVipECo22eTYTWkJVVg%3D%3D&p=1&plat_id=116&share_from=ugc&share_medium=android&share_plat=android&share_session_id=c42b6c60-9d22-4c75-90b8-48828e1168af&share_source=WEIXIN&share_tag=s_i×tamp=1676812375&unique_k=meHB9Xg&up_id=487788801&vd_source=1e55c5426b48b37e901ff0f78992e33f) -- [Philipp Schmid] Fine-tune FLAN-T5 XL/XXL using DeepSpeed & Hugging Face Transformers [Link](https://www.philschmid.de/fine-tune-flan-t5-deepspeed) -- [HuggingFace] Illustrating Reinforcement Learning from Human Feedback (RLHF) [Link](https://huggingface.co/blog/rlhf) -- [HuggingFace] What Makes a Dialog Agent Useful? [Link](https://huggingface.co/blog/dialog-agents) -- [张俊林]通向AGI之路：大型语言模型(LLM)技术精要 [Link](https://zhuanlan.zhihu.com/p/597586623) -- [大师兄]ChatGPT/InstructGPT详解 [Link](https://zhuanlan.zhihu.com/p/590311003) -- [HeptaAI]ChatGPT内核：InstructGPT，基于反馈指令的PPO强化学习 [Link](https://zhuanlan.zhihu.com/p/589747432) -- [Yao Fu] How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources [Link](https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1) -- [Stephen Wolfram] What Is ChatGPT Doing … and Why Does It Work? [Link](https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/) -- [Jingfeng Yang] Why did all of the public reproduction of GPT-3 fail? [Link](https://jingfengyang.github.io/gpt) -- [Hung-yi Lee] ChatGPT (可能)是怎麼煉成的 - GPT 社會化的過程 [Video](https://www.youtube.com/watch?v=e0aKI2GGZNg) -- [Keyvan Kambakhsh] Pure Rust implementation of a minimal Generative Pretrained Transformer [code](https://github.com/keyvank/femtoGPT) -- [过拟合] llm大模型训练知乎专栏 [Link](https://www.zhihu.com/column/c_1252604770952642560) -- [StatQuest] Sequence-to-Sequence (seq2seq) Encoder-Decoder Neural Networks [Link](https://www.youtube.com/watch?v=L8HKweZIOmg) -- [StatQuest] Transformer Neural Networks, ChatGPT's foundation [Link](https://www.youtube.com/watch?v=zxQyTK8quyY) -- [StatQuest] Decoder-Only Transformers, ChatGPTs specific Transformer [Link](https://www.youtube.com/watch?v=bQ5BoolX9Ag) -- [康斯坦丁] Understanding Language Processing Through Embedding [Link](https://zhuanlan.zhihu.com/p/643560252) - -## Courses - -- [UWaterloo] CS 886: Recent Advances on Foundation Models [Homepage](https://cs.uwaterloo.ca/~wenhuche/teaching/cs886/) -- [DeepLearning.AI] ChatGPT Prompt Engineering for Developers [Homepage](https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/) -- [Princeton] Understanding Large Language Models [Homepage](https://www.cs.princeton.edu/courses/archive/fall22/cos597G/) -- [OpenBMB] 大模型公开课 [主页](https://www.openbmb.org/community/course) -- [Stanford] CS224N-Lecture 11: Prompting, Instruction Finetuning, and RLHF [Slides](https://web.stanford.edu/class/cs224n/slides/cs224n-2023-lecture11-prompting-rlhf.pdf) -- [Stanford] CS324-Large Language Models [Homepage](https://stanford-cs324.github.io/winter2022/) -- [Stanford] CS25-Transformers United V2 [Homepage](https://web.stanford.edu/class/cs25/) -- [Stanford Webinar] GPT-3 & Beyond [Video](https://www.youtube.com/watch?v=-lnHHWRCDGk) -- [李沐] InstructGPT论文精读 [Bilibili](https://www.bilibili.com/video/BV1hd4y187CR/?spm_id_from=333.337.search-card.all.click&vd_source=1e55c5426b48b37e901ff0f78992e33f) [Youtube](https://www.youtube.com/watch?v=zfIGAwD1jOQ) -- [陳縕儂] OpenAI InstructGPT 從人類回饋中學習 ChatGPT 的前身 [Youtube](https://www.youtube.com/watch?v=ORHv8yKAV2Q) -- [李沐] HELM全面语言模型评测 [Bilibili](https://www.bilibili.com/video/BV1z24y1B7uX/?spm_id_from=333.337.search-card.all.click&vd_source=1e55c5426b48b37e901ff0f78992e33f) -- [李沐] GPT，GPT-2，GPT-3 论文精读 [Bilibili](https://www.bilibili.com/video/BV1AF411b7xQ/?spm_id_from=333.788&vd_source=1e55c5426b48b37e901ff0f78992e33f) [Youtube](https://www.youtube.com/watch?v=t70Bl3w7bxY&list=PLFXJ6jwg0qW-7UM8iUTj3qKqdhbQULP5I&index=18) -- [Aston Zhang] Chain of Thought论文 [Bilibili](https://www.bilibili.com/video/BV1t8411e7Ug/?spm_id_from=333.788&vd_source=1e55c5426b48b37e901ff0f78992e33f) [Youtube](https://www.youtube.com/watch?v=H4J59iG3t5o&list=PLFXJ6jwg0qW-7UM8iUTj3qKqdhbQULP5I&index=29) -- [MIT] Introduction to Data-Centric AI [Homepage](https://dcai.csail.mit.edu) -- [DeepLearning.AI] Building Applications with Vector Databases [Homepage](https://www.deeplearning.ai/short-courses/building-applications-vector-databases/) -- [DeepLearning.AI] Building Systems with the ChatGPT API [Homepage](https://www.deeplearning.ai/short-courses/building-systems-with-chatgpt/) -- [DeepLearning.AI] LangChain for LLM Application Development [Homepage](https://www.deeplearning.ai/short-courses/langchain-for-llm-application-development/) -- [DeepLearning.AI] LangChain: Chat with Your Data [Homepage](https://www.deeplearning.ai/short-courses/langchain-chat-with-your-data/) -- [DeepLearning.AI] Finetuning Large Language Models [Homepage](https://www.deeplearning.ai/short-courses/finetuning-large-language-models/) -- [DeepLearning.AI] Build LLM Apps with LangChain.js [Homepage](https://www.deeplearning.ai/short-courses/build-llm-apps-with-langchain-js/) -- [DeepLearning.AI] Large Language Models with Semantic Search [Homepage](https://www.deeplearning.ai/short-courses/large-language-models-semantic-search/) -- [DeepLearning.AI] LLMOps [Homepage](https://www.deeplearning.ai/short-courses/llmops/) -- [DeepLearning.AI] Building and Evaluating Advanced RAG Applications [Homepage](https://www.deeplearning.ai/short-courses/building-evaluating-advanced-rag/) -- [DeepLearning.AI] Quality and Safety for LLM Applications [Homepage](https://www.deeplearning.ai/short-courses/quality-safety-llm-applications/) -- [DeepLearning.AI] Vector Databases: from Embeddings to Applications [Homepage](https://www.deeplearning.ai/short-courses/vector-databases-embeddings-applications/) -- [DeepLearning.AI] Functions, Tools and Agents with LangChain [Homepage](https://www.deeplearning.ai/short-courses/functions-tools-agents-langchain/) -- [Arize] LLM Observability: Evaluations [Homepage](https://courses.arize.com/p/llm-evaluations/) -- [Arize] LLM Observability: Traces and Spans [Homepage](https://courses.arize.com/p/llm-observability-traces-spans/) - -## Books +## LLM Tutorials and Courses +- [llm-course](https://github.com/mlabonne/llm-course) - Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks. +- [UWaterloo CS 886](https://cs.uwaterloo.ca/~wenhuche/teaching/cs886/) - Recent Advances on Foundation Models. +- [CS25-Transformers United](https://web.stanford.edu/class/cs25/) +- [ChatGPT Prompt Engineering](https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/) +- [Princeton: Understanding Large Language Models](https://www.cs.princeton.edu/courses/archive/fall22/cos597G/) +- [CS324 - Large Language Models](https://stanford-cs324.github.io/winter2022/) +- [State of GPT](https://build.microsoft.com/en-US/sessions/db3f4859-cd30-4445-a0cd-553c3304f8e2) +- [A Visual Guide to Mamba and State Space Models](https://maartengrootendorst.substack.com/p/a-visual-guide-to-mamba-and-state?utm_source=multiple-personal-recommendations-email&utm_medium=email&open=false) +- [Let's build GPT: from scratch, in code, spelled out.](https://www.youtube.com/watch?v=kCc8FmEb1nY) +- [minbpe](https://www.youtube.com/watch?v=zduSFxRajkE&t=1157s) - Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. +- [femtoGPT](https://github.com/keyvank/femtoGPT) - Pure Rust implementation of a minimal Generative Pretrained Transformer. +- [Neurips2022-Foundational Robustness of Foundation Models](https://nips.cc/virtual/2022/tutorial/55796) +- [ICML2022-Welcome to the "Big Model" Era: Techniques and Systems to Train and Serve Bigger Models](https://icml.cc/virtual/2022/tutorial/18440) +- [GPT in 60 Lines of NumPy](https://jaykmody.com/blog/gpt-from-scratch/) + +## LLM Books - [Generative AI with LangChain: Build large language model (LLM) apps with Python, ChatGPT, and other LLMs](https://amzn.to/3GUlRng) - it comes with a [GitHub repository](https://github.com/benman1/generative_ai_with_langchain) that showcases a lot of the functionality - [Build a Large Language Model (From Scratch)](https://www.manning.com/books/build-a-large-language-model-from-scratch) - A guide to building your own working LLM. - [BUILD GPT: HOW AI WORKS](https://www.amazon.com/dp/9152799727?ref_=cm_sw_r_cp_ud_dp_W3ZHCD6QWM3DPPC0ARTT_1) - explains how to code a Generative Pre-trained Transformer, or GPT, from scratch. - -## Opinions - -- [A Stage Review of Instruction Tuning](https://yaofu.notion.site/June-2023-A-Stage-Review-of-Instruction-Tuning-f59dbfc36e2d4e12a33443bd6b2012c2) [2023-06-29] [Yao Fu] -- [LLM Powered Autonomous Agents](https://lilianweng.github.io/posts/2023-06-23-agent/) [2023-06-23] [Lilian] -- [Why you should work on AI AGENTS!](https://www.youtube.com/watch?v=fqVLjtvWgq8) [2023-06-22] [Andrej Karpathy] -- [Google "We Have No Moat, And Neither Does OpenAI"](https://www.semianalysis.com/p/google-we-have-no-moat-and-neither) [2023-05-05] -- [AI competition statement](https://petergabriel.com/news/ai-competition-statement/) [2023-04-20] [petergabriel] -- [我的大模型世界观](https://mp.weixin.qq.com/s/_ZvyxRpgIA4L4pqfcQtPTQ) [2023-04-23] [陆奇] -- [Prompt Engineering](https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/) [2023-03-15] [Lilian] -- [Noam Chomsky: The False Promise of ChatGPT](https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chatgpt-ai.html) \[2023-03-08][Noam Chomsky] -- [Is ChatGPT 175 Billion Parameters? Technical Analysis](https://orenleung.super.site/is-chatgpt-175-billion-parameters-technical-analysis) \[2023-03-04][Owen] -- [Towards ChatGPT and Beyond](https://zhuanlan.zhihu.com/p/607637180) \[2023-02-20][知乎][欧泽彬] -- [追赶ChatGPT的难点与平替](https://mp.weixin.qq.com/s/eYmssaPFODjC7xwh1jHydQ) \[2023-02-19][李rumor] -- [对话旷视研究院张祥雨｜ChatGPT的科研价值可能更大](https://zhuanlan.zhihu.com/p/606918875) \[2023-02-16][知乎][旷视科技] -- [关于ChatGPT八个技术问题的猜想](https://zhuanlan.zhihu.com/p/606478660) \[2023-02-15][知乎][张家俊] -- [ChatGPT发展历程、原理、技术架构详解和产业未来](https://zhuanlan.zhihu.com/p/590655677?utm_source=wechat_session&utm_medium=social&utm_oi=714896487502315520&s_r=0) \[2023-02-15][知乎][陈巍谈芯] -- [对ChatGPT的二十点看法](https://zhuanlan.zhihu.com/p/605882945?utm_medium=social&utm_oi=939485757606461440&utm_psn=1609870392121860096&utm_source=wechat_session) \[2023-02-13]\[知乎][熊德意] -- [ChatGPT-所见、所闻、所感](https://zhuanlan.zhihu.com/p/605331104) \[2023-02-11]\[知乎][刘聪NLP] -- [The Next Generation Of Large Language Models ](https://www.notion.so/Awesome-LLM-40c8aa3f2b444ecc82b79ae8bbd2696b) \[2023-02-07][Forbes] -- [Large Language Model Training in 2023](https://research.aimultiple.com/large-language-model-training/) \[2023-02-03][Cem Dilmegani] -- [What Are Large Language Models Used For? ](https://www.notion.so/Awesome-LLM-40c8aa3f2b444ecc82b79ae8bbd2696b) \[2023-01-26][NVIDIA] -- [Large Language Models: A New Moore's Law ](https://huggingface.co/blog/large-language-models) \[2021-10-26\]\[Huggingface\] - - -## Other Useful Resources +## Great thoughts about LLM +- [Why did all of the public reproduction of GPT-3 fail?](https://jingfengyang.github.io/gpt) +- [A Stage Review of Instruction Tuning](https://yaofu.notion.site/June-2023-A-Stage-Review-of-Instruction-Tuning-f59dbfc36e2d4e12a33443bd6b2012c2) +- [LLM Powered Autonomous Agents](https://lilianweng.github.io/posts/2023-06-23-agent/) +- [Why you should work on AI AGENTS!](https://www.youtube.com/watch?v=fqVLjtvWgq8) +- [Google "We Have No Moat, And Neither Does OpenAI"](https://www.semianalysis.com/p/google-we-have-no-moat-and-neither) +- [AI competition statement](https://petergabriel.com/news/ai-competition-statement/) +- [Prompt Engineering](https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/) +- [Noam Chomsky: The False Promise of ChatGPT](https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chatgpt-ai.html) +- [Is ChatGPT 175 Billion Parameters? Technical Analysis](https://orenleung.super.site/is-chatgpt-175-billion-parameters-technical-analysis) +- [The Next Generation Of Large Language Models ](https://www.notion.so/Awesome-LLM-40c8aa3f2b444ecc82b79ae8bbd2696b) +- [Large Language Model Training in 2023](https://research.aimultiple.com/large-language-model-training/) +- [How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources](https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1) +- [Open Pretrained Transformers](https://www.youtube.com/watch?v=p9IxoSkvZ-M&t=4s) +- [Scaling, emergence, and reasoning in large language models](https://docs.google.com/presentation/d/1EUV7W7X_w0BDrscDhPg7lMGzJCkeaPkGCJ3bN8dluXc/edit?pli=1&resourcekey=0-7Nz5A7y8JozyVrnDtcEKJA#slide=id.g16197112905_0_0) + +## Miscellaneous - [Arize-Phoenix](https://phoenix.arize.com/) - Open-source tool for ML observability that runs in your notebook environment. Monitor and fine tune LLM, CV and Tabular Models. - [Emergent Mind](https://www.emergentmind.com) - The latest AI news, curated & explained by GPT-4. @@ -527,11 +348,9 @@ The above tables coule be better summarized by this wonderful visualization from - [Cursor](https://www.cursor.so) - Write, edit, and chat about your code with a powerful AI. - [AutoGPT](https://github.com/Significant-Gravitas/Auto-GPT) - an experimental open-source application showcasing the capabilities of the GPT-4 language model. - [OpenAGI](https://github.com/agiresearch/OpenAGI) - When LLM Meets Domain Experts. -- [HuggingGPT](https://github.com/microsoft/JARVIS) - Solving AI Tasks with ChatGPT and its Friends in HuggingFace. - [EasyEdit](https://github.com/zjunlp/EasyEdit) - An easy-to-use framework to edit large language models. - [chatgpt-shroud](https://github.com/guyShilo/chatgpt-shroud) - A Chrome extension for OpenAI's ChatGPT, enhancing user privacy by enabling easy hiding and unhiding of chat history. Ideal for privacy during screen shares. -- [MTEB](https://huggingface.co/spaces/mteb/leaderboard) - Massive Text Embedding Benchmark Leaderboard -- [xFormer](https://github.com/facebookresearch/xformers) - A PyTorch based library which hosts flexible Transformers parts + ## Contributing This is an active repository and your contributions are always welcome! From 6c473beb0dd81172b2519a580ff7fa489d9e243d Mon Sep 17 00:00:00 2001 From: Vinicius Mesel Date: Thu, 23 May 2024 10:56:13 -0300 Subject: [PATCH 072/117] Adds talkd.ai --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 17fc832..98ae01d 100644 --- a/README.md +++ b/README.md @@ -269,6 +269,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [Tune Studio](https://studio.tune.app/) - Playground for devs to finetune & deploy LLMs - [LLocalSearch](https://github.com/nilsherzig/LLocalSearch) - Locally running websearch using LLM chains - [AI Gateway](https://github.com/Portkey-AI/gateway) — Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency. +- [talkd.ai dialog](https://github.com/talkdai/dialog) - Simple API for deploying any RAG or LLM that you want adding plugins. ## LLM Applications From d9b6517420171e0ccde0cf40b94adabca1bc9d72 Mon Sep 17 00:00:00 2001 From: Brandon Lockaby Date: Tue, 28 May 2024 08:23:08 -0400 Subject: [PATCH 073/117] Add Wllama --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 98ae01d..855684b 100644 --- a/README.md +++ b/README.md @@ -270,6 +270,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [LLocalSearch](https://github.com/nilsherzig/LLocalSearch) - Locally running websearch using LLM chains - [AI Gateway](https://github.com/Portkey-AI/gateway) — Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency. - [talkd.ai dialog](https://github.com/talkdai/dialog) - Simple API for deploying any RAG or LLM that you want adding plugins. +- [Wllama](https://github.com/ngxson/wllama) - WebAssembly binding for llama.cpp - Enabling in-browser LLM inference ## LLM Applications From 70b18cd25a85d09ed60f9465b435ed699cb11650 Mon Sep 17 00:00:00 2001 From: Hannibal046 Date: Wed, 5 Jun 2024 06:23:06 +0000 Subject: [PATCH 074/117] update trending project and add openai eval --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 855684b..3a6a226 100644 --- a/README.md +++ b/README.md @@ -6,11 +6,10 @@ 🔥 Large Language Models(LLM) have taken the ~~NLP community~~ ~~AI community~~ **the Whole World** by storm. Here is a curated list of papers about large language models, especially relating to ChatGPT. It also contains frameworks for LLM training, tools to deploy LLM, courses and tutorials about LLM and all publicly available LLM checkpoints and APIs. ## Trending LLM Projects -- [GPT-4o](https://openai.com/index/hello-gpt-4o/) - OpenAI's new flagship model that can reason across audio, vision, and text in real time. +- [Omost](https://github.com/lllyasviel/Omost) - a project to convert LLM's coding capability to image generation (or more accurately, image composing) capability. +- [llama-fs](https://github.com/iyaja/llama-fs) - A self-organizing file system with llama 3. - [MiniCPM-V](https://github.com/OpenBMB/MiniCPM-V) - A GPT-4V Level Multimodal LLM on Your Phone. -- [llama3-from-scratch](https://github.com/naklecha/llama3-from-scratch) - llama3 implementation one matrix multiplication at a time. -- [ChatGPT Desktop Application](https://github.com/lencx/ChatGPT) - ChatGPT Desktop Application (Mac, Windows and Linux). -- [llm.c](https://github.com/karpathy/llm.c) - LLM training in simple, raw C/CUDA. +- [fabric](https://github.com/danielmiessler/fabric) - an open-source framework for augmenting humans using AI. ## Table of Content @@ -210,6 +209,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [lighteval](https://github.com/huggingface/lighteval) - a lightweight LLM evaluation suite that Hugging Face has been using internally. - [OLMO-eval](https://github.com/allenai/OLMo-Eval) - a repository for evaluating open language models. - [instruct-eval](https://github.com/declare-lab/instruct-eval) - This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks. +- [simple-evals](https://github.com/openai/simple-evals) - Eval tools by OpenAI. ## LLM Training Frameworks From de210bf200527a9d071e92b3dd033b652d9b7496 Mon Sep 17 00:00:00 2001 From: Alex Combessie Date: Sat, 8 Jun 2024 00:13:28 +0200 Subject: [PATCH 075/117] Add Giskard for LLM app evaluation, in particular RAGs --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 3a6a226..8f3db85 100644 --- a/README.md +++ b/README.md @@ -210,6 +210,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [OLMO-eval](https://github.com/allenai/OLMo-Eval) - a repository for evaluating open language models. - [instruct-eval](https://github.com/declare-lab/instruct-eval) - This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks. - [simple-evals](https://github.com/openai/simple-evals) - Eval tools by OpenAI. +- [Giskard](https://github.com/Giskard-AI/giskard) - Testing & evaluation library for LLM applications, in particular RAGs ## LLM Training Frameworks From 9f3a0a005cf6827c1f8379b6a8f17a7eae49fdea Mon Sep 17 00:00:00 2001 From: Fran Santos <103037164+fsantosg@users.noreply.github.com> Date: Fri, 14 Jun 2024 10:53:30 +0200 Subject: [PATCH 076/117] add evaluation frameworks LangSmith and Ragas --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 8f3db85..37c57c9 100644 --- a/README.md +++ b/README.md @@ -211,6 +211,8 @@ If you're interested in the field of LLM, you may find the above list of milesto - [instruct-eval](https://github.com/declare-lab/instruct-eval) - This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks. - [simple-evals](https://github.com/openai/simple-evals) - Eval tools by OpenAI. - [Giskard](https://github.com/Giskard-AI/giskard) - Testing & evaluation library for LLM applications, in particular RAGs +- [LangSmith](https://www.langchain.com/langsmith) - a unified platform from LangChain framework for: evaluation, collaboration HITL (Human In The Loop), logging and monitoring LLM applications. +- [Ragas](https://github.com/explodinggradients/ragas) - a framework that helps you evaluate your Retrieval Augmented Generation (RAG) pipelines. ## LLM Training Frameworks From 45be887ef078210b3013ec8c13291e59ef6de7f3 Mon Sep 17 00:00:00 2001 From: wsl Date: Tue, 18 Jun 2024 16:19:21 +0800 Subject: [PATCH 077/117] update trending projects and add more open-sourced models like Qwen, Nemo --- README.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 37c57c9..e349ea3 100644 --- a/README.md +++ b/README.md @@ -6,10 +6,10 @@ 🔥 Large Language Models(LLM) have taken the ~~NLP community~~ ~~AI community~~ **the Whole World** by storm. Here is a curated list of papers about large language models, especially relating to ChatGPT. It also contains frameworks for LLM training, tools to deploy LLM, courses and tutorials about LLM and all publicly available LLM checkpoints and APIs. ## Trending LLM Projects -- [Omost](https://github.com/lllyasviel/Omost) - a project to convert LLM's coding capability to image generation (or more accurately, image composing) capability. -- [llama-fs](https://github.com/iyaja/llama-fs) - A self-organizing file system with llama 3. -- [MiniCPM-V](https://github.com/OpenBMB/MiniCPM-V) - A GPT-4V Level Multimodal LLM on Your Phone. -- [fabric](https://github.com/danielmiessler/fabric) - an open-source framework for augmenting humans using AI. +- [mistral.rs](https://github.com/EricLBuehler/mistral.rs) - Blazingly fast LLM inference. +- [dspy](https://github.com/stanfordnlp/dspy) - DSPy: The framework for programming—not prompting—foundation models. +- [QWen2](https://github.com/QwenLM/Qwen2) - Qwen2 is the large language model series developed by Qwen team, Alibaba Cloud. +- [DeepSeek-Coder-V2](https://github.com/deepseek-ai/DeepSeek-Coder-V2) - an open-source Mixture-of-Experts (MoE) code language model that achieves performance comparable to GPT4-Turbo in code-specific tasks. ## Table of Content @@ -81,7 +81,7 @@ | 2023-05 | ToT | Google&Princeton | [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://arxiv.org/pdf/2305.10601.pdf) | NeurIPS
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F2f3822eb380b5e753a6d579f31dfc3ec4c4a0820%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-07 | LLaMA 2 | Meta | [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/pdf/2307.09288.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F104b0bb1da562d53cbda87aec79ef6a2827d191a%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| |2023-10| Mistral 7B| Mistral |[Mistral 7B](https://arxiv.org/pdf/2310.06825.pdf)|
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fdb633c6b1c286c0386f0078d8a2e6224e03a6227%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-12 | Mamba | CMU&Princeton | [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/ftp/arxiv/papers/2312/2312.00752.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F432bef8e34014d726c674bc458008ac895297b51%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| +| 2023-12 | Mamba | CMU&Princeton | [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/ftp/arxiv/papers/2312/2312.00752.pdf) |ICML
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F432bef8e34014d726c674bc458008ac895297b51%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2024-03 | Jamba | AI21 Labs | [Jamba: A Hybrid Transformer-Mamba Language Model](https://arxiv.org/pdf/2403.19887) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fcbaf689fd9ea9bc939510019d90535d6249b3367%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | @@ -157,14 +157,16 @@ If you're interested in the field of LLM, you may find the above list of milesto - DeepSeek - [DeepSeek-Math-7B](https://huggingface.co/collections/deepseek-ai/deepseek-math-65f2962739da11599e441681) - [DeepSeek-Coder-1.3|6.7|7|33B](https://huggingface.co/collections/deepseek-ai/deepseek-coder-65f295d7d8a0a29fe39b4ec4) - - [DeepSeek-VL-1.3B|7B](https://huggingface.co/collections/deepseek-ai/deepseek-vl-65f295948133d9cf92b706d3) + - [DeepSeek-VL-1.3|7B](https://huggingface.co/collections/deepseek-ai/deepseek-vl-65f295948133d9cf92b706d3) - [DeepSeek-MoE-16B](https://huggingface.co/collections/deepseek-ai/deepseek-moe-65f29679f5cf26fe063686bf) - [DeepSeek-v2-236B-MoE](https://arxiv.org/abs/2405.04434) + - [DeepSeek-Coder-v2-16|236B-MOE](https://github.com/deepseek-ai/DeepSeek-Coder-V2) - Alibaba - [Qwen-1.8|7|14|72B](https://huggingface.co/collections/Qwen/qwen-65c0e50c3f1ab89cb8704144) - [Qwen1.5-1.8|4|7|14|32|72|110B](https://huggingface.co/collections/Qwen/qwen15-65c0a2f577b1ecb76d786524) - [CodeQwen-7B](https://huggingface.co/Qwen/CodeQwen1.5-7B) - [Qwen-VL-7B](https://huggingface.co/Qwen/Qwen-VL) + - [Qwen2-0.5|1.5|7|57-MOE|72B](https://qwenlm.github.io/blog/qwen2/) - 01-ai - [Yi-34B](https://huggingface.co/collections/01-ai/yi-2023-11-663f3f19119ff712e176720f) - [Yi1.5-6|9|34B](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) @@ -172,6 +174,8 @@ If you're interested in the field of LLM, you may find the above list of milesto - Baichuan - [Baichuan-7|13B](https://huggingface.co/baichuan-inc) - [Baichuan2-7|13B](https://huggingface.co/baichuan-inc) +- Nvidia + - [Nemotron-4-340B](https://huggingface.co/nvidia/Nemotron-4-340B-Instruct) - BLOOM - [BLOOMZ&mT0](https://huggingface.co/bigscience/bloomz) - Zhipu AI From 318dfa703f6894135ca8e85c1fac27ec91c9ccd0 Mon Sep 17 00:00:00 2001 From: -LAN- Date: Mon, 24 Jun 2024 00:42:28 +0800 Subject: [PATCH 078/117] doc(README): Add `dify` as a LLM Application --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index e349ea3..e10524e 100644 --- a/README.md +++ b/README.md @@ -305,7 +305,8 @@ If you're interested in the field of LLM, you may find the above list of milesto - [OneKE](https://openspg.yuque.com/ndx6g9/ps5q6b/vfoi61ks3mqwygvy) — A bilingual Chinese-English knowledge extraction model with knowledge graphs and natural language processing technologies. - [llm-ui](https://github.com/llm-ui-kit/llm-ui) - A React library for building LLM UIs. - [Wordware](https://www.wordware.ai) - A web-hosted IDE where non-technical domain experts work with AI Engineers to build task-specific AI agents. We approach prompting as a new programming language rather than low/no-code blocks. -- [Wallaroo.AI](https://github.com/WallarooLabs) - Deploy, manage, optimize any model at scale across any environment from cloud to edge. Let's you go from python notebook to inferencing in minutes. +- [Wallaroo.AI](https://github.com/WallarooLabs) - Deploy, manage, optimize any model at scale across any environment from cloud to edge. Let's you go from python notebook to inferencing in minutes. +- [Dify](https://github.com/langgenius/dify) - An open-source LLM app development platform with an intuitive interface that streamlines AI workflows, model management, and production deployment. ## LLM Tutorials and Courses - [llm-course](https://github.com/mlabonne/llm-course) - Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks. From 16165539c373aed07da03bed1172557ddf0e5e29 Mon Sep 17 00:00:00 2001 From: Sharath Bennur Date: Wed, 26 Jun 2024 11:10:25 -0400 Subject: [PATCH 079/117] Added DBRX & Berkeley Function Calling Leaderboard.md --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index e10524e..499d6b1 100644 --- a/README.md +++ b/README.md @@ -127,6 +127,7 @@ If you're interested in the field of LLM, you may find the above list of milesto using Nous benchmark suite. - [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) - aims to track, rank and evaluate LLMs and chatbots as they are released. - [OpenCompass 2.0 LLM Leaderboard](https://rank.opencompass.org.cn/leaderboard-llm-v2) - OpenCompass is an LLM evaluation platform, supporting a wide range of models (InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets. +- [Berkeley Function-Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html) - evaluates LLM's ability to call external functions / tools ## Open LLM - Meta @@ -199,6 +200,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [StarCoder2-3|7|15B](https://huggingface.co/collections/bigcode/starcoder2-65de6da6e87db3383572be1a) - DataBricks - [MPT-7B](https://www.databricks.com/blog/mpt-7b) + - [DBRX Base|Instruct](https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm) - Shanghai AI Laboratory - [InternLM2-1.8|7|20B](https://huggingface.co/collections/internlm/internlm2-65b0ce04970888799707893c) - [InternLM-Math-7B|20B](https://huggingface.co/collections/internlm/internlm2-math-65b0ce88bf7d3327d0a5ad9f) From 7eceade2d1968c1f4a6e1c75e9c47a1aa28956ce Mon Sep 17 00:00:00 2001 From: Hannibal046 <38466901+Hannibal046@users.noreply.github.com> Date: Fri, 28 Jun 2024 16:42:41 +0800 Subject: [PATCH 080/117] Update README.md --- README.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 499d6b1..b723b1d 100644 --- a/README.md +++ b/README.md @@ -6,10 +6,11 @@ 🔥 Large Language Models(LLM) have taken the ~~NLP community~~ ~~AI community~~ **the Whole World** by storm. Here is a curated list of papers about large language models, especially relating to ChatGPT. It also contains frameworks for LLM training, tools to deploy LLM, courses and tutorials about LLM and all publicly available LLM checkpoints and APIs. ## Trending LLM Projects -- [mistral.rs](https://github.com/EricLBuehler/mistral.rs) - Blazingly fast LLM inference. -- [dspy](https://github.com/stanfordnlp/dspy) - DSPy: The framework for programming—not prompting—foundation models. -- [QWen2](https://github.com/QwenLM/Qwen2) - Qwen2 is the large language model series developed by Qwen team, Alibaba Cloud. -- [DeepSeek-Coder-V2](https://github.com/deepseek-ai/DeepSeek-Coder-V2) - an open-source Mixture-of-Experts (MoE) code language model that achieves performance comparable to GPT4-Turbo in code-specific tasks. + +- [LibreChat](https://github.com/danny-avila/LibreChat) - All-In-One AI Conversations with LibreChat. +- [Open-Sora](https://github.com/hpcaitech/Open-Sora) - Democratizing Efficient Video Production for All. +- [LLM101n](https://github.com/karpathy/LLM101n) - Let's build a Storyteller. +- [Gemma 2](https://blog.google/technology/developers/google-gemma-2/) - A new open model standard for efficiency and performance from Google. ## Table of Content @@ -140,6 +141,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [Mixtral-8x7B](https://mistral.ai/news/mixtral-of-experts/) - [Mixtral-8x22B](https://mistral.ai/news/mixtral-8x22b/) - Google + - [Gemma2-9|27B](https://blog.google/technology/developers/google-gemma-2/) - [Gemma-2|7B](https://blog.google/technology/developers/gemma-open-models/) - [RecurrentGemma-2B](https://github.com/google-deepmind/recurrentgemma) - [T5](https://arxiv.org/abs/1910.10683) @@ -200,7 +202,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [StarCoder2-3|7|15B](https://huggingface.co/collections/bigcode/starcoder2-65de6da6e87db3383572be1a) - DataBricks - [MPT-7B](https://www.databricks.com/blog/mpt-7b) - - [DBRX Base|Instruct](https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm) + - [DBRX-132B-MoE](https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm) - Shanghai AI Laboratory - [InternLM2-1.8|7|20B](https://huggingface.co/collections/internlm/internlm2-65b0ce04970888799707893c) - [InternLM-Math-7B|20B](https://huggingface.co/collections/internlm/internlm2-math-65b0ce88bf7d3327d0a5ad9f) @@ -245,6 +247,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [ollama](https://github.com/ollama/ollama) - Get up and running with Llama 3, Mistral, Gemma, and other large language models. - [Langfuse](https://github.com/langfuse/langfuse) - Open Source LLM Engineering Platform 🪢 Tracing, Evaluations, Prompt Management, Evaluations and Playground. - [FastChat](https://github.com/lm-sys/FastChat) - A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs. +- [mistral.rs](https://github.com/EricLBuehler/mistral.rs) - Blazingly fast LLM inference. - [MindSQL](https://github.com/Mindinventory/MindSQL) - A python package for Txt-to-SQL with self hosting functionalities and RESTful APIs compatible with proprietary as well as open source LLM. - [SkyPilot](https://github.com/skypilot-org/skypilot) - Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface. - [Haystack](https://haystack.deepset.ai/) - an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data. @@ -282,7 +285,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [Wllama](https://github.com/ngxson/wllama) - WebAssembly binding for llama.cpp - Enabling in-browser LLM inference ## LLM Applications - +- [dspy](https://github.com/stanfordnlp/dspy) - DSPy: The framework for programming—not prompting—foundation models. - [YiVal](https://github.com/YiVal/YiVal) — Evaluate and Evolve: YiVal is an open-source GenAI-Ops tool for tuning and evaluating prompts, configurations, and model parameters using customizable datasets, evaluation methods, and improvement strategies. - [Guidance](https://github.com/microsoft/guidance) — A handy looking Python library from Microsoft that uses Handlebars templating to interleave generation, prompting, and logical control. - [LangChain](https://github.com/hwchase17/langchain) — A popular Python/JavaScript library for chaining sequences of language model prompts. From 7150cb71f64a61869990aa431fc1f94ae6e1d40e Mon Sep 17 00:00:00 2001 From: Hill Ma Date: Fri, 5 Jul 2024 19:53:00 -0700 Subject: [PATCH 081/117] Fix the link to Mamba paper. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b723b1d..8de1932 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,7 @@ | 2023-05 | ToT | Google&Princeton | [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://arxiv.org/pdf/2305.10601.pdf) | NeurIPS
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F2f3822eb380b5e753a6d579f31dfc3ec4c4a0820%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2023-07 | LLaMA 2 | Meta | [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/pdf/2307.09288.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F104b0bb1da562d53cbda87aec79ef6a2827d191a%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| |2023-10| Mistral 7B| Mistral |[Mistral 7B](https://arxiv.org/pdf/2310.06825.pdf)|
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fdb633c6b1c286c0386f0078d8a2e6224e03a6227%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-12 | Mamba | CMU&Princeton | [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/ftp/arxiv/papers/2312/2312.00752.pdf) |ICML
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F432bef8e34014d726c674bc458008ac895297b51%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| +| 2023-12 | Mamba | CMU&Princeton | [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/pdf/2312.00752) |ICML
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F432bef8e34014d726c674bc458008ac895297b51%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| | 2024-03 | Jamba | AI21 Labs | [Jamba: A Hybrid Transformer-Mamba Language Model](https://arxiv.org/pdf/2403.19887) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fcbaf689fd9ea9bc939510019d90535d6249b3367%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | From 2139d88c5b2e9bf2964c46d47d494f7bf531fa8a Mon Sep 17 00:00:00 2001 From: wangzhihong Date: Tue, 9 Jul 2024 09:41:47 +0800 Subject: [PATCH 082/117] add LazyLLM to README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 8de1932..4eb1b8b 100644 --- a/README.md +++ b/README.md @@ -312,6 +312,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [Wordware](https://www.wordware.ai) - A web-hosted IDE where non-technical domain experts work with AI Engineers to build task-specific AI agents. We approach prompting as a new programming language rather than low/no-code blocks. - [Wallaroo.AI](https://github.com/WallarooLabs) - Deploy, manage, optimize any model at scale across any environment from cloud to edge. Let's you go from python notebook to inferencing in minutes. - [Dify](https://github.com/langgenius/dify) - An open-source LLM app development platform with an intuitive interface that streamlines AI workflows, model management, and production deployment. +- [LazyLLM](https://github.com/LazyAGI/LazyLLM) - An open-source LLM app for building multi-agent LLMs applications in an easy and lazy way, supports model deployment and fine-tuning. ## LLM Tutorials and Courses - [llm-course](https://github.com/mlabonne/llm-course) - Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks. From 62addd4cf4ef099b7551155e7d04f16681897c9c Mon Sep 17 00:00:00 2001 From: Jinjie Date: Wed, 10 Jul 2024 16:00:08 +0800 Subject: [PATCH 083/117] Add MixEval --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 8de1932..a7ccf77 100644 --- a/README.md +++ b/README.md @@ -124,6 +124,7 @@ If you're interested in the field of LLM, you may find the above list of milesto ## LLM Leaderboard - [Chatbot Arena Leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard) - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner. +- [MixEval Leaderboard](https://mixeval.github.io/#leaderboard) - a ground-truth-based dynamic benchmark derived from off-the-shelf benchmark mixtures, which evaluates LLMs with a highly capable model ranking (i.e., 0.96 correlation with Chatbot Arena) while running locally and quickly (6% the time and cost of running MMLU). - [AlpacaEval Leaderboard](https://tatsu-lab.github.io/alpaca_eval/) - An Automatic Evaluator for Instruction-following Language Models using Nous benchmark suite. - [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) - aims to track, rank and evaluate LLMs and chatbots as they are released. @@ -214,6 +215,7 @@ If you're interested in the field of LLM, you may find the above list of milesto ## LLM Evaluation: - [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) - A framework for few-shot evaluation of language models. +- [MixEval](https://github.com/Psycoy/MixEval) - A reliable click-and-go evaluation suite compatible with both open-source and proprietary models, supporting MixEval and other benchmarks. - [lighteval](https://github.com/huggingface/lighteval) - a lightweight LLM evaluation suite that Hugging Face has been using internally. - [OLMO-eval](https://github.com/allenai/OLMo-Eval) - a repository for evaluating open language models. - [instruct-eval](https://github.com/declare-lab/instruct-eval) - This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks. From c45a4c288e3865d07b16ce37da7e78e0f66289e7 Mon Sep 17 00:00:00 2001 From: Dasha Maliugina <105814287+dmaliugina@users.noreply.github.com> Date: Tue, 16 Jul 2024 13:31:05 -0300 Subject: [PATCH 084/117] Update README.md Added open-source Evidently library to LLM Applications list --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 17dcdd0..b55915a 100644 --- a/README.md +++ b/README.md @@ -291,6 +291,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [YiVal](https://github.com/YiVal/YiVal) — Evaluate and Evolve: YiVal is an open-source GenAI-Ops tool for tuning and evaluating prompts, configurations, and model parameters using customizable datasets, evaluation methods, and improvement strategies. - [Guidance](https://github.com/microsoft/guidance) — A handy looking Python library from Microsoft that uses Handlebars templating to interleave generation, prompting, and logical control. - [LangChain](https://github.com/hwchase17/langchain) — A popular Python/JavaScript library for chaining sequences of language model prompts. +- [Evidently](https://github.com/evidentlyai/evidently) — An open-source framework to evaluate, test and monitor ML and LLM-powered systems. - [FLAML (A Fast Library for Automated Machine Learning & Tuning)](https://microsoft.github.io/FLAML/docs/Getting-Started/): A Python library for automating selection of models, hyperparameters, and other tunable choices. - [Chainlit](https://docs.chainlit.io/overview) — A Python library for making chatbot interfaces. - [Guardrails.ai](https://www.guardrailsai.com/docs/) — A Python library for validating outputs and retrying failures. Still in alpha, so expect sharp edges and bugs. From 38a1089773ecac131b0c6f7a719f55d7285973cd Mon Sep 17 00:00:00 2001 From: ahaapple <151364778+ahaapple@users.noreply.github.com> Date: Sat, 20 Jul 2024 17:11:05 +0800 Subject: [PATCH 085/117] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index b55915a..85cb728 100644 --- a/README.md +++ b/README.md @@ -316,6 +316,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [Wallaroo.AI](https://github.com/WallarooLabs) - Deploy, manage, optimize any model at scale across any environment from cloud to edge. Let's you go from python notebook to inferencing in minutes. - [Dify](https://github.com/langgenius/dify) - An open-source LLM app development platform with an intuitive interface that streamlines AI workflows, model management, and production deployment. - [LazyLLM](https://github.com/LazyAGI/LazyLLM) - An open-source LLM app for building multi-agent LLMs applications in an easy and lazy way, supports model deployment and fine-tuning. +- [MemFree](https://github.com/memfreeme/memfree) - Open Source Hybrid AI Search Engine, Instantly Get Accurate Answers from the Internet, Bookmarks, Notes, and Docs. Support One-Click Deployment ## LLM Tutorials and Courses - [llm-course](https://github.com/mlabonne/llm-course) - Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks. From f0e90599f12b5da58aa0c301d70514fc297afdc2 Mon Sep 17 00:00:00 2001 From: Li Yin Date: Sat, 20 Jul 2024 06:34:36 -0700 Subject: [PATCH 086/117] Update README.md Add lightrag library for llm applications --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index b55915a..0cb312b 100644 --- a/README.md +++ b/README.md @@ -287,6 +287,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [Wllama](https://github.com/ngxson/wllama) - WebAssembly binding for llama.cpp - Enabling in-browser LLM inference ## LLM Applications +- [LightRAG](https://github.com/SylphAI-Inc/LightRAG) - LightRAG: The PyTorch library for LLM applications. - [dspy](https://github.com/stanfordnlp/dspy) - DSPy: The framework for programming—not prompting—foundation models. - [YiVal](https://github.com/YiVal/YiVal) — Evaluate and Evolve: YiVal is an open-source GenAI-Ops tool for tuning and evaluating prompts, configurations, and model parameters using customizable datasets, evaluation methods, and improvement strategies. - [Guidance](https://github.com/microsoft/guidance) — A handy looking Python library from Microsoft that uses Handlebars templating to interleave generation, prompting, and logical control. From 2f7d1b1704edbc5cb15229252b6ef30cb4896a1b Mon Sep 17 00:00:00 2001 From: Li Yin Date: Sun, 21 Jul 2024 14:20:13 +0000 Subject: [PATCH 087/117] rename lightRAG to adalflow due to libarary renaming --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 0cb312b..eb73349 100644 --- a/README.md +++ b/README.md @@ -287,7 +287,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [Wllama](https://github.com/ngxson/wllama) - WebAssembly binding for llama.cpp - Enabling in-browser LLM inference ## LLM Applications -- [LightRAG](https://github.com/SylphAI-Inc/LightRAG) - LightRAG: The PyTorch library for LLM applications. +- [AdalFlow](https://github.com/SylphAI-Inc/AdalFlow) - AdalFlow: The PyTorch library for LLM applications. - [dspy](https://github.com/stanfordnlp/dspy) - DSPy: The framework for programming—not prompting—foundation models. - [YiVal](https://github.com/YiVal/YiVal) — Evaluate and Evolve: YiVal is an open-source GenAI-Ops tool for tuning and evaluating prompts, configurations, and model parameters using customizable datasets, evaluation methods, and improvement strategies. - [Guidance](https://github.com/microsoft/guidance) — A handy looking Python library from Microsoft that uses Handlebars templating to interleave generation, prompting, and logical control. From c0b04ccdadfa01091195d1753bb101b27abf3ce0 Mon Sep 17 00:00:00 2001 From: utuncel Date: Thu, 1 Aug 2024 15:29:51 +0200 Subject: [PATCH 088/117] adding unlsothai, a fine-tuning-framework --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 84d2099..7bd4f59 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,7 @@ - [LLM Books](#llm-books) - [Great thoughts about LLM](#great-thoughts-about-llm) - [Miscellaneous](#miscellaneous) + - [Fine-Tuning](#fine-tuning) ## Milestone Papers @@ -318,6 +319,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [Dify](https://github.com/langgenius/dify) - An open-source LLM app development platform with an intuitive interface that streamlines AI workflows, model management, and production deployment. - [LazyLLM](https://github.com/LazyAGI/LazyLLM) - An open-source LLM app for building multi-agent LLMs applications in an easy and lazy way, supports model deployment and fine-tuning. - [MemFree](https://github.com/memfreeme/memfree) - Open Source Hybrid AI Search Engine, Instantly Get Accurate Answers from the Internet, Bookmarks, Notes, and Docs. Support One-Click Deployment +- [unslothai](https://github.com/unslothai/unsloth) - A framework that specializes in efficient fine-tuning. On its GitHub page, you can find ready-to-use fine-tuning templates for various LLMs, allowing you to easily train your own data for free on the Google Colab cloud. ## LLM Tutorials and Courses - [llm-course](https://github.com/mlabonne/llm-course) - Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks. From ee702703d840bedd7f3e214e451da6a764d09479 Mon Sep 17 00:00:00 2001 From: Yinlin Li Date: Tue, 6 Aug 2024 16:40:59 +0800 Subject: [PATCH 089/117] Add GPUStack to LLM Deployment --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 7bd4f59..91292d2 100644 --- a/README.md +++ b/README.md @@ -286,6 +286,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [AI Gateway](https://github.com/Portkey-AI/gateway) — Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency. - [talkd.ai dialog](https://github.com/talkdai/dialog) - Simple API for deploying any RAG or LLM that you want adding plugins. - [Wllama](https://github.com/ngxson/wllama) - WebAssembly binding for llama.cpp - Enabling in-browser LLM inference +- [GPUStack](https://github.com/gpustack/gpustack) - An open-source GPU cluster manager for running LLMs ## LLM Applications - [AdalFlow](https://github.com/SylphAI-Inc/AdalFlow) - AdalFlow: The PyTorch library for LLM applications. From 9b83b3c95d62e0ef6f7b540c510017f70e23497d Mon Sep 17 00:00:00 2001 From: Hannibal046 Date: Sun, 11 Aug 2024 06:10:48 +0000 Subject: [PATCH 090/117] update --- README.md | 124 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 65 insertions(+), 59 deletions(-) diff --git a/README.md b/README.md index 91292d2..8282ea5 100644 --- a/README.md +++ b/README.md @@ -7,10 +7,9 @@ ## Trending LLM Projects -- [LibreChat](https://github.com/danny-avila/LibreChat) - All-In-One AI Conversations with LibreChat. -- [Open-Sora](https://github.com/hpcaitech/Open-Sora) - Democratizing Efficient Video Production for All. -- [LLM101n](https://github.com/karpathy/LLM101n) - Let's build a Storyteller. -- [Gemma 2](https://blog.google/technology/developers/google-gemma-2/) - A new open model standard for efficiency and performance from Google. +- [Deep-Live-Cam](https://github.com/hacksider/Deep-Live-Cam) - real time face swap and one-click video deepfake with only a single image (uncensored). +- [MiniCPM-V 2.6](https://github.com/OpenBMB/MiniCPM-V) - A GPT-4V Level MLLM for Single Image, Multi Image and Video on Your Phone +- [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS) - 1 min voice data can also be used to train a good TTS model! (few shot voice cloning). ## Table of Content @@ -27,64 +26,69 @@ - [LLM Books](#llm-books) - [Great thoughts about LLM](#great-thoughts-about-llm) - [Miscellaneous](#miscellaneous) - - [Fine-Tuning](#fine-tuning) ## Milestone Papers -| Date | keywords | Institute | Paper | Publication | -| :-----: | :------------------: | :--------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------: | -| 2017-06 | Transformers | Google | [Attention Is All You Need](https://arxiv.org/pdf/1706.03762.pdf) | NeurIPS
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F204e3073870fae3d05bcbc2f6a8e263d9b72e776%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | -| 2018-06 | GPT 1.0 | OpenAI | [Improving Language Understanding by Generative Pre-Training](https://www.cs.ubc.ca/~amuham01/LING530/papers/radford2018improving.pdf) | ![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fcd18800a0fe0b668a1cc19f2ec95b5003d0a5035%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | -| 2018-10 | BERT | Google | [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://aclanthology.org/N19-1423.pdf) | NAACL
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fdf2b0e26d0599ce3e70df8a9da02e51594e0e992%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | -| 2019-02 | GPT 2.0 | OpenAI | [Language Models are Unsupervised Multitask Learners](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) | ![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F9405cc0d6169988371b2755e573cc28650d14dfe%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | -| 2019-09 | Megatron-LM | NVIDIA | [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/pdf/1909.08053.pdf) | ![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F8323c591e119eb09b28b29fd6c7bc76bd889df7a%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | -| 2019-10 | T5 | Google | [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://jmlr.org/papers/v21/20-074.html) | JMLR
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F3cfb319689f06bf04c2e28399361f414ca32c4b3%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | -| 2019-10 | ZeRO | Microsoft | [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://arxiv.org/pdf/1910.02054.pdf) | SC
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F00c957711b12468cb38424caccdf5291bb354033%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | -| 2020-01 | Scaling Law | OpenAI | [Scaling Laws for Neural Language Models](https://arxiv.org/pdf/2001.08361.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fe6c561d02500b2596a230b341a8eb8b921ca5bf2%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2020-05 | GPT 3.0 | OpenAI | [Language models are few-shot learners](https://papers.nips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf) | NeurIPS
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F6b85b63579a916f705a8e10a49bd8d849d91b1fc%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | -| 2021-01 | Switch Transformers | Google | [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://arxiv.org/pdf/2101.03961.pdf) | JMLR
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Ffdacf2a732f55befdc410ea927091cad3b791f13%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | -| 2021-08 | Codex | OpenAI | [Evaluating Large Language Models Trained on Code](https://arxiv.org/pdf/2107.03374.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Facbdbf49f9bc3f151b93d9ca9a06009f4f6eb269%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2021-08 | Foundation Models | Stanford | [On the Opportunities and Risks of Foundation Models](https://arxiv.org/pdf/2108.07258.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F4f68e07c6c3173480053fd52391851d6f80d651b%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2021-09 | FLAN | Google | [Finetuned Language Models are Zero-Shot Learners](https://openreview.net/forum?id=gEZrGCozdqR) | ICLR
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fff0b2681d7b05e16c46dfb71d980cc2f605907cd%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2021-10 | T0 | HuggingFace et al. | [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://arxiv.org/abs/2110.08207) | ICLR
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F17dd3555fd1ccf1141cf984347fa1b3fd6b009ca%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2021-12 | GLaM | Google | [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://arxiv.org/pdf/2112.06905.pdf) | ICML
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F80d0116d77beeded0c23cf48946d9d10d4faee14%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2021-12 | WebGPT | OpenAI | [WebGPT: Browser-assisted question-answering with human feedback](https://www.semanticscholar.org/paper/WebGPT%3A-Browser-assisted-question-answering-with-Nakano-Hilton/2f3efe44083af91cef562c1a3451eee2f8601d22) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F2f3efe44083af91cef562c1a3451eee2f8601d22%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2021-12 | Retro | DeepMind | [Improving language models by retrieving from trillions of tokens](https://www.deepmind.com/publications/improving-language-models-by-retrieving-from-trillions-of-tokens) | ICML
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F002c256d30d6be4b23d365a8de8ae0e67e4c9641%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | -| 2021-12 | Gopher | DeepMind | [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://arxiv.org/pdf/2112.11446.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F68f141724814839d556a989646194be88641b143%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-01 | COT | Google | [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/pdf/2201.11903.pdf) | NeurIPS
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F1b6e810ce0afd0dd093f789d2b2742d047e316d5%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-01 | LaMDA | Google | [LaMDA: Language Models for Dialog Applications](https://arxiv.org/pdf/2201.08239.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fb3848d32f7294ec708627897833c4097eb4d8778%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-01 | Minerva | Google | [Solving Quantitative Reasoning Problems with Language Models](https://arxiv.org/abs/2206.14858) | NeurIPS
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fab0e3d3e4d42369de5933a3b4c237780b41c0d77%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | -| 2022-01 | Megatron-Turing NLG | Microsoft&NVIDIA | [Using Deep and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://arxiv.org/pdf/2201.11990.pdf) | ![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F7cbc2a7843411a1768ab762930707af0a3c33a19%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-03 | InstructGPT | OpenAI | [Training language models to follow instructions with human feedback](https://arxiv.org/pdf/2203.02155.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fd766bffc357127e0dc86dd69561d5aeb520d6f4c%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-04 | PaLM | Google | [PaLM: Scaling Language Modeling with Pathways](https://arxiv.org/pdf/2204.02311.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F094ff971d6a8b8ff870946c9b3ce5aa173617bfb%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-04 | Chinchilla | DeepMind | [An empirical analysis of compute-optimal large language model training](https://www.deepmind.com/publications/an-empirical-analysis-of-compute-optimal-large-language-model-training) | NeurIPS
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fbb0656031cb17adf6bac5fd0fe8d53dd9c291508%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | -| 2022-05 | OPT | Meta | [OPT: Open Pre-trained Transformer Language Models](https://arxiv.org/pdf/2205.01068.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F13a0d8bb38f739990c8cd65a44061c6534f17221%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-05 | UL2 | Google | [Unifying Language Learning Paradigms](https://arxiv.org/abs/2205.05131v1) | ICLR
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Ff40aeae3e522ada1f6a9f326841b01ef5c8657b6%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-06 | Emergent Abilities | Google | [Emergent Abilities of Large Language Models](https://openreview.net/pdf?id=yzkSU5zdwD) | TMLR
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fdac3a172b504f4e33c029655e9befb3386e5f63a%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-06 | BIG-bench | Google | [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://github.com/google/BIG-bench) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F34503c0b6a615124eaf82cb0e4a1dab2866e8980%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-06 | METALM | Microsoft | [Language Models are General-Purpose Interfaces](https://arxiv.org/pdf/2206.06336.pdf) | ![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fa8fd9c1625011741f74401ff9bdc1c584e25c86d%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | -| 2022-09 | Sparrow | DeepMind | [Improving alignment of dialogue agents via targeted human judgements](https://arxiv.org/pdf/2209.14375.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F74eae12620bd1c1393e268bddcb6f129a5025166%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-10 | Flan-T5/PaLM | Google | [Scaling Instruction-Finetuned Language Models](https://arxiv.org/pdf/2210.11416.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F5484d228bfc50efbac6e86677bc2ec2ee4ede1a6%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-10 | GLM-130B | Tsinghua | [GLM-130B: An Open Bilingual Pre-trained Model](https://arxiv.org/pdf/2210.02414.pdf) | ICLR
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F1d26c947406173145a4665dd7ab255e03494ea28%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-11 | HELM | Stanford | [Holistic Evaluation of Language Models](https://arxiv.org/pdf/2211.09110.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F5032c0946ee96ff11a292762f23e6377a6cf2731%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-11 | BLOOM | BigScience | [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://arxiv.org/pdf/2211.05100.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F964bd39b546f0f6625ff3b9ef1083f797807ef2e%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-11 | Galactica | Meta | [Galactica: A Large Language Model for Science](https://arxiv.org/pdf/2211.09085.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F7d645a3fd276918374fd9483fd675c28e46506d1%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2022-12 | OPT-IML | Meta | [OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization](https://arxiv.org/pdf/2212.12017) | ![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fe965e93e76a9e6c4e4863d145b5c007b540d575d%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-01 | Flan 2022 Collection | Google | [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://arxiv.org/pdf/2301.13688.pdf) | ICML
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Ff2b0017ddd77fa38760a18145e63553105a1a236%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-02 | LLaMA|Meta|[LLaMA: Open and Efficient Foundation Language Models](https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/)|![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F57e849d0de13ed5f91d086936296721d4ff75a75%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-02 | Kosmos-1|Microsoft|[Language Is Not All You Need: Aligning Perception with Language Models](https://arxiv.org/abs/2302.14045)|![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Ffbfef4723d8c8467d7bd523e1d0b703cce0e0f9c%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-03 | PaLM-E | Google | [PaLM-E: An Embodied Multimodal Language Model](https://palm-e.github.io)| ICML
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F38fe8f324d2162e63a967a9ac6648974fc4c66f3%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-03 | GPT 4 | OpenAI | [GPT-4 Technical Report](https://openai.com/research/gpt-4)|![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F8ca62fdf4c276ea3052dc96dcfd8ee96ca425a48%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-04 | Pythia | EleutherAI et al. | [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://arxiv.org/abs/2304.01373)|ICML
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fbe55e8ec4213868db08f2c3168ae666001bea4b8%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-05 | Dromedary | CMU et al. | [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://arxiv.org/abs/2305.03047)| NeurIPS
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fe01515c6138bc525f7aec30fc85f2adf028d4156%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-05 | PaLM 2 | Google | [PaLM 2 Technical Report](https://ai.google/static/documents/palm2techreport.pdf)|![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Feccee350691708972370b7a12c2a78ad3bddd159%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-05 | RWKV | Bo Peng | [RWKV: Reinventing RNNs for the Transformer Era](https://arxiv.org/abs/2305.13048) |EMNLP
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F026b3396a63ed5772329708b7580d633bb86bec9%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-05 | DPO | Stanford | [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/pdf/2305.18290.pdf) |Neurips
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F0d1c76d45afa012ded7ab741194baf142117c495%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-05 | ToT | Google&Princeton | [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://arxiv.org/pdf/2305.10601.pdf) | NeurIPS
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F2f3822eb380b5e753a6d579f31dfc3ec4c4a0820%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-07 | LLaMA 2 | Meta | [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/pdf/2307.09288.pdf) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F104b0bb1da562d53cbda87aec79ef6a2827d191a%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -|2023-10| Mistral 7B| Mistral |[Mistral 7B](https://arxiv.org/pdf/2310.06825.pdf)|
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fdb633c6b1c286c0386f0078d8a2e6224e03a6227%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2023-12 | Mamba | CMU&Princeton | [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/pdf/2312.00752) |ICML
![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2F432bef8e34014d726c674bc458008ac895297b51%3Ffields%3DcitationCount&query=%24.citationCount&label=citation)| -| 2024-03 | Jamba | AI21 Labs | [Jamba: A Hybrid Transformer-Mamba Language Model](https://arxiv.org/pdf/2403.19887) |![Dynamic JSON Badge](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fapi.semanticscholar.org%2Fgraph%2Fv1%2Fpaper%2Fcbaf689fd9ea9bc939510019d90535d6249b3367%3Ffields%3DcitationCount&query=%24.citationCount&label=citation) | +| Date | keywords | Institute | Paper | +|:-------:|:--------------------:|:------------------:|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 2017-06 | Transformers | Google | [Attention Is All You Need](https://arxiv.org/pdf/1706.03762.pdf) | +| 2018-06 | GPT 1.0 | OpenAI | [Improving Language Understanding by Generative Pre-Training](https://www.cs.ubc.ca/~amuham01/LING530/papers/radford2018improving.pdf) | +| 2018-10 | BERT | Google | [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://aclanthology.org/N19-1423.pdf) | +| 2019-02 | GPT 2.0 | OpenAI | [Language Models are Unsupervised Multitask Learners](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) | +| 2019-09 | Megatron-LM | NVIDIA | [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/pdf/1909.08053.pdf) | +| 2019-10 | T5 | Google | [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://jmlr.org/papers/v21/20-074.html) | +| 2019-10 | ZeRO | Microsoft | [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://arxiv.org/pdf/1910.02054.pdf) | +| 2020-01 | Scaling Law | OpenAI | [Scaling Laws for Neural Language Models](https://arxiv.org/pdf/2001.08361.pdf) | +| 2020-05 | GPT 3.0 | OpenAI | [Language models are few-shot learners](https://papers.nips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf) | +| 2021-01 | Switch Transformers | Google | [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://arxiv.org/pdf/2101.03961.pdf) | +| 2021-08 | Codex | OpenAI | [Evaluating Large Language Models Trained on Code](https://arxiv.org/pdf/2107.03374.pdf) | +| 2021-08 | Foundation Models | Stanford | [On the Opportunities and Risks of Foundation Models](https://arxiv.org/pdf/2108.07258.pdf) | +| 2021-09 | FLAN | Google | [Finetuned Language Models are Zero-Shot Learners](https://openreview.net/forum?id=gEZrGCozdqR) | +| 2021-10 | T0 | HuggingFace et al. | [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://arxiv.org/abs/2110.08207) | +| 2021-12 | GLaM | Google | [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://arxiv.org/pdf/2112.06905.pdf) | +| 2021-12 | WebGPT | OpenAI | [WebGPT: Browser-assisted question-answering with human feedback](https://www.semanticscholar.org/paper/WebGPT%3A-Browser-assisted-question-answering-with-Nakano-Hilton/2f3efe44083af91cef562c1a3451eee2f8601d22) | +| 2021-12 | Retro | DeepMind | [Improving language models by retrieving from trillions of tokens](https://www.deepmind.com/publications/improving-language-models-by-retrieving-from-trillions-of-tokens) | +| 2021-12 | Gopher | DeepMind | [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://arxiv.org/pdf/2112.11446.pdf) | +| 2022-01 | COT | Google | [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/pdf/2201.11903.pdf) | +| 2022-01 | LaMDA | Google | [LaMDA: Language Models for Dialog Applications](https://arxiv.org/pdf/2201.08239.pdf) | +| 2022-01 | Minerva | Google | [Solving Quantitative Reasoning Problems with Language Models](https://arxiv.org/abs/2206.14858) | +| 2022-01 | Megatron-Turing NLG | Microsoft&NVIDIA | [Using Deep and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://arxiv.org/pdf/2201.11990.pdf) | +| 2022-03 | InstructGPT | OpenAI | [Training language models to follow instructions with human feedback](https://arxiv.org/pdf/2203.02155.pdf) | +| 2022-04 | PaLM | Google | [PaLM: Scaling Language Modeling with Pathways](https://arxiv.org/pdf/2204.02311.pdf) | +| 2022-04 | Chinchilla | DeepMind | [An empirical analysis of compute-optimal large language model training](https://arxiv.org/abs/2408.00724) | +| 2022-05 | OPT | Meta | [OPT: Open Pre-trained Transformer Language Models](https://arxiv.org/pdf/2205.01068.pdf) | +| 2022-05 | UL2 | Google | [Unifying Language Learning Paradigms](https://arxiv.org/abs/2205.05131v1) | +| 2022-06 | Emergent Abilities | Google | [Emergent Abilities of Large Language Models](https://openreview.net/pdf?id=yzkSU5zdwD) | +| 2022-06 | BIG-bench | Google | [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://github.com/google/BIG-bench) | +| 2022-06 | METALM | Microsoft | [Language Models are General-Purpose Interfaces](https://arxiv.org/pdf/2206.06336.pdf) | +| 2022-09 | Sparrow | DeepMind | [Improving alignment of dialogue agents via targeted human judgements](https://arxiv.org/pdf/2209.14375.pdf) | +| 2022-10 | Flan-T5/PaLM | Google | [Scaling Instruction-Finetuned Language Models](https://arxiv.org/pdf/2210.11416.pdf) | +| 2022-10 | GLM-130B | Tsinghua | [GLM-130B: An Open Bilingual Pre-trained Model](https://arxiv.org/pdf/2210.02414.pdf) | +| 2022-11 | HELM | Stanford | [Holistic Evaluation of Language Models](https://arxiv.org/pdf/2211.09110.pdf) | +| 2022-11 | BLOOM | BigScience | [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://arxiv.org/pdf/2211.05100.pdf) | +| 2022-11 | Galactica | Meta | [Galactica: A Large Language Model for Science](https://arxiv.org/pdf/2211.09085.pdf) | +| 2022-12 | OPT-IML | Meta | [OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization](https://arxiv.org/pdf/2212.12017) | +| 2023-01 | Flan 2022 Collection | Google | [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://arxiv.org/pdf/2301.13688.pdf) | +| 2023-02 | LLaMA | Meta | [LLaMA: Open and Efficient Foundation Language Models](https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/) | +| 2023-02 | Kosmos-1 | Microsoft | [Language Is Not All You Need: Aligning Perception with Language Models](https://arxiv.org/abs/2302.14045) | +| 2023-03 | LRU | DeepMind | [Resurrecting Recurrent Neural Networks for Long Sequences](https://arxiv.org/abs/2303.06349) | +| 2023-03 | PaLM-E | Google | [PaLM-E: An Embodied Multimodal Language Model](https://palm-e.github.io) | +| 2023-03 | GPT 4 | OpenAI | [GPT-4 Technical Report](https://openai.com/research/gpt-4) | +| 2023-04 | LLaVA | UW–Madison&Microsoft | [Visual Instruction Tuning](https://arxiv.org/abs/2304.08485) | +| 2023-04 | Pythia | EleutherAI et al. | [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://arxiv.org/abs/2304.01373) | +| 2023-05 | Dromedary | CMU et al. | [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://arxiv.org/abs/2305.03047) | +| 2023-05 | PaLM 2 | Google | [PaLM 2 Technical Report](https://ai.google/static/documents/palm2techreport.pdf) | +| 2023-05 | RWKV | Bo Peng | [RWKV: Reinventing RNNs for the Transformer Era](https://arxiv.org/abs/2305.13048) | +| 2023-05 | DPO | Stanford | [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/pdf/2305.18290.pdf) | +| 2023-05 | ToT | Google&Princeton | [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://arxiv.org/pdf/2305.10601.pdf) | +| 2023-07 | LLaMA2 | Meta | [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/pdf/2307.09288.pdf) | +| 2023-10 | Mistral 7B | Mistral | [Mistral 7B](https://arxiv.org/pdf/2310.06825.pdf) | +| 2023-12 | Mamba | CMU&Princeton | [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/pdf/2312.00752) | +| 2024-01 | DeepSeek-v2 | DeepSeek | [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://arxiv.org/abs/2405.04434) | +| 2024-03 | Jamba | AI21 Labs | [Jamba: A Hybrid Transformer-Mamba Language Model](https://arxiv.org/pdf/2403.19887) | +| 2024-05 | Mamba2 | CMU&Princeton | [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://arxiv.org/abs/2405.21060)| +| 2024-05 | Llama3 | Meta | [The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783) | + ## Other Papers @@ -134,11 +138,13 @@ If you're interested in the field of LLM, you may find the above list of milesto ## Open LLM - Meta + - [Llama 3.1-8|70|405B](https://llama.meta.com/) - [Llama 3-8|70B](https://llama.meta.com/llama3/) - [Llama 2-7|13|70B](https://llama.meta.com/llama2/) - [Llama 1-7|13|33|65B](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) - [OPT-1.3|6.7|13|30|66B](https://arxiv.org/abs/2205.01068) - Mistral AI + - [Codestral-7|22B](https://mistral.ai/news/codestral/) - [Mistral-7B](https://mistral.ai/news/announcing-mistral-7b/) - [Mixtral-8x7B](https://mistral.ai/news/mixtral-of-experts/) - [Mixtral-8x22B](https://mistral.ai/news/mixtral-8x22b/) From 41122780085cd7fd7061e262f6c8c982d82fdb72 Mon Sep 17 00:00:00 2001 From: Yineng Zhang Date: Wed, 4 Sep 2024 21:42:18 +1000 Subject: [PATCH 091/117] dos: add SGLang blog https://lmsys.org/blog/2024-07-25-sglang-llama3/ https://lmsys.org/blog/2024-09-04-sglang-v0-3/ --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 8282ea5..3648c40 100644 --- a/README.md +++ b/README.md @@ -249,6 +249,7 @@ If you're interested in the field of LLM, you may find the above list of milesto ## LLM Deployment > Reference: [llm-inference-solutions](https://github.com/mani-kantap/llm-inference-solutions) +- [SGLang](https://github.com/sgl-project/sglang) - SGLang is a fast serving framework for large language models and vision language models. - [vLLM](https://github.com/vllm-project/vllm) - A high-throughput and memory-efficient inference and serving engine for LLMs. - [TGI](https://huggingface.co/docs/text-generation-inference/en/index) - a toolkit for deploying and serving Large Language Models (LLMs). - [exllama](https://github.com/turboderp/exllama) - A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights. From edc83dc7ebe02e89dfd2de2c4a8057841a413e76 Mon Sep 17 00:00:00 2001 From: Shivdeep Singh <98897052+shivdeep-singh-ibm@users.noreply.github.com> Date: Tue, 17 Sep 2024 14:15:12 +0530 Subject: [PATCH 092/117] Update README.md --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index 3648c40..e94d489 100644 --- a/README.md +++ b/README.md @@ -220,6 +220,9 @@ If you're interested in the field of LLM, you may find the above list of milesto ## LLM Data - [LLMDataHub](https://github.com/Zjh-819/LLMDataHub) +## LLM Data Preparation +- [IBM data-prep-kit](https://github.com/IBM/data-prep-kit) - Open-Source Toolkit for Efficient Unstructured Data Processing with Pre-built Modules and Local to Cluster Scalability. + ## LLM Evaluation: - [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) - A framework for few-shot evaluation of language models. - [MixEval](https://github.com/Psycoy/MixEval) - A reliable click-and-go evaluation suite compatible with both open-source and proprietary models, supporting MixEval and other benchmarks. From d7689286b14ccb0c1d3b89cc4c9594e7a784fce9 Mon Sep 17 00:00:00 2001 From: Hannibal046 <38466901+Hannibal046@users.noreply.github.com> Date: Tue, 17 Sep 2024 22:01:44 +0800 Subject: [PATCH 093/117] Update README.md --- README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/README.md b/README.md index e94d489..bb8f779 100644 --- a/README.md +++ b/README.md @@ -219,8 +219,6 @@ If you're interested in the field of LLM, you may find the above list of milesto ## LLM Data - [LLMDataHub](https://github.com/Zjh-819/LLMDataHub) - -## LLM Data Preparation - [IBM data-prep-kit](https://github.com/IBM/data-prep-kit) - Open-Source Toolkit for Efficient Unstructured Data Processing with Pre-built Modules and Local to Cluster Scalability. ## LLM Evaluation: From f94bb1c1be5dca1ac029e3822e90f4a470e34837 Mon Sep 17 00:00:00 2001 From: Shiguang WU Date: Tue, 17 Sep 2024 23:17:23 +0800 Subject: [PATCH 094/117] doc(README): Add a paper list about LM analysis --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index bb8f779..736a1df 100644 --- a/README.md +++ b/README.md @@ -126,6 +126,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [LLM4Opt](https://github.com/FeiLiu36/LLM4Opt) - Applying Large language models (LLMs) for diverse optimization tasks (Opt) is an emerging research area. This is a collection of references and papers of LLM4Opt. +- [awesome-language-model-analysis](https://github.com/Furyton/awesome-language-model-analysis) - This paper list focuses on the theoretical or empirical analysis of language models, e.g., the learning dynamics, expressive capacity, interpretability, generalization, and other interesting topics. ## LLM Leaderboard - [Chatbot Arena Leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard) - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner. From c04482d665e129dcf242fa38f7ae93998742ff90 Mon Sep 17 00:00:00 2001 From: "Lex.Chen" Date: Thu, 19 Sep 2024 09:49:09 +0800 Subject: [PATCH 095/117] Add Qwen2.5 --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index bb8f779..53c8972 100644 --- a/README.md +++ b/README.md @@ -178,6 +178,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [CodeQwen-7B](https://huggingface.co/Qwen/CodeQwen1.5-7B) - [Qwen-VL-7B](https://huggingface.co/Qwen/Qwen-VL) - [Qwen2-0.5|1.5|7|57-MOE|72B](https://qwenlm.github.io/blog/qwen2/) + - [Qwen2.5-0.5B|1.5B|3B|7B|14B|32B|72B](https://qwenlm.github.io/blog/qwen2.5/) - 01-ai - [Yi-34B](https://huggingface.co/collections/01-ai/yi-2023-11-663f3f19119ff712e176720f) - [Yi1.5-6|9|34B](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) From 1dda2f15be35a22025e70e48e69142fe7c1b650a Mon Sep 17 00:00:00 2001 From: "Lex.Chen" Date: Thu, 19 Sep 2024 13:41:12 +0800 Subject: [PATCH 096/117] Complete Qwen series information 1. Resort content 2. Add Qwen2.5-Coder Qwen2.5-Math Qwen2-Math Qwen2-VL Qwen2-Audio 3. Adjust some description URLs --- README.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 4ef02ad..cd1afe7 100644 --- a/README.md +++ b/README.md @@ -174,12 +174,17 @@ If you're interested in the field of LLM, you may find the above list of milesto - [DeepSeek-v2-236B-MoE](https://arxiv.org/abs/2405.04434) - [DeepSeek-Coder-v2-16|236B-MOE](https://github.com/deepseek-ai/DeepSeek-Coder-V2) - Alibaba - - [Qwen-1.8|7|14|72B](https://huggingface.co/collections/Qwen/qwen-65c0e50c3f1ab89cb8704144) - - [Qwen1.5-1.8|4|7|14|32|72|110B](https://huggingface.co/collections/Qwen/qwen15-65c0a2f577b1ecb76d786524) - - [CodeQwen-7B](https://huggingface.co/Qwen/CodeQwen1.5-7B) - - [Qwen-VL-7B](https://huggingface.co/Qwen/Qwen-VL) - - [Qwen2-0.5|1.5|7|57-MOE|72B](https://qwenlm.github.io/blog/qwen2/) + - [Qwen-1.8B|7B|14B|72B](https://huggingface.co/collections/Qwen/qwen-65c0e50c3f1ab89cb8704144) + - [Qwen1.5-0.5B|1.8B|4B|7B|14B|32B|72B|110B|MoE-A2.7B](https://qwenlm.github.io/blog/qwen1.5/) + - [Qwen2-0.5B|1.5B|7B|57B-A14B-MoE|72B](https://qwenlm.github.io/blog/qwen2) - [Qwen2.5-0.5B|1.5B|3B|7B|14B|32B|72B](https://qwenlm.github.io/blog/qwen2.5/) + - [CodeQwen1.5-7B](https://qwenlm.github.io/blog/codeqwen1.5/) + - [Qwen2.5-Coder-1.5B|7B|32B](https://qwenlm.github.io/blog/qwen2.5-coder/) + - [Qwen2-Math-1.5B|7B|72B](https://qwenlm.github.io/blog/qwen2-math/) + - [Qwen2.5-Math-1.5B|7B|72B](https://qwenlm.github.io/blog/qwen2.5-math/) + - [Qwen-VL-7B](https://huggingface.co/Qwen/Qwen-VL) + - [Qwen2-VL-2B|7B|72B](https://qwenlm.github.io/blog/qwen2-vl/) + - [Qwen2-Audio-7B](https://qwenlm.github.io/blog/qwen2-audio/) - 01-ai - [Yi-34B](https://huggingface.co/collections/01-ai/yi-2023-11-663f3f19119ff712e176720f) - [Yi1.5-6|9|34B](https://huggingface.co/collections/01-ai/yi-15-2024-05-663f3ecab5f815a3eaca7ca8) From 7396959aab74e4a046b00e636b1314155c0bd99b Mon Sep 17 00:00:00 2001 From: Abby Morgan <86856445+anmorgan24@users.noreply.github.com> Date: Fri, 20 Sep 2024 12:16:05 -0400 Subject: [PATCH 097/117] Add Opik Comet recently sunset CometLLM and in its place launched Opik, a tool with most of the same capabilities as the old CometLLM but with a whole host of additional features and capabilities. This PR updates the readme accordingly. Signed-off-by: Abby Morgan abigailm@comet.com --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index cd1afe7..d20632d 100644 --- a/README.md +++ b/README.md @@ -283,7 +283,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [Serge](https://github.com/serge-chat/serge) - a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted! - [Langroid](https://github.com/langroid/langroid) - Harness LLMs with Multi-Agent Programming - [Embedchain](https://github.com/embedchain/embedchain) - Framework to create ChatGPT like bots over your dataset. -- [CometLLM](https://github.com/comet-ml/comet-llm) - A 100% opensource LLMOps platform to log, manage, and visualize your LLM prompts and chains. Track prompt templates, prompt variables, prompt duration, token usage, and other metadata. Score prompt outputs and visualize chat history all within a single UI. +- [Opik](https://github.com/comet-ml/opik) - Confidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle. - [IntelliServer](https://github.com/intelligentnode/IntelliServer) - simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models. - [OpenLLM](https://github.com/bentoml/OpenLLM) - Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at [BentoML](https://bentoml.com/) for LLMs-based applications. - [DeepSpeed-Mii](https://github.com/microsoft/DeepSpeed-MII) - MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed. From 188eaf18cc44a8a1105daa63cc5576c35d3a547b Mon Sep 17 00:00:00 2001 From: JIMMY ZHAO Date: Mon, 30 Sep 2024 09:18:45 -0400 Subject: [PATCH 098/117] refine the leaderboards --- README.md | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index d20632d..8337ee8 100644 --- a/README.md +++ b/README.md @@ -129,13 +129,34 @@ If you're interested in the field of LLM, you may find the above list of milesto - [awesome-language-model-analysis](https://github.com/Furyton/awesome-language-model-analysis) - This paper list focuses on the theoretical or empirical analysis of language models, e.g., the learning dynamics, expressive capacity, interpretability, generalization, and other interesting topics. ## LLM Leaderboard +- [AlpacaEval Leaderboard](https://tatsu-lab.github.io/alpaca_eval/) - An Automatic Evaluator for Instruction-following Language Models using Nous benchmark suite. +- [BeHonest](https://tatsu-lab.github.io/alpaca_eval/) - A pioneering benchmark specifically designed to assess honesty in LLMs comprehensively. +- [Berkeley Function-Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html) - evaluates LLM's ability to call external functions/tools. +- [Chinese Large Model Leaderboard](https://github.com/jeinlee1991/chinese-llm-benchmark) - an expert-driven benchmark for Chineses LLMs. - [Chatbot Arena Leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard) - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner. +- [CompassBench Large Language Model Leaderboard +](https://rank.opencompass.org.cn/leaderboard-llm) - OpenCompass is an LLM evaluation platform, supporting a wide range of models (InternLM2, GPT-4, LLaMa 2, Qwen, GLM, Claude, etc) over 100+ datasets. +- [CompMix](https://qa.mpi-inf.mpg.de/compmix) - a benchmark evaluating QA methods that operate over a mixture of heterogeneous input sources (KB, text, tables, infoboxes). +- [DreamBench++](https://dreambenchplus.github.io/#leaderboard) - a benchmark for evaluating the performance of large language models (LLMs) in various tasks related to both textual and visual imagination. +- [FELM](https://hkust-nlp.github.io/felm) - a meta-benchmark that evaluates how well factuality evaluators assess the outputs of large language models (LLMs). +- [InfiBench](https://infi-coder.github.io/infibench) - a benchmark designed to evaluate large language models (LLMs) specifically in their ability to answer real-world coding-related questions. +- [LawBench](https://lawbench.opencompass.org.cn/leaderboard) - a benchmark designed to evaluate large language models in the legal domain. +- [LLMEval](http://llmeval.com) - focuses on understanding how these models perform in various scenarios and analyzing results from an interpretability perspective. +- [M3CoT](https://lightchen233.github.io/m3cot.github.io/leaderboard.html) - a benchmark that evaluates large language models on a variety of multimodal reasoning tasks, including language, natural and social sciences, physical and social commonsense, temporal reasoning, algebra, and geometry. +- [MathEval](https://matheval.ai) - a comprehensive benchmarking platform designed to evaluate large models' mathematical abilities across 20 fields and nearly 30,000 math problems. - [MixEval Leaderboard](https://mixeval.github.io/#leaderboard) - a ground-truth-based dynamic benchmark derived from off-the-shelf benchmark mixtures, which evaluates LLMs with a highly capable model ranking (i.e., 0.96 correlation with Chatbot Arena) while running locally and quickly (6% the time and cost of running MMLU). -- [AlpacaEval Leaderboard](https://tatsu-lab.github.io/alpaca_eval/) - An Automatic Evaluator for Instruction-following Language Models - using Nous benchmark suite. -- [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) - aims to track, rank and evaluate LLMs and chatbots as they are released. -- [OpenCompass 2.0 LLM Leaderboard](https://rank.opencompass.org.cn/leaderboard-llm-v2) - OpenCompass is an LLM evaluation platform, supporting a wide range of models (InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets. -- [Berkeley Function-Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html) - evaluates LLM's ability to call external functions / tools +- [MMedBench](https://henrychur.github.io/MultilingualMedQA) - a benchmark that evaluates large language models' ability to answer medical questions across multiple languages. +- [MMToM-QA](https://chuanyangjin.com/mmtom-qa-leaderboard) - a multimodal question-answering benchmark designed to evaluate AI models' cognitive ability to understand human beliefs and goals. +- [OlympicArena](https://gair-nlp.github.io/OlympicArena/#leaderboard) - a benchmark for evaluating AI models across multiple academic disciplines like math, physics, chemistry, biology, and more. +- [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) - aims to track, rank, and evaluate LLMs and chatbots as they are released. +- [PubMedQA](https://pubmedqa.github.io) - a biomedical question-answering benchmark designed for answering research-related questions using PubMed abstracts. +- [SciBench](https://scibench-ucla.github.io/#leaderboard) - benchmark designed to evaluate large language models (LLMs) on solving complex, college-level scientific problems from domains like chemistry, physics, and mathematics. +- [SuperBench](https://fm.ai.tsinghua.edu.cn/superbench/#/leaderboard) - a benchmark platform designed for evaluating large language models (LLMs) on a range of tasks, particularly focusing on their performance in different aspects such as natural language understanding, reasoning, and generalization. +- [SuperLim](https://lab.kb.se/leaderboard/results) - a Swedish language understanding benchmark that evaluates natural language processing (NLP) models on various tasks such as argumentation analysis, semantic similarity, and textual entailment. +- [TAT-DQA](https://nextplusplus.github.io/TAT-DQA) - a large-scale Document Visual Question Answering (VQA) dataset designed for complex document understanding, particularly in financial reports. +- [TAT-QA](https://nextplusplus.github.io/TAT-QA) - a large-scale question-answering benchmark focused on real-world financial data, integrating both tabular and textual information. +- [We-Math](https://we-math.github.io/#leaderboard) - a benchmark that evaluates large multimodal models (LMMs) on their ability to perform human-like mathematical reasoning. +- [WHOOPS!](https://whoops-benchmark.github.io) - a benchmark dataset aimed at testing AI's ability to reason about visual commonsense through images that defy normal expectations. ## Open LLM - Meta From 95ecbc0fa86539655a7a2bff1d537ac46ec3a05c Mon Sep 17 00:00:00 2001 From: JIMMY ZHAO Date: Wed, 2 Oct 2024 00:29:47 -0400 Subject: [PATCH 099/117] add new --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 8337ee8..fefd4e4 100644 --- a/README.md +++ b/README.md @@ -155,8 +155,9 @@ If you're interested in the field of LLM, you may find the above list of milesto - [SuperLim](https://lab.kb.se/leaderboard/results) - a Swedish language understanding benchmark that evaluates natural language processing (NLP) models on various tasks such as argumentation analysis, semantic similarity, and textual entailment. - [TAT-DQA](https://nextplusplus.github.io/TAT-DQA) - a large-scale Document Visual Question Answering (VQA) dataset designed for complex document understanding, particularly in financial reports. - [TAT-QA](https://nextplusplus.github.io/TAT-QA) - a large-scale question-answering benchmark focused on real-world financial data, integrating both tabular and textual information. +- [VisualWebArena](https://jykoh.com/vwa) - a benchmark designed to assess the performance of multimodal web agents on realistic visually grounded tasks. - [We-Math](https://we-math.github.io/#leaderboard) - a benchmark that evaluates large multimodal models (LMMs) on their ability to perform human-like mathematical reasoning. -- [WHOOPS!](https://whoops-benchmark.github.io) - a benchmark dataset aimed at testing AI's ability to reason about visual commonsense through images that defy normal expectations. +- [WHOOPS!](https://whoops-benchmark.github.io) - a benchmark dataset testing AI's ability to reason about visual commonsense through images that defy normal expectations. ## Open LLM - Meta From 49639c7d46b1c0747985fbf750370adc6a19a97f Mon Sep 17 00:00:00 2001 From: Hannibal046 <38466901+Hannibal046@users.noreply.github.com> Date: Wed, 2 Oct 2024 15:23:19 +0800 Subject: [PATCH 100/117] Update README.md reorder the leaderboard --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index fefd4e4..92a80fe 100644 --- a/README.md +++ b/README.md @@ -129,11 +129,12 @@ If you're interested in the field of LLM, you may find the above list of milesto - [awesome-language-model-analysis](https://github.com/Furyton/awesome-language-model-analysis) - This paper list focuses on the theoretical or empirical analysis of language models, e.g., the learning dynamics, expressive capacity, interpretability, generalization, and other interesting topics. ## LLM Leaderboard +- [Chatbot Arena Leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard) - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner. +- [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) - aims to track, rank, and evaluate LLMs and chatbots as they are released. - [AlpacaEval Leaderboard](https://tatsu-lab.github.io/alpaca_eval/) - An Automatic Evaluator for Instruction-following Language Models using Nous benchmark suite. - [BeHonest](https://tatsu-lab.github.io/alpaca_eval/) - A pioneering benchmark specifically designed to assess honesty in LLMs comprehensively. - [Berkeley Function-Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html) - evaluates LLM's ability to call external functions/tools. - [Chinese Large Model Leaderboard](https://github.com/jeinlee1991/chinese-llm-benchmark) - an expert-driven benchmark for Chineses LLMs. -- [Chatbot Arena Leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard) - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner. - [CompassBench Large Language Model Leaderboard ](https://rank.opencompass.org.cn/leaderboard-llm) - OpenCompass is an LLM evaluation platform, supporting a wide range of models (InternLM2, GPT-4, LLaMa 2, Qwen, GLM, Claude, etc) over 100+ datasets. - [CompMix](https://qa.mpi-inf.mpg.de/compmix) - a benchmark evaluating QA methods that operate over a mixture of heterogeneous input sources (KB, text, tables, infoboxes). @@ -148,7 +149,6 @@ If you're interested in the field of LLM, you may find the above list of milesto - [MMedBench](https://henrychur.github.io/MultilingualMedQA) - a benchmark that evaluates large language models' ability to answer medical questions across multiple languages. - [MMToM-QA](https://chuanyangjin.com/mmtom-qa-leaderboard) - a multimodal question-answering benchmark designed to evaluate AI models' cognitive ability to understand human beliefs and goals. - [OlympicArena](https://gair-nlp.github.io/OlympicArena/#leaderboard) - a benchmark for evaluating AI models across multiple academic disciplines like math, physics, chemistry, biology, and more. -- [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) - aims to track, rank, and evaluate LLMs and chatbots as they are released. - [PubMedQA](https://pubmedqa.github.io) - a biomedical question-answering benchmark designed for answering research-related questions using PubMed abstracts. - [SciBench](https://scibench-ucla.github.io/#leaderboard) - benchmark designed to evaluate large language models (LLMs) on solving complex, college-level scientific problems from domains like chemistry, physics, and mathematics. - [SuperBench](https://fm.ai.tsinghua.edu.cn/superbench/#/leaderboard) - a benchmark platform designed for evaluating large language models (LLMs) on a range of tasks, particularly focusing on their performance in different aspects such as natural language understanding, reasoning, and generalization. From b4fb0df085ff75262c58cf1dcc5bc14b94a899a7 Mon Sep 17 00:00:00 2001 From: JIMMY ZHAO Date: Wed, 2 Oct 2024 08:53:11 -0400 Subject: [PATCH 101/117] refined typos and linkages --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 92a80fe..92e30a4 100644 --- a/README.md +++ b/README.md @@ -130,9 +130,9 @@ If you're interested in the field of LLM, you may find the above list of milesto ## LLM Leaderboard - [Chatbot Arena Leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard) - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner. -- [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) - aims to track, rank, and evaluate LLMs and chatbots as they are released. +- [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) - aims to track, rank, and evaluate LLMs and chatbots as they are released. - [AlpacaEval Leaderboard](https://tatsu-lab.github.io/alpaca_eval/) - An Automatic Evaluator for Instruction-following Language Models using Nous benchmark suite. -- [BeHonest](https://tatsu-lab.github.io/alpaca_eval/) - A pioneering benchmark specifically designed to assess honesty in LLMs comprehensively. +- [BeHonest](https://gair-nlp.github.io/BeHonest/#leaderboard) - A pioneering benchmark specifically designed to assess honesty in LLMs comprehensively. - [Berkeley Function-Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html) - evaluates LLM's ability to call external functions/tools. - [Chinese Large Model Leaderboard](https://github.com/jeinlee1991/chinese-llm-benchmark) - an expert-driven benchmark for Chineses LLMs. - [CompassBench Large Language Model Leaderboard @@ -145,7 +145,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [LLMEval](http://llmeval.com) - focuses on understanding how these models perform in various scenarios and analyzing results from an interpretability perspective. - [M3CoT](https://lightchen233.github.io/m3cot.github.io/leaderboard.html) - a benchmark that evaluates large language models on a variety of multimodal reasoning tasks, including language, natural and social sciences, physical and social commonsense, temporal reasoning, algebra, and geometry. - [MathEval](https://matheval.ai) - a comprehensive benchmarking platform designed to evaluate large models' mathematical abilities across 20 fields and nearly 30,000 math problems. -- [MixEval Leaderboard](https://mixeval.github.io/#leaderboard) - a ground-truth-based dynamic benchmark derived from off-the-shelf benchmark mixtures, which evaluates LLMs with a highly capable model ranking (i.e., 0.96 correlation with Chatbot Arena) while running locally and quickly (6% the time and cost of running MMLU). +- [MixEval](https://mixeval.github.io/#leaderboard) - a ground-truth-based dynamic benchmark derived from off-the-shelf benchmark mixtures, which evaluates LLMs with a highly capable model ranking (i.e., 0.96 correlation with Chatbot Arena) while running locally and quickly (6% the time and cost of running MMLU). - [MMedBench](https://henrychur.github.io/MultilingualMedQA) - a benchmark that evaluates large language models' ability to answer medical questions across multiple languages. - [MMToM-QA](https://chuanyangjin.com/mmtom-qa-leaderboard) - a multimodal question-answering benchmark designed to evaluate AI models' cognitive ability to understand human beliefs and goals. - [OlympicArena](https://gair-nlp.github.io/OlympicArena/#leaderboard) - a benchmark for evaluating AI models across multiple academic disciplines like math, physics, chemistry, biology, and more. From 7c11557753058a60373b42ca594c84a37bfb3803 Mon Sep 17 00:00:00 2001 From: JIMMY ZHAO Date: Wed, 2 Oct 2024 08:59:55 -0400 Subject: [PATCH 102/117] Update README.md --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 92e30a4..03262a5 100644 --- a/README.md +++ b/README.md @@ -135,8 +135,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [BeHonest](https://gair-nlp.github.io/BeHonest/#leaderboard) - A pioneering benchmark specifically designed to assess honesty in LLMs comprehensively. - [Berkeley Function-Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html) - evaluates LLM's ability to call external functions/tools. - [Chinese Large Model Leaderboard](https://github.com/jeinlee1991/chinese-llm-benchmark) - an expert-driven benchmark for Chineses LLMs. -- [CompassBench Large Language Model Leaderboard -](https://rank.opencompass.org.cn/leaderboard-llm) - OpenCompass is an LLM evaluation platform, supporting a wide range of models (InternLM2, GPT-4, LLaMa 2, Qwen, GLM, Claude, etc) over 100+ datasets. +- [CompassRank](https://rank.opencompass.org.cn) - CompassRank is dedicated to exploring the most advanced language and visual models, offering a comprehensive, objective, and neutral evaluation reference for the industry and research. - [CompMix](https://qa.mpi-inf.mpg.de/compmix) - a benchmark evaluating QA methods that operate over a mixture of heterogeneous input sources (KB, text, tables, infoboxes). - [DreamBench++](https://dreambenchplus.github.io/#leaderboard) - a benchmark for evaluating the performance of large language models (LLMs) in various tasks related to both textual and visual imagination. - [FELM](https://hkust-nlp.github.io/felm) - a meta-benchmark that evaluates how well factuality evaluators assess the outputs of large language models (LLMs). From b29bb972fd280ae479f1245105d15dab3a6d912e Mon Sep 17 00:00:00 2001 From: JIMMY ZHAO Date: Sat, 5 Oct 2024 11:15:24 -0400 Subject: [PATCH 103/117] Update README.md --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 03262a5..6d350c3 100644 --- a/README.md +++ b/README.md @@ -131,7 +131,8 @@ If you're interested in the field of LLM, you may find the above list of milesto ## LLM Leaderboard - [Chatbot Arena Leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard) - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner. - [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) - aims to track, rank, and evaluate LLMs and chatbots as they are released. -- [AlpacaEval Leaderboard](https://tatsu-lab.github.io/alpaca_eval/) - An Automatic Evaluator for Instruction-following Language Models using Nous benchmark suite. +- [ACLUE](https://tatsu-lab.github.io/alpaca_eval/) - an evaluation benchmark focused on ancient Chinese language comprehension. +- [AlpacaEval](https://tatsu-lab.github.io/alpaca_eval/) - An Automatic Evaluator for Instruction-following Language Models using Nous benchmark suite. - [BeHonest](https://gair-nlp.github.io/BeHonest/#leaderboard) - A pioneering benchmark specifically designed to assess honesty in LLMs comprehensively. - [Berkeley Function-Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html) - evaluates LLM's ability to call external functions/tools. - [Chinese Large Model Leaderboard](https://github.com/jeinlee1991/chinese-llm-benchmark) - an expert-driven benchmark for Chineses LLMs. From 032129e671f9749edfc6042421792109f1325dda Mon Sep 17 00:00:00 2001 From: JIMMY ZHAO Date: Sat, 5 Oct 2024 11:17:49 -0400 Subject: [PATCH 104/117] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6d350c3..fd34d52 100644 --- a/README.md +++ b/README.md @@ -131,7 +131,7 @@ If you're interested in the field of LLM, you may find the above list of milesto ## LLM Leaderboard - [Chatbot Arena Leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard) - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner. - [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) - aims to track, rank, and evaluate LLMs and chatbots as they are released. -- [ACLUE](https://tatsu-lab.github.io/alpaca_eval/) - an evaluation benchmark focused on ancient Chinese language comprehension. +- [ACLUE](https://github.com/isen-zhang/ACLUE) - an evaluation benchmark focused on ancient Chinese language comprehension. - [AlpacaEval](https://tatsu-lab.github.io/alpaca_eval/) - An Automatic Evaluator for Instruction-following Language Models using Nous benchmark suite. - [BeHonest](https://gair-nlp.github.io/BeHonest/#leaderboard) - A pioneering benchmark specifically designed to assess honesty in LLMs comprehensively. - [Berkeley Function-Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html) - evaluates LLM's ability to call external functions/tools. From e3a676fc0a9aa810a8d9395cdcd2856126b2e498 Mon Sep 17 00:00:00 2001 From: Xizao Wang Date: Mon, 14 Oct 2024 14:50:40 +0800 Subject: [PATCH 105/117] Add book "Hands-On Large Language Models" --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index fd34d52..57e92a8 100644 --- a/README.md +++ b/README.md @@ -379,6 +379,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [Generative AI with LangChain: Build large language model (LLM) apps with Python, ChatGPT, and other LLMs](https://amzn.to/3GUlRng) - it comes with a [GitHub repository](https://github.com/benman1/generative_ai_with_langchain) that showcases a lot of the functionality - [Build a Large Language Model (From Scratch)](https://www.manning.com/books/build-a-large-language-model-from-scratch) - A guide to building your own working LLM. - [BUILD GPT: HOW AI WORKS](https://www.amazon.com/dp/9152799727?ref_=cm_sw_r_cp_ud_dp_W3ZHCD6QWM3DPPC0ARTT_1) - explains how to code a Generative Pre-trained Transformer, or GPT, from scratch. +- [Hands-On Large Language Models: Language Understanding and Generation](https://www.llm-book.com/) - Explore the world of Large Language Models with over 275 custom made figures in this illustrated guide! ## Great thoughts about LLM - [Why did all of the public reproduction of GPT-3 fail?](https://jingfengyang.github.io/gpt) From c60c6f5152cdd2a062b130f810592fb79e8a630d Mon Sep 17 00:00:00 2001 From: fzyzcjy <5236035+fzyzcjy@users.noreply.github.com> Date: Mon, 21 Oct 2024 15:30:28 +0800 Subject: [PATCH 106/117] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 57e92a8..d59b6bd 100644 --- a/README.md +++ b/README.md @@ -161,6 +161,7 @@ If you're interested in the field of LLM, you may find the above list of milesto ## Open LLM - Meta + - [Llama 3.2-1|3B](https://llama.meta.com/) - [Llama 3.1-8|70|405B](https://llama.meta.com/) - [Llama 3-8|70B](https://llama.meta.com/llama3/) - [Llama 2-7|13|70B](https://llama.meta.com/llama2/) From f6e84a056ec38b773e04dc2616c5066530cd55fb Mon Sep 17 00:00:00 2001 From: fzyzcjy <5236035+fzyzcjy@users.noreply.github.com> Date: Wed, 23 Oct 2024 10:24:54 +0800 Subject: [PATCH 107/117] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d59b6bd..8806ba0 100644 --- a/README.md +++ b/README.md @@ -161,7 +161,7 @@ If you're interested in the field of LLM, you may find the above list of milesto ## Open LLM - Meta - - [Llama 3.2-1|3B](https://llama.meta.com/) + - [Llama 3.2-1|3|11|90B](https://llama.meta.com/) - [Llama 3.1-8|70|405B](https://llama.meta.com/) - [Llama 3-8|70B](https://llama.meta.com/llama3/) - [Llama 2-7|13|70B](https://llama.meta.com/llama2/) From 7bb22840702fbe6c63cd50d4bb523cf383383958 Mon Sep 17 00:00:00 2001 From: tianbu Date: Tue, 29 Oct 2024 15:21:28 +0800 Subject: [PATCH 108/117] [LLM Inference] MNN-LLM: llm on device inference framework --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 8806ba0..4d201ed 100644 --- a/README.md +++ b/README.md @@ -325,6 +325,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [talkd.ai dialog](https://github.com/talkdai/dialog) - Simple API for deploying any RAG or LLM that you want adding plugins. - [Wllama](https://github.com/ngxson/wllama) - WebAssembly binding for llama.cpp - Enabling in-browser LLM inference - [GPUStack](https://github.com/gpustack/gpustack) - An open-source GPU cluster manager for running LLMs +- [MNN-LLM](https://github.com/alibaba/MNN) -- A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT) ## LLM Applications - [AdalFlow](https://github.com/SylphAI-Inc/AdalFlow) - AdalFlow: The PyTorch library for LLM applications. From 1c0cead77f01b083a28543a7f9ccc573867749db Mon Sep 17 00:00:00 2001 From: Li Yin Date: Mon, 4 Nov 2024 12:50:38 -0800 Subject: [PATCH 109/117] Update README.md Update the AdalFlow desc --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4d201ed..1895b8d 100644 --- a/README.md +++ b/README.md @@ -328,7 +328,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [MNN-LLM](https://github.com/alibaba/MNN) -- A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT) ## LLM Applications -- [AdalFlow](https://github.com/SylphAI-Inc/AdalFlow) - AdalFlow: The PyTorch library for LLM applications. +- [AdalFlow](https://github.com/SylphAI-Inc/AdalFlow) - AdalFlow: The library to build&auto-optimize LLM applications. - [dspy](https://github.com/stanfordnlp/dspy) - DSPy: The framework for programming—not prompting—foundation models. - [YiVal](https://github.com/YiVal/YiVal) — Evaluate and Evolve: YiVal is an open-source GenAI-Ops tool for tuning and evaluating prompts, configurations, and model parameters using customizable datasets, evaluation methods, and improvement strategies. - [Guidance](https://github.com/microsoft/guidance) — A handy looking Python library from Microsoft that uses Handlebars templating to interleave generation, prompting, and logical control. From 5daef1bd044836b770b65c20605d72f67303c097 Mon Sep 17 00:00:00 2001 From: Si-si Qu <39686395+sallyqus@users.noreply.github.com> Date: Wed, 20 Nov 2024 04:07:31 +0000 Subject: [PATCH 110/117] Add CAMEL to LLM deployment --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 1895b8d..673b828 100644 --- a/README.md +++ b/README.md @@ -326,6 +326,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [Wllama](https://github.com/ngxson/wllama) - WebAssembly binding for llama.cpp - Enabling in-browser LLM inference - [GPUStack](https://github.com/gpustack/gpustack) - An open-source GPU cluster manager for running LLMs - [MNN-LLM](https://github.com/alibaba/MNN) -- A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT) +- [CAMEL](https://www.camel-ai.org/) - First LLM Multi-agent framework. ## LLM Applications - [AdalFlow](https://github.com/SylphAI-Inc/AdalFlow) - AdalFlow: The library to build&auto-optimize LLM applications. From a6ebb3a3bfb50e1ab28cb199d3793c4c284a1015 Mon Sep 17 00:00:00 2001 From: Santosh Bhavani Date: Wed, 20 Nov 2024 15:32:13 -0800 Subject: [PATCH 111/117] Update README.md 1. Removed alpa - no longer actively maintained 2. Updated URL for maxtext 3. Added NeMo Framework --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 673b828..cf7f98e 100644 --- a/README.md +++ b/README.md @@ -268,12 +268,12 @@ If you're interested in the field of LLM, you may find the above list of milesto - [Megatron-DeepSpeed](https://github.com/microsoft/Megatron-DeepSpeed) - DeepSpeed version of NVIDIA's Megatron-LM that adds additional support for several features such as MoE model training, Curriculum Learning, 3D Parallelism, and others. - [torchtune](https://github.com/pytorch/torchtune) - A Native-PyTorch Library for LLM Fine-tuning. - [torchtitan](https://github.com/pytorch/torchtitan) - A native PyTorch Library for large model training. +- [NeMo Framework](https://github.com/NVIDIA/NeMo) - Generative AI framework built for researchers and PyTorch developers working on Large Language Models (LLMs), Multimodal Models (MMs), Automatic Speech Recognition (ASR), Text to Speech (TTS), and Computer Vision (CV) domains. - [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) - Ongoing research training transformer models at scale. - [Colossal-AI](https://github.com/hpcaitech/ColossalAI) - Making large AI models cheaper, faster, and more accessible. - [BMTrain](https://github.com/OpenBMB/BMTrain) - Efficient Training for Big Models. - [Mesh Tensorflow](https://github.com/tensorflow/mesh) - Mesh TensorFlow: Model Parallelism Made Easier. -- [maxtext](https://github.com/google/maxtext) - A simple, performant and scalable Jax LLM! -- [Alpa](https://alpa.ai/index.html) - Alpa is a system for training and serving large-scale neural networks. +- [maxtext](https://github.com/AI-Hypercomputer/maxtext) - A simple, performant and scalable Jax LLM! - [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) - An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library. From d643877353642e6a360a6ad63a66ea0b5677ef50 Mon Sep 17 00:00:00 2001 From: Yuki Watanabe <31463517+B-Step62@users.noreply.github.com> Date: Sun, 1 Dec 2024 19:06:36 +0900 Subject: [PATCH 112/117] Add MLflow to LLM Applications section Signed-off-by: B-Step62 --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index cf7f98e..9f4c5ca 100644 --- a/README.md +++ b/README.md @@ -331,6 +331,7 @@ If you're interested in the field of LLM, you may find the above list of milesto ## LLM Applications - [AdalFlow](https://github.com/SylphAI-Inc/AdalFlow) - AdalFlow: The library to build&auto-optimize LLM applications. - [dspy](https://github.com/stanfordnlp/dspy) - DSPy: The framework for programming—not prompting—foundation models. +- [MLflow](https://mlflow.org/) - MLflow: An open-source framework for the end-to-end machine learning lifecycle, helping developers track experiments, evaluate models/prompts, deploy models, and add observability with tracing. - [YiVal](https://github.com/YiVal/YiVal) — Evaluate and Evolve: YiVal is an open-source GenAI-Ops tool for tuning and evaluating prompts, configurations, and model parameters using customizable datasets, evaluation methods, and improvement strategies. - [Guidance](https://github.com/microsoft/guidance) — A handy looking Python library from Microsoft that uses Handlebars templating to interleave generation, prompting, and logical control. - [LangChain](https://github.com/hwchase17/langchain) — A popular Python/JavaScript library for chaining sequences of language model prompts. From 10f9379b913da7f17d413de0b1b55a4be37f5817 Mon Sep 17 00:00:00 2001 From: "Jeffrey (Dongkyu) Kim" Date: Sat, 7 Dec 2024 16:35:58 +0900 Subject: [PATCH 113/117] Add AutoRAG to the README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 9f4c5ca..56c42c4 100644 --- a/README.md +++ b/README.md @@ -362,6 +362,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [LazyLLM](https://github.com/LazyAGI/LazyLLM) - An open-source LLM app for building multi-agent LLMs applications in an easy and lazy way, supports model deployment and fine-tuning. - [MemFree](https://github.com/memfreeme/memfree) - Open Source Hybrid AI Search Engine, Instantly Get Accurate Answers from the Internet, Bookmarks, Notes, and Docs. Support One-Click Deployment - [unslothai](https://github.com/unslothai/unsloth) - A framework that specializes in efficient fine-tuning. On its GitHub page, you can find ready-to-use fine-tuning templates for various LLMs, allowing you to easily train your own data for free on the Google Colab cloud. +- [AutoRAG](https://github.com/Marker-Inc-Korea/AutoRAG) - Open source AutoML tool for RAG. Optimize the RAG answer quality automatically. From generation evaluation datset to deploying optimized RAG pipeline. ## LLM Tutorials and Courses - [llm-course](https://github.com/mlabonne/llm-course) - Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks. From 33349aa174b6a7b0cfa0e279f1bd3eaf886b087b Mon Sep 17 00:00:00 2001 From: Vanshika Gupta <67424390+vg11072001@users.noreply.github.com> Date: Thu, 19 Dec 2024 20:46:19 +0530 Subject: [PATCH 114/117] Transformer Engine lib added on llm training --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 56c42c4..3103328 100644 --- a/README.md +++ b/README.md @@ -275,7 +275,7 @@ If you're interested in the field of LLM, you may find the above list of milesto - [Mesh Tensorflow](https://github.com/tensorflow/mesh) - Mesh TensorFlow: Model Parallelism Made Easier. - [maxtext](https://github.com/AI-Hypercomputer/maxtext) - A simple, performant and scalable Jax LLM! - [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) - An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library. - +- [Transformer Engine](https://github.com/NVIDIA/TransformerEngine) - A library for accelerating Transformer model training on NVIDIA GPUs. ## LLM Deployment From 96be6a59b627596086a5fa199d571a17162d5f55 Mon Sep 17 00:00:00 2001 From: chengxin Date: Mon, 23 Dec 2024 09:55:11 +0800 Subject: [PATCH 115/117] update --- README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 3103328..c31bbd3 100644 --- a/README.md +++ b/README.md @@ -7,12 +7,12 @@ ## Trending LLM Projects -- [Deep-Live-Cam](https://github.com/hacksider/Deep-Live-Cam) - real time face swap and one-click video deepfake with only a single image (uncensored). -- [MiniCPM-V 2.6](https://github.com/OpenBMB/MiniCPM-V) - A GPT-4V Level MLLM for Single Image, Multi Image and Video on Your Phone -- [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS) - 1 min voice data can also be used to train a good TTS model! (few shot voice cloning). +- [OpenAI o3 preview](https://openai.com/12-days/) - AGI, maybe? +- [Qwen2.5 Technical Report](https://huggingface.co/papers/2412.15115) - This report introduces Qwen2.5, a comprehensive series of large language models (LLMs) designed to meet diverse needs. +- [Genesis](https://github.com/Genesis-Embodied-AI/Genesis) - A generative world for general-purpose robotics & embodied AI learning. +- [ModernBERT](https://github.com/AnswerDotAI/ModernBERT) - Bringing BERT into modernity via both architecture changes and scaling. ## Table of Content - - [Awesome-LLM ](#awesome-llm-) - [Milestone Papers](#milestone-papers) - [Other Papers](#other-papers) @@ -85,10 +85,9 @@ | 2023-10 | Mistral 7B | Mistral | [Mistral 7B](https://arxiv.org/pdf/2310.06825.pdf) | | 2023-12 | Mamba | CMU&Princeton | [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/pdf/2312.00752) | | 2024-01 | DeepSeek-v2 | DeepSeek | [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://arxiv.org/abs/2405.04434) | -| 2024-03 | Jamba | AI21 Labs | [Jamba: A Hybrid Transformer-Mamba Language Model](https://arxiv.org/pdf/2403.19887) | | 2024-05 | Mamba2 | CMU&Princeton | [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://arxiv.org/abs/2405.21060)| | 2024-05 | Llama3 | Meta | [The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783) | - +| 2024-12 | Qwen2.5 | Alibaba | [Qwen2.5 Technical Report](https://arxiv.org/abs/2412.15115) | ## Other Papers @@ -130,9 +129,10 @@ If you're interested in the field of LLM, you may find the above list of milesto ## LLM Leaderboard - [Chatbot Arena Leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard) - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner. +- [LiveBench](https://livebench.ai/#/) - A Challenging, Contamination-Free LLM Benchmark. - [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) - aims to track, rank, and evaluate LLMs and chatbots as they are released. -- [ACLUE](https://github.com/isen-zhang/ACLUE) - an evaluation benchmark focused on ancient Chinese language comprehension. - [AlpacaEval](https://tatsu-lab.github.io/alpaca_eval/) - An Automatic Evaluator for Instruction-following Language Models using Nous benchmark suite. +- [ACLUE](https://github.com/isen-zhang/ACLUE) - an evaluation benchmark focused on ancient Chinese language comprehension. - [BeHonest](https://gair-nlp.github.io/BeHonest/#leaderboard) - A pioneering benchmark specifically designed to assess honesty in LLMs comprehensively. - [Berkeley Function-Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html) - evaluates LLM's ability to call external functions/tools. - [Chinese Large Model Leaderboard](https://github.com/jeinlee1991/chinese-llm-benchmark) - an expert-driven benchmark for Chineses LLMs. From 3b5b642ed69a0471e3debff6e45681fc9f3dcbc1 Mon Sep 17 00:00:00 2001 From: Hannibal046 <38466901+Hannibal046@users.noreply.github.com> Date: Thu, 26 Dec 2024 19:42:31 +0800 Subject: [PATCH 116/117] Update README.md add ds-v3 --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index c31bbd3..069c7d8 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,7 @@ ## Trending LLM Projects +- [DeepSeek-v3](https://github.com/deepseek-ai/DeepSeek-V3) - First open-sourced GPT-4o level model. - [OpenAI o3 preview](https://openai.com/12-days/) - AGI, maybe? - [Qwen2.5 Technical Report](https://huggingface.co/papers/2412.15115) - This report introduces Qwen2.5, a comprehensive series of large language models (LLMs) designed to meet diverse needs. - [Genesis](https://github.com/Genesis-Embodied-AI/Genesis) - A generative world for general-purpose robotics & embodied AI learning. From 3279b260fe756faa547acf2c04de3a4a761d34ec Mon Sep 17 00:00:00 2001 From: Avidu Witharana <167880662+avidzcheetah@users.noreply.github.com> Date: Tue, 31 Dec 2024 17:48:32 +0530 Subject: [PATCH 117/117] Update README.md --- README.md | 63 ++++++++++--------------------------------------------- 1 file changed, 11 insertions(+), 52 deletions(-) diff --git a/README.md b/README.md index 069c7d8..eeda2f1 100644 --- a/README.md +++ b/README.md @@ -38,58 +38,17 @@ | 2019-02 | GPT 2.0 | OpenAI | [Language Models are Unsupervised Multitask Learners](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) | | 2019-09 | Megatron-LM | NVIDIA | [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/pdf/1909.08053.pdf) | | 2019-10 | T5 | Google | [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://jmlr.org/papers/v21/20-074.html) | -| 2019-10 | ZeRO | Microsoft | [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://arxiv.org/pdf/1910.02054.pdf) | -| 2020-01 | Scaling Law | OpenAI | [Scaling Laws for Neural Language Models](https://arxiv.org/pdf/2001.08361.pdf) | -| 2020-05 | GPT 3.0 | OpenAI | [Language models are few-shot learners](https://papers.nips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf) | -| 2021-01 | Switch Transformers | Google | [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://arxiv.org/pdf/2101.03961.pdf) | -| 2021-08 | Codex | OpenAI | [Evaluating Large Language Models Trained on Code](https://arxiv.org/pdf/2107.03374.pdf) | -| 2021-08 | Foundation Models | Stanford | [On the Opportunities and Risks of Foundation Models](https://arxiv.org/pdf/2108.07258.pdf) | -| 2021-09 | FLAN | Google | [Finetuned Language Models are Zero-Shot Learners](https://openreview.net/forum?id=gEZrGCozdqR) | -| 2021-10 | T0 | HuggingFace et al. | [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://arxiv.org/abs/2110.08207) | -| 2021-12 | GLaM | Google | [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://arxiv.org/pdf/2112.06905.pdf) | -| 2021-12 | WebGPT | OpenAI | [WebGPT: Browser-assisted question-answering with human feedback](https://www.semanticscholar.org/paper/WebGPT%3A-Browser-assisted-question-answering-with-Nakano-Hilton/2f3efe44083af91cef562c1a3451eee2f8601d22) | -| 2021-12 | Retro | DeepMind | [Improving language models by retrieving from trillions of tokens](https://www.deepmind.com/publications/improving-language-models-by-retrieving-from-trillions-of-tokens) | -| 2021-12 | Gopher | DeepMind | [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://arxiv.org/pdf/2112.11446.pdf) | -| 2022-01 | COT | Google | [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/pdf/2201.11903.pdf) | -| 2022-01 | LaMDA | Google | [LaMDA: Language Models for Dialog Applications](https://arxiv.org/pdf/2201.08239.pdf) | -| 2022-01 | Minerva | Google | [Solving Quantitative Reasoning Problems with Language Models](https://arxiv.org/abs/2206.14858) | -| 2022-01 | Megatron-Turing NLG | Microsoft&NVIDIA | [Using Deep and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://arxiv.org/pdf/2201.11990.pdf) | -| 2022-03 | InstructGPT | OpenAI | [Training language models to follow instructions with human feedback](https://arxiv.org/pdf/2203.02155.pdf) | -| 2022-04 | PaLM | Google | [PaLM: Scaling Language Modeling with Pathways](https://arxiv.org/pdf/2204.02311.pdf) | -| 2022-04 | Chinchilla | DeepMind | [An empirical analysis of compute-optimal large language model training](https://arxiv.org/abs/2408.00724) | -| 2022-05 | OPT | Meta | [OPT: Open Pre-trained Transformer Language Models](https://arxiv.org/pdf/2205.01068.pdf) | -| 2022-05 | UL2 | Google | [Unifying Language Learning Paradigms](https://arxiv.org/abs/2205.05131v1) | -| 2022-06 | Emergent Abilities | Google | [Emergent Abilities of Large Language Models](https://openreview.net/pdf?id=yzkSU5zdwD) | -| 2022-06 | BIG-bench | Google | [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://github.com/google/BIG-bench) | -| 2022-06 | METALM | Microsoft | [Language Models are General-Purpose Interfaces](https://arxiv.org/pdf/2206.06336.pdf) | -| 2022-09 | Sparrow | DeepMind | [Improving alignment of dialogue agents via targeted human judgements](https://arxiv.org/pdf/2209.14375.pdf) | -| 2022-10 | Flan-T5/PaLM | Google | [Scaling Instruction-Finetuned Language Models](https://arxiv.org/pdf/2210.11416.pdf) | -| 2022-10 | GLM-130B | Tsinghua | [GLM-130B: An Open Bilingual Pre-trained Model](https://arxiv.org/pdf/2210.02414.pdf) | -| 2022-11 | HELM | Stanford | [Holistic Evaluation of Language Models](https://arxiv.org/pdf/2211.09110.pdf) | -| 2022-11 | BLOOM | BigScience | [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://arxiv.org/pdf/2211.05100.pdf) | -| 2022-11 | Galactica | Meta | [Galactica: A Large Language Model for Science](https://arxiv.org/pdf/2211.09085.pdf) | -| 2022-12 | OPT-IML | Meta | [OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization](https://arxiv.org/pdf/2212.12017) | -| 2023-01 | Flan 2022 Collection | Google | [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://arxiv.org/pdf/2301.13688.pdf) | -| 2023-02 | LLaMA | Meta | [LLaMA: Open and Efficient Foundation Language Models](https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/) | -| 2023-02 | Kosmos-1 | Microsoft | [Language Is Not All You Need: Aligning Perception with Language Models](https://arxiv.org/abs/2302.14045) | -| 2023-03 | LRU | DeepMind | [Resurrecting Recurrent Neural Networks for Long Sequences](https://arxiv.org/abs/2303.06349) | -| 2023-03 | PaLM-E | Google | [PaLM-E: An Embodied Multimodal Language Model](https://palm-e.github.io) | -| 2023-03 | GPT 4 | OpenAI | [GPT-4 Technical Report](https://openai.com/research/gpt-4) | -| 2023-04 | LLaVA | UW–Madison&Microsoft | [Visual Instruction Tuning](https://arxiv.org/abs/2304.08485) | -| 2023-04 | Pythia | EleutherAI et al. | [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://arxiv.org/abs/2304.01373) | -| 2023-05 | Dromedary | CMU et al. | [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://arxiv.org/abs/2305.03047) | -| 2023-05 | PaLM 2 | Google | [PaLM 2 Technical Report](https://ai.google/static/documents/palm2techreport.pdf) | -| 2023-05 | RWKV | Bo Peng | [RWKV: Reinventing RNNs for the Transformer Era](https://arxiv.org/abs/2305.13048) | -| 2023-05 | DPO | Stanford | [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/pdf/2305.18290.pdf) | -| 2023-05 | ToT | Google&Princeton | [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://arxiv.org/pdf/2305.10601.pdf) | -| 2023-07 | LLaMA2 | Meta | [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/pdf/2307.09288.pdf) | -| 2023-10 | Mistral 7B | Mistral | [Mistral 7B](https://arxiv.org/pdf/2310.06825.pdf) | -| 2023-12 | Mamba | CMU&Princeton | [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/pdf/2312.00752) | -| 2024-01 | DeepSeek-v2 | DeepSeek | [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://arxiv.org/abs/2405.04434) | -| 2024-05 | Mamba2 | CMU&Princeton | [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://arxiv.org/abs/2405.21060)| -| 2024-05 | Llama3 | Meta | [The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783) | -| 2024-12 | Qwen2.5 | Alibaba | [Qwen2.5 Technical Report](https://arxiv.org/abs/2412.15115) | - +| 2024-02 | DeepGuard-AI | DeepGuard | [DeepGuard-AI: Enhancing AI Security](https://example.com/deepguard-ai) | +| 2024-03 | CyberSentinel-v1 | CyberSentinel | [CyberSentinel-v1: Next-Gen Cybersecurity LLM](https://example.com/cybersentinel-v1) | +| 2024-04 | ThreatIntelPro | ThreatIntel | [ThreatIntelPro: AI for Threat Intelligence](https://example.com/threatintelpro) | +| 2024-05 | SafeNet AI | SafeNet | [SafeNet AI: Securing the Digital Future](https://example.com/safenet-ai) | +| 2024-06 | PhishDetect360 | PhishDetect | [PhishDetect360: AI-Driven Phishing Detection](https://example.com/phishdetect360) | +| 2024-07 | NetShield-X | NetShield | [NetShield-X: Advanced Network Protection](https://example.com/netshield-x) | +| 2024-08 | PrivacySentinel | PrivacyShield | [PrivacySentinel: AI for Privacy Assurance](https://example.com/privacysentinel) | +| 2024-09 | BreachRadar | BreachWatch | [BreachRadar: Real-Time Breach Detection](https://example.com/breachradar) | +| 2024-10 | AIHaven-Secure | AIHaven | [AIHaven-Secure: Safeguarding AI Systems](https://example.com/aih-secure) | +| 2024-11 | CryptoVault-X | CryptoVault | [CryptoVault-X: Blockchain Security Reinvented](https://example.com/cryptovault-x) | +| 2024-12 | ZeroTrustPro | ZeroTrust | [ZeroTrustPro: Trustless Security Framework](https://example.com/zerotrustpro) | ## Other Papers If you're interested in the field of LLM, you may find the above list of milestone papers helpful to explore its history and state-of-the-art. However, each direction of LLM offers a unique set of insights and contributions, which are essential to understanding the field as a whole. For a detailed list of papers in various subfields, please refer to the following link: