Code

LLM.int8() Refactoring: Part 1#1401

Code

Merged

matthewdouglas merged 72 commits intomainbitsandbytes-foundation/bitsandbytes:mainfrom int8bitsandbytes-foundation/bitsandbytes:int8Copy head branch name to clipboardDec 5, 2024

+2,626-2,323

Commits

Commits on Oct 7, 2024

• Start of int8 refactor: remove col32/col_ampere/col_turing transforms in new igemmlt implementation

  matthewdouglas

  committed
  0cc5c95

  Copy full SHA for 0cc5c95

Commits on Oct 8, 2024

• Fix unintended change

  matthewdouglas

  committed
  0f2dc34

  Copy full SHA for 0f2dc34

Commits on Oct 9, 2024

• New naive mm_dequant kernel for row-major; cleanup

  matthewdouglas

  committed
  50fe50e

  Copy full SHA for 50fe50e

• fix

  matthewdouglas

  committed
  57e6427

  Copy full SHA for 57e6427

Commits on Oct 14, 2024

• int8 refactor: initial sparse decomp, cleanup

  matthewdouglas

  committed
  ca372f2

  Copy full SHA for ca372f2

• Int8 refactoring: remove separate NO_CUBLASLT build; more cleanup

  matthewdouglas

  committed
  510a880

  Copy full SHA for 510a880

Commits on Oct 18, 2024

• int8: inference optimizations, some cleanup

  matthewdouglas

  committed
  0ab14fe

  Copy full SHA for 0ab14fe

• int8: more tests passing, cleanup

  matthewdouglas

  committed
  fdf4745

  Copy full SHA for fdf4745

Commits on Oct 21, 2024

• int8 - more cleanup, most tests passing

  matthewdouglas

  committed
  d231db7

  Copy full SHA for d231db7

Commits on Oct 22, 2024

• int8: specify CUDA stream for int8 ops

  matthewdouglas

  committed
  dfc4668

  Copy full SHA for dfc4668

Commits on Oct 24, 2024

• perf: reduce overhead from getting cudaStream ptr

  matthewdouglas

  committed
  01bf54e

  Copy full SHA for 01bf54e

• Mark some functions for deprecation.

  matthewdouglas

  committed
  32979b4

  Copy full SHA for 32979b4

Commits on Oct 30, 2024

• int8 sparse decomp: small perf improvement

  matthewdouglas

  committed
  521da0c

  Copy full SHA for 521da0c

• Merge branch 'main' into int8

  matthewdouglas

  authored
  c75eecd

  Copy full SHA for c75eecd

• update setup.py

  matthewdouglas

  committed
  217cf8e

  Copy full SHA for 217cf8e

Commits on Oct 31, 2024

• Update bitsandbytes/autograd/_functions.py

  Show description for b9cb5c9
  

  matthewdouglas

  and

  akx

  authored
  b9cb5c9

  Copy full SHA for b9cb5c9

• Update bitsandbytes/functional.py

  Show description for 6fa7905
  

  matthewdouglas

  and

  akx

  authored
  6fa7905

  Copy full SHA for 6fa7905

• Update bitsandbytes/functional.py

  Show description for e929df0
  

  matthewdouglas

  and

  akx

  authored
  e929df0

  Copy full SHA for e929df0

• Update bitsandbytes/research/autograd/_functions.py

  Show description for c7b31df
  

  matthewdouglas

  and

  akx

  authored
  c7b31df

  Copy full SHA for c7b31df

Commits on Nov 4, 2024

• int8 - perf improvement for sparse decomposition inference; deprecate get_tensor_stream() in favor of new private fn

  matthewdouglas

  committed
  57300e7

  Copy full SHA for 57300e7

• int8 cleanup

  matthewdouglas

  committed
  0460d2e

  Copy full SHA for 0460d2e

• Merge branch 'int8' of https://github.com/TimDettmers/bitsandbytes into int8

  matthewdouglas

  committed
  437a17e

  Copy full SHA for 437a17e

• Ignore ruff rule ISC001 (incompatible with formatter)

  matthewdouglas

  committed
  762daf4

  Copy full SHA for 762daf4

• add comment

  matthewdouglas

  committed
  875414e

  Copy full SHA for 875414e

• int8 more cleanup

  matthewdouglas

  committed
  0aefeb0

  Copy full SHA for 0aefeb0

• Update bitsandbytes/functional.py

  Show description for bfb42d1
  

  matthewdouglas

  and

  akx

  authored
  bfb42d1

  Copy full SHA for bfb42d1

• Merge branch 'int8' of https://github.com/TimDettmers/bitsandbytes into int8

  matthewdouglas

  committed
  1ae2476

  Copy full SHA for 1ae2476

• int8: rename / deprecate old fn signatures

  matthewdouglas

  committed
  bf002db

  Copy full SHA for bf002db

• Update bitsandbytes/functional.py

  Show description for 7f6fb60
  

  matthewdouglas

  and

  akx

  authored
  7f6fb60

  Copy full SHA for 7f6fb60

• type annotation

  matthewdouglas

  committed
  135b336

  Copy full SHA for 135b336

• format update

  matthewdouglas

  committed
  5388877

  Copy full SHA for 5388877

• Update bitsandbytes/research/autograd/_functions.py

  Show description for be2e98f
  

  matthewdouglas

  and

  akx

  authored
  be2e98f

  Copy full SHA for be2e98f

• cleanup

  matthewdouglas

  committed
  4c849bb

  Copy full SHA for 4c849bb

• Merge branch 'int8' of https://github.com/TimDettmers/bitsandbytes into int8

  matthewdouglas

  committed
  32a60c5

  Copy full SHA for 32a60c5

• Add comment to explain division optimization

  matthewdouglas

  committed
  b954474

  Copy full SHA for b954474

• more cleanup

  matthewdouglas

  committed
  35dbb2e

  Copy full SHA for 35dbb2e

Commits on Nov 5, 2024

• Update bitsandbytes/functional.py

  Show description for b36003f
  

  matthewdouglas

  and

  akx

  authored
  b36003f

  Copy full SHA for b36003f

• Update bitsandbytes/functional.py

  Show description for 03a1963
  

  matthewdouglas

  and

  akx

  authored
  03a1963

  Copy full SHA for 03a1963

• Update bitsandbytes/functional.py

  Show description for 980279f
  

  matthewdouglas

  and

  akx

  authored
  980279f

  Copy full SHA for 980279f

• cleanup

  matthewdouglas

  committed
  a72c463

  Copy full SHA for a72c463

• Merge branch 'int8' of https://github.com/TimDettmers/bitsandbytes into int8

  matthewdouglas

  committed
  b5d6135

  Copy full SHA for b5d6135

• Type annotations, cleanup

  matthewdouglas

  committed
  b1c4adc

  Copy full SHA for b1c4adc

• remove unused kernels; improved type annotations

  matthewdouglas

  committed
  ed922b8

  Copy full SHA for ed922b8

• small perf optimization for single-GPU systems

  matthewdouglas

  committed
  a93b91f

  Copy full SHA for a93b91f

• small perf optimization for single-GPU systems

  matthewdouglas

  committed
  4bced86

  Copy full SHA for 4bced86

Commits on Nov 18, 2024

• update docstrings

  matthewdouglas

  committed
  f61d8bc

  Copy full SHA for f61d8bc

• Improve docs and tests

  matthewdouglas

  committed
  eed9c3c

  Copy full SHA for eed9c3c

• Update docstring

  matthewdouglas

  committed
  6e0a4b3

  Copy full SHA for 6e0a4b3

• Update test

  matthewdouglas

  committed
  161c194

  Copy full SHA for 161c194

Commits on Nov 19, 2024

• Merge branch 'main' into int8

  matthewdouglas

  authored
  0ac1452

  Copy full SHA for 0ac1452

Commits on Nov 20, 2024

• add benchmarking script

  matthewdouglas

  committed
  e3051fa

  Copy full SHA for e3051fa

• test cleanup: add deprecated marker, move benchmarks out

  matthewdouglas

  committed
  56abdc2

  Copy full SHA for 56abdc2

Commits on Nov 25, 2024

• Add int8 dequant function; misc improvements

  matthewdouglas

  committed
  df941ec

  Copy full SHA for df941ec

• int8 matmul fallback for inner dims not divisible by 4

  matthewdouglas

  committed
  73f02e8

  Copy full SHA for 73f02e8

Commits on Nov 27, 2024

• improve register usage of kInt8VectorQuant - especially for A100/H100

  matthewdouglas

  committed
  ebb6797

  Copy full SHA for ebb6797

• disable fail-fast for package build

  matthewdouglas

  committed
  196c8e0

  Copy full SHA for 196c8e0

• maxwell compat

  matthewdouglas

  committed
  fa6f597

  Copy full SHA for fa6f597

• ptxas verbose

  matthewdouglas

  committed
  498d8de

  Copy full SHA for 498d8de

Commits on Nov 29, 2024

• docs update

  matthewdouglas

  committed
  a2ee1c4

  Copy full SHA for a2ee1c4

Commits on Dec 2, 2024

• Merge branch 'main' into int8

  matthewdouglas

  authored
  ac09570

  Copy full SHA for ac09570

• doc update

  matthewdouglas

  committed
  15f1661

  Copy full SHA for 15f1661

• backward fix

  matthewdouglas

  committed
  5d536c6

  Copy full SHA for 5d536c6

Commits on Dec 3, 2024

• Bugfix sparse decomp

  matthewdouglas

  committed
  5b2348b

  Copy full SHA for 5b2348b

• Int8 fix for PEFT OLoRA init

  matthewdouglas

  committed
  bbb7063

  Copy full SHA for bbb7063

• Fix test for deprecated spmm_coo

  matthewdouglas

  committed
  d25ebb4

  Copy full SHA for d25ebb4

• test improvement

  matthewdouglas

  committed
  3d595f1

  Copy full SHA for 3d595f1

Commits on Dec 4, 2024

• doc update

  matthewdouglas

  committed
  03fcabd

  Copy full SHA for 03fcabd

• typo

  matthewdouglas

  committed
  582bf22

  Copy full SHA for 582bf22

• doc cleanup

  matthewdouglas

  committed
  1ae7c6b

  Copy full SHA for 1ae7c6b

• docs

  matthewdouglas

  committed
  213b10b

  Copy full SHA for 213b10b

• add inference benchmark script

  matthewdouglas

  committed
  ca6fd44

  Copy full SHA for ca6fd44

• Add benchmarks, doc update

  matthewdouglas

  committed
  b8c736b

  Copy full SHA for b8c736b