Backup of BitNet (No. 7)

• Backup list
• View the diff.
• View the diff current.
• View the diff for visual.
• View the source.
• Go to BitNet.
  • 1 (2024-02-28 (Wed) 17:37:28)
  • 2 (2024-02-29 (Thu) 10:24:39)
  • 3 (2024-02-29 (Thu) 13:50:55)
  • 4 (2024-03-01 (Fri) 11:44:59)
  • 5 (2024-03-02 (Sat) 14:26:39)
  • 6 (2024-03-22 (Fri) 12:23:23)
  • 7 (2024-03-30 (Sat) 21:44:04)

• https://www.microsoft.com/en-us/research/publication/bitnet-scaling-1-bit-transformers-for-large-language-models/
• https://arxiv.org/abs/2310.11453
  • https://arxiv.org/pdf/2310.11453.pdf

• https://github.com/microsoft/unilm/blob/master/bitnet/The-Era-of-1-bit-LLMs__Training_Tips_Code_FAQ.pdf

• https://github.com/kyegomez/BitNet
• https://github.com/shi3z/BitNet

1.58 Bits

• https://arxiv.org/abs/2402.17764
  • https://arxiv.org/pdf/2402.17764.pdf

• https://github.com/Beomi/BitNet-Transformers/
• https://github.com/frodo821/BitNet-Transformers

  https://twitter.com/BoufrawFrodo2/status/1763435835935047789 オリジナルのBitNetを1.58bの論文に従って3値にするように修正しました

• Microsoftが1.58ビットの大規模言語モデルをリリース、行列計算を足し算にできて計算コスト激減へ
  • https://gigazine.net/news/20240229-microsoft-1bit-llm/

• 1ビットLLMの衝撃! 70Bで8.9倍高速　全ての推論を加算のみで!GPU不要になる可能性も
  • https://topics.smt.docomo.ne.jp/article/wirelesswire/business/wirelesswire-20240286094?

• https://twitter.com/andrew_n_carr/status/1770487200234213758?s=12

  1.58 bit code is out (in an appendix) https://github.com/microsoft/unilm/blob/master/bitnet/The-Era-of-1-bit-LLMs__Training_Tips_Code_FAQ.pdf

.

• BitNet
  • https://qiita.com/fuyu_quant/items/fa4fc3930ddfd2ec6f16

    ※BitNetそのものを検証した記事ではなく，BitNetで使われているレイヤーを使った簡単なNNモデルを作成し検証しています．