paper_loadmap.md

markmap
colorFreezeLevel maxWidth 10 800

NLP

Bahdanau attention

Luong attention

Weight tying

Transformer

Encoder-Only BERT

• 성능 향상

  • RoBERTa
  • Big Bird
  • XLNet
  • ELECTRA
  • SpanBERT

• domain 적용

  • BioBERT
  • LogBERT
  • ClinicalBERT

• 경량화

  • DistilBERT
  • ALBERT

Decoder-Only GPT-1

• GPT-2
  • GPT-3
    • Self-Instruction
      • Self-Instruct
    • Open Access
      • LLAMA
        • Instruction-follwing
          • Alpaca
          • LIMA
      • BLOOM
        • BLOOMZ
    • RLHF (Reinformcement Learning Human Feedback)
      • InstructGPT
        • GPT-4
    • PALM
      • PALM2
      • Flan-PALM

Encoder-Decoder BART

• 성능 향상
  • T5
    • FLAN-T5
    • T0
• multimodal
  • METALM
    • KOSMOS-1
  • Flamingo

Transformer 구조 수정

• Layernorm + initialization
  • DeepNet
  • MAGNETO
• Self-attention 관련
  • Reformer
  • Relative positional encoding
  • Longformer
  • FNet

LM with External Knowledge

• with Document

  • DPR
    • Contriever
    • SamToNe
  • ORQA
    • REALM
      • Reader -> Generator RAG
        • Re2G
        • FiD
        • Atlas
        • REPLUG
        • EAR

• with KG

  • DiFaR
  • Retrieve-Rerank Framework

• Document + KG

  • UniK-QA

Benchmark

GLUE

• SQuAD
• SuperGLUE

DecaNLP

BIG-Bench

SUPER-NATURALINSTRUCTIONS

MMLU

Knowledge-Intensive Task

• KILT
  • FEVER
• BEIR

Effficient Training

• Mixed Precision
• Distributed learning
  • Megatron-LM
  • Zero
    • Zero-Infinity
• LoRA
• Prompt Tuning