The text-to-text unification

from transformers import T5ForConditionalGeneration, T5Tokenizer

tokenizer = T5Tokenizer.from_pretrained('t5-small', legacy=False)
model     = T5ForConditionalGeneration.from_pretrained('t5-small')

def t5_inference(input_text: str, max_length: int = 100) -> str:
    inputs  = tokenizer(input_text, return_tensors='pt', max_length=512, truncation=True)
    outputs = model.generate(**inputs, max_new_tokens=max_length,
                              num_beams=4, early_stopping=True)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# ── Multiple tasks, one model ──
tasks = [
    "translate English to French: The weather is beautiful today.",
    "summarize: The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris. It is named after the engineer Gustave Eiffel, whose company designed and built the tower. Constructed from 1887 to 1889, it was originally criticized by some of France's leading artists and intellectuals for its design.",
    "question: What year was the Eiffel Tower completed? context: The Eiffel Tower was constructed from 1887 to 1889 by engineer Gustave Eiffel.",
    "grammar: Me want to goes to the store.",
]
for task in tasks:
    result = t5_inference(task)
    print(f"Input:  {task[:70]}...")
    print(f"Output: {result}")
    print()

# ── BART: another popular encoder-decoder (better for summarization) ──
from transformers import BartForConditionalGeneration, BartTokenizer
bart_tok = BartTokenizer.from_pretrained('facebook/bart-large-cnn')
bart_mdl = BartForConditionalGeneration.from_pretrained('facebook/bart-large-cnn')

article = """The transformer architecture was introduced in 2017 by Vaswani et al.
It replaced recurrent networks with self-attention, enabling parallel training.
BERT and GPT are both based on transformers but use different attention patterns."""

inputs = bart_tok(article, return_tensors='pt', max_length=1024, truncation=True)
summary_ids = bart_mdl.generate(**inputs, max_new_tokens=60, num_beams=4)
print("BART summary:", bart_tok.decode(summary_ids[0], skip_special_tokens=True))

Encoder-decoder architecture deep dive

Encoder: Processes the full input sequence with bidirectional self-attention — every token can attend to every other input token. Produces a rich contextual representation of the input. Decoder: Generates output tokens auto-regressively. Has two types of attention per block: (1) Masked self-attention over previously generated tokens. (2) Cross-attention over all encoder states — the decoder 'reads' the encoder output to inform generation.

Span corruption: how T5 is actually pre-trained

T5's pre-training objective is span corruption — a deliberate middle ground between BERT's single-token masking and full text generation. Random contiguous spans (~15% of tokens, mean span length 3) are replaced with unique sentinel tokens, and the model must generate the missing spans in order:

Original:  "Thank you for inviting me to your party last week."

Encoder input:   "Thank you <X> me to your party <Y> week."
Decoder target:  "<X> for inviting <Y> last <Z>"

Sentinels <X>, <Y> mark WHERE spans were removed; the target
interleaves each sentinel with the span it replaces, ending
with a final sentinel <Z>. The model learns both bidirectional
understanding (encoder must grasp the corrupted sentence) and
fluent generation (decoder must produce the missing text).

Why spans instead of single tokens?: Masking single tokens (BERT-style) makes generation targets trivially short, while generating full text (GPT-style) wastes compute re-predicting what is already known. Span corruption forces multi-token generation conditioned on deep bidirectional context — and because targets are short, pre-training is cheap per example. The same idea returned years later in fill-in-the-middle (FIM) training for code models.

Does encoder-decoder still matter in 2026?

Decoder-only won the LLM race — simpler to scale, one attention pattern, and the prompt's KV cache is reusable across a growing conversation, whereas an encoder must re-encode changed input from scratch. But encoder-decoder never died; it dominates wherever the task is a clean input→output transformation with input much longer than output:

• Speech recognition: Whisper is an encoder-decoder — the encoder digests the audio spectrogram bidirectionally, the decoder writes the transcript.
• Machine translation: production MT systems (MarianMT/OPUS models, Google's MADLAD-400) remain overwhelmingly encoder-decoder.
• Instruction-tuned T5 lives on: Flan-T5 remains a workhorse for cheap, fine-tunable classification and structured NLP at sizes from 80M to 11B — often the right answer when an LLM API is overkill.
• The idea is being revived: Google's T5Gemma (2025) rebuilt encoder-decoder models from modern Gemma decoder weights, showing the architecture's quality-per-FLOP advantage on understanding-heavy tasks still holds with modern training.
• Embeddings and reranking: encoder(-decoder) models power much of retrieval — the encoder half is exactly what you want when no text needs to be generated at all.

Practice questions

1. T5 frames classification as text-to-text. How does it output a class label? (Answer: The model generates the label as a text string. For sentiment: output is literally the word "positive" or "negative". For cola acceptability: output is "acceptable" or "unacceptable". The text output is then mapped to a class. This allows a single model to handle all tasks without task-specific output heads.)
2. What is cross-attention in the encoder-decoder and what does it connect? (Answer: Cross-attention in the decoder attends to the ENCODER hidden states (keys and values come from encoder, queries come from decoder current state). At each decoder step, the decoder reads the most relevant parts of the encoded input. This is how the model accesses the input meaning while generating output.)
3. BART vs T5 — what is the main architectural difference? (Answer: BART uses a standard transformer encoder-decoder with a denoising pre-training objective (corrupting input text and learning to reconstruct it). T5 uses span masking (a form of masked LM) as pre-training and frames everything as text-to-text. BART is better for generation tasks (summarization); T5 is better for understanding tasks.)
4. When would you choose T5 over GPT for a business NLP task? (Answer: T5 when: (1) You have a specific seq2seq task (translation, summarization, data-to-text). (2) Input understanding is as important as generation. (3) You want to fine-tune on a specific task efficiently. GPT when: (1) You need flexible generation. (2) Zero/few-shot prompting is sufficient. (3) Chat interface is needed.)
5. Whisper uses an encoder-decoder transformer for speech recognition. What does each part do? (Answer: Encoder: processes the mel spectrogram (audio features) with full bidirectional attention — understands the entire audio context. Decoder: auto-regressively generates transcript tokens while attending to encoder audio representations via cross-attention. Same architecture as T5/BART applied to audio-to-text.)