Content summarization sounds simple: take a long document, produce a short version. In practice, building a summarization system that works reliably across document types, handles edge cases without hallucinating, respects length constraints consistently, and scales to thousands of documents per day is a genuine engineering challenge. We built a summarization pipeline for a media company that processes 3,000 articles per day, generating summaries for daily email newsletters, social media posts, internal executive briefings, and RSS feed descriptions. This post covers the architecture, the model selection trade-offs with real numbers, and the quality assurance mechanisms that make it production-ready rather than demo-ready.

The Summarization Taxonomy

Not all summaries are the same, and understanding the types matters because the optimal approach differs significantly for each:

• Extractive summarization: Selects and concatenates the most important sentences from the original text verbatim. No new words are generated. Extremely faithful to the source (it literally copies sentences) but often reads awkwardly because selected sentences may lack transitions and coherence.
• Abstractive summarization: Generates entirely new text that captures the key points. Reads naturally because the model writes complete, coherent prose. But it can hallucinate details not in the source: inventing quotes, misattributing statements, or interpolating numbers between what the source actually says.
• Query-focused summarization: Summarizes the document from a specific angle or perspective. “Summarize this earnings call focusing on guidance for next quarter” produces a fundamentally different summary than “Summarize this earnings call focusing on competitive positioning.” The same source document yields different (correct) summaries depending on the query.
• Multi-document summarization: Synthesizes information across multiple documents into a single coherent summary. Required for daily news briefings (10 articles about the same event), literature reviews (50 papers on a topic), and market intelligence (15 competitor announcements this week). This is the hardest variant because it requires deduplication (do not repeat the same fact from 5 sources), contradiction resolution (different sources report different numbers), and source attribution (the reader needs to know where each claim originates).

Our production system uses abstractive summarization as the primary approach because extractive summaries read like bullet points rather than prose, and our client’s use cases (newsletters, executive briefings) require readable narrative text. We maintain an extractive fallback for two scenarios: when the abstractive model’s faithfulness score drops below our threshold (more on this below), and for highly technical or numerical content where paraphrasing risks distorting precise figures.

Model Selection: Cost vs. Quality vs. Latency

We evaluated five approaches on a benchmark of 500 articles with human-written reference summaries. The benchmark articles span news reporting (200), opinion/analysis (150), technical writing (100), and corporate communications (50). The reference summaries were written by 3 professional editors and cross-validated for consistency. Here are the results:

# Benchmark: 500 articles, avg 2,100 words each
# Reference: 3 professional editors, cross-validated

| Model                  | ROUGE-L | Faithful | Cost/1K | p50 Lat |
|------------------------|---------|----------|---------|--------|
| GPT-4o                 | 0.42    | 96.8%    | $18.50  | 4.2s   |
| GPT-4o-mini            | 0.39    | 94.1%    | $1.85   | 2.1s   |
| Claude 3.5 Sonnet      | 0.41    | 97.2%    | $13.50  | 3.8s   |
| Mistral Large          | 0.37    | 91.3%    | $7.20   | 3.1s   |
| BART-large-CNN (local) | 0.34    | 99.1%    | $0.12*  | 0.3s   |

* GPU compute cost (A10G spot), amortized over batch

Faithfulness: % of summaries where ALL claims are 
supported by the source document (binary per-summary, 
not per-claim)

Several results surprised us. BART-large-CNN, a fine-tuned 400M parameter model from 2020, has the highest faithfulness score at 99.1%. This is because BART-large-CNN trained on CNN/DailyMail is primarily extractive in practice: it copies and lightly rearranges source sentences rather than generating truly novel text. It rarely halluccinates because it rarely generates anything the source did not contain. But its ROUGE-L score is lowest because it does not paraphrase effectively, producing summaries that are technically accurate but stylistically poor (repetitive phrasing, awkward transitions, no synthesis).

Claude 3.5 Sonnet produces the most faithful abstractive summaries (97.2%) while maintaining strong ROUGE-L and readability. GPT-4o is close behind at 96.8% with slightly better stylistic quality. GPT-4o-mini hits 94.1% faithfulness, meaning roughly 6% of its summaries contain at least one unsupported claim. For our use case (media newsletters where every claim needs to be verifiable), that 6% error rate requires a mitigation strategy, which is where the faithfulness checking pipeline comes in.

Our production choice

We use GPT-4o-mini as the primary model. At $1.85 per 1,000 articles, it is 10x cheaper than GPT-4o with 94% faithfulness. Every summary passes through a faithfulness check (described below), and the 6% that fail are either regenerated with a stricter prompt (which resolves 70% of failures) or escalated to GPT-4o (which resolves the remaining 30%). The blended cost including retries and escalations is approximately $2.40 per 1,000 articles, still far cheaper than using GPT-4o for everything.

The Summarization Pipeline

from dataclasses import dataclass
from enum import Enum
import tiktoken

class SummaryLength(Enum):
    TWEET = 280         # Characters (not tokens)
    SOCIAL_POST = 150   # Tokens (~100 words)
    NEWSLETTER = 300    # Tokens (~200 words)
    BRIEFING = 600      # Tokens (~400 words)
    EXECUTIVE = 1000    # Tokens (~700 words)

@dataclass
class SummaryRequest:
    content: str
    length: SummaryLength
    focus: str | None = None     # Query-focused angle
    audience: str = "general"    # Audience context
    style: str = "informative"   # informative, analytical, casual

class SummarizationPipeline:
    def __init__(self, config: PipelineConfig):
        self.tokenizer = tiktoken.encoding_for_model(
            "gpt-4o-mini"
        )
        self.primary_model = "gpt-4o-mini"
        self.fallback_model = "gpt-4o"
        self.max_input_tokens = 120_000
    
    async def summarize(
        self, request: SummaryRequest
    ) -> SummaryResult:
        # Step 1: Preprocess
        content = self._clean_and_normalize(request.content)
        token_count = len(self.tokenizer.encode(content))
        
        if token_count > self.max_input_tokens:
            return await self._hierarchical_summarize(request)
        
        # Step 2: Generate with primary model
        prompt = self._build_prompt(request)
        raw_summary = await self._call_llm(
            prompt, model=self.primary_model
        )
        
        # Step 3: Faithfulness verification
        faith_result = await self._check_faithfulness(
            content, raw_summary
        )
        
        if faith_result.score < 0.90:
            # Step 3a: Retry with stricter prompt
            strict_summary = await self._retry_strict(
                request, faith_result.unsupported_claims
            )
            faith_retry = await self._check_faithfulness(
                content, strict_summary
            )
            
            if faith_retry.score < 0.90:
                # Step 3b: Escalate to better model
                raw_summary = await self._call_llm(
                    prompt, model=self.fallback_model
                )
                faith_result = await self._check_faithfulness(
                    content, raw_summary
                )
            else:
                raw_summary = strict_summary
                faith_result = faith_retry
        
        # Step 4: Length enforcement
        final = self._enforce_length(
            raw_summary, request.length
        )
        
        return SummaryResult(
            summary=final,
            faithfulness_score=faith_result.score,
            model_used=self._last_model_used,
            input_tokens=token_count,
            output_tokens=len(
                self.tokenizer.encode(final)
            ),
            unsupported_claims=faith_result.unsupported_claims
        )

The prompt engineering matters enormously for summarization quality. Our production prompt has been through 14 iterations over 6 months. The key elements that improved quality most were: explicitly stating length in words ("approximately 200 words") rather than vague instructions ("brief summary"), requiring the model to lead with the most important information (inverted pyramid structure), prohibiting phrases like "This article discusses" or "The author argues" in favor of direct statements, and including a "Rules" section that explicitly says "Include only facts stated in the article. Do not add analysis or context from your training data."

Hierarchical Summarization for Long Documents

Documents that exceed the model's effective context window (or where you want to minimize cost for very long inputs) require hierarchical summarization: split into chunks, summarize each chunk, then summarize the chunk summaries into a final output.

async def _hierarchical_summarize(
    self, request: SummaryRequest
) -> SummaryResult:
    chunks = self._split_into_chunks(
        request.content, 
        chunk_size=30_000,
        overlap_size=2_000  # 15% overlap at boundaries
    )
    
    # Stage 1: Summarize each chunk independently
    # Include document title and first paragraph in every
    # chunk prompt for global context
    global_context = self._extract_header(request.content)
    chunk_summaries = []
    
    for i, chunk in enumerate(chunks):
        chunk_prompt = (
            f"Document title and introduction:n"
            f"{global_context}nn"
            f"Section {i+1} of {len(chunks)}:n"
            f"{chunk}"
        )
        chunk_request = SummaryRequest(
            content=chunk_prompt,
            length=SummaryLength.BRIEFING,
            focus=request.focus,
            audience=request.audience
        )
        result = await self._single_pass_summarize(
            chunk_request
        )
        chunk_summaries.append({
            "section": i + 1,
            "summary": result.summary
        })
    
    # Stage 2: Synthesize chunk summaries into final
    combined = "nn".join(
        f"[Section {cs['section']}]n{cs['summary']}" 
        for cs in chunk_summaries
    )
    
    synthesis_request = SummaryRequest(
        content=f"The following are summaries of sequential "
                f"sections of a single document. Synthesize "
                f"them into one coherent summary.nn"
                f"{combined}",
        length=request.length,
        focus=request.focus,
        audience=request.audience,
        style=request.style
    )
    
    return await self._single_pass_summarize(
        synthesis_request
    )

The main risk with hierarchical summarization is information loss at chunk boundaries. If a key point starts in one chunk and concludes in the next, neither chunk summary may capture it fully. The 15% overlap between chunks mitigates this: boundary content appears in both chunks, so at least one chunk summary will capture it. Including the document title and opening paragraph in every chunk's prompt provides global context that helps the model understand the significance of section-level details. In our testing, the overlap + global context approach recovered 92% of cross-boundary key points compared to 71% without these mitigations.

Faithfulness Checking: The Critical Quality Gate

This is the most important component of the entire pipeline. A faithfulness check verifies that every factual claim in the summary is supported by the source document. Without it, roughly 6% of GPT-4o-mini summaries and 3% of GPT-4o summaries contain at least one hallucinated detail: a number that was close but not exactly what the source said, a quote attributed to the wrong person, or a causal claim that the source merely implied but did not state.

@dataclass
class FaithfulnessResult:
    score: float  # 0.0 to 1.0
    claims: list[dict]  # Each claim with verdict + evidence
    unsupported_claims: list[str]

async def _check_faithfulness(
    self, source: str, summary: str
) -> FaithfulnessResult:
    prompt = f"""You are a meticulous fact-checker. Your task is to 
verify every factual claim in the Summary against the Source.

For EACH sentence in the Summary, extract individual factual claims
and classify each as:
- SUPPORTED: The claim is explicitly stated in or directly 
  implied by the Source. Quote the supporting evidence.
- UNSUPPORTED: The claim cannot be verified from the Source.
  This includes claims that are plausible but not stated.
- CONTRADICTED: The claim contradicts the Source. 
  Quote the contradicting evidence.

Be strict. "The company reported strong growth" is UNSUPPORTED 
if the Source says "The company reported 12% growth" - the summary 
added the subjective word "strong" which is not in the Source.

Return JSON:
{{
  "claims": [
    {{
      "claim": "exact text from summary",
      "verdict": "SUPPORTED|UNSUPPORTED|CONTRADICTED",
      "evidence": "quoted text from source or null",
      "reasoning": "brief explanation"
    }}
  ],
  "overall_score": 0.0-1.0
}}

Source (first 50,000 characters):
{source[:50000]}

Summary to verify:
{summary}"""
    
    response = await self._call_llm(
        prompt, 
        model="gpt-4o-mini",  # Cheaper model is fine for checking
        response_format={"type": "json_object"}
    )
    result = json.loads(response)
    
    unsupported = [
        c["claim"] for c in result["claims"] 
        if c["verdict"] in ("UNSUPPORTED", "CONTRADICTED")
    ]
    
    return FaithfulnessResult(
        score=result["overall_score"],
        claims=result["claims"],
        unsupported_claims=unsupported
    )

The faithfulness check adds $0.50 per 1,000 articles when using GPT-4o-mini as the checker. We run it on every summary and flag anything below 0.90 for retry. The specific unsupported claims are passed to the retry prompt so the model can correct them specifically rather than regenerating from scratch. This targeted retry resolves 70% of faithfulness failures without requiring escalation to the more expensive model.

An important calibration point: we found that using GPT-4o-mini to check GPT-4o-mini summaries is effective because the failure modes are different. The summarizer hallucinates because it is trying to generate coherent narrative text. The checker is looking for specific claims and comparing them against specific source text, which is a simpler, more reliable task. When we tested the checker against human fact-checkers on 200 summaries, the checker agreed with humans 94% of the time at the claim level.

Multi-Document Summarization

The hardest variant and the most commercially valuable. When summarizing 5-10 articles about the same breaking story into a single executive briefing, you need to handle four challenges that do not exist in single-document summarization. First, overlapping information: 8 out of 10 articles report the same core facts, and simply concatenating summaries would repeat the same information 8 times. Second, contradictions: different sources report different figures (one says revenue was $4.2B, another says $4.3B). Third, attribution: the reader needs to know which source each claim comes from, especially for forward-looking statements and opinions. Fourth, narrative synthesis: the final summary must read as a coherent story, not a list of disconnected facts from different sources.

async def multi_document_summarize(
    self, articles: list[Article], 
    length: SummaryLength,
    focus: str | None = None
) -> MultiDocSummaryResult:
    # Phase 1: Individual summaries with source tracking
    article_summaries = []
    for article in articles:
        summary = await self.summarize(SummaryRequest(
            content=article.content,
            length=SummaryLength.BRIEFING,
            focus=focus
        ))
        article_summaries.append({
            "source": article.publication_name,
            "title": article.title,
            "url": article.url,
            "summary": summary.summary,
            "published": article.published_at.isoformat()
        })
    
    # Phase 2: Synthesis with attribution
    synthesis_prompt = f"""You have summaries from 
{len(articles)} different sources covering the same topic. 
Write a unified briefing {self._length_instruction(length)}.

Critical rules:
- Synthesize information, do not concatenate summaries
- When all sources agree on a fact, state it without attribution
- When sources disagree on specifics, note the disagreement: 
  "According to [Source A], revenue was $4.2B, while 
  [Source B] reported $4.3B."
- For opinions, analysis, and forward-looking statements, 
  always attribute: "[Source name] suggests that..."
- Lead with the consensus facts, then note divergences
- Preserve specific numbers and direct quotes with attribution
- End with a brief note on what to watch for next

Article summaries:
{json.dumps(article_summaries, indent=2)}"""
    
    synthesis = await self._call_llm(synthesis_prompt)
    
    return MultiDocSummaryResult(
        summary=synthesis,
        sources=[
            {"name": a["source"], "url": a["url"]} 
            for a in article_summaries
        ],
        article_count=len(articles)
    )

The two-phase approach (individual summaries first, then synthesis) is significantly better than the naive approach of concatenating all article texts and summarizing the combined text. The naive approach fails for three reasons: the combined text often exceeds context limits, the model loses track of which facts came from which source, and the model tends to favor the first few articles in the concatenation (recency bias in long contexts). By summarizing individually and then synthesizing from structured summaries with labeled sources, the model maintains clear attribution and produces more balanced output.

Production Performance and Cost

Processing 3,000 articles per day requires careful optimization across the pipeline. We send LLM requests in parallel batches of 20, respecting the per-minute rate limits of each provider. With GPT-4o-mini's 10,000 RPM limit, we can process 200 articles per minute (one summary request + one faithfulness check per article, plus occasional retries). The full 3,000-article daily batch completes in approximately 25 minutes wall-clock time, compared to 3.5 hours if processed sequentially.

Summaries are cached by a SHA-256 hash of the article content. If an article is updated with minor edits (typo fixes, added hyperlinks, updated publication time), the content hash changes. Before regenerating, we compute the cosine similarity between the old and new article embeddings. If similarity exceeds 0.98 (meaning the semantic content barely changed), we reuse the cached summary. This saves approximately 15% of API calls per day from minor article updates.

Total API cost for 3,000 articles per day with GPT-4o-mini primary, faithfulness checking, and selective GPT-4o escalation: approximately $180/month. That is $0.002 per article. For comparison, a human editor producing equivalent-quality summaries at freelance rates ($30/hour, 3 minutes per summary) would cost approximately $4,500/month for the same volume. The AI pipeline is 25x cheaper and produces summaries in 2 seconds rather than 3 minutes, though it requires the faithfulness checking infrastructure that a human editor does not need.

What We Got Wrong Initially

1. We underestimated length control. LLMs are bad at counting their own output tokens. "Write a 200-word summary" produces anywhere from 140 to 380 words depending on the complexity of the source material. Simple articles with clear conclusions produce short summaries; complex articles with multiple themes produce long ones. We now enforce length programmatically: if the output exceeds the target by more than 20%, we send it back with a truncation prompt that says "This summary is {n} words. Shorten it to approximately {target} words while keeping the most important information." This adds one LLM call for about 25% of summaries but brings length variance under control.
2. We over-relied on ROUGE scores during development. ROUGE measures n-gram overlap with reference summaries. A summary can have a low ROUGE score but be perfectly good because it paraphrased effectively. Conversely, an extractive summary that copies sentences verbatim gets high ROUGE but reads poorly. We now use ROUGE only as a sanity check (flag summaries with ROUGE-L below 0.20, which usually indicates the model went off-topic) and rely on faithfulness checking + human evaluation for actual quality assessment.
3. We did not handle tables, lists, and structured data well. Articles with significant tabular data (quarterly results tables, comparison matrices) or long bulleted lists ("10 features of the new product") need different summarization strategies than narrative text. For tabular content, we include explicit instructions to "preserve the 3 most important data points from the table with exact figures." For list-heavy content, we instruct the model to "identify the 3-4 most significant items and summarize them, noting the total count." These special cases required detecting structured content during preprocessing (regex for HTML tables and lists, heuristics for content that is >40% bullet points) and branching the prompt accordingly.
4. Multi-language content caused silent failures. About 8% of our client's monitored sources publish in French, German, or Spanish. GPT-4o-mini handles multilingual summarization reasonably well, but the faithfulness checker was trained (via our prompt) to compare English claims against potentially non-English sources. We added a translation step for non-English content and language detection to route the prompt appropriately.

Content summarization is deceptively simple to prototype and genuinely hard to productionize. The gap between a working demo and a system that handles 3,000 articles per day with consistent quality, verifiable faithfulness, and predictable cost is filled with edge cases that you only discover by running on real data at scale. Build the faithfulness checking pipeline first, before anything else. It is the foundation that makes everything else trustworthy.