How to summarize a text using the Chain of Density and GPT-4 ? ​

import PyPDF2

# Open the PDF file
with open('path_to_your_pdf.pdf', 'rb') as pdf_file:
    pdf_reader = PyPDF2.PdfReader(pdf_file)

    # Extract the text from the PDF
    pages = []
    for page in pdf_reader.pages:
        pages.append(page.extract_text())

    text = ''.join(pages)

import unicodedata


def cleanup_unicode(text):
    corrected_chars = []
    for char in text:
        corrected_char = unicodedata.normalize("NFKC", char)
        corrected_chars.append(corrected_char)
    return "".join(corrected_chars)


text = cleanup_unicode(text)

from llm_core.splitters import TokenSplitter


# We'll take the first 6 000 tokens
splitter = TokenSplitter(chunk_size=6_000, chunk_overlap=0)

first_10_pages = next(splitter.chunkify(text))

print(first_10_pages)

from typing import List
from dataclasses import dataclass
from llm_core.assistants import OpenAIAssistant


@dataclass
class DenseSummary:
    denser_summary: str
    missing_entities: List[str]


@dataclass
class DenserSummaryCollection:
  system_prompt = """
  You are an expert in writing rich and dense summaries in broad domains.
  """

  prompt = """
  Article:
  
  {article}

  ----

  You will generate increasingly concise, entity-dense summaries of the above
  Article.

  Repeat the following 2 steps 5 times.

  - Step 1: Identify 1-3 informative Entities from the Article
  which are missing from the previously generated summary and are the most
  relevant.

  - Step 2: Write a new, denser summary of identical length which covers
  every entity and detail from the previous summary plus the missing entities

  A Missing Entity is:

  - Relevant: to the main story
  - Specific: descriptive yet concise (5 words or fewer)
  - Novel: not in the previous summary
  - Faithful: present in the Article
  - Anywhere: located anywhere in the Article

  Guidelines:
  - The first summary should be long (4-5 sentences, approx. 80 words) yet
  highly non-specific, containing little information beyond the entities
  marked as missing.

  - Use overly verbose language and fillers (e.g. "this article discusses") to
  reach approx. 80 words.

  - Make every word count: re-write the previous summary to improve flow and
  make space for additional entities.

  - Make space with fusion, compression, and removal of uninformative phrases
  like "the article discusses"

  - The summaries should become highly dense and concise yet self-contained,
  e.g., easily understood without the Article.

  - Missing entities can appear anywhere in the new summary.

  - Never drop entities from the previous summary. If space cannot be made,
  add fewer new entities.

  > Remember to use the exact same number of words for each summary.
  Answer in JSON.

  > The JSON in `summaries_per_step` should be a list (length 5) of
  dictionaries whose keys are "missing_entities" and "denser_summary".

  """

  summaries: List[DenseSummary]


  @classmethod
  def summarize(cls, article):
      with OpenAIAssistant(cls, model='gpt-4') as assistant:
          return assistant.process(article=article)

>>> summary_collection = DenserSummaryCollection.summarize(first_10_pages)
>>> print(len(summary_collection.summaries))
5
>>> print(summary_collection.summaries[0].missing_entities)
['Yann LeCun', 'autonomous intelligent agents', 'self-supervised learning']

>>> print(summary_collection.summaries[0].denser_summary)
This article discusses a position paper by Yann LeCun, which proposes an
architecture for autonomous intelligent agents.

The paper combines concepts such as configurable predictive world models, 
behavior driven through intrinsic motivation, and hierarchical joint embedding
architectures trained with self-supervised learning.

The paper aims to address three main challenges in AI research: learning to
represent the world, reasoning and planning compatible with gradient-based
learning, and learning to represent percepts and action plans in a hierarchical
manner.


>>> print(summary_collection.summaries[1].missing_entities)
['JEPA', 'Hierarchical JEPA', 'non-generative architectures']

>>> print(summary_collection.summaries[1].denser_summary)
Yann LeCun's position paper proposes an architecture for autonomous
intelligent agents, combining concepts like configurable predictive world
models and behavior driven by intrinsic motivation.

It addresses three AI research challenges: learning world representation,
reasoning and planning compatible with gradient-based learning, and
hierarchical representation of percepts and action plans.

The paper also introduces JEPA and Hierarchical JEPA,
non-generative architectures for predictive world models.