How to run Nougat with an API | Clarifai Blog

Nougat is a visual transformer model from Meta AI that converts document images, including complex math equations, into structured text, offering advancements in academic paper parsing.

You can now try out nougat-base in the Clarifai Platform and access it through the API.

Introduction
Model Architecture
Running Nougat model with Python
Running Nougat model with Javascript
Best Use Cases

Introduction

Nougat is a visual transformer model developed by researchers at Meta AI that can convert images of document pages into structured text. It takes a scanned image of a document page as input and outputs text in a lightweight markup language.

The key advantage of Nougat is that it relies solely on the document image and does not need any OCR text. This allows it to recover semantic structure like math equations properly. It is trained on millions of academic papers from arXiv and PubMed to learn the patterns of research paper formatting and language.

Model Architecture

Nougat uses a visual transformer encoder-decoder architecture. The encoder uses a Swin Transformer to encode the document image into latent embeddings. The Swin Transformer processes the image in a hierarchical fashion using shifted windows. The decoder then generates the output text tokens autoregressive using self-attention over the encoder outputs.

Running Nougat model with Python

You can run Nougat with Clarifai’s Python SDK in just a few lines of code. To get started, Signup to Clarifai and get your Personal Access Token(PAT) following the instructions here.

Export your PAT as an environment variable

export CLARIFAI_PAT={your personal access token}

Check out the Code below to run the Model:

Running Nougat model with Javascript

You can also run it with our Javascript Client:

You can also run Nougat using other Clarifai Client Libraries like Java, cURL, NodeJS, PHP, etc.

Model Demo in the Clarifai Platform:

Try out the Nougat model here: https://clarifai.com/facebook/nougat/models/nougat-base

Best Use Cases

Nougat Model has a wide range of applications in the field of document understanding and extraction. Some key use cases include:

Research Paper Parsing: Nougat can accurately parse research papers, extracting text, tables, figures, and equations from document images. This capability is crucial for making the information in research papers more accessible for various applications.
Data Extraction: The model's ability to convert documented images into structured text makes it valuable for extracting valuable data from academic papers, which can be used for research, analysis, and data-driven decision-making.
Summarization: Nougat can be integrated into text summarization pipelines to extract and summarize the content of research papers automatically, saving time and effort for researchers.

Table of Contents