This web app is an educational tool designed for those interested in machine learning, natural language processing (NLP), and artificial intelligence (AI). It offers detailed insights into how BERT processes language, helping users gain a deeper understanding of transformers—deep learning models that use attention mechanisms to process sequential data. This makes them highly effective for tasks such as language translation, text generation, and comprehension.
This web app allows you to visualize and analyze the attention layers of the BERT transformer model. It provides a unique and insightful representation of how BERT processes language, focusing particularly on its attention mechanisms.
When you input a sentence containing a masked token (denoted as [MASK]), the website generates a detailed report showing how different attention layers in the BERT model attend to various tokens in the sentence. The report includes visualizations of the attention from all 12 attention heads and 12 layers, helping you better understand the underlying mechanisms behind BERT’s language comprehension.
This website stands out from other projects because it focuses specifically on NLP (Natural Language Processing), an area that has not been covered in previous projects. From a technical perspective, this project generates files on demand and sends them through an API, creates caches for faster loading when the same sentence is entered twice, and uses asynchronous functions for efficiency—features not typically found in other projects.
-
Interactive Attention Visualizations: The app provides real-time, interactive visualizations of how BERT’s attention layers process a given sentence with a [MASK]. Users can explore the attention mechanisms across all 12 attention heads and layers, gaining insights into how each part of the sentence is attended to by different layers of the model.
-
Detailed Attention Reports: After submitting a sentence with a masked token, the web app generates a detailed report showing how each attention head and layer interacts with different tokens in the sentence. These reports provide a comprehensive breakdown of the attention patterns, helping users understand BERT’s decision-making process.
-
Caching for faster loading : Since processing a new sentence can take a few minutes, the website saves the report for previously entered sentences. This allows for faster loading times when visualizing them again in the "Recent Visualizations" section. It also loads cached versions of sentences that have been entered before.
-
Downloadable Attention Reports in PDF Format: The website allows users to download the detailed attention reports in PDF format. After generating the report, users can easily download it for offline viewing or sharing, making it convenient for research, documentation, or further analysis.
Simply input a sentence with a [MASK] (e.g., "The sun is [MASK].") and press "GO."
-
The web server may take about a minute to start because it needs to load TensorFlow and the transformer models beforehand.
-
Cached files are saved in a folder called 'saved' within the ai_transformer directory. If these files are deleted, you will need to manually clear the history model. Otherwise, the sentence will still appear in the "Recent Visualizations" page. If you try to enter a sentence that has been deleted from the cache, the entry will be automatically removed when attempting to generate the visualizations.
- Capstone
- ai_transformer
- saved - this directory contain cached pdfs created in the website(empty at start).
- static - this directory contain CSS and Java script for the website.
- templates - this directory contain html templates for the website.
- models.py - contain models for the website.
- admin.py - registed the models here.
- views.py - handles web requests and api requests for file creation.
- urls.py - added url patterns.
- ai_transformer