Getting Started

Getting Started§

Requirements§

  • torch

  • einops

  • tiktoken [optional] (tokenizer used for pre-trained language models)

  • huggingface_hub [optional] (for loading pre-trained model checkpoints from HF)

  • safetensors [optional] (again, for loading pre-trained model checkpoints from HF)

  • gradio [optional] (for creating graphical interfaces for visualization and running inference)

  • bertviz [optional] (for visualizing attention heads)

Installation§

The dual-attention package is hosted on the Python Package Index (PyPI) and can be installed using pip:

pip install dual-attention

We recommend installing the package in a virtual environment.

You can also install from source by cloning the repository:

git clone https://github.com/Awni00/dual-attention.git
cd dual-attention
pip install .
Contents