FRIdata

Imagine regularly downloading new releases of protein databases (PDB, UniProt, AFDB, ESMAtlas, etc.) and having to process them efficiently while avoiding redundant computations. It's a surprisingly frustrating problem.

FRIdata (free data) is a protein data generation and storage workflow that produces non-redundant protein derivatives, e.g.:

• 3D coordinates and distograms
• Sequences and protein language model embeddings (ESM-2, ESM-C, gLM-2, etc.)
• HTML reports showing data dependencies Leveraging Dask, it is highly efficient and scalable

In deepFRI2 training, we use FRIdata to manage different releases of the Gene Ontology Annotation (GOA) database, stratified by annotation quality.

Full documentation may be found here.

The repository is currently under active development. If you run into installation problems, find a bug, or would like to propose an improvement, please raise an issue or write directly to p.szczerbiak[at]sanoscience.org.

Installation and activation

1. Download the repo

git clone https://github.com/Tomasz-Lab/FRIdata.git
cd FRIdata

2. Install miniconda

3. Install mamba

## prioritize 'conda-forge' channel
conda config --add channels conda-forge

## update existing packages to use 'conda-forge' channel
conda update -n base --all

## install 'mamba'
conda install -n base mamba

4. Create the environment (recommended)

Use the setup script to create the conda environment, install pip dependencies, and install a PyTorch build matched to your GPU driver:

./scripts/setup_env.sh

Use a custom environment name for a separate install:

./scripts/setup_env.sh -n fridata_gpu_verify

For CPU-only systems:

./scripts/setup_env.sh --cpu

5. Activate the environment

# Choose your shell type. Could be one of these: {bash,cmd.exe,dash,fish,nu,posix,powershell,tcsh,xonsh,zsh}
eval "$(mamba shell hook --shell <replace with shell type>)"
mamba activate fridata_env

Manual installation

If you prefer to run the steps yourself:

mamba env create -f fridata_env_conda.yml -n fridata_env
mamba run -n fridata_env python -m pip install -r requirements-fridata.txt
./scripts/install_pytorch.sh

Pip dependencies live in requirements-fridata.txt instead of fridata_env_conda.yml so environment creation works from read-only repository checkouts.

Troubleshooting

If you see Error opening for writing ".../mambaf..." while creating the environment, libmamba is trying to write a temporary pip requirements file next to the environment YAML. Use ./scripts/setup_env.sh instead, or ensure the directory containing fridata_env_conda.yml is writable.

Running tests

pytest ./tests

Usage examples

Running on AFDB structures locally

Requires having a directory with AFDB structures and a text file containing list of AFDB IDs with \n delimeter. Assuming all steps from Installation and activation succeded

FRIDATA_PATH="<repository path>"
AFDB_PATH="<AFDB structures directory path>"
IDS_PATH="<AFDB IDs file path>"

cd ${FRIDATA_PATH}

EMBEDDER_TYPE=esm2_t33_650M_UR50D

# (MACOS only) Fix for OpenMP multiple runtime error
export KMP_DUPLICATE_LIB_OK=TRUE

PYTHONPATH='.' python3 -u ${FRIDATA_PATH}/fridata.py \
 generate_data \
 -t sequences,coordinates,distograms,embeddings \
 -d AFDB \
 -c subset \
 --version test  \
 -i ${IDS_PATH} \
 --input-path ${AFDB_PATH} \
 -e ${EMBEDDER_TYPE}

For subset runs with --input-path, new datasets store canonical keys as {line_from_ids_file}_{chain} (for example A0A2K6V5L6_A), not the full AlphaFold CIF filename stem. The dataset’s input_structures.idx maps each canonical key to the source structure filename. Older datasets created before this convention may still use long AF-style keys.

Running as a CLI tool

Assuming all Installation and activation steps succeeded.

0. Go into FRIdata directory

cd <path into FRIdata>

1. Install as a CLI tool

python3 -m pip install -e .

2. Now FRIdata can be run as a CLI tool

fridata <...>

(Use ids_file tokens (e.g. plain UniProt) plus chain as the canonical dataset index keys)

Running on HPC

Running FRIdata on HPC differs on CPU and GPU nodes. This instruction set is valid for HPC hosted in PLGrid infrastructure. Running on other infrastructures may require additional adjustments.

Prerequisites:

• Having active grant valid on the HPC
• Having a full list of mandatory ENV vars set (ideally in .bashrc):
  • DEEPFRI_PATH: should always refer to a parent directory of this repo
  • IDS_PATH: path to a text file with AFDB indexes listed
  • AFDB_PATH: path to AFDB structures (can be empty directory - structures will be fetched there)
  • DATA_PATH: path to the parent diretory of all generated output data
  • Optional ENV vars with default values:
    • COMMON_SLURM_PATH: path to common_slurm.sh, defaults to $DEEPFRI_PATH/FRIdata/scripts/hpc/common_slurm.sh
    • LAUNCH_WORKER_SLURM_PATH: path to launch_worker_slurm.sh, defaults to $DEEPFRI_PATH/FRIdata/scripts/hpc/launch_workers_slurm.sh
    • MEMORY_LIMIT: memory limit per Dask worker, defaults to 288GiB
    • IP_INTERFACE: network unix interface, where dask workers are connected. Defaults to ens1f0
    • CONDA_ENV_PATH: path to conda environment, defaults to $DEEPFRI_PATH/conda_dev
• Have installed module miniconda3
• Have installed module gcc

Steps:

1. Download the repo

git clone https://github.com/Tomasz-Lab/FRIdata.git
cd FRIdata

2. Update run permissions

chmod u+x -R scripts/hpc/cpu

3. Run initialize_slurm.sh. As an argument put the path into directory, where .conda directory should be installed and specify --cpu flag if the script is run on CPU cluster.

./scripts/hpc/initialize_slurm.sh <path to .conda> [--cpu]

4. Schedule sbatch script into the HPC with all the args specified. Operations to be chosen are: sequences, coordinates, embeddings

For CPU:

sbatch --cpus-per-task=<cpus> --time=<HH:MM:SS> --nodes=<nodes> --account=<grant name> scripts/hpc/run_slurm.sh sequences,coordinates

For GPU:

sbatch --gres=gpu[:gpu-number] --time=<HH:MM:SS> --account=<grant name> --nodes=1 --partition=<partition name> --cpus-per-task=<cpus> scripts/hpc/run_slurm.sh embeddings