# Project README

This project is based on `Python 3.10`. To get started, you can create an environment using conda with the following command:

```bash
conda create -n superpos python=3.10
```

After setting up the environment, install all the required packages with:

```bash
pip install -r requirements.txt
```

## Project Structure

The entry point of this project is located in the `./09_Cluster` directory. The most important files in this directory are the `config.yaml` files. Below is an example of a configuration file:

```yaml
default: &default
  use_tqdm: true 
  random_seed: 42
  base_save_path: /home/msadraei/trained_final
  model_name: google/t5-base-lm-adapt
  project_name_prefix: iclr_attempt_lmt5
  experiment_name_suffix: null
  train_batch_size: 32
  valid_batch_size: 32
  remove_dropout: true
  learning_rate: 0.01
  weight_decay: 0.01
  num_epochs: 40
  peft_params: null  # no mutation
  hot_modules:
  - sadcl
  best_finder:
    save: true
    metric: valid_mean
    higher_better: true
  tasks:
  - glue:cola
  - glue:mrpc
  - glue:stsb
  - superglue:rte
  - superglue:cb
  - superglue:wic
  - superglue:copa
  - superglue:boolq
  - superglue:multirc
  
pp: &pp
  - /home/msadraei/trained_final/hzi_cluster_t5_base_glue-mnli/10_combine_128
  - /home/msadraei/trained_final/hzi_cluster_t5_base_glue-sst2/10_combine_128
  - /home/msadraei/trained_final/hzi_cluster_t5_base_glue-qqp/10_combine_128
  - /home/msadraei/trained_final/hzi_cluster_t5_base_glue-qnli/10_combine_128

run_configs:
- <<: *default
  learning_rate: 0.3
  weight_decay: 0.00001
  peft_params:
    kind: attempt
    n_tokens: 10
    g_bottleneck: 100
    pretrained_paths: *pp
```

## PEFT Support

This project supports different kinds of Parameter-Efficient Fine-Tuning (PEFT) methods. The valid values for PEFT types are `'combine'`, `'residual'`, `'simple'`, `'spot'`, and `'attempt'`. Each run configuration will be executed over each dataset in the list of tasks.

## Running the Project

To run a configuration, use the following command:

```bash
python train.py config.yaml
```

This will start the training process based on the settings defined in `config.yaml`.