|
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980 |
- # Project README
-
- This project is based on `Python 3.10`. To get started, you can create an environment using conda with the following command:
-
- ```bash
- conda create -n superpos python=3.10
- ```
-
- After setting up the environment, install all the required packages with:
-
- ```bash
- pip install -r requirements.txt
- ```
-
- ## Project Structure
-
- The entry point of this project is located in the `./09_Cluster` directory. The most important files in this directory are the `config.yaml` files. Below is an example of a configuration file:
-
- ```yaml
- default: &default
- use_tqdm: true
- random_seed: 42
- base_save_path: /home/msadraei/trained_final
- model_name: google/t5-base-lm-adapt
- project_name_prefix: iclr_attempt_lmt5
- experiment_name_suffix: null
- train_batch_size: 32
- valid_batch_size: 32
- remove_dropout: true
- learning_rate: 0.01
- weight_decay: 0.01
- num_epochs: 40
- peft_params: null # no mutation
- hot_modules:
- - sadcl
- best_finder:
- save: true
- metric: valid_mean
- higher_better: true
- tasks:
- - glue:cola
- - glue:mrpc
- - glue:stsb
- - superglue:rte
- - superglue:cb
- - superglue:wic
- - superglue:copa
- - superglue:boolq
- - superglue:multirc
-
- pp: &pp
- - /home/msadraei/trained_final/hzi_cluster_t5_base_glue-mnli/10_combine_128
- - /home/msadraei/trained_final/hzi_cluster_t5_base_glue-sst2/10_combine_128
- - /home/msadraei/trained_final/hzi_cluster_t5_base_glue-qqp/10_combine_128
- - /home/msadraei/trained_final/hzi_cluster_t5_base_glue-qnli/10_combine_128
-
- run_configs:
- - <<: *default
- learning_rate: 0.3
- weight_decay: 0.00001
- peft_params:
- kind: attempt
- n_tokens: 10
- g_bottleneck: 100
- pretrained_paths: *pp
- ```
-
- ## PEFT Support
-
- This project supports different kinds of Parameter-Efficient Fine-Tuning (PEFT) methods. The valid values for PEFT types are `'combine'`, `'residual'`, `'simple'`, `'spot'`, and `'attempt'`. Each run configuration will be executed over each dataset in the list of tasks.
-
- ## Running the Project
-
- To run a configuration, use the following command:
-
- ```bash
- python train.py config.yaml
- ```
-
- This will start the training process based on the settings defined in `config.yaml`.
|