LAP

An Attention-Based Module for Faithful Interpretation and Knowledge Injection in Convolutional Neural Networks

In this supplementary material, we have provided a pdf containing more details for some of the sections mentioned in the paper, and the code of our work.

Data Preparation

Metadata

First, download the metadata from here

RSNA

CelebA

1. Download the dataset from here

2. Extract the data

   unzip -q -j celeba-dataset.zip '**/*.jpg' -d data/celeba
   

3. Extract celeba.tsv from this metadata file to dataset_metadata/celeba.tsv

Imagenet

1. Download the ImageNet ILSVRC2012 dataset from here.

2. Extract ILSVRC2012_img_train_t3.tar to data/imagenet/tars.

3. Run the following command to extract the train images per class:

   python data_preparation/imagenet/extract_train.py -s data/imagenet/tars/ -n <num-threads>
   

4. Run the following command to extract the validation images per class (replace directory with the directory containing ILSVRC2012_img_val.tar, and extract imagenet_val_maps.pklz from this metadata file):

   python data_preparation/imagenet/extract_val.py -s <directory> -m <imagenet_val_maps.pklz>
   

5. download the localizations bounding-boxes from here. Put LOC_train_solution.csv and LOC_val_solution.csv in data/imagenet/extracted.

Model checkpoints

Download all the checkpoints from here.

Run

Evaluate the checkpoints

RSNA

Vanilla ResNet 18

Evaluate the model performance on the test set

python evaluate.py rsna.org_resnet --device cuda:0 --samples-dir test --final-model-dir checkpoints/rsna.org_resnet.pth

Vanilla Inception V3

Evaluate the model performance on the test set

python evaluate.py rsna.org_inception --device cuda:0 --samples-dir test --final-model-dir checkpoints/rsna.org_inception.pth

WS ResNet 18

Evaluate the model performance on the test set

python evaluate.py rsna.ws_lap_resnet --device cuda:0 --samples-dir test --final-model-dir checkpoints/rsna.ws_resnet.pth

Generate the LAP interpretations for positive samples

python interpret.py rsna.ws_lap_resnet --device cuda:0 --samples-dir test --final-model-dir checkpoints/rsna.ws_resnet.pth --skip-raw --interpretation-method Attention --global-threshold --cut-threshold 0 --mapped-labels-to-use 1 --save-by-file-name --report-dir rsna/ws_resnet_interpretations

Generate the LAP interpretations for negative samples

python interpret.py rsna.ws_lap_resnet --device cuda:0 --samples-dir test --final-model-dir checkpoints/rsna.ws_resnet.pth --skip-raw --interpretation-method Attention --global-threshold --cut-threshold 0 --mapped-labels-to-use 0 --save-by-file-name --report-dir rsna/ws_resnet_interpretations

WS Inception V3

Evaluate the model performance on the test set

python evaluate.py rsna.ws_lap_inception --device cuda:0 --samples-dir test --final-model-dir checkpoints/rsna.ws_inception.pth

Generate the LAP interpretations for positive samples

python interpret.py rsna.ws_lap_inception --device cuda:0 --samples-dir test --final-model-dir checkpoints/rsna.ws_inception.pth --skip-raw --interpretation-method Attention --global-threshold --cut-threshold 0 --mapped-labels-to-use 1 --save-by-file-name --report-dir rsna/ws_inception_interpretations

Generate the LAP interpretations for negative samples

python interpret.py rsna.ws_lap_inception --device cuda:0 --samples-dir test --final-model-dir checkpoints/rsna.ws_inception.pth --skip-raw --interpretation-method Attention --global-threshold --cut-threshold 0 --mapped-labels-to-use 0 --save-by-file-name --report-dir rsna/ws_inception_interpretations

BB ResNet 18

Evaluate the model performance on the test set

python evaluate.py rsna.bb_lap_resnet --device cuda:0 --samples-dir test --final-model-dir checkpoints/rsna.bb_resnet.pth

Generate the LAP interpretations for positive samples

python interpret.py rsna.bb_lap_resnet --device cuda:0 --samples-dir test --final-model-dir checkpoints/rsna.bb_resnet.pth --skip-raw --interpretation-method Attention --global-threshold --cut-threshold 0 --mapped-labels-to-use 1 --save-by-file-name --report-dir rsna/bb_resnet_interpretations

Generate the LAP interpretations for negative samples

python interpret.py rsna.bb_lap_resnet --device cuda:0 --samples-dir test --final-model-dir checkpoints/rsna.bb_resnet.pth --skip-raw --interpretation-method Attention --global-threshold --cut-threshold 0 --mapped-labels-to-use 0 --save-by-file-name --report-dir rsna/bb_resnet_interpretations

BB Inception V3

Evaluate the model performance on the test set

python evaluate.py rsna.bb_lap_inception --device cuda:0 --samples-dir test --final-model-dir checkpoints/rsna.bb_inception.pth

Generate the LAP interpretations for positive samples

python interpret.py rsna.bb_lap_inception --device cuda:0 --samples-dir test --final-model-dir checkpoints/rsna.bb_inception.pth --skip-raw --interpretation-method Attention --global-threshold --cut-threshold 0 --mapped-labels-to-use 1 --save-by-file-name --report-dir rsna/bb_inception_interpretations

Generate the LAP interpretations for negative samples

python interpret.py rsna.ws_lap_inception --device cuda:0 --samples-dir test --final-model-dir checkpoints/rsna.ws_inception.pth --skip-raw --interpretation-method Attention --global-threshold --cut-threshold 0 --mapped-labels-to-use 0 --save-by-file-name --report-dir rsna/ws_inception_interpretations

CelebA

Vanilla ResNet 18

Evaluate the model performance on the test set

python evaluate.py celeba.org_resnet --device cuda:0 --samples-dir test --final-model-dir checkpoints/celeba.org_resnet.pth

Vanilla Inception V3

Evaluate the model performance on the test set

python evaluate.py celeba.org_inception --device cuda:0 --samples-dir test --final-model-dir checkpoints/celeba.org_inception.pth

WS ResNet 18

Evaluate the model performance on the test set

python evaluate.py celeba.ws_lap_resnet --device cuda:0 --samples-dir test --final-model-dir checkpoints/celeba.ws_resnet.pth

Generate the LAP interpretations for positive samples

python interpret.py celeba.ws_lap_resnet --device cuda:0 --samples-dir test --final-model-dir checkpoints/celeba.ws_resnet.pth --skip-raw --interpretation-method Attention --global-threshold --cut-threshold 0 --mapped-labels-to-use 1 --save-by-file-name --report-dir celeba/ws_resnet_interpretations

Generate the LAP interpretations for negative samples

python interpret.py celeba.ws_lap_resnet --device cuda:0 --samples-dir test --final-model-dir checkpoints/celeba.ws_resnet.pth --skip-raw --interpretation-method Attention --global-threshold --cut-threshold 0 --mapped-labels-to-use 0 --save-by-file-name --report-dir celeba/ws_resnet_interpretations

WS Inception V3

Evaluate the model performance on the test set

python evaluate.py celeba.ws_lap_inception --device cuda:0 --samples-dir test --final-model-dir checkpoints/celeba.ws_inception.pth

Generate the LAP interpretations for positive samples

python interpret.py celeba.ws_lap_inception --device cuda:0 --samples-dir test --final-model-dir checkpoints/celeba.ws_inception.pth --skip-raw --interpretation-method Attention --global-threshold --cut-threshold 0 --mapped-labels-to-use 1 --save-by-file-name --report-dir celeba/ws_inception_interpretations

Generate the LAP interpretations for negative samples

python interpret.py celeba.ws_lap_inception --device cuda:0 --samples-dir test --final-model-dir checkpoints/celeba.ws_inception.pth --skip-raw --interpretation-method Attention --global-threshold --cut-threshold 0 --mapped-labels-to-use 0 --save-by-file-name --report-dir celeba/ws_inception_interpretations

ImageNet

WS ResNet 50 (Fine-tuned)

Evaluate the model performance on the validation set

python evaluate.py imagenet.lap_resnet50_ft --device cuda:0 --samples-dir val --final-model-dir checkpoints/imagenet.ws_resnet_ft.pth