Skip to content

Commit

Permalink
update readme
Browse files Browse the repository at this point in the history
  • Loading branch information
Guangxuan-Xiao committed Feb 9, 2023
1 parent 807d25e commit ced9774
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 5 deletions.
8 changes: 3 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Offsite-Tuning: Transfer Learning without Full Model [[paper]()]
![overview](figures/overview.png)
<img src="figures/overview.png" width="600">

## Abstract

Expand Down Expand Up @@ -38,19 +38,17 @@ In this repository, you will find all the necessary components to reproduce the
## Results

- Comparing existing fine-tuning approaches (top and middle) and Offsite-Tuning (bottom). (a) Traditionally, users send labeled data to model owners for fine-tuning, raising privacy concerns and incurring high computational costs. (b) Model owner sending the full model to the data owner is not practical, which threatens the ownership of the proprietary model, and it's not affordable for users to fine-tune the huge foundation model due to resource constraints. (c) Offsite-tuning offers a privacy-preserving and efficient alternative to traditional fine-tuning methods that require access to full model weights.
![paradigm](figures/paradigm.png)
<img src="figures/paradigm.png" width="600">

- On 1-billion scale language models, Offsite-tuning (OT Plug-in) improves zero-shot (ZS) performance across all tasks, with only slight decreases compared to full fine-tuning (FT). Also, a consistent performance gap is observed between the emulator fine-tuning and plug-in, indicating offsite-tuning effectively preserves the privacy of the original proprietary model (users can not use the emulator to achieve the same performance).
![lm_results](figures/lm_results.png)

- Offsite-tuning also works on language models over 6-billion parameters.
![llm_results](figures/llm_results.png)

- Offsite-tuning also works on vision foundation models.
![cv_results](figures/cv_results.png)

- Offsite-Tuning significantly increase the fine-tuning throughput and reduce the memory footprint compared to the existing fine-tuning methods.
![efficiency](figures/efficiency.png)
<img src="figures/efficiency.png" width="600">

## Citation

Expand Down
Binary file removed figures/cv_results.png
Binary file not shown.

0 comments on commit ced9774

Please sign in to comment.