Add new feature of SafeLoRA #2201

chiayi-hsu · 2024-11-06T17:27:15Z

The pull request was closed due to syncing with the latest version of PEFT, so I have requested the pull request again.
I have made all the necessary changes based on our previous conversations in this version.

If there are any issues, please let me know.

Thank you.

…method of loading the peft config.

BenjaminBossan

Thanks for the update to the SafeLoRA PR. I did another review and found a few areas to improve. Please take a look. Also, please run make style once you're finished with your changed.

examples/safelora/README.md

BenjaminBossan · 2024-11-07T18:17:47Z

examples/safelora/README.md

+                            save_weights=True)
+
+final_lora_weight = apply_safelora(config)
+


Can we add a bit more to the example. For instance, how to save and load these weights?

I have added more descriptions to the example.
If you feel there are still any missing parts, please let me know.

BenjaminBossan · 2024-11-07T18:18:11Z

examples/safelora/README.md

+config = SafeLoraConfig(base_model_path='../LLM_Models/llama-2-7b-hf/',\
+                            aligned_model_path='../LLM_Models/llama-2-7b-chat-fp16/',


Let's use the HF model ids for these two.

Has been modified.

BenjaminBossan · 2024-11-07T18:20:13Z

src/peft/utils/safelora.py

+            peft_weights = {name: f.get_tensor(name).to(safelora_config.dtype) for name in f.keys()}
+        else:
+            peft_weights = {name: f.get_tensor(name).to(safelora_config.dtype) for name in f.keys()}


These 2 lines are identical

Has been modified.

- if (safelora_config.devices).lower() == "cpu": - peft_weights = {name: f.get_tensor(name).to(safelora_config.dtype) for name in f.keys()} - else: - peft_weights = {name: f.get_tensor(name).to(safelora_config.dtype) for name in f.keys()} + peft_weights = {name: f.get_tensor(name).to(safelora_config.dtype) for name in f.keys()}

BenjaminBossan · 2024-11-07T18:31:17Z

src/peft/utils/safelora.py

+    ]
+    align_model_parameters = [
+        name for name in sl_align.weight_map.keys() if any(v in name for v in list(peft_config.target_modules))
+    ]


Should we also check that base_model_parameters and align_model_parameters are the same?

I have added a check to verify if the model weights are the same.

+ if (sl_base.get_tensor(name_base) == sl_align.get_tensor(name_align)).all(): + raise ValueError("The weights of the base Model and the aligned Model should be different.")

I meant something else. Would we expect that base_model_parameters == align_model_parameters? If not, under what circumstances would they differ?

BenjaminBossan · 2024-11-07T18:32:14Z

src/peft/utils/safelora.py

+    return safety_vector
+
+
+def project_weights(configs, peft_weights, v):


Let's rename configs to config or safelora_config.

Has been modified.

BenjaminBossan · 2024-11-07T18:33:18Z

src/peft/utils/safelora.py

+        metadata={"help": "The path of the LoRA wieghts and configs."},
+    )
+
+    select_layers_type: str = field(


Instead of str, we can annotate this as Literal["threshold", "number"].

Has been modified.

src/peft/utils/safelora.py

BenjaminBossan · 2024-11-07T18:35:15Z

examples/safelora/safelora_inference.py

+                            select_layers_type='threshold',
+                            save_weights=True)
+
+final_lora_weight = apply_safelora(config)


The example should show inference, here we only create the weights. What are the next steps?

I have added more explanations in the README.md and also included code on how to use the SafeLoRA model.

Co-authored-by: Benjamin Bossan <[email protected]>

…lora.py

BenjaminBossan · 2024-11-15T16:22:04Z

@chiayi-hsu Once you're finished with your changes and want me to give another review, please ping me.

chiayi-hsu · 2024-11-19T06:57:26Z

@BenjaminBossan I have completed the modifications. Please help review them. Thanks!

BenjaminBossan

Thanks a lot for the updates. I did another review. Most of what I found are just smaller things like docs, please take a look.

Now as a next step, it is important that we also add some unit tests. This not going to be very straightforward, because we cannot easily test model alignment and we also don't want to use any big models during unit testing.

One proposal for this would be to use a small model like hf-internal-testing/tiny-random-OPTForCausalLM as the base model. Then let's modify some weights (setting them to 0?) and save this as the "aligned" model. Then call apply_safelora with these 2 models and various options to see if those tests pass. This would not really check the alignment though.

In addition, we could think about adding a true alignment test for the nightly run with GPU. For this test, it would be okay to use a bigger model (but ideally still not too big).

LMK what you think about this testing strategy and if you have further questions.

Apart from this, please call make style on your PR, as this is a prerequisite for the CI to pass.

BenjaminBossan · 2024-11-20T10:07:41Z

src/peft/utils/safelora.py

+    This is the configuration class to store the configuration of a safeLora.
+
+
+    Args:


Could you please format the docstring to be in line with the other docstrings used in PEFT. As an example, check here:

peft/src/peft/tuners/lora/config.py

Line 128 in 8874ab5

Args:

BenjaminBossan · 2024-11-20T10:08:08Z

src/peft/utils/safelora.py

+        default="meta-llama/Llama-2-7b-hf",
+        metadata={"help": "The path of the base model for obtaining the aligned matrix."},
+    )
+
+    aligned_model_path: str = field(
+        default="TheBloke/Llama-2-7B-Chat-fp16",
+        metadata={"help": "The path of the aligned model for obtaining the aligned matrix."},
+    )
+
+    peft_model_path: str = field(
+        default="LisaSchunke/llama-2-7b-peft-finetuned-20000-dataset",


IMO, it doesn't make sense to set default values here, I would remove them. WDYT?

BenjaminBossan · 2024-11-20T10:08:32Z

src/peft/utils/safelora.py

+
+    peft_model_path: str = field(
+        default="LisaSchunke/llama-2-7b-peft-finetuned-20000-dataset",
+        metadata={"help": "The path of the LoRA wieghts and configs."},


Suggested change

metadata={"help": "The path of the LoRA wieghts and configs."},

metadata={"help": "The path of the LoRA weights and config."},

BenjaminBossan · 2024-11-20T10:09:14Z

src/peft/utils/safelora.py

+
+    device: str = field(
+        default="cuda",
+        metadata={"help": "Device is used in SafeLoRA. (cuda or cpu)"},


Suggested change

metadata={"help": "Device is used in SafeLoRA. (cuda or cpu)"},

metadata={"help": "Device that is used for SafeLoRA (cuda or cpu)."},

src/peft/utils/safelora.py

BenjaminBossan · 2024-11-20T10:48:15Z

src/peft/utils/safelora.py

+    After fine-tuning large language models (LLMs) using LoRA, the alignment of the resulting models may decrease.
+    Therefore, applying `apply_safelora()` is intended to help preserve the alignment of the final models.
+
+    It is important to note that the model weights of the aligned model and the base model must be of the same size.


Let's also mention that right now, only safetensors format is supported.

BenjaminBossan · 2024-11-20T10:49:00Z

src/peft/utils/safelora.py

+    )
+
+    with safe_open(
+        f"{os.path.join(safelora_config.peft_model_path, 'adapter_model.safetensors')}",


Let's not hard-code adapter_model.safetensors, let's use peft.utils.constants.SAFETENSORS_WEIGHTS_NAME.

BenjaminBossan · 2024-11-20T10:49:04Z

src/peft/utils/safelora.py

+        final_weights, _ = project_weights(safelora_config, peft_weights, projected_matrix)
+
+    if safelora_config.save_weights:
+        save_file(final_weights, f"{os.path.join(safelora_config.peft_model_path, 'adapter_model.safetensors')}")


Let's not hard-code adapter_model.safetensors, let's use peft.utils.constants.SAFETENSORS_WEIGHTS_NAME.

examples/safelora/README.md

Co-authored-by: Benjamin Bossan <[email protected]>

chiayi-hsu added 7 commits November 6, 2024 14:35

change variablle names and modify the class of _SafetensorLoader

842a424

modify safelora.py

7610aa1

docs, refactor: Add the config and function description./ Modify the …

00dac0b

…method of loading the peft config.

fix: Adding the dtype argument that users can select.

d962bdf

style: Adding the annotation of SafeLoraConfig.

609891e

docs: Adding an example of safelora.

65ad744

Style: Change READEME of safelora.

c682be3

BenjaminBossan requested changes Nov 7, 2024

View reviewed changes

chiayi-hsu and others added 6 commits November 9, 2024 02:20

Update examples/safelora/README.md

8d7ea67

Co-authored-by: Benjamin Bossan <[email protected]>

Update examples/safelora/README.md

9be2429

Co-authored-by: Benjamin Bossan <[email protected]>

Update src/peft/utils/safelora.py

5ee3c83

Co-authored-by: Benjamin Bossan <[email protected]>

Merge remote-tracking branch 'upstream/main' into main

e8ab799

docs/refactors: Add more steps of the inference example./ Modify safe…

b27c9e2

…lora.py

docs: Change README.md

71e9467

BenjaminBossan requested changes Nov 20, 2024

View reviewed changes

chiayi-hsu and others added 4 commits November 25, 2024 21:32

Update examples/safelora/README.md

9b0b06e

Co-authored-by: Benjamin Bossan <[email protected]>

Update examples/safelora/README.md

4e1e702

Co-authored-by: Benjamin Bossan <[email protected]>

Update src/peft/utils/safelora.py

fdb7af5

Co-authored-by: Benjamin Bossan <[email protected]>

Update src/peft/utils/safelora.py

7ec02e7

Co-authored-by: Benjamin Bossan <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new feature of SafeLoRA #2201

Add new feature of SafeLoRA #2201

chiayi-hsu commented Nov 6, 2024

BenjaminBossan left a comment

BenjaminBossan Nov 7, 2024

chiayi-hsu Nov 14, 2024

BenjaminBossan Nov 7, 2024

chiayi-hsu Nov 14, 2024

BenjaminBossan Nov 7, 2024

chiayi-hsu Nov 14, 2024

BenjaminBossan Nov 7, 2024

chiayi-hsu Nov 14, 2024

BenjaminBossan Nov 20, 2024

BenjaminBossan Nov 7, 2024

chiayi-hsu Nov 14, 2024

BenjaminBossan Nov 7, 2024

chiayi-hsu Nov 14, 2024

BenjaminBossan Nov 7, 2024

chiayi-hsu Nov 14, 2024

BenjaminBossan commented Nov 15, 2024

chiayi-hsu commented Nov 19, 2024

BenjaminBossan left a comment

BenjaminBossan Nov 20, 2024

BenjaminBossan Nov 20, 2024

BenjaminBossan Nov 20, 2024

BenjaminBossan Nov 20, 2024

BenjaminBossan Nov 20, 2024

BenjaminBossan Nov 20, 2024

BenjaminBossan Nov 20, 2024

		save_weights=True)

		final_lora_weight = apply_safelora(config)

		config = SafeLoraConfig(base_model_path='../LLM_Models/llama-2-7b-hf/',\
		aligned_model_path='../LLM_Models/llama-2-7b-chat-fp16/',

		return safety_vector


		def project_weights(configs, peft_weights, v):

		This is the configuration class to store the configuration of a safeLora.


		Args:

	metadata={"help": "The path of the LoRA wieghts and configs."},
	metadata={"help": "The path of the LoRA weights and config."},

	metadata={"help": "Device is used in SafeLoRA. (cuda or cpu)"},
	metadata={"help": "Device that is used for SafeLoRA (cuda or cpu)."},

Add new feature of SafeLoRA #2201

Are you sure you want to change the base?

Add new feature of SafeLoRA #2201

Conversation

chiayi-hsu commented Nov 6, 2024

BenjaminBossan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenjaminBossan commented Nov 15, 2024

chiayi-hsu commented Nov 19, 2024

BenjaminBossan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment