Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Instance anonymization #17

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Open

Instance anonymization #17

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
9b1214b
WIP Instance anonymization
maks-operlejn-ds Sep 11, 2023
a95e424
Refactor mapping creation, adding it to anonymizer
maks-operlejn-ds Sep 11, 2023
866831f
Adjust tests
maks-operlejn-ds Sep 11, 2023
429b8dc
CR
maks-operlejn-ds Sep 11, 2023
9503094
test instance anonymization
maks-operlejn-ds Sep 12, 2023
9ee9ae5
CR
maks-operlejn-ds Sep 12, 2023
2ce5256
Add descriptions about instance anonymization
maks-operlejn-ds Sep 12, 2023
90fe863
Update docs
maks-operlejn-ds Sep 12, 2023
a95d703
CR 2
maks-operlejn-ds Sep 12, 2023
8f2ac21
CR 3
maks-operlejn-ds Sep 12, 2023
ddb5cda
Add possibility to omit using faker values
maks-operlejn-ds Sep 21, 2023
bd09a82
Adjust tests
maks-operlejn-ds Sep 21, 2023
0db3ab8
Add two more tests for non faker values
maks-operlejn-ds Sep 21, 2023
97767bd
Better faker operators handling
maks-operlejn-ds Oct 5, 2023
5e70b20
Better duplicate handling
maks-operlejn-ds Oct 5, 2023
a8bfb3a
fmt
baskaryan Oct 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 85 additions & 13 deletions docs/extras/guides/privacy/presidio_data_anonymization/index.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@
{
"data": {
"text/plain": [
"'My name is Laura Ruiz, call me at +1-412-982-8374x13414 or email me at javierwatkins@example.net'"
"'My name is James Martinez, call me at (576)928-1972x679 or email me at lisa44@example.com'"
]
},
"execution_count": 2,
Expand Down Expand Up @@ -114,11 +114,11 @@
"text": [
"Dear Sir/Madam,\n",
"\n",
"We regret to inform you that Richard Fields has recently misplaced his wallet, which contains a sum of cash and his credit card bearing the number 30479847307774. \n",
"We regret to inform you that Mr. Dennis Cooper has recently misplaced his wallet. The wallet contains a sum of cash and his credit card, bearing the number 3588895295514977. \n",
"\n",
"Should you happen to come across it, we kindly request that you contact us immediately at 6439182672 or via email at frank45@example.com.\n",
"Should you happen to come across the aforementioned wallet, kindly contact us immediately at (428)451-3494x4110 or send an email to perryluke@example.com.\n",
"\n",
"Thank you for your attention to this matter.\n",
"Your prompt assistance in this matter would be greatly appreciated.\n",
"\n",
"Yours faithfully,\n",
"\n",
Expand Down Expand Up @@ -159,7 +159,7 @@
{
"data": {
"text/plain": [
"'My name is Adrian Fleming, call me at 313-666-7440 or email me at [email protected]'"
"'My name is Shannon Steele, call me at 313-666-7440 or email me at [email protected]'"
]
},
"execution_count": 6,
Expand Down Expand Up @@ -190,7 +190,7 @@
{
"data": {
"text/plain": [
"'My name is Justin Miller, call me at 761-824-1889 or email me at [email protected]'"
"'My name is Wesley Flores, call me at (498)576-9526 or email me at [email protected]'"
]
},
"execution_count": 7,
Expand Down Expand Up @@ -225,7 +225,7 @@
{
"data": {
"text/plain": [
"'My name is Dr. Jennifer Baker, call me at (508)839-9329x232 or email me at ehamilton@example.com'"
"'My name is Carla Fisher, call me at 001-683-324-0721x0644 or email me at krausejeremy@example.com'"
]
},
"execution_count": 8,
Expand Down Expand Up @@ -256,7 +256,7 @@
{
"data": {
"text/plain": [
"'My polish phone number is NRGN41434238921378'"
"'My polish phone number is QESQ21234635370499'"
]
},
"execution_count": 9,
Expand Down Expand Up @@ -361,7 +361,7 @@
{
"data": {
"text/plain": [
"'511 622 683'"
"'665 631 080'"
]
},
"execution_count": 13,
Expand Down Expand Up @@ -422,7 +422,7 @@
{
"data": {
"text/plain": [
"'My polish phone number is +48 734 630 977'"
"'My polish phone number is 538 521 657'"
]
},
"execution_count": 16,
Expand All @@ -438,8 +438,80 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Future works\n",
"- **instance anonymization** - at this point, each occurrence of PII is treated as a separate entity and separately anonymized. Therefore, two occurrences of the name John Doe in the text will be changed to two different names. It is therefore worth introducing support for full instance detection, so that repeated occurrences are treated as a single object."
"## Important considerations\n",
"\n",
"### Anonymizer detection rates\n",
"\n",
"**The level of anonymization and the precision of detection are just as good as the quality of the recognizers implemented.**\n",
"\n",
"Texts from different sources and in different languages have varying characteristics, so it is necessary to test the detection precision and iteratively add recognizers and operators to achieve better and better results.\n",
"\n",
"Microsoft Presidio gives a lot of freedom to refine anonymization. The library's author has provided his [recommendations and a step-by-step guide for improving detection rates](https://github.com/microsoft/presidio/discussions/767#discussion-3567223)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Instance anonymization\n",
"\n",
"`PresidioAnonymizer` has no built-in memory. Therefore, two occurrences of the entity in the subsequent texts will be replaced with two different fake values:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"My name is Robert Morales. Hi Robert Morales!\n",
"My name is Kelly Mccoy. Hi Kelly Mccoy!\n"
]
}
],
"source": [
"print(anonymizer.anonymize(\"My name is John Doe. Hi John Doe!\"))\n",
"print(anonymizer.anonymize(\"My name is John Doe. Hi John Doe!\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To preserve previous anonymization results, use `PresidioReverseAnonymizer`, which has built-in memory:"
maks-operlejn-ds marked this conversation as resolved.
Show resolved Hide resolved
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"My name is Ashley Cervantes. Hi Ashley Cervantes!\n",
"My name is Ashley Cervantes. Hi Ashley Cervantes!\n"
]
}
],
"source": [
"from langchain_experimental.data_anonymizer import PresidioReversibleAnonymizer\n",
"\n",
"anonymizer_with_memory = PresidioReversibleAnonymizer()\n",
"\n",
"print(anonymizer_with_memory.anonymize(\"My name is John Doe. Hi John Doe!\"))\n",
"print(anonymizer_with_memory.anonymize(\"My name is John Doe. Hi John Doe!\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can learn more about `PresidioReverseAnonymizer` in the next section."
]
}
],
Expand All @@ -459,7 +531,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.1"
"version": "3.11.4"
}
},
"nbformat": 4,
Expand Down
75 changes: 58 additions & 17 deletions docs/extras/guides/privacy/presidio_data_anonymization/reversible.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,14 +185,13 @@
"text": [
"Dear Sir/Madam,\n",
"\n",
"We regret to inform you that Mr. Dana Rhodes has reported the loss of his wallet. The wallet contains a sum of cash and his credit card, bearing the number 4397528473885757. \n",
"We regret to inform you that Monique Turner has recently misplaced his wallet, which contains a sum of cash and his credit card with the number 213152056829866. \n",
"\n",
"If you happen to come across the aforementioned wallet, we kindly request that you contact us immediately at 258-481-7074x714 or via email at laurengoodman@example.com.\n",
"If you happen to come across this wallet, kindly contact us at (770)908-7734x2835 or send an email to barbara25@example.net.\n",
"\n",
"Your prompt assistance in this matter would be greatly appreciated.\n",
"\n",
"Yours faithfully,\n",
"Thank you for your cooperation.\n",
"\n",
"Sincerely,\n",
"[Your Name]\n"
]
}
Expand Down Expand Up @@ -232,14 +231,13 @@
"text": [
"Dear Sir/Madam,\n",
"\n",
"We regret to inform you that Mr. Slim Shady has recently misplaced his wallet. The wallet contains a sum of cash and his credit card, bearing the number 4916 0387 9536 0861. \n",
"\n",
"If by any chance you come across the lost wallet, kindly contact us immediately at 313-666-7440 or send an email to [email protected].\n",
"We regret to inform you that Slim Shady has recently misplaced his wallet, which contains a sum of cash and his credit card with the number 4916 0387 9536 0861. \n",
"\n",
"Your prompt assistance in this matter would be greatly appreciated.\n",
"If you happen to come across this wallet, kindly contact us at 313-666-7440 or send an email to [email protected].\n",
"\n",
"Yours faithfully,\n",
"Thank you for your cooperation.\n",
"\n",
"Sincerely,\n",
"[Your Name]\n"
]
}
Expand Down Expand Up @@ -356,13 +354,57 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We can save the mapping itself to a file for future use: "
"Thanks to the built-in memory, entities that have already been detected and anonymised will take the same form in subsequent processed texts, so no duplicates will exist in the mapping:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"My VISA card number is 3537672423884966 and my name is William Bowman.\n"
]
},
{
"data": {
"text/plain": [
"{'PERSON': {'Maria Lynch': 'Slim Shady', 'William Bowman': 'John Doe'},\n",
" 'PHONE_NUMBER': {'7344131647': '313-666-7440'},\n",
" 'EMAIL_ADDRESS': {'[email protected]': '[email protected]'},\n",
" 'CREDIT_CARD': {'4838637940262': '4916 0387 9536 0861',\n",
" '3537672423884966': '4001 9192 5753 7193'}}"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"print(\n",
" anonymizer.anonymize(\n",
" \"My VISA card number is 4001 9192 5753 7193 and my name is John Doe.\"\n",
" )\n",
")\n",
"\n",
"anonymizer.deanonymizer_mapping"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can save the mapping itself to a file for future use: "
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"# We can save the deanonymizer mapping as a JSON or YAML file\n",
Expand All @@ -380,7 +422,7 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 12,
"metadata": {},
"outputs": [
{
Expand All @@ -389,7 +431,7 @@
"{}"
]
},
"execution_count": 11,
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
Expand All @@ -402,7 +444,7 @@
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": 13,
"metadata": {},
"outputs": [
{
Expand All @@ -415,7 +457,7 @@
" '3537672423884966': '4001 9192 5753 7193'}}"
]
},
"execution_count": 12,
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
Expand All @@ -432,7 +474,6 @@
"source": [
"## Future works\n",
"\n",
"- **instance anonymization** - at this point, each occurrence of PII is treated as a separate entity and separately anonymized. Therefore, two occurrences of the name John Doe in the text will be changed to two different names. It is therefore worth introducing support for full instance detection, so that repeated occurrences are treated as a single object.\n",
"- **better matching and substitution of fake values for real ones** - currently the strategy is based on matching full strings and then substituting them. Due to the indeterminism of language models, it may happen that the value in the answer is slightly changed (e.g. *John Doe* -> *John* or *Main St, New York* -> *New York*) and such a substitution is then no longer possible. Therefore, it is worth adjusting the matching for your needs."
]
}
Expand All @@ -453,7 +494,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.1"
"version": "3.11.4"
}
},
"nbformat": 4,
Expand Down
Loading