Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QA with private data - demo notebook #21

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
+1,018 −9
Open

QA with private data - demo notebook #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
4765da0
WIP Add basic version of notebook
maks-operlejn-ds Oct 10, 2023
9479344
Updated notebook with descriptions and better LCEL
maks-operlejn-ds Oct 10, 2023
1b98040
Change file format
maks-operlejn-ds Oct 10, 2023
2db632f
Revert "Change file format"
maks-operlejn-ds Oct 11, 2023
e4aec93
Add text to codespell ignore list
maks-operlejn-ds Oct 11, 2023
36a0296
Change PII name
maks-operlejn-ds Oct 11, 2023
4261a2a
Rerun notebook
maks-operlejn-ds Oct 11, 2023
c511915
Add image, correct colab link
maks-operlejn-ds Oct 11, 2023
1f14d71
Trigger Build
maks-operlejn-ds Oct 11, 2023
c866c49
merge
baskaryan Oct 11, 2023
d7e4d98
merge
baskaryan Oct 11, 2023
be48f56
cr
baskaryan Oct 11, 2023
afbf24b
Reorder notebooks in docs
maks-operlejn-ds Oct 11, 2023
4094d29
Add note about visibility of markers
maks-operlejn-ds Oct 11, 2023
e965ccd
Add sidebar title to notebook
maks-operlejn-ds Oct 11, 2023
02854a5
Small fixes to docs
maks-operlejn-ds Oct 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 20 additions & 8 deletions docs/docs/guides/privacy/presidio_data_anonymization/multi_language.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
{
"cells": [
{
"cell_type": "raw",
"metadata": {},
"source": [
"---\n",
"sidebar_position: 2\n",
"title: Multi-language anonymization\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down Expand Up @@ -248,15 +258,15 @@
"import langdetect\n",
"from langchain.schema import runnable\n",
"\n",
"\n",
"def detect_language(text: str) -> dict:\n",
" language = langdetect.detect(text)\n",
" print(language)\n",
" return {\"text\": text, \"language\": language}\n",
"\n",
"\n",
"chain = (\n",
" runnable.RunnableLambda(detect_language)\n",
" | (lambda x: anonymizer.anonymize(x[\"text\"], language=x[\"language\"]))\n",
"chain = runnable.RunnableLambda(detect_language) | (\n",
" lambda x: anonymizer.anonymize(x[\"text\"], language=x[\"language\"])\n",
")"
]
},
Expand Down Expand Up @@ -345,15 +355,17 @@
"import fasttext\n",
"\n",
"model = fasttext.load_model(\"lid.176.ftz\")\n",
"\n",
"\n",
"def detect_language(text: str) -> dict:\n",
" language = model.predict(text)[0][0].replace('__label__', '')\n",
" language = model.predict(text)[0][0].replace(\"__label__\", \"\")\n",
" print(language)\n",
" return {\"text\": text, \"language\": language}\n",
"\n",
"chain = (\n",
" runnable.RunnableLambda(detect_language)\n",
" | (lambda x: anonymizer.anonymize(x[\"text\"], language=x[\"language\"]))\n",
")\n"
"\n",
"chain = runnable.RunnableLambda(detect_language) | (\n",
" lambda x: anonymizer.anonymize(x[\"text\"], language=x[\"language\"])\n",
")"
]
},
{
Expand Down
Loading
Loading