diff --git a/README.md b/README.md index 0581a5fdd..6da4ba032 100644 --- a/README.md +++ b/README.md @@ -154,6 +154,8 @@ Various tutorials are given for [built-in models](docs/built-in-models/ml_exampl - [Health diagnosis](use_case_examples/disease_prediction/): based on a patient's symptoms, history and other health factors, give a diagnosis using FHE to preserve the privacy of the patient. +- [Private inference for federated learned models](use_case_examples/federated_learning/): private training of a Logistic Regression model and then import the model into Concrete ML and perform encrypted prediction + - [Titanic](use_case_examples/titanic/KaggleTitanic.ipynb): solve the [Kaggle Titanic competition](https://www.kaggle.com/c/titanic/). Implemented with XGBoost from Concrete ML, this example comes as a companion of the [Kaggle notebook](https://www.kaggle.com/code/concretemlteam/titanic-with-privacy-preserving-machine-learning), and was the subject of a blogpost in [KDnuggets](https://www.kdnuggets.com/2022/08/machine-learning-encrypted-data.html). - [Sentiment analysis with transformers](use_case_examples/sentiment_analysis_with_transformer): predict if an encrypted tweet / short message is positive, negative or neutral, using FHE. The [live interactive](https://huggingface.co/spaces/zama-fhe/encrypted_sentiment_analysis) demo is available on Hugging Face. This [blog post](https://huggingface.co/blog/sentiment-analysis-fhe) explains how this demo works! diff --git a/docs/README.md b/docs/README.md index e477d1444..0a4a44f09 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,7 +8,7 @@ Concrete ML is an open source, privacy-preserving, machine learning framework ba Fully Homomorphic Encryption is an encryption technique that allows computing directly on encrypted data, without needing to decrypt it. With FHE, you can build private-by-design applications without compromising on features. You can learn more about FHE in [this introduction](https://www.zama.ai/post/tfhe-deep-dive-part-1) or by joining the [FHE.org](https://fhe.org) community. -Training on encrypted data provides the highest level of privacy but is slower than training on clear data. Federated learning is an alternative approach, where data privacy can be ensured through _differential privacy_ instead of encryption. Concrete ML +Training on encrypted data provides the highest level of privacy but is slower than training on clear data. Federated learning is an alternative approach, where data privacy can be ensured by using a trusted gradient aggregator, coupled with optional _differential privacy_ instead of encryption. Concrete ML can import linear models, including logistic regression, that are trained using federated learning using the [`from_sklearn` function](./built-in-models/linear.md#pre-trained-models). ## Example usage @@ -89,7 +89,7 @@ This example shows the typical flow of a Concrete ML model: To make a model work with FHE, the only constraint is to make it run within the supported precision limitations of Concrete ML (currently 16-bit integers). Thus, machine learning models must be quantized, which sometimes leads to a loss of accuracy versus the original model, which operates on plaintext. -Additionally, Concrete ML currently only supports training on encrypted data for some models, while it supports _inference_ for a large variety of models. +Additionally, Concrete ML currently only supports training on encrypted data for some models, while it supports _inference_ for a large variety of models. Finally, there is currently no support for pre-processing model inputs and post-processing model outputs. These processing stages may involve text-to-numerical feature transformation, dimensionality reduction, KNN or clustering, featurization, normalization, and the mixing of results of ensemble models. diff --git a/docs/advanced-topics/advanced_features.md b/docs/advanced-topics/advanced_features.md index ab8fdd837..91761b4dd 100644 --- a/docs/advanced-topics/advanced_features.md +++ b/docs/advanced-topics/advanced_features.md @@ -14,8 +14,8 @@ Concrete ML has a _simulation_ mode where the impact of approximate computation In Concrete ML, there are three different ways to define the tolerance to off-by-one errors for each TLU operation: -- setting `p_error`, the error probability of an individual TLU (see [here](advanced_features.md#an-error-probability-for-an-individual-tlu)) -- setting `global_p_error`, the error probability of the full circuit (see [here](advanced_features.md#a-global-error-probability-for-the-entire-model)) +- setting `p_error`, the error probability of an individual TLU (see [here](advanced_features.md#tolerance-to-off-by-one-error-for-an-individual-tlu)) +- setting `global_p_error`, the error probability of the full circuit (see [here](advanced_features.md#a-global-tolerance-for-one-off-errors-for-the-entire-model)) - not setting `p_error` nor `global_p_error`, and using default parameters (see [here](advanced_features.md#using-default-error-probability)) {% hint style="warning" %} diff --git a/docs/built-in-models/training.md b/docs/built-in-models/training.md index 57aee642e..4cd2c5e27 100644 --- a/docs/built-in-models/training.md +++ b/docs/built-in-models/training.md @@ -27,7 +27,7 @@ model.fit(X_binary, y_binary, fhe="execute") ``` {% hint style="info" %} -Training on encrypted data provides the highest level of privacy but is slower than training on clear data. Federated learning is an alternative approach, where data privacy can be ensured through _differential privacy_ instead of encryption. Concrete ML +Training on encrypted data provides the highest level of privacy but is slower than training on clear data. Federated learning is an alternative approach, where data privacy can be ensured by using a trusted gradient aggregator, coupled with optional _differential privacy_ instead of encryption. Concrete ML can import linear models, including logistic regression, that are trained using federated learning using the [`from_sklearn` function](linear.md#pre-trained-models). {% endhint %} @@ -42,5 +42,5 @@ The `parameters_range` parameter determines the initialization of the coefficien The logistic model that can be trained uses Stochastic Gradient Descent (SGD) and quantizes for data, weights, gradients and the error measure. It currently supports training 6-bit models, training both the coefficients and the bias. -The `SGDClassifier` does not currently support training models with other values for the bit-widths. The time to train the model -is proportional to the number of features and the number of training examples. The `SGDClassifier` training does not currently support client/server deployment for training. +The `SGDClassifier` does not currently support training models with other values for the bit-widths. The execution time to train a model +is proportional to the number of features and the number of training examples in the batch. The `SGDClassifier` training does not currently support client/server deployment for training. diff --git a/docs/deep-learning/fhe_assistant.md b/docs/deep-learning/fhe_assistant.md index 4f306605b..f0ecf46c2 100644 --- a/docs/deep-learning/fhe_assistant.md +++ b/docs/deep-learning/fhe_assistant.md @@ -58,8 +58,9 @@ concrete_clf.compile(X, debug_config) ## Compilation error debugging Compilation errors that signal that the ML model is not FHE compatible are usually of two types: + 1. TLU input maximum bit-width is exceeded -2. No crypto-parameters can be found for the ML model: `RuntimeError: NoParametersFound` is raised by the compiler +1. No crypto-parameters can be found for the ML model: `RuntimeError: NoParametersFound` is raised by the compiler The following produces a neural network that is not FHE-compatible: @@ -115,10 +116,12 @@ Function you are trying to compile cannot be compiled: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this 17-bit value is used as an input to a table lookup ``` -The error `this 17-bit value is used as an input to a table lookup` indicates that the 16-bit limit on the input of the Table Lookup (TLU) operation has been exceeded. To pinpoint the model layer that causes the error, Concrete ML provides the [bitwidth_and_range_report](../developer-guide/api/concrete.ml.quantization.quantized_module.md#method-bitwidth_and_range_report) helper function. First, the model must be compiled so that it can be [simulated](#simulation). +The error `this 17-bit value is used as an input to a table lookup` indicates that the 16-bit limit on the input of the Table Lookup (TLU) operation has been exceeded. To pinpoint the model layer that causes the error, Concrete ML provides the [bitwidth_and_range_report](../developer-guide/api/concrete.ml.quantization.quantized_module.md#method-bitwidth_and_range_report) helper function. First, the model must be compiled so that it can be [simulated](#simulation). To make this network FHE-compatible one can apply several techniques: + 1. use [rounded accumulators](../advanced-topics/advanced_features.md#rounded-activations-and-quantizers) by specifying the `rounding_threshold_bits` parameter. Please evaluate the accuracy of the model using simulation if you use this feature, as it may impact accuracy. Setting a value 2-bit higher than the quantization `n_bits` should be a good start. + ```python @@ -132,7 +135,6 @@ quantized_numpy_module = compile_torch_model( ) ``` - 2. reduce the accumulator bit-width of the second layer named `fc2`. To do this, a simple solution is to reduce the number of neurons, as it is proportional to the bit-width. diff --git a/docs/deep-learning/torch_support.md b/docs/deep-learning/torch_support.md index dd2cefac2..578a8e9fc 100644 --- a/docs/deep-learning/torch_support.md +++ b/docs/deep-learning/torch_support.md @@ -6,7 +6,6 @@ As [Quantization Aware Training (QAT)](../advanced-topics/quantization.md) is th The following example uses a simple QAT PyTorch model that implements a fully connected neural network with two hidden layers. Due to its small size, making this model respect FHE constraints is relatively easy. - {% hint style="info" %} Converting neural networks to use FHE can be done with `compile_brevitas_qat_model` or with `compile_torch_model` for post-training quantization. If the model can not be converted to FHE two types of errors can be raised: (1) crypto-parameters can not be found and, (2) table look-up bit-width limit is exceeded. See the [debugging section](./fhe_assistant.md#compilation-error-debugging) if you encounter these errors. {% endhint %}