Skip to content

Commit

Permalink
chore: fix forbidden words
Browse files Browse the repository at this point in the history
  • Loading branch information
RomanBredehoft committed Jun 17, 2024
1 parent 4904c7b commit 27b37de
Show file tree
Hide file tree
Showing 10 changed files with 20 additions and 20 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@
"This step is easy to achieve as we use a built-in Fully Connected Neural Network. Only a few input parameters are needed:\n",
"- `module__n_layers`: number of Fully Connected layers to use in the model\n",
"- `module__n_w_bits` and `module__n_a_bits`: respectively the number of bits to use for quantizing the weight and input/activation values as the FHE can currently only compute integers. These numbers should not become too large as it can cause the compilation step to fail (see Compile section below)\n",
"- `module__n_accum_bits`: The maximal allowed bit-width to target for intermediate accumulators. It is currently set to 15 as the actual maximum bit width reached during compilation can be up to one bit higher than this targeted value, in this case 16, which is the maximal value that Concrete ML currently supports.\n",
"- `module__n_accum_bits`: The maximal allowed bit-width to target for intermediate accumulators. It is currently set to 15 as the actual maximum bit-width reached during compilation can be up to one bit higher than this targeted value, in this case 16, which is the maximal value that Concrete ML currently supports.\n",
"- `module__n_hidden_neurons_multiplier`: A factor that is multiplied by the maximal number of active (non-zero weight) neurons for every layer. Default to 4 but set to 1 here in order to speed up all executions without changing the test accuracy by much. More detail in the qqn documentation. \n",
"- `module__activation_function`: The activation function to use\n",
"- `max_epochs`: The number of epochs to consider"
Expand Down
2 changes: 1 addition & 1 deletion docs/advanced_examples/LogisticRegression.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,7 @@
"metadata": {},
"source": [
"### Generate the keys\n",
"We can determine the maximum bit width reached within the FHE circuit."
"We can determine the maximum bit-width reached within the FHE circuit."
]
},
{
Expand Down
4 changes: 2 additions & 2 deletions docs/advanced_examples/PoissonRegression.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -893,7 +893,7 @@
"id": "de58b9eb",
"metadata": {},
"source": [
"### Test the multivariate GLM with multiple quantization bit widths for inputs"
"### Test the multivariate GLM with multiple quantization bit-widths for inputs"
]
},
{
Expand Down Expand Up @@ -940,7 +940,7 @@
"id": "6dcb5f7e",
"metadata": {},
"source": [
"We then plot the Poisson deviance with respect to the quantized bit width in order to show how performance degrades with quantization:"
"We then plot the Poisson deviance with respect to the quantized bit-width in order to show how performance degrades with quantization:"
]
},
{
Expand Down
4 changes: 2 additions & 2 deletions docs/advanced_examples/SVMClassifier.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -433,7 +433,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"FHE Accuracy: 0.9765 (bit width: 17)\n"
"FHE Accuracy: 0.9765 (bit-width: 17)\n"
]
}
],
Expand All @@ -444,7 +444,7 @@
"y_pred = svm_concrete.predict(X_test, fhe=\"execute\")\n",
"accuracy = accuracy_score(y_test, y_pred)\n",
"# print the accuracy\n",
"print(f\"FHE Accuracy: {accuracy:.4f} (bit width: {circuit.graph.maximum_integer_bit_width()})\")"
"print(f\"FHE Accuracy: {accuracy:.4f} (bit-width: {circuit.graph.maximum_integer_bit_width()})\")"
]
},
{
Expand Down
2 changes: 1 addition & 1 deletion docs/built-in-models/encrypted_dataframe.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This document introduces how to construct and perform operations on encrypted Da

Encrypted DataFrames are a storage format for encrypted tabular data. You can exchange encrypted DataFrames with third parties to collaborate without privacy risks. Potential applications include:

- Encrypt storage of tabular datasets
- Encrypt storage of tabular data-sets
- Joint data analysis efforts between multiple parties
- Data preparation steps before machine learning tasks, such as inference or training
- Secure outsourcing of data analysis to untrusted third parties
Expand Down
4 changes: 2 additions & 2 deletions docs/built-in-models/linear.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The use of the following models in FHE is very similar to the use of scikit-lear

## Quantization parameters

The `n_bits` parameter controls the bit width of the inputs and weights of the linear models. Linear models do not use table lookups and thus alllows weight and inputs to be high precision integers.
The `n_bits` parameter controls the bit-width of the inputs and weights of the linear models. Linear models do not use table lookups and thus alllows weight and inputs to be high precision integers.

For models with input dimensions up to `300`, the parameter `n_bits` can be set to `8` or more. When the input dimensions are larger, `n_bits` must be reduced to `6-7`. In many cases, quantized models can preserve all performance metrics compared to the non-quantized float models from scikit-learn when `n_bits` is down to `6`. You should validate accuracy on held-out test sets and adjust `n_bits` accordingly.

Expand All @@ -40,7 +40,7 @@ You can convert an already trained scikit-learn linear model to a Concrete ML on

## Example

The following example shows how to train a LogisticRegression model on a simple dataset and then use FHE to perform inference on encrypted data. You can find a more complete example in the [LogisticRegression notebook](../tutorials/ml_examples.md).
The following example shows how to train a LogisticRegression model on a simple data-set and then use FHE to perform inference on encrypted data. You can find a more complete example in the [LogisticRegression notebook](../tutorials/ml_examples.md).

```python
from sklearn.datasets import make_classification
Expand Down
6 changes: 3 additions & 3 deletions docs/built-in-models/nearest-neighbors.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ concrete_classifier = KNeighborsClassifier(n_bits=2, n_neighbors=3)

## Quantization parameters

The `KNeighborsClassifier` class quantizes the training dataset provided to `.fit` using the specified number of bits (`n_bits`). To comply with [accumulator size constraints](../getting-started/concepts.md#model-accuracy-considerations-under-fhe-constraints), you must keep this value low. The model's accuracy will depend significantly on a well-chosen `n_bits` value and the dimensionality of the data.
The `KNeighborsClassifier` class quantizes the training data-set provided to `.fit` using the specified number of bits (`n_bits`). To comply with [accumulator size constraints](../getting-started/concepts.md#model-accuracy-considerations-under-fhe-constraints), you must keep this value low. The model's accuracy will depend significantly on a well-chosen `n_bits` value and the dimensionality of the data.

The `predict` method of the `KNeighborsClassifier` performs the following steps:

Expand All @@ -26,6 +26,6 @@ The `predict` method of the `KNeighborsClassifier` performs the following steps:

## Inference time considerations

The FHE inference latency of this model is heavily influenced by the `n_bits` and the dimensionality of the data. Additionally, the dataset size has a linear impact on the data complexity. The number of nearest neighbors (`n_neighbors`) also affects performance.
The FHE inference latency of this model is heavily influenced by the `n_bits` and the dimensionality of the data. Additionally, the data-set size has a linear impact on the data complexity. The number of nearest neighbors (`n_neighbors`) also affects performance.

The KNN computation executes in FHE in $$O(Nlog^2k)$$ steps, where $$N$$ is the training data-set size and $$k$$ is `n_neighbors`. Each step requires several [PBS operations](/docs/getting-started/concepts.md#cryptography-concepts), with their runtime affected by the factors listed above. These factors determine the precision needed to represent the distances between test vectors and training dataset vectors. The PBS input precision required by the circuit is related to the precision of the distance values.
The KNN computation executes in FHE in $$O(Nlog^2k)$$ steps, where $$N$$ is the training data-set size and $$k$$ is `n_neighbors`. Each step requires several [PBS operations](/docs/getting-started/concepts.md#cryptography-concepts), with their runtime affected by the factors listed above. These factors determine the precision needed to represent the distances between test vectors and training data-set vectors. The PBS input precision required by the circuit is related to the precision of the distance values.
4 changes: 2 additions & 2 deletions docs/built-in-models/neural-networks.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,8 @@ The folowing figure shows the Concrete ML neural network trained with Quantizati

- `n_w_bits` (default 3): number of bits for weights
- `n_a_bits` (default 3): number of bits for activations and inputs
- `n_accum_bits`: maximum accumulator bit width that is desired
- By default, this is unbounded, which, for weight and activation bit-width settings, [may make the trained networks fail in compilation](neural-networks.md#overflow-errors). When used, the implementation will attempt to keep accumulators under this bit width through [pruning](../explanations/pruning.md) (for example, setting some weights to zero).
- `n_accum_bits`: maximum accumulator bit-width that is desired
- By default, this is unbounded, which, for weight and activation bit-width settings, [may make the trained networks fail in compilation](neural-networks.md#overflow-errors). When used, the implementation will attempt to keep accumulators under this bit-width through [pruning](../explanations/pruning.md) (for example, setting some weights to zero).
- `power_of_two_scaling` (default True): forces quantization scales to be powers-of-two
- When coupled with the ReLU activation, this optimize strongly the FHE inference time.
- See this [section](../explanations/quantization.md#quantization-special-cases) in the quantization documentation for more details.
Expand Down
10 changes: 5 additions & 5 deletions docs/built-in-models/tree.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Using the maximum depth parameter of decision trees and tree-ensemble models str

## Example

Here's an example of how to use this model in FHE on a popular dataset using some of scikit-learn's pre-processing tools. You can find a more complete example in the [XGBClassifier notebook](../tutorials/ml_examples.md).
Here's an example of how to use this model in FHE on a popular data-set using some of scikit-learn's pre-processing tools. You can find a more complete example in the [XGBClassifier notebook](../tutorials/ml_examples.md).

```python
from sklearn.datasets import load_breast_cancer
Expand Down Expand Up @@ -118,18 +118,18 @@ We can plot and compare the decision boundaries of the Concrete ML model and the

## Quantization parameters

When using a sufficiently high bit width, quantization has little impact on the decision boundaries of the Concrete ML FHE decision tree model, as quantization is done individually on each input feature. It means FHE models can achieve similar accuracy levels as floating point models. Using 6 bits for quantization is effective in reaching or even exceeding floating point accuracy.
When using a sufficiently high bit-width, quantization has little impact on the decision boundaries of the Concrete ML FHE decision tree model, as quantization is done individually on each input feature. It means FHE models can achieve similar accuracy levels as floating point models. Using 6 bits for quantization is effective in reaching or even exceeding floating point accuracy.

To adjust the number of bits for quantization, use the `n_bits` parameter. Setting `n_bits` to a low value may introduce artifacts, potentially reducing accuracy. However, the execution speed in FHE could improve. This adjustment allows you to manage the accuracy/speed trade-off. Additionally, you can recover some accuracy by increasing the `n_estimators` parameter.

The following graph shows that using 5-6 bits of quantization is usually sufficient to reach the performance of a non-quantized XGBoost model on floating point data. The metrics plotted are accuracy and F1-score on the `spambase` dataset.
The following graph shows that using 5-6 bits of quantization is usually sufficient to reach the performance of a non-quantized XGBoost model on floating point data. The metrics plotted are accuracy and F1-score on the `spambase` data-set.

![XGBoost n_bits comparison](../figures/XGBClassifier_nbits.png)

## FHE Inference time considerations

The inference time in FHE is strongly dependant on the maximum circuit bit width. For trees, in most cases, the quantization bit width will be the same as the circuit bit width. Therefore, reducing the quantization bit width to 4 or less will result in fast inference times. Adding more bits will increase FHE inference time exponentially.
The inference time in FHE is strongly dependant on the maximum circuit bit-width. For trees, in most cases, the quantization bit-width will be the same as the circuit bit-width. Therefore, reducing the quantization bit-width to 4 or less will result in fast inference times. Adding more bits will increase FHE inference time exponentially.

In some rare cases, the bit width of the circuit can be higher than the quantization bit width. This could happen when the quantization bit width is low but the tree-depth is high. In such cases, the circuit bit width is upper bounded by `ceil(log2(max_depth + 1) + 1)`.
In some rare cases, the bit-width of the circuit can be higher than the quantization bit-width. This could happen when the quantization bit-width is low but the tree-depth is high. In such cases, the circuit bit-width is upper bounded by `ceil(log2(max_depth + 1) + 1)`.

For more information on the inference time of FHE decision trees and tree-ensemble models please see [Privacy-Preserving Tree-Based Inference with Fully Homomorphic Encryption, arXiv:2303.01254](https://arxiv.org/abs/2303.01254).
2 changes: 1 addition & 1 deletion docs/getting-started/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,6 @@ FHE requires all inputs, constants, and intermediate values to be integers of ma

- **Quantization**: Concrete ML quantizes inputs, outputs, weights, and activations to meet FHE limitations. See [the quantization documentation](<(../explanations/quantization.md)>) for details.

- **Accuracy trade-off**: Quantization may reduce accuracy, but careful selection of quantization parameters or of the training approach can mitigate this. Concrete ML offers built-in quantized models; users only configure parameters like bit width. For more details of quantization configurations, see [the advanced quantization guide](../explanations/quantization.md#configuring-model-quantization-parameters).
- **Accuracy trade-off**: Quantization may reduce accuracy, but careful selection of quantization parameters or of the training approach can mitigate this. Concrete ML offers built-in quantized models; users only configure parameters like bit-width. For more details of quantization configurations, see [the advanced quantization guide](../explanations/quantization.md#configuring-model-quantization-parameters).

- **Additional methods**: Dimensionality reduction and pruning are additional ways to make programs compatible for FHE. See [Poisson regression example](../advanced_examples/PoissonRegression.ipynb) for dimensionality reduction and [built-in neural networks](../built-in-models/neural-networks.md) for pruning.

0 comments on commit 27b37de

Please sign in to comment.