-
Notifications
You must be signed in to change notification settings - Fork 833
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
draft: Spark33rebase #2075
draft: Spark33rebase #2075
Conversation
Hey @JessicaXYWang 👋! We use semantic commit messages to streamline the release process. Examples of commit messages with semantic prefixes:
To test your commit locally, please follow our guild on building from source. |
71a2acf
to
27ea1ae
Compare
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
Codecov Report
@@ Coverage Diff @@
## master #2075 +/- ##
===========================================
- Coverage 87.06% 43.18% -43.89%
===========================================
Files 306 306
Lines 16162 16162
Branches 824 824
===========================================
- Hits 14072 6980 -7092
- Misses 2090 9182 +7092
|
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Summary by GPT-4
This commit updates the versions of several dependencies in the SynapseML project. The Spark version is updated from 3.2.3 to 3.3.2, and the Horovod version is updated from 0.25.0 to 0.27.0.
In addition, the versions of Torch and Torchvision are updated, and the Transformers library is updated from version 4.15.0 to 4.32.1.
The commit also updates some test files to reflect these changes, and modifies some environment setup scripts and Dockerfiles accordingly.
Overall, this commit aims to keep the project up-to-date with its dependencies, which can bring performance improvements, bug fixes, and new features from these libraries into SynapseML.
Suggestions
The changes in this PR are generally good, but there are a few areas that could be improved:
-
In the
build.sbt
file, the spark version is updated from "3.2.3" to "3.3.2". It would be helpful to include a comment explaining why this change was made. -
In the
core/src/main/scala/com/microsoft/azure/synapse/ml/codegen/PyCodegen.scala
file, several package versions are updated. Again, it would be helpful to include comments explaining why these changes were made. -
In the
core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/DatabricksUtilities.scala
file, several package versions are updated in theGPULibraries
string. It would be helpful to include comments explaining why these changes were made. -
In the
deep-learning/src/main/python/horovod_installation.sh
file, several commands are changed and some are removed entirely. It would be helpful to include comments explaining why these changes were made. -
In the
docs/Explore Algorithms/Hyperparameter Tuning/HyperOpt.ipynb
file, the mlflow version is downgraded from 2.6.0 to 2.5.0 without explanation. -
In multiple files (e.g.,
DeepTextClassifier.py
,DeepVisionClassifier.py
, etc.), checks for specific versions of packages (e.g., transformers and horovod) are updated without explanation. -
The Dockerfiles in
tools/docker/demo/Dockerfile
andtools/docker/minimal/Dockerfile
have their Spark and Hadoop versions updated without explanation. -
The script in
tools/dotnet/dotnetSetup.sh
has its Spark version updated without explanation.
In general, it's important to explain why version updates are being made so that other developers can understand the rationale behind these changes.
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
94ec40a
to
3f6c04b
Compare
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
57c8674
to
ce9b01f
Compare
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
8b76583
to
afb0054
Compare
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
Related Issues/PRs
#xxx
What changes are proposed in this pull request?
Briefly describe the changes included in this Pull Request.
How is this patch tested?
Does this PR change any dependencies?
Does this PR add a new feature? If so, have you added samples on website?
website/docs/documentation
folder.Make sure you choose the correct class
estimators/transformers
and namespace.DocTable
points to correct API link.yarn run start
to make sure the website renders correctly.<!--pytest-codeblocks:cont-->
before each python code blocks to enable auto-tests for python samples.WebsiteSamplesTests
job pass in the pipeline.