Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add gpu demo 3 #2271

Open
wants to merge 93 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
0d76e67
Feat: Add custom embedder
vonodiripsa Jun 12, 2024
9afa431
Corrected Names and file location
Jun 13, 2024
ee8f6f9
Code style corrections
Jun 14, 2024
09af0e0
Source temp fixes
Jun 14, 2024
a485010
Formating
Jun 14, 2024
eae2aee
First test
Jun 14, 2024
2cdfb59
Name changes
Jun 14, 2024
88d34b8
With two models
Jun 14, 2024
052f39b
Source style corrections
Jun 16, 2024
ac7bc67
Name change
Jun 16, 2024
52a1581
Name change
Jun 16, 2024
1bd45ab
Merge init scripts
Jun 17, 2024
cce365a
Removed extra file
Jun 17, 2024
8d2ed06
Added result output and _ correction
Jun 18, 2024
6b7798d
Formatted
Jun 18, 2024
a8b4dc9
Merge branch 'microsoft:master' into add-demo
bvonodiripsa Jun 18, 2024
dff46fe
Runtime flag update and load class from file (not from synapse.ml..)
Jun 18, 2024
5f569ce
Use built synapse.ml package instead of file
Jun 18, 2024
3098645
Clean imports and the class
Jun 27, 2024
e2e2014
Corrected edge cases (slam dataframe or no gpu)
Jun 28, 2024
710d9a6
Added check for cuda
Jun 28, 2024
42d8a07
Added synapse.ml.nn.KNN to run on CPU
Jul 1, 2024
7280ea7
add some small fixes to namespaces
mhamilton723 Jul 3, 2024
c63ab5b
Merge branch 'master' into add-demo
bvonodiripsa Jul 10, 2024
ac5828c
formatted
Jul 10, 2024
ff7b42d
Merge branch 'master' into add-demo
mhamilton723 Jul 17, 2024
96b81e2
Merge branch 'master' into add-demo
mhamilton723 Jul 17, 2024
f0c3b49
corrected default batch size
Jul 19, 2024
4047867
Added test
Jul 19, 2024
7894e9d
Corrected build errors
Jul 20, 2024
48433cd
style fixes
Jul 20, 2024
6989322
Style corrections
Jul 20, 2024
c58d929
More style corrections
Jul 20, 2024
9c54a66
Added extra row
Jul 20, 2024
776dfeb
Corrected comparison results image
Jul 22, 2024
52d52df
Corrected init and test
Jul 26, 2024
959f04e
Style correction
Jul 26, 2024
9fefefe
Updated notebook image link
Jul 26, 2024
b39218b
Added sentence_transformers for testing
Jul 31, 2024
f058cb2
trying to fix testing
Jul 31, 2024
ab0895b
style change
Jul 31, 2024
9c096ed
removed style spaces
Jul 31, 2024
62d10fc
Style again...
Jul 31, 2024
538c05c
added pyspark
Aug 1, 2024
4006266
remove pyspark
Aug 1, 2024
63ef878
corrected utest
Aug 1, 2024
72b4595
Reverse style change
Aug 1, 2024
575f020
change data size
Aug 1, 2024
fc66d0d
comment a line
Aug 1, 2024
e8d308d
Corrected init_spark()
Aug 1, 2024
07a67d3
Style and added SQLContext
Aug 1, 2024
2599ae0
Corrected result_df and remove old image
Aug 1, 2024
2b3f3d4
Corrected sidebars.js
Aug 2, 2024
be95231
Merge branch 'master' into add-demo
mhamilton723 Aug 2, 2024
b19a895
match web names
Aug 6, 2024
5a993cf
Merge branch 'master' into add-demo
mhamilton723 Aug 7, 2024
050ba59
Add initial demo 3
Aug 23, 2024
a0fabc0
Changes for demo3
Aug 23, 2024
164d8e5
Corrections in demo description
Aug 23, 2024
1186e71
Merge remote-tracking branch 'origin/add-demo' into add-demo
Aug 26, 2024
5dcd252
added pdf comments
Aug 26, 2024
c59e262
Added demo3 and demo2 text corrections
Aug 26, 2024
5b3a0a9
Delete docs/Explore Algorithms/AI Services/QuickStart - Distributed Q…
bvonodiripsa Aug 26, 2024
63096a9
Demo 3 File rename
Aug 26, 2024
81b0705
Clean imports
Aug 26, 2024
50df271
style changes
Aug 27, 2024
b3820dc
More style changes
Aug 27, 2024
5cda463
Added to the web
Aug 27, 2024
8b72f0d
Initial Demo 3 (RAG on GPU)
bvonodiripsa Aug 27, 2024
c2e0646
Update init-rapidsml-cuda-11.8.sh
bvonodiripsa Aug 27, 2024
c17be8f
Update sidebars.js
bvonodiripsa Aug 27, 2024
2f408f4
change text in Embeddings and Approximate KNN on GPU.ipynb
bvonodiripsa Aug 27, 2024
3d13751
Demo3 with correct name
bvonodiripsa Aug 28, 2024
18fd9b3
Delete docs/Explore Algorithms/AI Services/QuickStart - Distributed Q…
bvonodiripsa Aug 28, 2024
deae6ce
Merge branch 'master' into add-demo
mhamilton723 Sep 10, 2024
a063796
Merge remote-tracking branch 'origin/add-demo' into add-demo
Oct 18, 2024
59568a0
Fix batch size
Oct 18, 2024
fcd7ede
Merge branch 'master' into add-demo
bvonodiripsa Oct 19, 2024
9cf90b1
Update OpenAIPrompt.scala to match main
bvonodiripsa Oct 19, 2024
9567a7b
Update OpenAIChatCompletion.scala to sync with main
bvonodiripsa Oct 19, 2024
129c510
Update OpenAICompletion.scala to match main
bvonodiripsa Oct 19, 2024
d2c6446
Update OpenAICompletion.scala
bvonodiripsa Oct 19, 2024
c112182
Update OpenAICompletion.scala
bvonodiripsa Oct 19, 2024
597151a
Update OpenAIChatCompletion.scala
bvonodiripsa Oct 19, 2024
f2a21a4
Update OpenAIChatCompletion.scala
bvonodiripsa Oct 19, 2024
711ae07
Update OpenAIChatCompletion.scala
bvonodiripsa Oct 19, 2024
232a743
Update OpenAIPromptSuite.scala
bvonodiripsa Oct 19, 2024
51bab74
Update SynapseMLLogger.py
bvonodiripsa Oct 19, 2024
70d2dca
Update test_logging.py
bvonodiripsa Oct 19, 2024
0159897
Update DatabricksUtilities.scala
bvonodiripsa Oct 19, 2024
9dab2d6
Update environment.yml
bvonodiripsa Oct 19, 2024
dd6cffa
Update conda.yml
bvonodiripsa Oct 19, 2024
ebb2976
Delete docs/Explore Algorithms/AI Services/QuickStart - Distributed Q…
bvonodiripsa Oct 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update OpenAIPrompt.scala to match main
  • Loading branch information
bvonodiripsa authored Oct 19, 2024
commit 9cf90b117a00e1f38625addb2f1510764d5b5696
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import org.apache.spark.ml.{ComplexParamsReadable, ComplexParamsWritable, Transf
import org.apache.spark.sql.catalyst.encoders.RowEncoder
import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.types.{DataType, StructType}
import org.apache.spark.sql.{Column, DataFrame, Dataset, functions => F, types => T}
import org.apache.spark.sql.{Column, DataFrame, Dataset, Row, functions => F, types => T}

import scala.collection.JavaConverters._

Expand Down Expand Up @@ -200,6 +200,16 @@ class OpenAIPrompt(override val uid: String) extends Transformer
completion
}

override protected def prepareEntity: Row => Option[AbstractHttpEntity] = {
r =>
openAICompletion match {
case chatCompletion: OpenAIChatCompletion =>
chatCompletion.prepareEntity(r)
case completion: OpenAICompletion =>
completion.prepareEntity(r)
}
}

private def getParser: OutputParser = {
val opts = getPostProcessingOptions

Expand Down