[6.0] Global.deserializeTo[] method #979

kushti · 2024-05-13T09:24:24Z

This PR is introducing new Global.deserializeRaw method which is allowing to deserialize a typed object from underlying bytes. It is compatible (has roundtrip with) Global.serialize from #989 .

example:

{
     val src = getVar[Coll[Short]](21).get
     val ba = serialize(src)
     val restored = deserializeTo[Coll[Short]](ba)
     src == restored
}

aslesarenko · 2024-05-13T15:00:18Z

data/shared/src/main/scala/org/ergoplatform/ErgoHeader.scala

+  * @param votes - votes for changing system parameters
+  * @param _bytes - serialized bytes of the header when not `null`
+  */
+class ErgoHeader(override val version: ErgoHeader.Version,


What is the motivation for having this class. Why sigma.data.CHeader is not enough?
I think it is better to have unification of domain types if possible.

data/shared/src/main/scala/org/ergoplatform/HeaderWithoutPow.scala

aslesarenko · 2024-05-13T15:05:13Z

data/shared/src/main/scala/org/ergoplatform/HeaderWithoutPow.scala

+
+  override def parse(r: SigmaByteReader): Long = {
+    readUint32BE(r.getBytes(4))
+  }


Array allocation can be avoided.

aslesarenko · 2024-05-13T15:11:35Z

data/shared/src/main/scala/sigma/ast/methods.scala

+  lazy val checkPowMethod = SMethod(
+    this, "checkPow", SFunc(Array(SHeader), SBoolean), 16, GroupGenerator.costKind) // todo: cost
+    .withIRInfo(MethodCallIrBuilder)
+    .withInfo(Xor, "Check PoW of this header") // todo: desc


Can use .withInfo overload without opDesc. Note, opDesc should point to the operation this Method can be lowered to. In this case there is no such operation.

aslesarenko · 2024-05-13T15:23:31Z

data/shared/src/main/scala/sigma/ast/methods.scala

+  lazy val deserializeRawMethod = SMethod(
+    this, "deserializeRaw", SFunc(Array(SGlobal, SByteArray), tT, Array(paramT)), 3, Xor.costKind) // todo: cost
+    .copy(irInfo = MethodIRInfo(None, Some(desJava), None))
+    .withInfo(Xor, "Byte-wise XOR of two collections of bytes",  // todo: desc


Use different .withInfo overload.

aslesarenko · 2024-05-13T16:01:01Z

data/shared/src/main/scala/sigma/serialization/MethodCallSerializer.scala

@@ -23,6 +25,9 @@ case class MethodCallSerializer(cons: (Value[SType], SMethod, IndexedSeq[Value[S
    w.putValue(mc.obj, objInfo)
    assert(mc.args.nonEmpty)
    w.putValues(mc.args, argsInfo, argsItemInfo)
+    if(mc.method.name == SGlobalMethods.deserializeRawMethod.name){
+      w.putType(mc.tpe, typeInfo)
+    }


Better leave MethodCall unchanged and use PolyMethodCall as described in #980
This will also give both simplicity of code and efficiency of execution, because specialization will happen at compile time and most of the method calls are simple method calls.

aslesarenko · 2024-05-13T16:05:19Z

interpreter/shared/src/test/scala/sigma/serialization/DataSerializerSpecification.scala


+    an[SerializerException] should be thrownBy (


For negative cases there is assertExceptionThrown which allows testing not only exception type but also the message. See other usages of it as an example.

aslesarenko · 2024-05-13T16:10:15Z

sc/shared/src/main/scala/sigmastate/eval/GraphBuilding.scala

@@ -19,6 +20,25 @@ import sigma.serialization.OpCodes

 import scala.collection.mutable.ArrayBuffer

+
+case class DeserializeRawBytes[V <: SType](bytes: Value[SByteArray], tpe: V) extends NotReadyValue[V] {


It's not necessary to introduce new operation with opcode for each method, MethodCall is enough and can be translated to MethodCall from GraphBuilding.

aslesarenko · 2024-05-13T16:11:05Z

sc/shared/src/main/scala/sigmastate/eval/GraphBuilding.scala

@@ -536,6 +556,9 @@ trait GraphBuilding extends SigmaLibrary { IR: IRContext =>
        val e = stypeToElem(optTpe.elemType)
        ctx.getVar(id)(e)

+      case DeserializeRawBytes(bytes, tpe) =>
+        sigmaDslBuilder.deserializeRaw(asRep[Coll[Byte]](eval(bytes)))(stypeToElem(tpe))


This is not necessary.

aslesarenko · 2024-05-13T16:14:06Z

sc/shared/src/main/scala/sigmastate/lang/SigmaTyper.scala

+              }
+            case _ =>
+              error(s"Cannot find Global method: $n", bound.sourceContext)
+          }


This shouldn't be necessary.

aslesarenko

LGTM, see also 1 comment

aslesarenko · 2024-11-20T22:12:22Z

data/shared/src/main/scala/sigma/ast/methods.scala

@@ -1824,6 +1825,16 @@ case object SGlobalMethods extends MonoTypeMethods {
    .withInfo(Xor, "Byte-wise XOR of two collections of bytes",
      ArgInfo("left", "left operand"), ArgInfo("right", "right operand"))

+  private val deserializeCostKind = PerItemCost(
+    baseCost = JitCost(20), perChunkCost = JitCost(7), chunkSize = 128)


Feels like two big chunkSize (or small perChunkCost) considering that deserialization is many allocations.

Makes sense, raised cost to
baseCost = JitCost(30), perChunkCost = JitCost(20), chunkSize = 32

kushti changed the base branch from develop to v6.0.0 May 13, 2024 09:24

This was referenced May 13, 2024

Header deserialization #964

Closed

Box deserialization #963

Closed

[v6.0] Implement new CreateAvlTree operation #907

Open

kushti changed the title ~~[6.0] Global.deserialize method~~ [6.0] Global.deserializeRaw method May 13, 2024

kushti modified the milestone: v6.0 May 13, 2024

kushti mentioned this pull request May 13, 2024

Implement binary serialization and deserialization for every type #945

Open

aslesarenko requested changes May 13, 2024

View reviewed changes

kushti mentioned this pull request May 16, 2024

Global.serialize function to serialize instance of any type #988

Open

ergoplatform deleted a comment from kushti May 17, 2024

kushti changed the title ~~[6.0] Global.deserializeRaw method~~ [6.0] Global.deserializeTo[] method May 20, 2024

aslesarenko mentioned this pull request May 22, 2024

[6.0] Global.serialize method #989

Merged

This was referenced Jun 5, 2024

[6.0.0] Header.bytes method implementation #975

Closed

[6.0] Common SMethod and MethodCallSerializer new features #1002

Merged

kushti mentioned this pull request Aug 14, 2024

6.0.0 candidate for testnet #1025

Closed

SethDusek mentioned this pull request Sep 17, 2024

Global.deserialize method ergoplatform/sigma-rust#768

Open

kushti requested a review from aslesarenko September 24, 2024 16:07

kushti mentioned this pull request Oct 9, 2024

6.0.0-RC1 candidate for devnets #1027

Closed

aslesarenko approved these changes Nov 20, 2024

View reviewed changes

Global.deserializeTo[] method

f195072

kushti force-pushed the 6.0-deserialize branch from ae351a0 to f195072 Compare November 22, 2024 09:48

kushti merged commit 47b5558 into v6.0.0 Nov 22, 2024
3 of 4 checks passed

kushti mentioned this pull request Nov 22, 2024

Candidate for 6.0.0 release #930

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[6.0] Global.deserializeTo[] method #979

[6.0] Global.deserializeTo[] method #979

kushti commented May 13, 2024 •

edited

Loading

aslesarenko May 13, 2024

aslesarenko May 13, 2024

aslesarenko May 13, 2024

aslesarenko May 13, 2024

aslesarenko May 13, 2024

aslesarenko May 13, 2024

aslesarenko May 13, 2024

aslesarenko May 13, 2024

aslesarenko May 13, 2024

aslesarenko left a comment

aslesarenko Nov 20, 2024

kushti Nov 22, 2024

		@@ -19,6 +20,25 @@ import sigma.serialization.OpCodes

		import scala.collection.mutable.ArrayBuffer


		case class DeserializeRawBytes[V <: SType](bytes: Value[SByteArray], tpe: V) extends NotReadyValue[V] {

[6.0] Global.deserializeTo[] method #979

[6.0] Global.deserializeTo[] method #979

Conversation

kushti commented May 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aslesarenko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kushti commented May 13, 2024 •

edited

Loading