Decoding error for aws.route53.getZone

[grzegorz-bielski]

When using:

//> using scala "3.4.2"
//> using options -Wunused:all -Wvalue-discard -Wnonunit-statement
//> using dep "org.virtuslab::besom-core:0.3.2"
//> using dep org.virtuslab::besom-aws:6.39.0-core.0.3

@main def main = Pulumi.run:
   Stack(
       aws.route53
          .getZone(aws.route53.GetZoneArgs(name = "example.com."))
          .map(_.zoneId)
   )

I'm getting following error.

Exception in thread "main" besom.internal.AggregatedDecodingError: Decoding Errors [3]:
      aws:route53/getZone:getZone().linkedServiceDescription: Encountered an error - possible secret - Known(Set(),false,Some(Value(NullValue(NULL_VALUE),UnknownFieldSet(Map()))))
      aws:route53/getZone:getZone().linkedServicePrincipal: Encountered an error - possible secret - Known(Set(),false,Some(Value(NullValue(NULL_VALUE),UnknownFieldSet(Map()))))
      aws:route53/getZone:getZone().vpcId: Encountered an error - possible secret - Known(Set(),false,Some(Value(NullValue(NULL_VALUE),UnknownFieldSet(Map()))))
    (with aggregate stack trace)
        at besom.internal.AggregatedDecodingError$.apply(codecs.scala:99)
        at besom.internal.ResourceOps.decodeResponse$1$$anonfun$1(ResourceOps.scala:91)
        at besom.internal.Result.flatMap$$anonfun$1(Result.scala:166)
        at besom.internal.Result.runM$$anonfun$7(Result.scala:273)
        at besom.internal.Runtime.flatMapBothM$$anonfun$1$$anonfun$1(Result.scala:118)
        at besom.internal.FutureRuntime.flatMapBoth$$anonfun$1(Result.scala:404)
        at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
    Caused by: besom.internal.DecodingError: [aws:route53/getZone:getZone().linkedServiceDescription] aws:route53/getZone:getZone().linkedServiceDescription: Encountered an error - possible secret - Known(Set(),false,Some(Value(NullValue(NULL_VALUE),UnknownFieldSet(Map()))))
        at besom.internal.DecodingError$.apply(codecs.scala:80)
        at besom.internal.Decoder.decode$$anonfun$1$$anonfun$2(codecs.scala:132)
        at besom.util.NonEmptyVector.map(Validated.scala:167)
        at besom.util.Validated.lmap(Validated.scala:47)
        at besom.internal.Decoder.decode$$anonfun$1(codecs.scala:134)
        at besom.util.Validated$.flatMap$$anonfun$1(Validated.scala:112)
        ... 11 more
        Suppressed: besom.internal.DecodingError: [aws:route53/getZone:getZone().linkedServicePrincipal] aws:route53/getZone:getZone().linkedServicePrincipal: Encountered an error - possible secret - Known(Set(),false,Some(Value(NullValue(NULL_VALUE),UnknownFieldSet(Map()))))
                ... 17 more
        Caused by: besom.internal.DecodingError: [aws:route53/getZone:getZone().linkedServicePrincipal] aws:route53/getZone:getZone().linkedServicePrincipal: Expected a string, got: 'NullValue(NULL_VALUE)'
                at besom.internal.DecodingError$.apply(codecs.scala:80)
                at besom.internal.Decoder$stringDecoder$.mapping(codecs.scala:217)
                at besom.internal.Decoder.decode$$anonfun$1$$anonfun$1(codecs.scala:128)
                at besom.internal.OutputData.traverseValidated(OutputData.scala:93)
                at besom.internal.Decoder.decode$$anonfun$1(codecs.scala:128)
                ... 12 more
        Suppressed: besom.internal.DecodingError: [aws:route53/getZone:getZone().vpcId] aws:route53/getZone:getZone().vpcId: Encountered an error - possible secret - Known(Set(),false,Some(Value(NullValue(NULL_VALUE),UnknownFieldSet(Map()))))
                ... 17 more
        Caused by: besom.internal.DecodingError: [aws:route53/getZone:getZone().vpcId] aws:route53/getZone:getZone().vpcId: Expected a string, got: 'NullValue(NULL_VALUE)'
                at besom.internal.DecodingError$.apply(codecs.scala:80)
                at besom.internal.Decoder$stringDecoder$.mapping(codecs.scala:217)
                at besom.internal.Decoder.decode$$anonfun$1$$anonfun$1(codecs.scala:128)
                at besom.internal.OutputData.traverseValidated(OutputData.scala:93)
                at besom.internal.Decoder.decode$$anonfun$1(codecs.scala:128)
                ... 12 more
    Caused by: besom.internal.DecodingError: [aws:route53/getZone:getZone().linkedServiceDescription] aws:route53/getZone:getZone().linkedServiceDescription: Expected a string, got: 'NullValue(NULL_VALUE)'
        at besom.internal.DecodingError$.apply(codecs.scala:80)
        at besom.internal.Decoder$stringDecoder$.mapping(codecs.scala:217)
        at besom.internal.Decoder.decode$$anonfun$1$$anonfun$1(codecs.scala:128)
        at besom.internal.OutputData.traverseValidated(OutputData.scala:93)
        at besom.internal.Decoder.decode$$anonfun$1(codecs.scala:128)
        ... 12 more

Possibly the same thing as in: #432

Is there any known workaround?

[lbialy]

If you don't plan to use any of the following fields:

linkedServiceDescription
linkedServicePrincipal
vpcId

then the answer is yes as the workaround in form of more granular, field level decoding is already in main branch awaiting 0.4.0 release. Actual fix for this problem would be either to:
a) get Pulumi to fix the schema to mark these fields as optional if they are mistakenly marked as non-optional or
b) if they are indeed non-optional but terraform provider is buggy, to get AWS to fix the provider ooooor
c) for us to maintain per-package registry of field optionality overrides in our codegen that would be consulted during package generation to apply type adjustments for fields that do indeed show up as nulls in runtime and crash user apps.

I am strongly biased towards c) currently even though my co-maintainers voiced vocal opposition to this as it's basically a hack that requires maintenance effort on our side. I think we should do special case handling for unexpected nullable fields as they are so common (thanks to null-unsafe langs being used in the whole infra stack beneath us) and not do any special handling for actual type errors, ie we expect a Double but there's a String in data - these things have to be raised as issues upstream. For nullable fields I think we should just implement the overrides and post issues to Pulumi / TF concurrently, this should solve the problem on our side quickly and once the upstream fixes things on their side we can just drop the override. I'll make an issue and see if we can fit it for 0.4 release.

[grzegorz-bielski]

Thanks for the quick and detailed response!

I'm won't be using these fields, so I'll just wait for 0.4.0.
In the meantime I'll settle for hardcoding / using aws sdk to get that zone id.

I have no opinion on internals of Besom, but looking at issues I can see that such situations are not that rare and some form of documented escape hatch would be useful.

[lbialy]

Yes, granular decoding alleviates the problem to "if you don't need that field, it won't block you" level but I think we should improve the error messages so that they are more actionable (I think it also was partially implemented already, error messages will contain a clickable link to open an issue on gh that is already filled with relevant context from the error).