How to combine generative observations with an explicit observation model?

[stefan-f-bucher]

Hi all, and thank you so much for your work on this incredible framework! 🙏

I'm hoping for some help with my conceptual understanding of generative and explicit models and how they can be combined, especially in relation to the changes in #269 by @zsunberg. To make things concrete, let's say I'm trying to implement a lunar lander with noisy sensors.

I have an observation model implemented as POMDPs.observation(m::myProblem, s, a, sp).

Now let's say I'm trying to adapt the framework for use with a physical environment sending observations. Transition models and reward are still implemented using the explicit interface. My understanding is that I can implement this observation using the generative framework, along the lines of

function POMDPs.gen(m::myProblem, s, a, sp, rng)
    measurement = obtainSomePhysicalMeasurement()
    return (o=measurement,)  #NamedTuple 
end

First question: If both implementations are present, when does POMDPs.jl invoke the generative vs explicit implementation? Here I would guess that the generative implementation of o is used, but the explicit implementations of the transition and reward (since sp and r are lacking in the NamedTuple returned by gen). Is that correct?

Second question: I would like model-based solvers to have access to my explicit observer model in POMDPs.observation(m::myProblem, s, a, sp) rather than just samples as would be the case if I would implement it as an ImplicitDistribution. The generative model is only meant to provide observations when interacting sequentially with a physical environment. How can I ensure that? Currently it appears as though POMSPs.observation takes precedence over POMDPs.gen.

I hope that my questions are clear. I'm really just hoping for a better conceptual understanding of how generative and explicit implementations can/should be combined. Thank you so much for your kind help!

[zsunberg]

Hi @stefan-f-bucher,

I think the solution is to create multiple POMDP models. For example ExplicitLunarLander which returns an explicit distribution from observation and PhysicalLunarLander which gets observations from the physical environment.

You can use the explicit model to find a policy and then simulate with physical data, e.g.

m = ExplicitLunarLander()
policy = solve(solver, m)

sim_m = PhysicalLunarLander()
history = simulate(HistoryRecorder(), sim_m, policy) 

(As an alternative to multiple types, you could implement this using a compositional approach with a single type that has an observation model field and associated type parameter - ask if you are interested in further explanation)

One more note: As mentioned in the docs, gen is intended only for the case where two or more of the next state, observation, and reward need to be generated at the same time. If the state transition model can be separated from the reward and observation models, you should implement each separately with an ImplicitDistribution instead of gen. In particular, only gen(m, s, a, rng) is part of the POMDPs.jl interface gen(m, s, a, sp, rng) (the observation version that you wrote above) is not part of the interface and will never be called.

[stefan-f-bucher]

Thank you so much for your kind and quick response @zsunberg!

This is very helpful. I was indeed considering separate models but was hoping that's not necessary. But I'm glad you think that's the sensible approach, because I was confused about coexisting gen and observation implementations (rightly so, it seems? they're meant to be exclusive?).

The compositional type approach seems great, though I wouldn't mind some clarification as I'm new to Julia. Are you proposing something along the following lines?

abstract type ObservationModel end
struct GenerativeObservationModel <: ObservationModel end
struct ExplicitObservationModel <: ObservationModel end

struct myProblem{T<:ObservationModel} <: POMDP{ typeof(𝒮), typeof(ℋ), typeof(𝒜)}  
    m::Int 
    myProblem(T=ExplicitObservationModel, m=2) = new{T}(m)
end

function POMDPs.observation(m::myProblem{ExplicitObservationModel}, s, a, sp)
    ImplicitDistribution(s, a) do s, a
        return obtainSomePhysicalMeasurement()
    end
end

function POMDPs.observation(m::myProblem{GenerativeObservationModel}, s, a, sp)
    # bla
    return SparseCat(foo,bar)
end

# DON'T USE THIS:
function POMDPs.gen(m::myProblem{GenerativeObservationModel}, s, a, rng)
    measurement = obtainSomePhysicalMeasurement()
    return (o=measurement,)  
end

So now I should be able to do the following?

m = myProblem(ExplicitObservationModel, 2)
planner = solve(solver, m)
m_phys = myProblem(GenerativeObservationModel, 2)
simulate(simulator, simulate, planner, belief_updater, init_belief, s0)

And that should work even with planners (as opposed to policies)?

I did read that gen should only be implemented if two or more of those are generated simultaneously (though I did not quite understand why), but I violated it because I thought the ImplicitDistribution approach would "crowd out" my model. With your suggested approach, that is no longer of concern. How do I want to think about this design choice? For context, my reward depends on o, so I could either specify the reward explicitly as a function of o, or generate them together. I guess the former is preferable whenever feasible so that the reward function is available explicitly?

Again, thanks a lot!

[zsunberg]

Great! Yeah, I think the compositional approach is the best way to go (the only reason I didn't suggest this first is that I wasn't sure about your familiarity with parametric types).

You are on the right track with your code snippet. I don't have time to give detailed feedback now, but a fully implemented (though unfortunately somewhat complex) model that uses this approach is here: https://github.com/JuliaPOMDP/LaserTag.jl . One note for understanding it: I originally expected to implement both continuous and discrete observation models, but now there is only one ObsModel, which is designed to emulate the observation model from the DESPOT paper. Hence the DESPOTEmu name.

Hope that is helpful enough.

I guess the former is preferable whenever feasible so that the reward function is available explicitly?

Yes, it is best to implement reward(m, s, a, sp, o), but some solvers are not designed to handle observation-dependent rewards. This is usually a relatively easy problem to overcome, but I will need to know which solver you are planning to use to advise you further.

[stefan-f-bucher]

Thank you! Let me take a look, digest, and get back to you in case of further problems/questions!