Illumintation usage

[kerim371]

Hi,

I'm testing illumintation feature on my data.
It seems it improves the result of the FWI inversion but requires more time to compute.

My data consist of about 1000 3D shots.

My attempt:

F = judiModeling(model0, q.geometry, d_obs.geometry; options=jopt)
J = judiJacobian(F(model0), q)
Il = inv(judiIllumination(model0; mode="uv"))
Ml_ref = judiDataMute(q.geometry, d_obs.geometry, vp=2000f0, t0=0.70f0, mode=:reflection, taperwidth=8) # keep reflections

d0 = F(model0)[indsrc]*q[indsrc]
r = Ml_ref[indsrc]*d0 - Ml_ref[indsrc]*d_obs[indsrc]
gradient = Il*J'[indsrc] * r
fval = .5*norm(r)^2

This produces better gradient but the problem is the computational time.

With illumination I spend about 7-8 hours per iteration on SLURM cluster with 16 tasks and 14 CPU per task on 6 nodes (each node consists of 56 CPU): 2 full nodes and 4 half of power of a node. Even though I see that operator forward/adjoint runs in 15 sec in average (but sometimes it requres about 150 sec don't know why). But first it computes the forward operator for all the shots.

Without illumination it requires about 1 hour to compute single iteration with similar cluster configuration. And it also takes about 15 sec per forward/adjoint operator.

Here is som part of output (I also attach all output in .ZIP file):

Operator `forward` ran in 14.44 s
Operator `gradient` ran in 12.38 s
Operator `forward` ran in 11.99 s
Operator `gradient` ran in 12.69 s
Operator `forward` ran in 12.23 s
Operator `gradient` ran in 13.49 s
Operator `gradient` ran in 12.29 s
Operator `gradient` ran in 12.10 s
Operator `gradient` ran in 13.11 s
Operator `gradient` ran in 11.95 s
Operator `gradient` ran in 8.37 s
Operator `gradient` ran in 16.02 s
Operator `gradient` ran in 17.08 s
Operator `gradient` ran in 11.69 s
Operator `gradient` ran in 15.74 s
Operator `gradient` ran in 15.06 s
Operator `forward` ran in 149.49 s
Operator `forward` ran in 151.08 s
Operator `forward` ran in 151.76 s
Operator `gradient` ran in 127.36 s
Operator `gradient` ran in 126.00 s
Operator `gradient` ran in 121.85 s
Operator `forward` ran in 5.86 s
Operator `forward` ran in 7.05 s
Operator `forward` ran in 9.04 s
Operator `forward` ran in 11.05 s
Operator `forward` ran in 11.48 s
Operator `forward` ran in 11.75 s
Operator `forward` ran in 8.59 s
Operator `forward` ran in 10.40 s
Operator `forward` ran in 11.81 s

slurm script:

#!/bin/bash

#SBATCH --ntasks=16
#SBATCH --cpus-per-task=14
#SBATCH --mem-per-cpu=8G
#SBATCH --time="30-00"

julia fwi_spg.jl

As the computation are still running and don't want to interrupt them I didn't do all the tests to investigate this issue. But maybe @mloubout you have some thoughts?

JUDI: v4.0.2
Devito: 4.8.11

slurm-9532.zip

[mloubout]

The illumination has near zero overhead so the only thing that could explain it is something weird happening in the reduction(sum over shots) phase which I'm not sure why it would.

I can try to have a quick look next week but from the log the propagation does show now overhead (it's part of the forward) so it has to be something in the postprocessing

[kerim371]

Would be good, thank you.
I will finish the computation (at least 4 iterations yet) and do some tests with cluster configuration as well.

[kerim371]

@mloubout is it now possible to use preconditionners with fwi_objective?
Because doing that in a way:

d0 = F(model0)[indsrc]*q[indsrc]
r = Ml_ref[indsrc]*d0 - Ml_ref[indsrc]*d_obs[indsrc]
gradient = Il*J'[indsrc] * r
fval = .5*norm(r)^2

computes forward operator twice and thus spend 1.5 time more.

[mloubout]

Yes it is, forgot to point it out here. See

https://slimgroup.github.io/JUDI.jl/dev/preconditioners/#Inversion-wrappers

And some tests

JUDI.jl/test/test_gradients.jl

Line 43 in 5a2cf8f

@testset "FWI preconditionners test with $(nlayer) layers and tti $(tti) and viscoacoustic $(viscoacoustic) and freesurface $(fs)" begin

[kerim371]

And for model preconditionners can I use fwi_objective(model, q, dobs; model_precon=dPrec) like in lsrtm example?

[mloubout]

There is no model preconditionner for FWI only for LSRTM where you need J * Precon * dm in the objective. For fwi you just apply your model preconditioned to the gradient you get back

f,  g= fwi_objective
gp = Precon * g
```