'ragavi-vis' -- Unexpected Interruption During Plotting -- 'inspect' worker

[bazhaoyu]

Hi,

I'm currently processing MeerKAT data using CARACAL and encountered an unexpected issue with 'ragavi-vis' during the plotting phase. The process gets interrupted, and I am unsure why. Here's the error message I encountered (screenshot attached below):

My environment is set up with CARACAL version 1.0.7 and STIMELA 1.7.7, using APPTAINER 1.2.4. For further information, here are my configure file and output log file here.

Previously, I successfully processed data of around 5 GB, but this issue arises with a larger dataset of approximately 2 TB. Could this issue be related to the large size of the data? Has anyone experienced something similar, or could you suggest potential solutions? Thanks in advance for your help!

[KshitijT]

@bazhaoyu most probably if this works with smaller datasets the size - probably RAM? might be the issue. @Mulan-94 any suggestions here?
Does using shadems instead of ragavi give you similar issues ?

[bazhaoyu]

@bazhaoyu most probably if this works with smaller datasets the size - probably RAM? might be the issue. @Mulan-94 any suggestions here? Does using shadems instead of ragavi give you similar issues ?

Thank you for your reply. I noticed that maybe I could set a larger 'mem_limit' with a value of ~ 80 GB (default is 8GB) for 'ragavi-vis'. I will let you know if it works. If not, I would like to try 'shadems' to see what happens.

[bazhaoyu]

Hi,
Using 'ragavi-vis' with an 80 GB memory limit reported the same error.
Using 'shadems' reported another error: "NoneType" object has no attribute 'upper'. I am not clear if there are any necessary settings that I didn't set when choosing 'shadems'？

This is the configuration info:

##########
inspect:  
  enable: true
  label_in: cal 
  uvrange: '>150'
  dirname: 'crosscal'
  label_plot: '1gc'
  mem_limit: '80GB'
  shadems:
    enable: true 
    default_column: CORRECTED_DATA 
  standard_plotter: shadems  
  field: calibrators   # target  # bpcal  # gcal 
  real_imag:
    enable: true
    avgtime: '60'
    avgchan: '10'
  amp_chan:
    enable: true
    avgtime: '60'
    avgchan: '1'
  phase_chan:
    enable: true
    avgtime: '60'
    avgchan: '1'
  amp_uvwave:
    enable: true
    avgtime: '60'
    avgchan: '10'
  phase_uvwave:
    enable: true
    avgtime: '60'
    avgchan: '10'
################

This is the screenshot of the error:

I also tried the configuration as your sample -- 'meerkat-continuum-defaults.yml', but set the 'standard_plotter: shadems'. However, I got the same error "NoneType" object has no attribute 'upper'.

Do you happen to know what the problem is or what I should do to fix it? Thanks a lot!

[Athanaseus]

My environment is set up with CARACAL version 1.0.7 and STIMELA 1.7.7, using APPTAINER 1.2.4. For further information, here are my configure file and output log file here.

Please provide the shadems full log. The link above doesn't work.

[Fil8]

Maybe the ragavi memory issue is due to the fact that averaging in time or in chans is turned off in the code?

https://github.com/caracal-pipeline/caracal/blob/2d338e2afc5ed8fc723be95e74b071890d4a7bed/caracal/workers/inspect_worker.py#L519C1-L520C35