SPLIT_LAST parameter commented out in the code

[ainagaya]

Hi @kinow @dbeltrankyl ,

I was checking the available variables with calendar information about the splits. I wanted a variable that tells me if that SPLIT is the last split of the chunk or not. For that, what I tried was SPLIT_LAST (even though I thought that maybe it meant the last split of the whole simulation, meaning the last SPLIT of the last CHUNK), but I saw that the block of code defining that variable is commented out: https://github.com/search?q=repo%3ABSC-ES%2Fautosubmit+SPLIT_LAST&type=code

I think that is a nice parameter that could be (re)included.

Thanks!

[dbeltrankyl]

Uh, not sure why it is commented.

You can do this meanwhile:

[ %SPLIT% -eq %SPLITSIZE% ] (%SPLIT% starts in 1 )

[ainagaya]

Hi @dbeltrankyl , thanks! But this gives the equivalent to SPLIT_FIRST, doesn't it? I think that I will compare SPLIT_END_DATE with CHUNK_END_DATE, and that should work.

[dbeltrankyl]

I think, it is the last, no?

Having a job_date_member_chunk configured like this

job_date_member_chunk_1
..
job_date_member_chunk_31
%JOBS.JOB.SPLITS%: 31 ( or auto) 

template:

%SPLITS% == 31 ( or auto ) # This is what you define, it returns the value as it was set 
%SPLITSIZE% == 31  # this the actual definitive  size

%SPLIT% == 1 for job_date_member_chunk_1

%SPLIT% == 31 for job_date_member_chunk_31

[ 1 -eq 31 ] # job_date_member_chunk_1
 don't enter

[ 31 -eq 31 ] # job_date_member_chunk_31
 enter

[ainagaya]

In my case I have this values for the latest split:

SPLIT=${27:-31}
SPLITSIZE=${28:-1}

That comes from:

JOBS:
  TRANSFER:
    FILE: templates/transfer.sh
    PLATFORM: lumi-transfer
    DEPENDENCIES: SIM TRANSFER-1
    RUNNING: chunk
    SPLITS: 'auto'
EXPERIMENT:
  SPLITSIZEUNIT: day # default CHUNKSIZEUNIT-1 (month-1 == day)
  # Split size. NUMERIC: 4, 6, 12
  SPLITSIZE: 2
  # Split policy. STRING: flexible, strict
  SPLITPOLICY: flexible

So I think that since I'm using auto the splitsize varies among splits.

What I want to know is if the current split is the latest in that chunk or not.

[ainagaya]

Maybe comparing with SPLIT with CHUNKSIZE is what makes sense?

[dbeltrankyl]

I've tested it and %SPLITS% -eq %SPLIT% should be enough

  a:
    SCRIPT: |
      echo "Hello World with id=FAILED + wrappers"
      echo "SPLITS: %SPLITS%"
      echo "SPLIT: %SPLIT%"
      echo "SPLITSIZE: %SPLITSIZE%"
    PLATFORM: MARENOSTRUM5
    DEPENDENCIES: a-1
    RUNNING: chunk
    wallclock: 00:02
    threads: 4
    EXCLUSIVE: True
    splits: auto

What I want to know is if the current split is the latest in that chunk or not.

chunk 1

cat a000_20221101_fc1_1_61_A.cmd

echo "SPLITS: 61"
echo "SPLIT: 61"

[ 61 -eq 61 ] is true, and it is the last split of the current chunk

cat a000_20221101_fc1_1_1_A.cmd

echo "SPLITS: 61"
echo "SPLIT: 1"

chunk 2:

cat a000_20221101_fc1_2_59_A.cmd* of the current chunk

echo "SPLITS: 59"
echo "SPLIT: 59"

[ 59 -eq 59 ] is true, and it is the last split of the current chunk

cat a000_20221101_fc1_2_1_A.cmd

echo "SPLITS: 59"
echo "SPLIT: 1"

While for the first is False ( and the rest )

So I think that since I'm using auto the splitsize varies among splits.

It shouldn't vary until you go to the next chunk, I think %SPLITS% is more reliable after the testing.

[ainagaya]

Nice, thanks Dani!! So SPLITS is the total number of splits in a chunk, right? Linking #1463 , because it is documented, but I did not understand the variable meaning.

It shouldn't vary until you go to the next chunk, I think %SPLITS% is more reliable after the testing.

But for example if the splitsize is 2 and the policy is flexible, for January I get 15 splits of 2 days and 1 split of 1 day, right?

Sorry for all these questions!! Trying to find the best way to handle https://earth.bsc.es/gitlab/digital-twins/de_340-2/workflow/-/issues/827 :)

[dbeltrankyl]

Nice, thanks Dani!! So SPLITS is the total number of splits in a chunk, right?

Yes,

But for example if the splitsize is 2 and the policy is flexible, for January I get 15 splits of 2 days and 1 split of 1 day, right?

if I recall well, what happens is that you will get 15 splits but the last split will contain 3 days