Sweeps

[cosmo3769]

Sweeps Code here. This is to resolve issue 23, issue 24, issue 20, issue 26.

[cosmo3769]

@ayulockin I managed to produce some results. Please review the code. Things I am unsure of:

• I ran the main function which also runs the train function. In the train function, the configs for epochs is only 1 to quickly see the results. But while running sweeps, should the epoch be more than 1? Does this epoch for train config also depend on the sweep_config epoch values?
• Does the sweeps result show at the end of the wandb run? I got one sweep result at the end of the run. Here is the link showing the result: sweep result
• Why is every sweep ran is showing in pending state: all sweeps
• I don't have the permission to delete the sweeps. The sweeps are increasing in number. Could you check from your side if you can do so?
• The whole process is running completely with !python sweep_train.py --configs configs/config.py with no errors in quick sight. But when I look closely the output of the result, I see a peculiar error. I am attaching a screenshot of the error down below. By looking at this error, I am in doubt that am i getting the correct result for sweep or not. Have a look at the error(can't find 'main' module in ''):

[cosmo3769]

• The whole process is running completely with !python sweep_train.py --configs configs/config.py with no errors in quick sight. But when I look closely the output of the result, I see a peculiar error. I am attaching a screenshot of the error down below. By looking at this error, I am in doubt that am i getting the correct result for sweep or not. Have a look at the error(can't find 'main' module in ''):

I think there is an error.(ERROR - Detected 5 failed runs in a row, shutting down)

[cosmo3769]

I coded keeping the main function. I think I have to go for other alternative. But I also think there is some slight error using the main function and it can be fixed. Not very sure about this.

[ayulockin]

First thing first, we will have to remove the code from #21 from this PR.

[ayulockin]

I don't have the permission to delete the sweeps. The sweeps are increasing in number. Could you check from your side if you can do so?

You have admin access now, you can delete the runs/sweeps/artifacts.

[cosmo3769]

Manual configuring sweep error

[cosmo3769]

Will this example work while using FLAGS?

[ayulockin]

I don't know actually.

[cosmo3769]

wandb.config resolved for sweeps to work as given in the documentaion and this example

with wandb.init(config=CONFIG.value.to_dict(), entity="wandb_fc", project="ssl-study"):
      config = wandb.config

[cosmo3769]

Now, the issue to resolve is to fix the sweeps config file so it can take the parameters value from the .yaml file or the .py file.

[cosmo3769]

Manual configuring sweep error

Now, the issue to resolve is to fix the sweeps config file so it can take the parameters value from the .yaml file or the .py file.

The recent commit resolves these issues. wandb.agent properly working with no errors. I have to remove FLAGS from the sweep_train.py to make it work.

With this, I still have some questions:

• Is the sweeps showing in the w&b portal correct?
• wandb.agent is taking the parameters value from sweep_config.yaml file. The main config file has epochs of 3. wandb.agent is taking epoch value of 5. So, should it have to run for 3 epochs that's given in the main config file config.py or 5 epochs that's given in the sweep_config.py file?

[ayulockin]

Is the sweeps showing in the w&b portal correct?

can you share the sweep dashboard?

The main config file has epochs of 3. wandb.agent is taking epoch value of 5

The agent will pick from epoch values assigned in the sweep_config.yaml. In your sweep yaml file you have [5, 10, etc].

[cosmo3769]

can you share the sweep dashboard?

@ayulockin Here is the sweep dashboard.

[ayulockin]

This looks perfect. @cosmo3769

Sabash.

[ayulockin]

If you think code refactoring is required, do it.

[cosmo3769]

If you think code refactoring is required, do it.

Done.

[cosmo3769]

@ayulockin Should we merge this branch into master branch now or after fixing the wandb.Table logging everytime issue?

[cosmo3769]

Should we do something like this to fix wandb.Table logging everytime issue?

if wandb.run is not None:
      if self.args.train_config.use_validation_table_log:
            wandb.log({
                'val_eval_loss': val_eval_loss,
                'val_top@1': val_top_1_acc,
                'val_top@5': val_top_5_acc,
                'val_table': validation_table
            })
       else: 
            wandb.log({
                'val_eval_loss': val_eval_loss,
                'val_top@1': val_top_1_acc,
                'val_top@5': val_top_5_acc,
            })

[cosmo3769]

Should we do something like this to fix wandb.Table logging everytime issue?

Yes, it works.

• wandb.Table logging everytime issue fixed.

@ayulockin Is there any other way to do it?

[cosmo3769]

Work done:

• Sweeps
• wandb.Table
• Code Refactored
• Some augmentations used (We have to create more robust pipeline)
• Class Weights
• LR Scheduling (Not working correctly)
• README updated

[ayulockin]

So in favor of this PR we should close #21 PR? Given there's overlap of the code and everything in #21 is also present here.

Also LGTM. I will give your code a try and merge it.

[cosmo3769]

So in favor of this PR we should close #21 PR?

I think when we will merge this #22 PR, the #21 PR will get closed too.

[ayulockin]

I think the way you are doing sweep is correct. It's not working with train.py as stated. LGTM.

I am merging the PR and we will fix the edge cased if we encounter (that we didn't so far) one PR at a time. :D