po_s3_del does not handle * or ? in filename

[ggalibert]

Using * or ? in filename for po_s3_del is deceptively working and a source of error.

ggalibert@10-aws-syd:~/$ po_s3_del IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/15/IMOS_ACORN_V_20160515T1?3000Z_CBG_FV00_1-hour-avg.nc
Deleting 'IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/15/IMOS_ACORN_V_20160515T1?3000Z_CBG_FV00_1-hour-avg.nc' with index deletion
delete: 's3://imos-data/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/15/IMOS_ACORN_V_20160515T103000Z_CBG_FV00_1-hour-avg.nc'
delete: 's3://imos-data/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/15/IMOS_ACORN_V_20160515T113000Z_CBG_FV00_1-hour-avg.nc'
delete: 's3://imos-data/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/15/IMOS_ACORN_V_20160515T123000Z_CBG_FV00_1-hour-avg.nc'
delete: 's3://imos-data/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/15/IMOS_ACORN_V_20160515T133000Z_CBG_FV00_1-hour-avg.nc'
delete: 's3://imos-data/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/15/IMOS_ACORN_V_20160515T143000Z_CBG_FV00_1-hour-avg.nc'
delete: 's3://imos-data/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/15/IMOS_ACORN_V_20160515T153000Z_CBG_FV00_1-hour-avg.nc'
delete: 's3://imos-data/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/15/IMOS_ACORN_V_20160515T163000Z_CBG_FV00_1-hour-avg.nc'
delete: 's3://imos-data/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/15/IMOS_ACORN_V_20160515T173000Z_CBG_FV00_1-hour-avg.nc'
delete: 's3://imos-data/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/15/IMOS_ACORN_V_20160515T183000Z_CBG_FV00_1-hour-avg.nc'

In the above example the files have been deleted from S3 but not from the database...

Either this should be fixed or better documented somewhere that this functionality cannot be used.

[lbesnard]

I'm even surprised it half worked. I always thought that you had to give the full object path, and that no wildcard was allowed. Reason why you couldn't do things recursively and had to list the objects first and then delete them in a for loop
for f in object_path_list; do po_s3_del $f; done

Anyway, if you want to delete them from the database, you can run the command again with the full object path without wildcards. That should work

[ghost]

Looking at the code, it looks to me like it is intended to be per file with no wildcards. That said, it looks like it might half work if supplied with a wildcard:

https://github.com/aodn/data-services/blob/master/profile.d/util.sh#L26

It should also be logging if it fails the unindex operation, so this may be a way to help identify specifically the ones that it's deleted from S3 but not the database. I think the loop method from @lbesnard is probably the safest way to delete en masse but there are references to "bulk" functions there as well, so there may well be another way to trigger it...

[ggalibert]

Thanks, will do the loop method. However I tried these other 2 methods and didn't work unfortunately (would have been handy...):

ggalibert@10-aws-syd:/mnt/imos-data$ find IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/ -type f -wholename "*IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/*.nc" -print0 | xargs -0 po_s3_del
xargs: po_s3_del: No such file or directory
ggalibert@10-aws-syd:/mnt/imos-data$ find IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/ -type f -wholename "*IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/*.nc" -exec po_s3_del {} +
find: `po_s3_del': No such file or directory

while

ggalibert@10-aws-syd:/mnt/imos-data$ find IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/ -type f -wholename "*IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/*.nc" -print0 | xargs -0 -I {} echo {}
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T003000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T033000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T043000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T063000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T073000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T083000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T093000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T103000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T113000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T123000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T133000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T143000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T153000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T163000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T173000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T183000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T193000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T203000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T223000Z_CBG_FV00_1-hour-avg.nc
IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/16/IMOS_ACORN_V_20160516T233000Z_CBG_FV00_1-hour-avg.nc

works

[ggalibert]

yep, worked with:

ggalibert@10-aws-syd:/mnt/imos-data$ for file in `find IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/19/ -type f -wholename "*IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/19/*.nc"`; do po_s3_del $file; done
Deleting 'IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/19/IMOS_ACORN_V_20160519T023000Z_CBG_FV00_1-hour-avg.nc' with index deletion
delete: 's3://imos-data/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/19/IMOS_ACORN_V_20160519T023000Z_CBG_FV00_1-hour-avg.nc'
Deleting 'IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/19/IMOS_ACORN_V_20160519T033000Z_CBG_FV00_1-hour-avg.nc' with index deletion
delete: 's3://imos-data/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/19/IMOS_ACORN_V_20160519T033000Z_CBG_FV00_1-hour-avg.nc'
Deleting 'IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/19/IMOS_ACORN_V_20160519T043000Z_CBG_FV00_1-hour-avg.nc' with index deletion
delete: 's3://imos-data/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/19/IMOS_ACORN_V_20160519T043000Z_CBG_FV00_1-hour-avg.nc'
Deleting 'IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/19/IMOS_ACORN_V_20160519T053000Z_CBG_FV00_1-hour-avg.nc' with index deletion
delete: 's3://imos-data/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/19/IMOS_ACORN_V_20160519T053000Z_CBG_FV00_1-hour-avg.nc'
Deleting 'IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/19/IMOS_ACORN_V_20160519T063000Z_CBG_FV00_1-hour-avg.nc' with index deletion
delete: 's3://imos-data/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/19/IMOS_ACORN_V_20160519T063000Z_CBG_FV00_1-hour-avg.nc'
Deleting 'IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/19/IMOS_ACORN_V_20160519T103000Z_CBG_FV00_1-hour-avg.nc' with index deletion
delete: 's3://imos-data/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/19/IMOS_ACORN_V_20160519T103000Z_CBG_FV00_1-hour-avg.nc'
Deleting 'IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/19/IMOS_ACORN_V_20160519T113000Z_CBG_FV00_1-hour-avg.nc' with index deletion
delete: 's3://imos-data/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/CBG/2016/05/19/IMOS_ACORN_V_20160519T113000Z_CBG_FV00_1-hour-avg.nc'

[ghost]

Might be because it's defined as a function and for some reason xargs and find couldn't resolve it, whereas the bash loop keeps it in the shell environment.