issue creating directory

[madhavitippani]

I was able to make GEO images and create directories fine, until today. All of a sudden it throws the following errors and its stuck there.

this_sample.add_image(
...     tiff = img_fullres,
...     channels = img_channels,
...     defaultChannels = default_channels,
...     scale = m_per_px
... )
[14:47:01] INFO     Transforming /dcs04/lieber/lcolladotor/spatialHPC_LIBD4035/spatial_hpc/processed-data/VSPG_image_stitching/Br8325/combined_Br8325.tif to COG.                    logger.py:27
Sample(name=Br8325, path=/dcs04/lieber/lcolladotor/spatialHPC_LIBD4035/spatial_hpc/processed-data/VSPG_image_stitching/Br8325) with ['coords'] as coords and None as features.

this_sample.write()
[14:47:04] INFO     'Br8325' Executing queued functions                                                                                                                              logger.py:27
          INFO     Br8325 Adding coords 'coords'                                                                                                                                    logger.py:27
[14:47:05] INFO     Writing uncompressed GeoTIFF /dcs04/lieber/lcolladotor/spatialHPC_LIBD4035/spatial_hpc/processed-data/VSPG_image_stitching/Br8325/combined_Br8325_1.tif          logger.py:27
[14:47:27] INFO     GDAL signalled an error: err_no=1, msg='TIFFResetField:Value exceeds 32bit range of output type.'                                                                image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='TIFFWriteDirectorySec:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.'                                 image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='TIFFFetchDirectory:Sanity check on directory count failed, this is probably not a valid IFD offset'                      image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='TIFFReadDirectory:Failed to read directory at offset 160046330'                                                          image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='Cannot re-open TIFF handle for file                                                                                      image.py:168
                   /dcs04/lieber/lcolladotor/spatialHPC_LIBD4035/spatial_hpc/processed-data/VSPG_image_stitching/Br8325/combined_Br8325_1.tif_. Directory chaining may be corrupted             
                   !'                                                                                                                                                                           
          INFO     GDAL signalled an error: err_no=1, msg='TIFFFetchDirectory:Sanity check on directory count failed, this is probably not a valid IFD offset'                      image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='TIFFReadDirectory:Failed to read directory at offset 160046330'                                                          image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='Cannot re-open TIFF handle for file                                                                                      image.py:168
                   /dcs04/lieber/lcolladotor/spatialHPC_LIBD4035/spatial_hpc/processed-data/VSPG_image_stitching/Br8325/combined_Br8325_1.tif_. Directory chaining may be corrupted             
                   !'                                                                                                                                                                           
          INFO     GDAL signalled an error: err_no=1, msg='TIFFFetchDirectory:Sanity check on directory count failed, this is probably not a valid IFD offset'                      image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='TIFFReadDirectory:Failed to read directory at offset 160046330'                                                          image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='Cannot re-open TIFF handle for file                                                                                      image.py:168
                   /dcs04/lieber/lcolladotor/spatialHPC_LIBD4035/spatial_hpc/processed-data/VSPG_image_stitching/Br8325/combined_Br8325_1.tif_. Directory chaining may be corrupted             
                   !'                                                                                                                                                                           
          INFO     GDAL signalled an error: err_no=1, msg='TIFFFetchDirectory:Sanity check on directory count failed, this is probably not a valid IFD offset'                      image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='TIFFReadDirectory:Failed to read directory at offset 160046330'                                                          image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='Cannot re-open TIFF handle for file                                                                                      image.py:168
                   /dcs04/lieber/lcolladotor/spatialHPC_LIBD4035/spatial_hpc/processed-data/VSPG_image_stitching/Br8325/combined_Br8325_1.tif_. Directory chaining may be corrupted             
                   !'                                                                                                                                                                           
          INFO     GDAL signalled an error: err_no=1, msg='TIFFFetchDirectory:Sanity check on directory count failed, this is probably not a valid IFD offset'                      image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='TIFFReadDirectory:Failed to read directory at offset 160046330'                                                          image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='Cannot re-open TIFF handle for file                                                                                      image.py:168
                   /dcs04/lieber/lcolladotor/spatialHPC_LIBD4035/spatial_hpc/processed-data/VSPG_image_stitching/Br8325/combined_Br8325_1.tif_. Directory chaining may be corrupted             
                   !'                                                                                                                                                                           
          INFO     GDAL signalled an error: err_no=1, msg='TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.'                                     image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='TIFFResetField:Value exceeds 32bit range of output type.'                                                                image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='TIFFWriteDirectorySec:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.'                                 image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='TIFFFetchDirectory:Sanity check on directory count failed, this is probably not a valid IFD offset'                      image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='TIFFReadDirectory:Failed to read directory at offset 160046330'                                                          image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='Cannot re-open TIFF handle for file                                                                                      image.py:168
                   /dcs04/lieber/lcolladotor/spatialHPC_LIBD4035/spatial_hpc/processed-data/VSPG_image_stitching/Br8325/combined_Br8325_1.tif_. Directory chaining may be corrupted             
                   !'                                                                                                                                                                           
          INFO     GDAL signalled an error: err_no=1,                                                                                                                               image.py:168
                   msg='TIFFFetchDirectory:/dcs04/lieber/lcolladotor/spatialHPC_LIBD4035/spatial_hpc/processed-data/VSPG_image_stitching/Br8325/combined_Br8325_1.tif_: Can not                 
                   read TIFF directory count'                                                                                                                                                   
          INFO     GDAL signalled an error: err_no=1, msg='TIFFReadDirectory:Failed to read directory at offset 4455013626'                                                         image.py:168
          INFO     GDAL signalled an error: err_no=1,                                                                                                                               image.py:168
                   msg='TIFFResetField:/dcs04/lieber/lcolladotor/spatialHPC_LIBD4035/spatial_hpc/processed-data/VSPG_image_stitching/Br8325/combined_Br8325_1.tif_: Can not read                
                   TIFF directory count'                                                                                                                                                        
          INFO     GDAL signalled an error: err_no=1, msg='TIFFRewriteDirectory:tif->tif_diroff exceeds 32 bit range allowed for Classic TIFF'                                      image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.'                                     image.py:168
          INFO     GDAL signalled an error: err_no=1, msg='TIFFResetField:Value exceeds 32bit range of output type.'

[madhavitippani]

I tried running it with the old images which ran fine previously but throwing the same error today.

[chaichontat]

Could you send me the images?

[madhavitippani]

It's a 6 GB multichannel tiff image, how do I share it?

[madhavitippani]

It's throwing a different error today

>>>this_sample.write()
[09:23:06] INFO     'Br3942' Executing queued functions                                                                                                                    logger.py:27
           INFO     Br3942 Adding coords 'coords'                                                                                                                          logger.py:27
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /jhpce/shared/libd/core/samui/1.0.0-next.45/samui/loopy/sample.py:119 in write                   │
│                                                                                                  │
│   116 │   │   │   log(f"'{self.name}' Executing queued functions")                               │
│   117 │   │   │   for f in self.queue_:                                                          │
│   118 │   │   │   │   try:                                                                       │
│ ❱ 119 │   │   │   │   │   f[1]()                                                                 │
│   120 │   │   │   │   except Exception as e:                                                     │
│   121 │   │   │   │   │   raise Exception(f"Error executing queued functions: {f[0]}") from e    │
│   122 │   │   │   self.queue_ = []                                                               │
│                                                                                                  │
│ /jhpce/shared/libd/core/samui/1.0.0-next.45/samui/loopy/image.py:121 in run                      │
│                                                                                                  │
│   118 │   │                                                                                      │
│   119 │   │   def run():                                                                         │
│   120 │   │   │   for name, c in zip(names, chanlist):                                           │
│ ❱ 121 │   │   │   │   self._write_uncompressed_geotiff(                                          │
│   122 │   │   │   │   │   path=path_in.with_name(name),                                          │
│   123 │   │   │   │   │   channels=c,                                                            │
│   124 │   │   │   │   │   transform=rasterio.Affine(                                             │
│                                                                                                  │
│ /jhpce/shared/libd/core/samui/1.0.0-next.45/samui/loopy/image.py:168 in                          │
│ _write_uncompressed_geotiff                                                                      │
│                                                                                                  │
│   165 │   │   dst: DatasetWriter                                                                 │
│   166 │   │   # Not compressing here since we cannot control the compression level.              │
│   167 │   │   assert 0 < len(channels) <= 4                                                      │
│ ❱ 168 │   │   with rasterio.open(                                                                │
│   169 │   │   │   path.with_suffix(".tif_").as_posix(),                                          │
│   170 │   │   │   "w",                                                                           │
│   171 │   │   │   driver="GTiff",                                                                │
│                                                                                                  │
│ /jhpce/shared/libd/core/samui/1.0.0-next.45/samui_env/lib/python3.10/site-packages/rasterio/env. │
│ py:451 in wrapper                                                                                │
│                                                                                                  │
│   448 │   │   │   session = DummySession()                                                       │
│   449 │   │                                                                                      │
│   450 │   │   with env_ctor(session=session):                                                    │
│ ❱ 451 │   │   │   return f(*args, **kwds)                                                        │
│   452 │                                                                                          │
│   453 │   return wrapper                                                                         │
│   454                                                                                            │
│                                                                                                  │
│ /jhpce/shared/libd/core/samui/1.0.0-next.45/samui_env/lib/python3.10/site-packages/rasterio/__in │
│ it__.py:314 in open                                                                              │
│                                                                                                  │
│   311 │   │   │   │   driver = driver_from_extension(path)                                       │
│   312 │   │   │   writer = get_writer_for_driver(driver)                                         │
│   313 │   │   │   if writer is not None:                                                         │
│ ❱ 314 │   │   │   │   dataset = writer(                                                          │
│   315 │   │   │   │   │   path,                                                                  │
│   316 │   │   │   │   │   mode,                                                                  │
│   317 │   │   │   │   │   driver=driver,                                                         │
│                                                                                                  │
│ in rasterio._io.DatasetWriterBase.__init__:1464                                                  │
│                                                                                                  │
│ in rasterio._io._delete_dataset_if_exists:314                                                    │
│                                                                                                  │
│ in rasterio._io._delete_dataset_if_exists:315                                                    │
│                                                                                                  │
│ in rasterio._base.open_dataset:221                                                               │
│                                                                                                  │
│ in rasterio._err.exc_wrap_pointer:221                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CPLE_AppDefinedError: /dcs04/lieber/lcolladotor/spatialHPC_LIBD4035/spatial_hpc/processed-data/VSPG_image_stitching/Br3942/combined_Br3942_1.tif_: TIFFReadDirectory:Failed to read 
directory at offset 31853458

The above exception was the direct cause of the following exception:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <module>:1                                                                                    │
│                                                                                                  │
│ /jhpce/shared/libd/core/samui/1.0.0-next.45/samui/loopy/sample.py:61 in wrapper                  │
│                                                                                                  │
│    58 │   │   def wrapper(self: Sample, *args: P.args, **kwargs: P.kwargs) -> R:                 │
│    59 │   │   │   if self.path is None:                                                          │
│    60 │   │   │   │   raise ValueError("Path not set. Use Sample.set_path() first")              │
│ ❱  61 │   │   │   return func(self, *args, **kwargs)                                             │
│    62 │   │                                                                                      │
│    63 │   │   return wrapper                                                                     │
│    64                                                                                            │
│                                                                                                  │
│ /jhpce/shared/libd/core/samui/1.0.0-next.45/samui/loopy/sample.py:121 in write                   │
│                                                                                                  │
│   118 │   │   │   │   try:                                                                       │
│   119 │   │   │   │   │   f[1]()                                                                 │
│   120 │   │   │   │   except Exception as e:                                                     │
│ ❱ 121 │   │   │   │   │   raise Exception(f"Error executing queued functions: {f[0]}") from e    │
│   122 │   │   │   self.queue_ = []                                                               │
│   123 │   │                                                                                      │
│   124 │   │   (self.path / "sample.json").write_text(self.json())                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
Exception: Error executing queued functions: Add image: 
/dcs04/lieber/lcolladotor/spatialHPC_LIBD4035/spatial_hpc/processed-data/VSPG_image_stitching/combined_Br3942/combined_Br3942.tif
>>> session_info.show()
-----
PIL                 10.0.0
affine              2.4.0
anndata             0.9.1
loopy               NA
matplotlib          3.7.2
numpy               1.24.4
pandas              2.0.3
pyhere              1.0.0
scanpy              1.9.3
scipy               1.11.1
session_info        1.0.0
tifffile            2023.7.4
-----
Python 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:40:32) [GCC 12.3.0]
Linux-5.14.0-284.11.1.el9_2.x86_64-x86_64-with-glibc2.34
-----
Session information updated at 2024-02-21 09:24`

[madhavitippani]

I think Samui is running fine on images <6GB and throwing these above errors when the image size is >5.9GB.