The centralized nightlies job failed on Wednesday (2024-09-11)

[github-actions]

The centralized nightlies job failed on Wednesday (2024-09-11) in run 10822955068

[jdblischak]

@kounelisagis there are now 3 failing TileDB-Py tests when building the TileDB-Py dev branch against the TileDB dev branch.

• For the test test_array_schema_enumeration, I thought your PR Update HISTORY for release 0.32.0 against TileDB 2.26.0 and replace dump() calls with operator<< TileDB-Py#1975 was supposed to fix this (https://github.com/jdblischak/centralized-tiledb-nightlies/issues/22#issuecomment-2341464948)
• For the two test failures related to StorageManager, I suspect they are caused by this upstream TileDB PR Remove StorageManager from stats dumps. TileDB#5297 Will you be able to address these test failures in a branch? That way I can point my nightlies to a TileDB-Py branch that is meant to be built against TileDB dev

=========================== short test summary info ============================
FAILED tiledb/tests/test_enumeration.py::EnumerationTest::test_array_schema_enumeration - assert "ArraySchema: Unable to check if unknown enumeration is loaded. No enumeration named 'enmr3'." == "Array: Unable to get enumeration; Enumeration 'enmr3' does not exist."
  
  - Array: Unable to get enumeration; Enumeration 'enmr3' does not exist.
  + ArraySchema: Unable to check if unknown enumeration is loaded. No enumeration named 'enmr3'.
FAILED tiledb/tests/test_libtiledb.py::GetStatsTest::test_ctx - assert 'Context.StorageManager.write_store' in '{\n  "timers": {\n    "Context.write_store_frag_meta.sum": 2.40292,\n    "Context.write_store_frag_meta.avg": 0.0023558,\n    "Context.write_meta.sum": 1.82331,\n    "Context.write_meta.avg": 0.00116655,\n    "Context.subSubarray.sort_and_merge_ranges.sum": 0.00828159,\n    "Context.subSubarray.sort_and_merge_ranges.avg": 1.12675e-05,\n    "Context.subSubarray.read_load_relevant_rtrees.sum": 0.0592289,\n    "Context.subSubarray.read_load_relevant_rtrees.avg": 1.78508e-05,\n    "Context.subSubarray.read_compute_tile_overlap.sum": 0.260605,\n    "Context.subSubarray.read_compute_tile_overlap.avg": 5.08497e-05,\n    "Context.subSubarray.read_compute_simple_tile_overlap.sum": 0.0459882,\n    "Context.subSubarray.read_compute_simple_tile_overlap.avg": 8.34632e-05,\n    "Context.subSubarray.read_compute_relevant_tile_overlap.sum": 0.127294,\n    "Context.subSubarray.read_compute_relevant_tile_overlap.avg": 4.60044e-05,\n    "Context.subSubarray.read_compute_est_result_size.sum": 0.390431,\n    "Context.subSubarray.read_compute_est_result_size.avg": 0.000153956,\n    "Context.subSubarray.compute_relevant_frags.sum": 0.0588457,\n    "Context.subSubarray.compute_relevant_frags.avg": 1.77353e-05,\n    "Context.sm_read_load_consolidated_frag_meta.sum": 0.000299845,\n    "Context.sm_read_load_consolidated_frag_meta.avg": 9.99483e-05,\n    "Context.sm_load_fragment_metadata.sum": 0.0896338,\n    "Context.sm_load_fragment_metadata.avg": 6.55218e-05,\n    "Context.sm_load_filtered_fragment_uris.sum": 0.037451,\n    "Context.sm_load_filtered_fragment_uris.avg": 2.75578e-05,\n    "Context.sm_load_enumeration.sum": 0.000603242,\n    "Context.sm_load_enumeration.avg": 7.54052e-05,\n    "Context.sm_load_array_schemas_and_fragment_metadata.sum": 0.244309,\n    "Context.sm_load_array_schemas_and_fragment_metadata.avg": 0.000179771,\n    "Context.sm_load_array_schema_latest.sum": 0.00704617,\n    "Context.sm_load_array_schema_latest.avg": 7.82908e-05,\n    "Context.sm_load_array_schema_from_uri.sum": 0.190367,\n    "Context.sm_load_array_schema_from_uri.avg": 7.28816e-05,\n    "Context.sm_load_array_metadata.sum": 0.00572608,\n    "Context.sm_load_array_metadata.avg": 4.09006e-05,\n    "Context.sm_load_all_array_schemas.sum": 0.205236,\n    "Context.sm_load_all_array_schemas.avg": 8.18652e-05,\n    "Context.sm_array_load_fragments.sum": 0.00117922,\n    "Context.sm_array_load_fragments.avg": 0.000131025,\n    "Context.load_group_from_all_uris.sum": 0.0217312,\n    "Context.load_group_from_all_uris.avg": 0.000101075,\n    "Context.load_group_details.sum": 0.0221566,\n    "Context.load_group_details.avg": 2.02528e-05,\n    "Context.group_open_for_writes.sum": 0.00101028,\n    "Context.group_open_for_writes.avg": 2.2652e-06,\n    "Context.group_open_for_reads.sum": 0.0228723,\n    "Context.group_open_for_reads.avg": 3.52968e-05,\n    "Context.group_load_metadata_from_storage.sum": 1.54396,\n    "Context.group_load_metadata_from_storage.avg": 0.00356573,\n    "Context.array_reopen_directory.sum": 0.00055889,\n    "Context.array_reopen_directory.avg": 0.000186297,\n    "Context.array_reopen.sum": 0.0011362,\n    "Context.array_reopen.avg": 0.000284051,\n    "Context.array_open_write_load_schemas.sum": 0.094248,\n    "Context.array_open_write_load_schemas.avg": 8.28918e-05,\n    "Context.array_open_write_load_directory.sum": 0.069823,\n    "Context.array_open_write_load_directory.avg": 6.12483e-05,\n    "Context.array_open_without_fragments_load_directory.sum": 0.0029546,\n    "Context.array_open_without_fragments_load_directory.avg": 0.0002686,\n    "Context.array_open_without_fragments.sum": 0.00390507,\n    "Context.array_open_without_fragments.avg": 0.000355006,\n    "Context.array_open_read_without_fragments_load_schemas.sum": 0.000871825,\n    "Context.array_open_read_without_fragments_load_schemas.avg": 7.92568e-05,\n    "Context.array_open_read_load_schemas_and_fragment_meta.sum": 0.215166,\n    "Context.array_open_read_load_schemas_and_fragment_meta.avg": 0.000170767,\n    "Context.array_open_read_load_directory.sum": 0.274782,\n    "Context.array_open_read_load_directory.avg": 0.000169201,\n    "Context.array_open_WRITE.sum": 0.177096,\n    "Context.array_open_WRITE.avg": 0.000156583,\n    "Context.array_open_READ.sum": 0.505755,\n    "Context.array_open_READ.avg": 0.000311425,\n    "Context.array_open_MODIFY_EXCLUSIVE.sum": 0.00143571,\n    "Context.array_open_MODIFY_EXCLUSIVE.avg": 0.000159523,\n    "Context.VFS.ArrayDirectory.load_consolidated_commit_uris.sum": 0.0010817,\n    "Context.VFS.ArrayDirectory.load_consolidated_commit_uris.avg": 7.86687e-07,\n    "Context.VFS.ArrayDirectory.list_root_uris.sum": 0.0896568,\n    "Context.VFS.ArrayDirectory.list_root_uris.avg": 5.14678e-05,\n    "Context.VFS.ArrayDirectory.list_fragment_meta_uris.sum": 0.0435826,\n    "Context.VFS.ArrayDirectory.list_fragment_meta_uris.avg": 2.50187e-05,\n    "Context.VFS.ArrayDirectory.list_commit_uris.sum": 0.0702803,\n    "Context.VFS.ArrayDirectory.list_commit_uris.avg": 4.03446e-05,\n    "Context.VFS.ArrayDirectory.list_array_schema_uris.sum": 0.126122,\n    "Context.VFS.ArrayDirectory.list_array_schema_uris.avg": 4.24226e-05,\n    "Context.VFS.ArrayDirectory.list_array_meta_uris.sum": 0.0436618,\n    "Context.VFS.ArrayDirectory.list_array_meta_uris.avg": 2.50642e-05,\n    "Context.Subarray.read_compute_tile_coords.sum": 0.0467563,\n    "Context.Subarray.read_compute_tile_coords.avg": 1.69899e-05,\n    "Context.Query.Writer.write_tiles.sum": 0.892713,\n    "Context.Query.Writer.write_tiles.avg": 0.000469849,\n    "Context.Query.Writer.write_num_tiles.sum": 0.222797,\n    "Context.Query.Writer.write_num_tiles.avg": 0.000667058,\n    "Context.Query.Writer.split_coords_buff.sum": 8.6953e-05,\n    "Context.Query.Writer.split_coords_buff.avg": 8.5248e-08,\n    "Context.Query.Writer.sort_coords.sum": 0.0165743,\n    "Context.Query.Writer.sort_coords.avg": 5.11553e-05,\n    "Context.Query.Writer.prepare_tiles.sum": 0.0105029,\n    "Context.Query.Writer.prepare_tiles.avg": 3.14459e-05,\n    "Context.Query.Writer.prepare_filter_and_write_tiles.sum": 0.624974,\n    "Context.Query.Writer.prepare_filter_and_write_tiles.avg": 0.00088398,\n    "Context.Query.Writer.prepare_and_filter_tiles.sum": 0.193829,\n    "Context.Query.Writer.prepare_and_filter_tiles.avg": 0.000163156,\n    "Context.Query.Writer.ordered_write.sum": 2.2707,\n    "Context.Query.Writer.ordered_write.avg": 0.00331006,\n    "Context.Query.Writer.finalize.sum": 0.0378508,\n    "Context.Query.Writer.finalize.avg": 3.80028e-05,\n    "Context.Query.Writer.filter_tiles.sum": 0.0382752,\n    "Context.Query.Writer.filter_tiles.avg": 0.000114596,\n    "Context.Query.Writer.filter_tile.sum": 0.16523,\n    "Context.Query.Writer.filter_tile.avg": 4.4936e-05,\n    "Context.Query.Writer.dowork.sum": 3.54016,\n    "Context.Query.Writer.dowork.avg": 0.00347075,\n    "Context.Query.Writer.compute_coord_meta.sum": 0.00225656,\n    "Context.Query.Writer.compute_coord_meta.avg": 6.75616e-06,\n    "Context.Query.Writer.check_global_order.sum": 3.01e-07,\n    "Context.Query.Writer.check_global_order.avg": 5.01667e-08,\n    "Context.Query.Writer.check_coord_oob.sum": 0.0126484,\n    "Context.Query.Writer.check_coord_oob.avg": 1.24004e-05,\n    "Context.Query.Writer.check_coord_dups.sum": 0.00640162,\n    "Context.Query.Writer.check_coord_dups.avg": 1.93988e-05,\n    "Context.Query.Writer.check_attr_order.sum": 0.000130974,\n    "Context.Query.Writer.check_attr_order.avg": 1.90924e-07,\n    "Context.Query.Writer.DenseTiler.get_tile.sum": 0.114011,\n    "Context.Query.Writer.DenseTiler.get_tile.avg": 4.18235e-05,\n    "Context.Query.Subarray.sort_and_merge_ranges.sum": 0.000106882,\n    "Context.Query.Subarray.sort_and_merge_ranges.avg": 1.78137e-05,\n    "Context.Query.Subarray.read_load_relevant_rtrees.sum": 0.0075757,\n    "Context.Query.Subarray.read_load_relevant_rtrees.avg": 7.89136e-05,\n    "Context.Query.Subarray.read_compute_tile_overlap.sum": 0.0139519,\n    "Context.Query.Subarray.read_compute_tile_overlap.avg": 0.000145333,\n    "Context.Query.Subarray.read_compute_tile_coords.sum": 0.000221897,\n    "Context.Query.Subarray.read_compute_tile_coords.avg": 2.31143e-06,\n    "Context.Query.Subarray.read_compute_relevant_tile_overlap.sum": 0.00441067,\n    "Context.Query.Subarray.read_compute_relevant_tile_overlap.avg": 4.59445e-05,\n    "Context.Query.Subarray.compute_relevant_frags.sum": 0.000944027,\n    "Context.Query.Subarray.compute_relevant_frags.avg": 9.83361e-06,\n    "Context.Query.Reader.validate_attribute_order.sum": 2.6332e-05,\n    "Context.Query.Reader.validate_attribute_order.avg": 2.8314e-07,\n    "Context.Query.Reader.unfilter_coord_tiles.sum": 0.128977,\n    "Context.Query.Reader.unfilter_coord_tiles.avg": 4.30498e-05,\n    "Context.Query.Reader.unfilter_attr_tiles.sum": 0.644699,\n    "Context.Query.Reader.unfilter_attr_tiles.avg": 0.000185418,\n    "Context.Query.Reader.tile_offset_sizes.sum": 0.00546643,\n    "Context.Query.Reader.tile_offset_sizes.avg": 7.41714e-06,\n    "Context.Query.Reader.switch_sizes_to_offsets.sum": 7.399e-06,\n    "Context.Query.Reader.switch_sizes_to_offsets.avg": 1.42288e-07,\n    "Context.Query.Reader.sort_result_coords.sum": 7.2479e-05,\n    "Context.Query.Reader.sort_result_coords.avg": 8.05322e-06,\n    "Context.Query.Reader.resize_fixed_results_to_copy.sum": 3.748e-06,\n    "Context.Query.Reader.resize_fixed_results_to_copy.avg": 1.24933e-06,\n    "Context.Query.Reader.read_tiles.sum": 2.26349,\n    "Context.Query.Reader.read_tiles.avg": 0.000386196,\n    "Context.Query.Reader.read_coordinate_tiles.sum": 0.029578,\n    "Context.Query.Reader.read_coordinate_tiles.avg": 2.00122e-05,\n    "Context.Query.Reader.read_attribute_tiles.sum": 2.24222,\n    "Context.Query.Reader.read_attribute_tiles.avg": 0.000511573,\n    "Context.Query.Reader.read_and_unfilter_coords.sum": 0.168391,\n    "Context.Query.Reader.read_and_unfilter_coords.avg": 0.000231306,\n    "Context.Query.Reader.read_and_unfilter_attributes.sum": 0.0402968,\n    "Context.Query.Reader.read_and_unfilter_attributes.avg": 5.85709e-05,\n    "Context.Query.Reader.process_tiles.sum": 0.000994247,\n    "Context.Query.Reader.process_tiles.avg": 0.000331416,\n    "Context.Query.Reader.process_slabs.sum": 0.0932652,\n    "Context.Query.Reader.process_slabs.avg": 0.000136154,\n    "Context.Query.Reader.process_aggregates.sum": 0.011858,\n    "Context.Query.Reader.process_aggregates.avg": 3.41728e-05,\n    "Context.Query.Reader.merge_result_cell_slabs.sum": 0.00662612,\n    "Context.Query.Reader.merge_result_cell_slabs.avg": 9.02741e-06,\n    "Context.Query.Reader.max_num_cells_to_copy.sum": 0.000178206,\n    "Context.Query.Reader.max_num_cells_to_copy.avg": 2.42787e-07,\n    "Context.Query.Reader.load_tile_var_sizes.sum": 0.0305279,\n    "Context.Query.Reader.load_tile_var_sizes.avg": 8.24632e-06,\n    "Context.Query.Reader.load_tile_offsets.sum": 0.108525,\n    "Context.Query.Reader.load_tile_offsets.avg": 2.08102e-05,\n    "Context.Query.Reader.load_tile_metadata.sum": 0.0210189,\n    "Context.Query.Reader.load_tile_metadata.avg": 1.86504e-05,\n    "Context.Query.Reader.load_label_min_max_values.sum": 0.00976622,\n    "Context.Query.Reader.load_label_min_max_values.avg": 0.000105013,\n    "Context.Query.Reader.load_initial_data.sum": 0.0563024,\n    "Context.Query.Reader.load_initial_data.avg": 7.6394e-05,\n    "Context.Query.Reader.init_state.sum": 0.0362209,\n    "Context.Query.Reader.init_state.avg": 1.26602e-05,\n    "Context.Query.Reader.fix_offset_tiles.sum": 0.000123532,\n    "Context.Query.Reader.fix_offset_tiles.avg": 5.10463e-07,\n    "Context.Query.Reader.fill_dense_coords.sum": 0.000758379,\n    "Context.Query.Reader.fill_dense_coords.avg": 5.2302e-06,\n    "Context.Query.Reader.dowork.sum": 4.67674,\n    "Context.Query.Reader.dowork.avg": 0.00126569,\n    "Context.Query.Reader.dedup_tiles_with_timestamps.sum": 0.00619299,\n    "Context.Query.Reader.dedup_tiles_with_timestamps.avg": 8.61334e-06,\n    "Context.Query.Reader.dedup_fragments_with_timestamps.sum": 0.00821341,\n    "Context.Query.Reader.dedup_fragments_with_timestamps.avg": 1.12822e-05,\n    "Context.Query.Reader.create_result_tiles.sum": 0.0100629,\n    "Context.Query.Reader.create_result_tiles.avg": 1.21239e-05,\n    "Context.Query.Reader.copy_var_tiles.sum": 0.00784412,\n    "Context.Query.Reader.copy_var_tiles.avg": 2.66807e-05,\n    "Context.Query.Reader.copy_timestamps_tiles.sum": 0.000271972,\n    "Context.Query.Reader.copy_timestamps_tiles.avg": 4.53287e-05,\n    "Context.Query.Reader.copy_offsets_tiles.sum": 0.00224441,\n    "Context.Query.Reader.copy_offsets_tiles.avg": 4.31617e-05,\n    "Context.Query.Reader.copy_offset_tiles.sum": 0.0065389,\n    "Context.Query.Reader.copy_offset_tiles.avg": 2.70203e-05,\n    "Context.Query.Reader.copy_fixed_tiles.sum": 0.422514,\n    "Context.Query.Reader.copy_fixed_tiles.avg": 0.000188202,\n    "Context.Query.Reader.copy_fixed_data_tiles.sum": 0.0146291,\n    "Context.Query.Reader.copy_fixed_data_tiles.avg": 2.74467e-05,\n    "Context.Query.Reader.copy_fixed_coords.sum": 0.000221674,\n    "Context.Query.Reader.copy_fixed_coords.avg": 7.91693e-06,\n    "Context.Query.Reader.copy_fixed_attr_values.sum": 0.000101738,\n    "Context.Query.Reader.copy_fixed_attr_values.avg": 7.826e-06,\n    "Context.Query.Reader.copy_coordinates.sum": 0.000269706,\n    "Context.Query.Reader.copy_coordinates.avg": 1.79804e-05,\n    "Context.Query.Reader.copy_attribute.sum": 0.442087,\n    "Context.Query.Reader.copy_attribute.avg": 0.000177759,\n    "Context.Query.Reader.copy_attr_values.sum": 0.00157636,\n    "Context.Query.Reader.copy_attr_values.avg": 0.000105091,\n    "Context.Query.Reader.compute_tiles_cell_num.sum": 0.032372,\n    "Context.Query.Reader.compute_tiles_cell_num.avg": 1.13746e-05,\n    "Context.Query.Reader.compute_tile_bitmaps.sum": 0.027897,\n    "Context.Query.Reader.compute_tile_bitmaps.avg": 3.832e-05,\n    "Context.Query.Reader.compute_subarray_coords.sum": 9.0839e-05,\n    "Context.Query.Reader.compute_subarray_coords.avg": 6.98762e-06,\n    "Context.Query.Reader.compute_sparse_result_tiles.sum": 3.6017e-05,\n    "Context.Query.Reader.compute_sparse_result_tiles.avg": 2.40113e-06,\n    "Context.Query.Reader.compute_sparse_result_cell_slabs_sparse.sum": 4.76e-06,\n    "Context.Query.Reader.compute_sparse_result_cell_slabs_sparse.avg": 3.17333e-07,\n    "Context.Query.Reader.compute_results_count_sparse.sum": 0.000442457,\n    "Context.Query.Reader.compute_results_count_sparse.avg": 4.0114e-07,\n    "Context.Query.Reader.compute_result_coords.sum": 0.00605428,\n    "Context.Query.Reader.compute_result_coords.avg": 0.000403619,\n    "Context.Query.Reader.compute_range_result_coords.sum": 0.000275081,\n    "Context.Query.Reader.compute_range_result_coords.avg": 2.11601e-05,\n    "Context.Query.Reader.compute_hilbert_values.sum": 0.00750825,\n    "Context.Query.Reader.compute_hilbert_values.avg": 0.000144389,\n    "Context.Query.Reader.compute_array_tile_indexes_for_ranges.sum": 0.00191747,\n    "Context.Query.Reader.compute_array_tile_indexes_for_ranges.avg": 2.06179e-05,\n    "Context.Query.Reader.compute_and_copy_range_indexes.sum": 0.00010934,\n    "Context.Query.Reader.compute_and_copy_range_indexes.avg": 1.1757e-06,\n    "Context.Query.Reader.apply_query_condition.sum": 0.00627404,\n    "Context.Query.Reader.apply_query_condition.avg": 1.75547e-06,\n    "Context.Query.Reader.SubarrayPartitioner.read_next_partition.sum": 0.199962,\n    "Context.Query.Reader.SubarrayPartitioner.read_next_partition.avg": 6.98923e-05,\n    "Context.Consolidator.resize_buffers.sum": 0.505872,\n    "Context.Consolidator.resize_buffers.avg": 0.056208,\n    "Context.Consolidator.consolidate_internal.sum": 0.564998,\n    "Context.Consolidator.consolidate_internal.avg": 0.0627776,\n    "Context.Consolidator.consolidate_group_meta.sum": 0.00601419,\n    "Context.Consolidator.consolidate_group_meta.avg": 0.00200473,\n    "Context.Consolidator.consolidate_frags.sum": 0.597723,\n    "Context.Consolidator.consolidate_frags.avg": 0.0543385,\n    "Context.Consolidator.consolidate_frag_meta.sum": 0.00459178,\n    "Context.Consolidator.consolidate_frag_meta.avg": 0.00153059,\n    "Context.Consolidator.consolidate_create_queries.sum": 0.0019117,\n    "Context.Consolidator.consolidate_create_queries.avg": 0.000212411,\n    "Context.Consolidator.consolidate_copy_array.sum": 0.0136376,\n    "Context.Consolidator.consolidate_copy_array.avg": 0.00151529,\n    "Context.Consolidator.consolidate_compute_next.sum": 3.4616e-05,\n    "Context.Consolidator.consolidate_compute_next.avg": 6.9232e-06\n  },\n  "counters": {\n    "Context.write_tile_var_sizes_size": 341492,\n    "Context.write_tile_var_offsets_size": 340876,\n    "Context.write_tile_validity_offsets_size": 340828,\n    "Context.write_tile_offsets_size": 347144,\n    "Context.write_sums_size": 344599,\n    "Context.write_rtree_size": 105183,\n    "Context.write_processed_conditions_size": 100980,\n    "Context.write_null_counts_size": 464767,\n    "Context.write_mins_size": 345342,\n    "Context.write_meta_size": 104935410,\n    "Context.write_maxs_size": 345490,\n    "Context.write_group_size": 878,\n    "Context.write_frag_meta_footer_size": 444184,\n    "Context.write_filtered_byte_num": 106050622,\n    "Context.write_array_schema_size": 165745,\n    "Context.subSubarray.precompute_tile_overlap.tile_overlap_cache_hit": 2358,\n    "Context.subSubarray.precompute_tile_overlap.tile_overlap_byte_size": 192560,\n    "Context.subSubarray.precompute_tile_overlap.relevant_fragment_num": 2869,\n    "Context.subSubarray.precompute_tile_overlap.ranges_requested": 2769,\n    "Context.subSubarray.precompute_tile_overlap.ranges_computed": 2769,\n    "Context.subSubarray.precompute_tile_overlap.fragment_num": 2890,\n    "Context.subSubarray.add_range_dim_8": 2,\n    "Context.subSubarray.add_range_dim_7": 4,\n    "Context.subSubarray.add_range_dim_6": 6,\n    "Context.subSubarray.add_range_dim_5": 8,\n    "Context.subSubarray.add_range_dim_4": 10,\n    "Context.subSubarray.add_range_dim_3": 12,\n    "Context.subSubarray.add_range_dim_2": 1458,\n    "Context.subSubarray.add_range_dim_1": 1956,\n    "Context.subSubarray.add_range_dim_0": 4105,\n    "Context.read_unfiltered_byte_num": 315464527,\n    "Context.read_tile_var_sizes_size": 4192,\n    "Context.read_tile_var_offsets_size": 4056,\n    "Context.read_tile_validity_offsets_size": 528,\n    "Context.read_tile_sum_size": 944,\n    "Context.read_tile_offsets_size": 35992,\n    "Context.read_tile_null_count_size": 504,\n    "Context.read_tile_min_size": 3592,\n    "Context.read_tile_max_size": 3509,\n    "Context.read_rtree_size": 10726,\n    "Context.read_group_size": 44440,\n    "Context.read_frag_meta_size": 829870,\n    "Context.read_enumeration_size": 836,\n    "Context.read_array_schema_size": 537747,\n    "Context.read_array_meta_size": 16185,\n    "Context.group_read_group_meta_size": 314790247,\n    "Context.consolidated_frag_meta_size": [1102](https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/10822955068/job/30028085307#step:7:1103)9,\n    "Context.VFS.write_ops_num": 69525,\n    "Context.VFS.write_byte_num": 172241896,\n    "Context.VFS.read_ops_num": 29904,\n    "Context.VFS.read_byte_num": 4350385199,\n    "Context.VFS.ls_num": 3323,\n    "Context.VFS.is_object_num": 4564,\n    "Context.VFS.file_size_num": 1998,\n    "Context.Query.Writer.write_filtered_byte_num": 76958068,\n    "Context.Query.Writer.tile_num": 340,\n    "Context.Query.Writer.dim_var_num": 33,\n    "Context.Query.Writer.dim_num": 366,\n    "Context.Query.Writer.dim_fixed_num": 333,\n    "Context.Query.Writer.cell_num": 30815,\n    "Context.Query.Writer.attr_var_num": 153,\n    "Context.Query.Writer.attr_num": 1053,\n    "Context.Query.Writer.attr_nullable_num": 29,\n    "Context.Query.Writer.attr_fixed_num": 900,\n    "Context.Query.Subarray.precompute_tile_overlap.tile_overlap_byte_size": 7104,\n    "Context.Query.Subarray.precompute_tile_overlap.relevant_fragment_num": 111,\n    "Context.Query.Subarray.precompute_tile_overlap.ranges_requested": 96,\n    "Context.Query.Subarray.precompute_tile_overlap.ranges_computed": 96,\n    "Context.Query.Subarray.precompute_tile_overlap.fragment_num": 111,\n    "Context.Query.Subarray.add_range_dim_0": 20,\n    "Context.Query.Reader.tiles_unfiltered": 108151,\n    "Context.Query.Reader.tiles_allocated": 108028,\n    "Context.Query.Reader.result_num": 70164,\n    "Context.Query.Reader.read_unfiltered_byte_num": 9940788,\n    "Context.Query.Reader.overlap_tile_num": 13,\n    "Context.Query.Reader.num_tiles_read": 107525,\n    "Context.Query.Reader.num_tiles": 106099,\n    "Context.Query.Reader.loop_num": 3695,\n    "Context.Query.Reader.internal_loop_num": 3583,\n    "Context.Query.Reader.ignored_tiles": 61,\n    "Context.Query.Reader.dim_var_num": 12,\n    "Context.Query.Reader.dim_num": 596,\n    "Context.Query.Reader.dim_fixed_num": 584,\n    "Context.Query.Reader.cell_num": 299513,\n    "Context.Query.Reader.attr_var_num": 287,\n    "Context.Query.Reader.attr_num": 2838,\n    "Context.Query.Reader.attr_nullable_num": 109,\n    "Context.Query.Reader.attr_fixed_num": 2551,\n    "Context.Query.Reader.SubarrayPartitioner.compute_current_start_end.ranges": 2863,\n    "Context.Query.Reader.SubarrayPartitioner.compute_current_start_end.found": 2861,\n    "Context.Query.Reader.SubarrayPartitioner.compute_current_start_end.adjusted_ranges": 2863,\n    "Context.Consolidator.consolidate_step_num": 5\n  }\n}'
FAILED tiledb/tests/test_libtiledb.py::GetStatsTest::test_query - assert 'Context.StorageManager.Query' in '{\n  "timers": {\n    "Context.Query.Reader.unfilter_attr_tiles.sum": 5.359e-05,\n    "Context.Query.Reader.unfilter_attr_tiles.avg": 5.359e-05,\n    "Context.Query.Reader.read_tiles.sum": 3.9394e-05,\n    "Context.Query.Reader.read_tiles.avg": 3.9394e-05,\n    "Context.Query.Reader.read_attribute_tiles.sum": 4.0857e-05,\n    "Context.Query.Reader.read_attribute_tiles.avg": 4.0857e-05,\n    "Context.Query.Reader.load_tile_var_sizes.sum": 5.971e-06,\n    "Context.Query.Reader.load_tile_var_sizes.avg": 5.971e-06,\n    "Context.Query.Reader.load_tile_offsets.sum": 6.6725e-05,\n    "Context.Query.Reader.load_tile_offsets.avg": 6.6725e-05,\n    "Context.Query.Reader.init_state.sum": 1.071e-05,\n    "Context.Query.Reader.init_state.avg": 1.071e-05,\n    "Context.Query.Reader.dowork.sum": 0.000320793,\n    "Context.Query.Reader.dowork.avg": 0.000320793,\n    "Context.Query.Reader.copy_fixed_tiles.sum": 3.5547e-05,\n    "Context.Query.Reader.copy_fixed_tiles.avg": 3.5547e-05,\n    "Context.Query.Reader.copy_attribute.sum": 3.688e-05,\n    "Context.Query.Reader.copy_attribute.avg": 3.688e-05,\n    "Context.Query.Reader.compute_tiles_cell_num.sum": 8.336e-06,\n    "Context.Query.Reader.compute_tiles_cell_num.avg": 8.336e-06,\n    "Context.Query.Reader.apply_query_condition.sum": 2.7e-07,\n    "Context.Query.Reader.apply_query_condition.avg": 2.7e-07,\n    "Context.Query.Reader.SubarrayPartitioner.read_next_partition.sum": 4.1228e-05,\n    "Context.Query.Reader.SubarrayPartitioner.read_next_partition.avg": 4.1228e-05\n  },\n  "counters": {\n    "Context.Query.Reader.tiles_unfiltered": 1,\n    "Context.Query.Reader.tiles_allocated": 1,\n    "Context.Query.Reader.num_tiles_read": 1,\n    "Context.Query.Reader.num_tiles": 1,\n    "Context.Query.Reader.loop_num": 1,\n    "Context.Query.Reader.internal_loop_num": 1,\n    "Context.Query.Reader.attr_num": 1,\n    "Context.Query.Reader.attr_fixed_num": 1,\n    "Context.Query.Reader.SubarrayPartitioner.compute_current_start_end.ranges": 1,\n    "Context.Query.Reader.SubarrayPartitioner.compute_current_start_end.found": 1,\n    "Context.Query.Reader.SubarrayPartitioner.compute_current_start_end.adjusted_ranges": 1\n  }\n}'
== 3 failed, 968 passed, 5 skipped, 8 xfailed, 1 xpassed in 100.59s (0:01:40) ==

xref: https://github.com/jdblischak/centralized-tiledb-nightlies/issues/23#issuecomment-2344400003

[github-actions]

The centralized nightlies job failed on Thursday (2024-09-12) in run 10841905705

[jdblischak]

The same 3 TileDB-Py tests continue to fail. @kounelisagis could you please advise?

[kounelisagis]

So excinfo.value found here is:

• ArraySchema: Unable to check if unknown enumeration is loaded. No enumeration named 'enmr3'. when build is against 2.26.0
• Array: Unable to get enumeration; Enumeration 'enmr3' does not exist. when build is against libtiledb dev

I also checked locally using TileDB-Py 0.32.0. It's -internally- under discussion. Maybe it's an issue coming from core.

[github-actions]

The centralized nightlies job failed on Friday (2024-09-13) in run 10858700434

[github-actions]

The centralized nightlies job failed on Saturday (2024-09-14) in run 10867377569

[github-actions]

The centralized nightlies job failed on Sunday (2024-09-15) in run 10876527899

[github-actions]

The centralized nightlies job failed on Monday (2024-09-16) in run 10895298455

[github-actions]

The centralized nightlies job failed on Tuesday (2024-09-17) in run 10895298455

[jdblischak]

The same 3 TileDB-Py tests continue to fail. Interestingly, the Monday night tiledb-py-feedstock build passed. On Mondays it enables deprecated APIs (ie it will fail if any deprecated APIs are used). It is dev TileDB-Py built against dev TileDB, and it runs most of the test suite:

  commands:
    - python -c "import tiledb; tiledb.libtiledb.version()"
    - python -c "import tiledb; assert tiledb.VFS().supports('s3')"
    - pytest -v --ignore-glob='*_examples.py' --ignore-glob='*_dask.py'

The failing tests are in test_enumeration.py and test_libtiledb.py. I didn't expect these to be excluded via the --ignore-glob flags. However, I confirmed that none of the 3 failing tests are run in the feedstock build, which explains why it passed.

[jdblischak]

The TileDB-SOMA-R tests failed immediately with a segfault:

*** caught segfault ***
address 0x556dd4[9](https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/10895298455/job/30233783685#step:12:10)8db08, cause 'invalid permissions'
An irrecoverable exception occurred. R is aborting now ...
/home/runner/work/_temp/4b1240bf-8aa4-4295-b72d-a7d3abfafabf.sh: line 1:  6848 Segmentation fault      (core dumped) Rscript -e 'testthat::test_package("tiledbsoma")'

I restarted the build this morning, but the test immediately failed again, though with a different error:

terminate called after throwing an instance of 'std::bad_function_call'
  what():  bad_function_call
/home/runner/work/_temp/bc26d54a-33c5-40[8](https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/10895298455/job/30263799598#step:12:9)3-ac77-8251f037e8db.sh: line 1:  6[9](https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/10895298455/job/30263799598#step:12:10)19 Aborted                 (core dumped) Rscript -e 'testthat::test_package("tiledbsoma")'

There were various PRs merged to SOMA yesterday. Thus it would probably be necessary to rebuild libtiledbsoma too. Let's see what happens tonight.

[github-actions]

The centralized nightlies job failed on Tuesday (2024-09-17) in run 10914384366

[jdblischak]

There were various PRs merged to SOMA yesterday. Thus it would probably be necessary to rebuild libtiledbsoma too. Let's see what happens tonight.

This was the correct decision. The tiledbsoma-r tests passed last night:

Run Rscript -e 'testthat::test_package("tiledbsoma")'
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 3710 ]

Now we're back to only the known 3 failing TileDB-Py tests.

[github-actions]

The centralized nightlies job failed on Wednesday (2024-09-18) in run 10933290763

[github-actions]

The centralized nightlies job failed on Thursday (2024-09-19) in run 10952052698

[github-actions]

The centralized nightlies job failed on Friday (2024-09-20) in run 10968862997

[github-actions]

The centralized nightlies job failed on Saturday (2024-09-21) in run 10977537064

[github-actions]

The centralized nightlies job failed on Sunday (2024-09-22) in run 10986620240

[github-actions]

The centralized nightlies job failed on Monday (2024-09-23) in run 11005814946

[github-actions]

The centralized nightlies job failed on Tuesday (2024-09-24) in run 11005814946

[jdblischak]

The TileDB-R build failed last night but passed when restarted this morning. Back to only the failing TileDB-Py tests

[github-actions]

The centralized nightlies job failed on Tuesday (2024-09-24) in run 11025154430

[github-actions]

The centralized nightlies job failed on Wednesday (2024-09-25) in run 11044409073

[github-actions]

The centralized nightlies job failed on Thursday (2024-09-26) in run 11044409073

[github-actions]

The centralized nightlies job failed on Thursday (2024-09-26) in run 11044409073

[jdblischak]

In addition to the known TileDB-Py test failures, the tiledbsoma-r tests segfaulted last night and repeatedly segfaulted this morning after 2 restarts:

*** caught segfault ***
address 0x400000000, cause 'memory not mapped'
An irrecoverable exception occurred. R is aborting now ...
/home/runner/work/_temp/40e55f[9](https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/11044409073/job/30708188362#step:12:10)6-295d-4f21-acc4-07ffa737fbdd.sh: line 1: 12520 Segmentation fault      (core dumped) Rscript -e 'testthat::test_package("tiledbsoma")'

[github-actions]

The centralized nightlies job failed on Thursday (2024-09-26) in run 11063416207

[jdblischak]

Fortunately the tiledbsoma-r tests passed last night

[johnkerl]

single-cell-data/TileDB-SOMA#3178

[github-actions]

The centralized nightlies job failed on Wednesday (2024-10-16) in run 11377170824

[johnkerl]

TileDB-Inc/TileDB-R#770

[kounelisagis]

The only remaining PR to be merged for Python is TileDB-Inc/TileDB-Py#2088, which is expected to fix the last failing test we are encountering here.

[github-actions]

The centralized nightlies job failed on Thursday (2024-10-17) in run 11396305245

[jdblischak]

TileDB-R passed last night thanks to TileDB-Inc/TileDB-R#770

[kounelisagis]

@jdblischak, could you please rerun tiledb-py? There haven't been any recent changes in group.cc. However, I will look into the error.

[teo-tsirpanis]

Python failures were caused by TileDB-Inc/TileDB#5336 because the compiler cannot process Group::add_member getting an extra optional argument.

[jdblischak]

Python failures were caused by TileDB-Inc/TileDB#5336 because the compiler cannot process Group::add_member getting an extra optional argument.

@teo-tsirpanis Thanks for the diagnosis! That timing makes sense. That PR was merged early yesterday morning in TileDB-Inc/TileDB@8dbab75, and I first observed this build error later in the afternoon in 48f88d5.

Is there a fix we can apply (prior to the 2.27 release)?

[25/34] Building CXX object tiledb/cc/CMakeFiles/cc.dir/group.cc.o
FAILED: tiledb/cc/CMakeFiles/cc.dir/group.cc.o
/usr/bin/g++ -Dcc_EXPORTS -isystem /opt/hostedtoolcache/Python/3.11.10/x64/include/python3.11 -isystem /tmp/pip-build-env-ueaihjgn/overlay/lib/python3.11/site-packages/pybind11/include -isystem /home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/install-libtiledb/include -O3 -DNDEBUG -std=gnu++20 -fPIC -fvisibility=hidden -flto -fno-fat-lto-objects -MD -MT tiledb/cc/CMakeFiles/cc.dir/group.cc.o -MF tiledb/cc/CMakeFiles/cc.dir/group.cc.o.d -o tiledb/cc/CMakeFiles/cc.dir/group.cc.o -c /home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/tiledb/cc/group.cc
In file included from /tmp/pip-build-env-ueaihjgn/overlay/lib/python3.11/site-packages/pybind11/include/pybind11/numpy.h:12,
                 from /home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/tiledb/cc/group.cc:4:
/tmp/pip-build-env-ueaihjgn/overlay/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h: In instantiation of ‘void pybind11::cpp_function::initialize(Func&&, Return (*)(Args ...), const Extra& ...) [with Func = pybind11::cpp_function::cpp_function<void, tiledb::Group, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const bool&, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::optional<tiledb_object_t>, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v, pybind11::arg_v>(void (tiledb::Group::*)(const std::__cxx11::basic_string<char>&, const bool&, std::optional<std::__cxx11::basic_string<char> >, std::optional<tiledb_object_t>), const pybind11::name&, const pybind11::is_method&, const pybind11::sibling&, const pybind11::arg&, const pybind11::arg_v&, const pybind11::arg_v&)::<lambda(tiledb::Group*, const std::__cxx11::basic_string<char>&, const bool&, std::optional<std::__cxx11::basic_string<char> >, std::optional<tiledb_object_t>)>; Return = void; Args = {tiledb::Group*, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const bool&, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::optional<tiledb_object_t>}; Extra = {pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v, pybind11::arg_v}]’:
/tmp/pip-build-env-ueaihjgn/overlay/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:135:19:   required from ‘pybind11::cpp_function::cpp_function(Return (Class::*)(Arg ...), const Extra& ...) [with Return = void; Class = tiledb::Group; Arg = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const bool&, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::optional<tiledb_object_t>}; Extra = {pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v, pybind11::arg_v}]’
/tmp/pip-build-env-ueaihjgn/overlay/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:1621:22:   required from ‘pybind11::class_<type_, options>& pybind11::class_<type_, options>::def(const char*, Func&&, const Extra& ...) [with Func = void (tiledb::Group::*)(const std::__cxx11::basic_string<char>&, const bool&, std::optional<std::__cxx11::basic_string<char> >, std::optional<tiledb_object_t>); Extra = {pybind11::arg, pybind11::arg_v, pybind11::arg_v}; type_ = tiledb::Group; options = {}]’
/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/tiledb/cc/group.cc:122:11:   required from here
/tmp/pip-build-env-ueaihjgn/overlay/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:244:40: error: static assertion failed: The number of argument annotations does not match the number of function arguments
  244 |             expected_num_args<Extra...>(
      |             ~~~~~~~~~~~~~~~~~~~~~~~~~~~^
  245 |                 sizeof...(Args), cast_in::args_pos >= 0, cast_in::has_kwargs),
      |                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/pip-build-env-ueaihjgn/overlay/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:244:40: note: ‘pybind11::detail::expected_num_args<pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v, pybind11::arg_v>(5, (((int)pybind11::detail::argument_loader<tiledb::Group*, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const bool&, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::optional<tiledb_object_t> >::args_pos) >= 0), ((int)((bool)pybind11::detail::argument_loader<tiledb::Group*, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const bool&, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::optional<tiledb_object_t> >::has_kwargs)))’ evaluates to false

[teo-tsirpanis]

We can replace the &Group::add_member with a lambda.

[kounelisagis]

We can replace the &Group::add_member with a lambda.

Story created.

All the previous test failures are now fixed.

[github-actions]

The centralized nightlies job failed on Friday (2024-10-18) in run 11413743783

[github-actions]

The centralized nightlies job failed on Saturday (2024-10-19) in run 11422666729

[jdblischak]

Since TileDB-Inc/TileDB-Py#2093 was merged, there are now only 2 remaining TileDB-Py test failures:

=========================== short test summary info ============================
FAILED tiledb/tests/test_enumeration.py::EnumerationTest::test_array_schema_enumeration - assert "Array: Unable to get enumeration; Enumeration 'enmr3' does not exist." == "ArraySchema: Unable to check if unknown enumeration is loaded. No enumeration named 'enmr3'."
  
  - ArraySchema: Unable to check if unknown enumeration is loaded. No enumeration named 'enmr3'.
  + Array: Unable to get enumeration; Enumeration 'enmr3' does not exist.
FAILED tiledb/tests/test_libtiledb.py::GetStatsTest::test_ctx - assert 'Context.StorageManager.write_store' in '{\n  "timers": {\n    "Context.write_store_frag_meta.sum": 3.15797,\n    "Context.write_store_frag_meta.avg": 0.00315167,\n    "Context.write_meta.sum": 2.25259,\n    "Context.write_meta.avg": 0.0014561,\n    "Context.subSubarray.sort_and_merge_ranges.sum": 0.00801585,\n    "Context.subSubarray.sort_and_merge_ranges.avg": 1.09656e-05,\n    "Context.subSubarray.read_load_relevant_rtrees.sum": 0.0602286,\n    "Context.subSubarray.read_load_relevant_rtrees.avg": 1.82125e-05,\n    "Context.subSubarray.read_compute_tile_overlap.sum": 0.261911,\n    "Context.subSubarray.read_compute_tile_overlap.avg": 5.12445e-05,\n    "Context.subSubarray.read_compute_simple_tile_overlap.sum": 0.0469569,\n    "Context.subSubarray.read_compute_simple_tile_overlap.avg": 8.58444e-05,\n    "Context.subSubarray.read_compute_relevant_tile_overlap.sum": 0.127108,\n    "Context.subSubarray.read_compute_relevant_tile_overlap.avg": 4.60538e-05,\n    "Context.subSubarray.read_compute_est_result_size.sum": 0.391887,\n    "Context.subSubarray.read_compute_est_result_size.avg": 0.00015508,\n    "Context.subSubarray.compute_relevant_frags.sum": 0.0592539,\n    "Context.subSubarray.compute_relevant_frags.avg": 1.79177e-05,\n    "Context.sm_read_load_consolidated_frag_meta.sum": 0.000228155,\n    "Context.sm_read_load_consolidated_frag_meta.avg": 7.60517e-05,\n    "Context.sm_load_fragment_metadata.sum": 0.0906758,\n    "Context.sm_load_fragment_metadata.avg": 6.69194e-05,\n    "Context.sm_load_filtered_fragment_uris.sum": 0.0380716,\n    "Context.sm_load_filtered_fragment_uris.avg": 2.8264e-05,\n    "Context.sm_load_enumeration.sum": 0.000559655,\n    "Context.sm_load_enumeration.avg": 6.99569e-05,\n    "Context.sm_load_array_schemas_and_fragment_metadata.sum": 0.250189,\n    "Context.sm_load_array_schemas_and_fragment_metadata.avg": 0.000185738,\n    "Context.sm_load_array_schema_latest.sum": 0.00708142,\n    "Context.sm_load_array_schema_latest.avg": 7.69719e-05,\n    "Context.sm_load_array_schema_from_uri.sum": 0.195707,\n    "Context.sm_load_array_schema_from_uri.avg": 7.57085e-05,\n    "Context.sm_load_array_metadata.sum": 0.00603352,\n    "Context.sm_load_array_metadata.avg": 4.30966e-05,\n    "Context.sm_load_all_array_schemas.sum": 0.211492,\n    "Context.sm_load_all_array_schemas.avg": 8.5348e-05,\n    "Context.sm_array_load_fragments.sum": 0.00103976,\n    "Context.sm_array_load_fragments.avg": 0.00012997,\n    "Context.load_group_from_all_uris.sum": 0.0223198,\n    "Context.load_group_from_all_uris.avg": 0.000103813,\n    "Context.load_group_details.sum": 0.0228154,\n    "Context.load_group_details.avg": 2.0855e-05,\n    "Context.group_open_for_writes.sum": 0.00116902,\n    "Context.group_open_for_writes.avg": 2.62112e-06,\n    "Context.group_open_for_reads.sum": 0.0236023,\n    "Context.group_open_for_reads.avg": 3.64233e-05,\n    "Context.group_load_metadata_from_storage.sum": 1.52617,\n    "Context.group_load_metadata_from_storage.avg": 0.00352463,\n    "Context.array_reopen_directory.sum": 0.000560445,\n    "Context.array_reopen_directory.avg": 0.000186815,\n    "Context.array_reopen.sum": 0.00110986,\n    "Context.array_reopen.avg": 0.000277464,\n    "Context.array_open_write_load_schemas.sum": 0.0964461,\n    "Context.array_open_write_load_schemas.avg": 8.60357e-05,\n    "Context.array_open_write_load_directory.sum": 0.132407,\n    "Context.array_open_write_load_directory.avg": 0.0001178,\n    "Context.array_open_without_fragments_load_directory.sum": 0.00275761,\n    "Context.array_open_without_fragments_load_directory.avg": 0.000275761,\n    "Context.array_open_without_fragments.sum": 0.00432872,\n    "Context.array_open_without_fragments.avg": 0.000432872,\n    "Context.array_open_read_without_fragments_load_schemas.sum": 0.00148222,\n    "Context.array_open_read_without_fragments_load_schemas.avg": 0.000148222,\n    "Context.array_open_read_load_schemas_and_fragment_meta.sum": 0.219527,\n    "Context.array_open_read_load_schemas_and_fragment_meta.avg": 0.000176044,\n    "Context.array_open_read_load_directory.sum": 0.288231,\n    "Context.array_open_read_load_directory.avg": 0.000179248,\n    "Context.array_open_WRITE.sum": 0.241663,\n    "Context.array_open_WRITE.avg": 0.000216738,\n    "Context.array_open_READ.sum": 0.524461,\n    "Context.array_open_READ.avg": 0.000326157,\n    "Context.array_open_MODIFY_EXCLUSIVE.sum": 0.00145992,\n    "Context.array_open_MODIFY_EXCLUSIVE.avg": 0.000162213,\n    "Context.VFS.ArrayDirectory.load_consolidated_commit_uris.sum": 0.00130292,\n    "Context.VFS.ArrayDirectory.load_consolidated_commit_uris.avg": 9.56623e-07,\n    "Context.VFS.ArrayDirectory.list_root_uris.sum": 0.108211,\n    "Context.VFS.ArrayDirectory.list_root_uris.avg": 6.26945e-05,\n    "Context.VFS.ArrayDirectory.list_fragment_meta_uris.sum": 0.0459366,\n    "Context.VFS.ArrayDirectory.list_fragment_meta_uris.avg": 2.66145e-05,\n    "Context.VFS.ArrayDirectory.list_commit_uris.sum": 0.07187,\n    "Context.VFS.ArrayDirectory.list_commit_uris.avg": 4.16396e-05,\n    "Context.VFS.ArrayDirectory.list_array_schema_uris.sum": 0.193078,\n    "Context.VFS.ArrayDirectory.list_array_schema_uris.avg": 6.56059e-05,\n    "Context.VFS.ArrayDirectory.list_array_meta_uris.sum": 0.0442847,\n    "Context.VFS.ArrayDirectory.list_array_meta_uris.avg": 2.56574e-05,\n    "Context.Subarray.read_compute_tile_coords.sum": 0.044306,\n    "Context.Subarray.read_compute_tile_coords.avg": 1.61406e-05,\n    "Context.Query.Writer.write_tiles.sum": 1.89955,\n    "Context.Query.Writer.write_tiles.avg": 0.00102181,\n    "Context.Query.Writer.write_num_tiles.sum": 0.551096,\n    "Context.Query.Writer.write_num_tiles.avg": 0.00167506,\n    "Context.Query.Writer.split_coords_buff.sum": 0.000169372,\n    "Context.Query.Writer.split_coords_buff.avg": 1.69034e-07,\n    "Context.Query.Writer.sort_coords.sum": 0.0162209,\n    "Context.Query.Writer.sort_coords.avg": 5.06902e-05,\n    "Context.Query.Writer.prepare_tiles.sum": 0.0133484,\n    "Context.Query.Writer.prepare_tiles.avg": 4.05725e-05,\n    "Context.Query.Writer.prepare_filter_and_write_tiles.sum": 1.0277,\n    "Context.Query.Writer.prepare_filter_and_write_tiles.avg": 0.00149159,\n    "Context.Query.Writer.prepare_and_filter_tiles.sum": 0.191214,\n    "Context.Query.Writer.prepare_and_filter_tiles.avg": 0.000164273,\n    "Context.Query.Writer.ordered_write.sum": 3.13131,\n    "Context.Query.Writer.ordered_write.avg": 0.00465276,\n    "Context.Query.Writer.finalize.sum": 0.0533008,\n    "Context.Query.Writer.finalize.avg": 5.44998e-05,\n    "Context.Query.Writer.filter_tiles.sum": 0.0398798,\n    "Context.Query.Writer.filter_tiles.avg": 0.000121215,\n    "Context.Query.Writer.filter_tile.sum": 0.16488,\n    "Context.Query.Writer.filter_tile.avg": 4.5572e-05,\n    "Context.Query.Writer.dowork.sum": 5.09996,\n    "Context.Query.Writer.dowork.avg": 0.00508978,\n    "Context.Query.Writer.compute_coord_meta.sum": 0.00237481,\n    "Context.Query.Writer.compute_coord_meta.avg": 7.21827e-06,\n    "Context.Query.Writer.check_global_order.sum": 3.09e-07,\n    "Context.Query.Writer.check_global_order.avg": 5.15e-08,\n    "Context.Query.Writer.check_coord_oob.sum": 0.0130942,\n    "Context.Query.Writer.check_coord_oob.avg": 1.30681e-05,\n    "Context.Query.Writer.check_coord_dups.sum": 0.00855692,\n    "Context.Query.Writer.check_coord_dups.avg": 2.62482e-05,\n    "Context.Query.Writer.check_attr_order.sum": 0.000105554,\n    "Context.Query.Writer.check_attr_order.avg": 1.56841e-07,\n    "Context.Query.Writer.DenseTiler.get_tile.sum": 0.115461,\n    "Context.Query.Writer.DenseTiler.get_tile.avg": 4.30183e-05,\n    "Context.Query.Subarray.sort_and_merge_ranges.sum": 0.000109003,\n    "Context.Query.Subarray.sort_and_merge_ranges.avg": 1.81672e-05,\n    "Context.Query.Subarray.read_load_relevant_rtrees.sum": 0.0070948,\n    "Context.Query.Subarray.read_load_relevant_rtrees.avg": 7.46821e-05,\n    "Context.Query.Subarray.read_compute_tile_overlap.sum": 0.0131661,\n    "Context.Query.Subarray.read_compute_tile_overlap.avg": 0.000138591,\n    "Context.Query.Subarray.read_compute_tile_coords.sum": 0.000164749,\n    "Context.Query.Subarray.read_compute_tile_coords.avg": 1.7342e-06,\n    "Context.Query.Subarray.read_compute_relevant_tile_overlap.sum": 0.00427871,\n    "Context.Query.Subarray.read_compute_relevant_tile_overlap.avg": 4.50391e-05,\n    "Context.Query.Subarray.compute_relevant_frags.sum": 0.000798009,\n    "Context.Query.Subarray.compute_relevant_frags.avg": 8.40009e-06,\n    "Context.Query.Reader.validate_attribute_order.sum": 2.7198e-05,\n    "Context.Query.Reader.validate_attribute_order.avg": 2.92452e-07,\n    "Context.Query.Reader.unfilter_coord_tiles.sum": 0.132065,\n    "Context.Query.Reader.unfilter_coord_tiles.avg": 4.43768e-05,\n    "Context.Query.Reader.unfilter_attr_tiles.sum": 0.674519,\n    "Context.Query.Reader.unfilter_attr_tiles.avg": 0.000194891,\n    "Context.Query.Reader.tile_offset_sizes.sum": 0.0289716,\n    "Context.Query.Reader.tile_offset_sizes.avg": 8.11301e-06,\n    "Context.Query.Reader.switch_sizes_to_offsets.sum": 7.894e-06,\n    "Context.Query.Reader.switch_sizes_to_offsets.avg": 1.51808e-07,\n    "Context.Query.Reader.sort_result_coords.sum": 0.000139273,\n    "Context.Query.Reader.sort_result_coords.avg": 1.54748e-05,\n    "Context.Query.Reader.resize_fixed_results_to_copy.sum": 3.937e-06,\n    "Context.Query.Reader.resize_fixed_results_to_copy.avg": 1.31233e-06,\n    "Context.Query.Reader.read_tiles.sum": 2.25445,\n    "Context.Query.Reader.read_tiles.avg": 0.000386367,\n    "Context.Query.Reader.read_coordinate_tiles.sum": 0.0300752,\n    "Context.Query.Reader.read_coordinate_tiles.avg": 2.04593e-05,\n    "Context.Query.Reader.read_attribute_tiles.sum": 2.23476,\n    "Context.Query.Reader.read_attribute_tiles.avg": 0.000511973,\n    "Context.Query.Reader.read_and_unfilter_coords.sum": 0.172172,\n    "Context.Query.Reader.read_and_unfilter_coords.avg": 0.000237807,\n    "Context.Query.Reader.read_and_unfilter_attributes.sum": 0.0398229,\n    "Context.Query.Reader.read_and_unfilter_attributes.avg": 5.82206e-05,\n    "Context.Query.Reader.process_tiles.sum": 0.000951237,\n    "Context.Query.Reader.process_tiles.avg": 0.000317079,\n    "Context.Query.Reader.process_slabs.sum": 0.0929778,\n    "Context.Query.Reader.process_slabs.avg": 0.000136531,\n    "Context.Query.Reader.process_aggregates.sum": 0.0121577,\n    "Context.Query.Reader.process_aggregates.avg": 3.50367e-05,\n    "Context.Query.Reader.merge_result_cell_slabs.sum": 0.00679423,\n    "Context.Query.Reader.merge_result_cell_slabs.avg": 9.30717e-06,\n    "Context.Query.Reader.max_num_cells_to_copy.sum": 0.000185163,\n    "Context.Query.Reader.max_num_cells_to_copy.avg": 2.53648e-07,\n    "Context.Query.Reader.load_tile_var_sizes.sum": 0.0325715,\n    "Context.Query.Reader.load_tile_var_sizes.avg": 8.82696e-06,\n    "Context.Query.Reader.load_tile_offsets.sum": 0.109032,\n    "Context.Query.Reader.load_tile_offsets.avg": 2.09878e-05,\n    "Context.Query.Reader.load_tile_metadata.sum": 0.0222322,\n    "Context.Query.Reader.load_tile_metadata.avg": 1.97971e-05,\n    "Context.Query.Reader.load_label_min_max_values.sum": 0.00972506,\n    "Context.Query.Reader.load_label_min_max_values.avg": 0.00010457,\n    "Context.Query.Reader.load_initial_data.sum": 0.0572752,\n    "Context.Query.Reader.load_initial_data.avg": 7.81381e-05,\n    "Context.Query.Reader.init_state.sum": 0.0368733,\n    "Context.Query.Reader.init_state.avg": 1.29244e-05,\n    "Context.Query.Reader.fix_offset_tiles.sum": 0.000138514,\n    "Context.Query.Reader.fix_offset_tiles.avg": 5.72372e-07,\n    "Context.Query.Reader.fill_dense_coords.sum": 0.000918406,\n    "Context.Query.Reader.fill_dense_coords.avg": 6.33383e-06,\n    "Context.Query.Reader.dowork.sum": 4.75852,\n    "Context.Query.Reader.dowork.avg": 0.00129202,\n    "Context.Query.Reader.dedup_tiles_with_timestamps.sum": 0.00631389,\n    "Context.Query.Reader.dedup_tiles_with_timestamps.avg": 8.83062e-06,\n    "Context.Query.Reader.dedup_fragments_with_timestamps.sum": 0.00830216,\n    "Context.Query.Reader.dedup_fragments_with_timestamps.avg": 1.14671e-05,\n    "Context.Query.Reader.create_result_tiles.sum": 0.00996954,\n    "Context.Query.Reader.create_result_tiles.avg": 1.20697e-05,\n    "Context.Query.Reader.copy_var_tiles.sum": 0.00799831,\n    "Context.Query.Reader.copy_var_tiles.avg": 2.72051e-05,\n    "Context.Query.Reader.copy_timestamps_tiles.sum": 0.000169645,\n    "Context.Query.Reader.copy_timestamps_tiles.avg": 2.82742e-05,\n    "Context.Query.Reader.copy_offsets_tiles.sum": 0.00139348,\n    "Context.Query.Reader.copy_offsets_tiles.avg": 2.67978e-05,\n    "Context.Query.Reader.copy_offset_tiles.sum": 0.00659708,\n    "Context.Query.Reader.copy_offset_tiles.avg": 2.72607e-05,\n    "Context.Query.Reader.copy_fixed_tiles.sum": 0.429093,\n    "Context.Query.Reader.copy_fixed_tiles.avg": 0.000191988,\n    "Context.Query.Reader.copy_fixed_data_tiles.sum": 0.0145736,\n    "Context.Query.Reader.copy_fixed_data_tiles.avg": 2.78654e-05,\n    "Context.Query.Reader.copy_fixed_coords.sum": 0.000223146,\n    "Context.Query.Reader.copy_fixed_coords.avg": 7.9695e-06,\n    "Context.Query.Reader.copy_fixed_attr_values.sum": 0.000105099,\n    "Context.Query.Reader.copy_fixed_attr_values.avg": 8.08454e-06,\n    "Context.Query.Reader.copy_coordinates.sum": 0.000270935,\n    "Context.Query.Reader.copy_coordinates.avg": 1.80623e-05,\n    "Context.Query.Reader.copy_attribute.sum": 0.448835,\n    "Context.Query.Reader.copy_attribute.avg": 0.000181201,\n    "Context.Query.Reader.copy_attr_values.sum": 0.00161334,\n    "Context.Query.Reader.copy_attr_values.avg": 0.000107556,\n    "Context.Query.Reader.compute_tiles_cell_num.sum": 0.0344948,\n    "Context.Query.Reader.compute_tiles_cell_num.avg": 1.21546e-05,\n    "Context.Query.Reader.compute_tile_bitmaps.sum": 0.0284552,\n    "Context.Query.Reader.compute_tile_bitmaps.avg": 3.93028e-05,\n    "Context.Query.Reader.compute_subarray_coords.sum": 0.000161192,\n    "Context.Query.Reader.compute_subarray_coords.avg": 1.23994e-05,\n    "Context.Query.Reader.compute_sparse_result_tiles.sum": 3.6016e-05,\n    "Context.Query.Reader.compute_sparse_result_tiles.avg": 2.40107e-06,\n    "Context.Query.Reader.compute_sparse_result_cell_slabs_sparse.sum": 3.386e-06,\n    "Context.Query.Reader.compute_sparse_result_cell_slabs_sparse.avg": 2.25733e-07,\n    "Context.Query.Reader.compute_results_count_sparse.sum": 0.000487153,\n    "Context.Query.Reader.compute_results_count_sparse.avg": 4.41662e-07,\n    "Context.Query.Reader.compute_result_coords.sum": 0.00581849,\n    "Context.Query.Reader.compute_result_coords.avg": 0.000387899,\n    "Context.Query.Reader.compute_range_result_coords.sum": 0.000270255,\n    "Context.Query.Reader.compute_range_result_coords.avg": 2.07888e-05,\n    "Context.Query.Reader.compute_hilbert_values.sum": 0.00721426,\n    "Context.Query.Reader.compute_hilbert_values.avg": 0.000138736,\n    "Context.Query.Reader.compute_array_tile_indexes_for_ranges.sum": 0.00206105,\n    "Context.Query.Reader.compute_array_tile_indexes_for_ranges.avg": 2.21618e-05,\n    "Context.Query.Reader.compute_and_copy_range_indexes.sum": 0.0001007,\n    "Context.Query.Reader.compute_and_copy_range_indexes.avg": 1.0828e-06,\n    "Context.Query.Reader.apply_query_condition.sum": 0.00665784,\n    "Context.Query.Reader.apply_query_condition.avg": 1.86913e-06,\n    "Context.Query.Reader.SubarrayPartitioner.read_next_partition.sum": 0.201595,\n    "Context.Query.Reader.SubarrayPartitioner.read_next_partition.avg": 7.06608e-05,\n    "Context.Consolidator.resize_buffers.sum": 0.302461,\n    "Context.Consolidator.resize_buffers.avg": 0.0378076,\n    "Context.Consolidator.consolidate_internal.sum": 0.377679,\n    "Context.Consolidator.consolidate_internal.avg": 0.0472099,\n    "Context.Consolidator.consolidate_group_meta.sum": 0.00871836,\n    "Context.Consolidator.consolidate_group_meta.avg": 0.00290612,\n    "Context.Consolidator.consolidate_frags.sum": 0.403451,\n    "Context.Consolidator.consolidate_frags.avg": 0.0403451,\n    "Context.Consolidator.consolidate_frag_meta.sum": 0.00870596,\n    "Context.Consolidator.consolidate_frag_meta.avg": 0.00290199,\n    "Context.Consolidator.consolidate_create_queries.sum": 0.00180195,\n    "Context.Consolidator.consolidate_create_queries.avg": 0.000225243,\n    "Context.Consolidator.consolidate_copy_array.sum": 0.0123192,\n    "Context.Consolidator.consolidate_copy_array.avg": 0.0015399,\n    "Context.Consolidator.consolidate_compute_next.sum": 2.688e-05,\n    "Context.Consolidator.consolidate_compute_next.avg": 6.72e-06\n  },\n  "counters": {\n    "Context.write_tile_var_sizes_size": 334348,\n    "Context.write_tile_var_offsets_size": 333736,\n    "Context.write_tile_validity_offsets_size": 333688,\n    "Context.write_tile_offsets_size": 339944,\n    "Context.write_sums_size": 337360,\n    "Context.write_rtree_size": 103362,\n    "Context.write_processed_conditions_size": 99198,\n    "Context.write_null_counts_size": 455117,\n    "Context.write_mins_size": 337899,\n    "Context.write_meta_size": 104936158,\n    "Context.write_maxs_size": 338051,\n    "Context.write_group_size": 878,\n    "Context.write_frag_meta_footer_size": 435308,\n    "Context.write_filtered_byte_num": 106032026,\n    "Context.write_array_schema_size": 162495,\n    "Context.subSubarray.precompute_tile_overlap.tile_overlap_cache_hit": 2351,\n    "Context.subSubarray.precompute_tile_overlap.tile_overlap_byte_size": 192048,\n    "Context.subSubarray.precompute_tile_overlap.relevant_fragment_num": 2862,\n    "Context.subSubarray.precompute_tile_overlap.ranges_requested": 2762,\n    "Context.subSubarray.precompute_tile_overlap.ranges_computed": 2762,\n    "Context.subSubarray.precompute_tile_overlap.fragment_num": 2883,\n    "Context.subSubarray.add_range_dim_8": 2,\n    "Context.subSubarray.add_range_dim_7": 4,\n    "Context.subSubarray.add_range_dim_6": 6,\n    "Context.subSubarray.add_range_dim_5": 8,\n    "Context.subSubarray.add_range_dim_4": 10,\n    "Context.subSubarray.add_range_dim_3": 12,\n    "Context.subSubarray.add_range_dim_2": 1458,\n    "Context.subSubarray.add_range_dim_1": 1944,\n    "Context.subSubarray.add_range_dim_0": 4081,\n    "Context.read_unfiltered_byte_num": 315459847,\n    "Context.read_tile_var_sizes_size": 4192,\n    "Context.read_tile_var_offsets_size": 4056,\n    "Context.read_tile_validity_offsets_size": 528,\n    "Context.read_tile_sum_size": 944,\n    "Context.read_tile_offsets_size": 35544,\n    "Context.read_tile_null_count_size": 504,\n    "Context.read_tile_min_size": 3592,\n    "Context.read_tile_max_size": 3509,\n    "Context.read_rtree_size": 10510,\n    "Context.read_group_size": 44440,\n    "Context.read_frag_meta_size": 819094,\n    "Context.read_enumeration_size": 836,\n    "Context.read_array_schema_size": 530751,\n    "Context.read_array_meta_size": 16077,\n    "Context.group_read_group_meta_size": 314793335,\n    "Context.consolidated_frag_meta_size": 11029,\n    "Context.VFS.write_ops_num": 68145,\n    "Context.VFS.write_byte_num": 172082239,\n    "Context.VFS.read_ops_num": 29654,\n    "Context.VFS.read_byte_num": 4350122763,\n    "Context.VFS.ls_num": 3321,\n    "Context.VFS.is_object_num": 4508,\n    "Context.VFS.file_size_num": 1973,\n    "Context.Query.Writer.write_filtered_byte_num": 76875713,\n    "Context.Query.Writer.tile_num": 335,\n    "Context.Query.Writer.dim_var_num": 33,\n    "Context.Query.Writer.dim_num": 358,\n    "Context.Query.Writer.dim_fixed_num": 325,\n    "Context.Query.Writer.cell_num": 30803,\n    "Context.Query.Writer.attr_var_num": 153,\n    "Context.Query.Writer.attr_num": 1026,\n    "Context.Query.Writer.attr_nullable_num": 29,\n    "Context.Query.Writer.attr_fixed_num": 873,\n    "Context.Query.Subarray.precompute_tile_overlap.tile_overlap_byte_size": 6784,\n    "Context.Query.Subarray.precompute_tile_overlap.relevant_fragment_num": 106,\n    "Context.Query.Subarray.precompute_tile_overlap.ranges_requested": 95,\n    "Context.Query.Subarray.precompute_tile_overlap.ranges_computed": 95,\n    "Context.Query.Subarray.precompute_tile_overlap.fragment_num": 106,\n    "Context.Query.Subarray.add_range_dim_0": 20,\n    "Context.Query.Reader.tiles_unfiltered": [1081](https://github.com/TileDB-Inc/centralized-tiledb-nightlies/actions/runs/11422666729/job/31781139710#step:7:1082)12,\n    "Context.Query.Reader.tiles_allocated": 107992,\n    "Context.Query.Reader.result_num": 70144,\n    "Context.Query.Reader.read_unfiltered_byte_num": 9940598,\n    "Context.Query.Reader.overlap_tile_num": 13,\n    "Context.Query.Reader.num_tiles_read": 107489,\n    "Context.Query.Reader.num_tiles": 106083,\n    "Context.Query.Reader.loop_num": 3683,\n    "Context.Query.Reader.internal_loop_num": 3571,\n    "Context.Query.Reader.ignored_tiles": 61,\n    "Context.Query.Reader.dim_var_num": 12,\n    "Context.Query.Reader.dim_num": 592,\n    "Context.Query.Reader.dim_fixed_num": 580,\n    "Context.Query.Reader.cell_num": 299481,\n    "Context.Query.Reader.attr_var_num": 287,\n    "Context.Query.Reader.attr_num": 2822,\n    "Context.Query.Reader.attr_nullable_num": 109,\n    "Context.Query.Reader.attr_fixed_num": 2535,\n    "Context.Query.Reader.SubarrayPartitioner.compute_current_start_end.ranges": 2855,\n    "Context.Query.Reader.SubarrayPartitioner.compute_current_start_end.found": 2853,\n    "Context.Query.Reader.SubarrayPartitioner.compute_current_start_end.adjusted_ranges": 2855,\n    "Context.Consolidator.consolidate_step_num": 4\n  }\n}'
== 2 failed, 971 passed, 6 skipped, 8 xfailed, 1 xpassed in 112.10s (0:01:52) ==

[github-actions]

The centralized nightlies job failed on Sunday (2024-10-20) in run 11432033631

[github-actions]

The centralized nightlies job failed on Monday (2024-10-21) in run 11452048312

[github-actions]

The centralized nightlies job failed on Tuesday (2024-10-22) in run 11471844709

[github-actions]

The centralized nightlies job failed on Wednesday (2024-10-23) in run 11491453361

[kounelisagis]

The PRs referenced above should fix both test failures.

[jdblischak]

I manually triggered a build, and the TileDB-Py tests all passed! 🎉

There was a single tiledbsoma-py test failure, but I'll let the pipeline open a new Issue tonight to track that failing test. This Issue has already been open for a month.

[ihnorton]

Great work @kounelisagis @jdblischak -- thank you for pushing on this!