h5repack doesn't work when I use zfp filter for HDF5/1.14.3 installed by spack on Linux #148
Comments
|
Am capturing here trace output from running one of the |
|
Ok, the above error output is due to attempt to compress It looks to me like the command-line args for |
|
Ok, yeah. The cl argument format for |
|
I am not seeing expected behavior with filter optional flag (the |
|
Ok, yeah. I think a few things are going on here that require some attention in the filter. We should not error from |
|
@brtnfld or @qkoziol do you have advice on when a filter's |
|
In looking at the code, I do think the |
|
Yes, that's correct, the can_apply method can return three values: <0 is a failure, 0 is "can't apply", and 1 is "can apply". In this case, the filter should return 0 (can't apply). Some of the related code,, around line 800 in src/H5Z.c: |
|
I believe the When I specify |
|
I tried this command:
h5repack -n -f Pressure:UD=32013,1,4,3,0,3539053052,1062232653 -l
Pressure:CHUNK=10x20x5 mesh.h5 mesh_zfp.h5
It works on my Mac, I have not tried on Linux machine yet.
…On Wed, Nov 20, 2024 at 10:58 AM Mark C. Miller ***@***.***> wrote:
@halehawk <https://github.com/halehawk> have you re-tried your exercise
using the slightly modified way to invoke h5repack as explained in the
comment
<#148 (comment)>
above.
—
Reply to this email directly, view it on GitHub
<#148 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACAPEFGL4FARPB3AKIE6QVT2BTEVJAVCNFSM6AAAAABR5JHOT6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBZGIZDSNZYHE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Ok, great. 💪. Thanks. And, just to be sure, "works" means that you indeed did have |
|
Yes, I looked at the filter attribute of Pressure by h5dump. I can see the zfp id and other values.So it looks like I have to specify which variable to compress by -f options, otherwise it will compress all variables and will give me not compressed because some variables only have one value.Sent from my iPhoneWe On Nov 20, 2024, at 3:58 PM, Mark C. Miller ***@***.***> wrote:
I tried this command: h5repack -n -f Pressure:UD=32013,1,4,3,0,3539053052,1062232653 -l Pressure:CHUNK=10x20x5 mesh.h5 mesh_zfp.h5 It works on my Mac, I have not tried on Linux machine yet.
Ok, great. 💪. Thanks. And, just to be sure, "works" means that you indeed did have HDF5_PLUGIN_PATH set so that h5repack found the plugin and you confirmed via h5dump or h5ls that the Pressure dataset in mesh_zfp.h was indeed compressed with the H5Z-ZFP filter?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
DATASET "Pressure" {
DATATYPE H5T_STD_U32LE
DATASPACE SIMPLE { ( 10, 20, 30 ) / ( 10, 20, 30 ) }
STORAGE_LAYOUT {
CHUNKED ( 10, 20, 5 )
SIZE 24000 (1.000:1 COMPRESSION)
}
FILTERS {
USER_DEFINED_FILTER {
FILTER_ID 32013
PARAMS { 3 0 -755914244 1062232653 }
}
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_IFSET
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
…On Wed, Nov 20, 2024 at 9:24 PM Haiying Xu ***@***.***> wrote:
Yes, I looked at the filter attribute of Pressure by h5dump. I can see the
zfp id and other values.
So it looks like I have to specify which variable to compress by -f
options, otherwise it will compress all variables and will give me not
compressed because some variables only have one value.
Sent from my iPhone
We On Nov 20, 2024, at 3:58 PM, Mark C. Miller ***@***.***>
wrote:
I tried this command: h5repack -n -f
Pressure:UD=32013,1,4,3,0,3539053052,1062232653 -l Pressure:CHUNK=10x20x5
mesh.h5 mesh_zfp.h5 It works on my Mac, I have not tried on Linux machine
yet.
Ok, great. 💪. Thanks. And, just to be sure, "works" means that you indeed
did have HDF5_PLUGIN_PATH set so that h5repack found the plugin and you
confirmed via h5dump or h5ls that the Pressure dataset in mesh_zfp.h was
indeed compressed with the H5Z-ZFP filter?
—
Reply to this email directly, view it on GitHub
<#148 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACAPEFHG3LAJ24XEN6BUB5L2BUHXPAVCNFSM6AAAAABR5JHOT6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBZG4YDCOBWHE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Pressure Dataset {10/10, 20/20, 30/30}
Location: 1:31528
Links: 1
Chunks: {10, 20, 5} 4000 bytes
Storage: 24000 logical bytes, 24000 allocated bytes, 100.00%
utilization
Filter-0: method-32013 OPT {3, 0, 3539053052, 1062232653}
Type: native unsigned int
…On Wed, Nov 20, 2024 at 9:29 PM Haiying Xu ***@***.***> wrote:
DATASET "Pressure" {
DATATYPE H5T_STD_U32LE
DATASPACE SIMPLE { ( 10, 20, 30 ) / ( 10, 20, 30 ) }
STORAGE_LAYOUT {
CHUNKED ( 10, 20, 5 )
SIZE 24000 (1.000:1 COMPRESSION)
}
FILTERS {
USER_DEFINED_FILTER {
FILTER_ID 32013
PARAMS { 3 0 -755914244 1062232653 }
}
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_IFSET
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
On Wed, Nov 20, 2024 at 9:24 PM Haiying Xu ***@***.***> wrote:
> Yes, I looked at the filter attribute of Pressure by h5dump. I can see
> the zfp id and other values.
>
> So it looks like I have to specify which variable to compress by -f
> options, otherwise it will compress all variables and will give me not
> compressed because some variables only have one value.
>
>
> Sent from my iPhone
>
> We On Nov 20, 2024, at 3:58 PM, Mark C. Miller ***@***.***>
> wrote:
>
>
>
> I tried this command: h5repack -n -f
> Pressure:UD=32013,1,4,3,0,3539053052,1062232653 -l Pressure:CHUNK=10x20x5
> mesh.h5 mesh_zfp.h5 It works on my Mac, I have not tried on Linux machine
> yet.
>
> Ok, great. 💪. Thanks. And, just to be sure, "works" means that you
> indeed did have HDF5_PLUGIN_PATH set so that h5repack found the plugin
> and you confirmed via h5dump or h5ls that the Pressure dataset in
> mesh_zfp.h was indeed compressed with the H5Z-ZFP filter?
>
> —
> Reply to this email directly, view it on GitHub
> <#148 (comment)>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACAPEFHG3LAJ24XEN6BUB5L2BUHXPAVCNFSM6AAAAABR5JHOT6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBZG4YDCOBWHE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
>
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Recently, I asked our system admin to install HDF5/1.14.3 by spack, zfp, and H5Z-ZFP manually on our supercomputer. But when I tried h5repack to compress the mesh.h5 in the H5Z-ZFP test. I got errors on all variables in the h5 file as follows:
Before I used h5repack, I export HDF5_PLUGIN_PATH correctly. After I got the errors, I checked that all the libraries were correctly linked. There are more details in the attached log file.
Also if I used the different set of HDF5 from one of my conda environment, but used the same HDF5_PLUGIN_PATH, I can run the h5repack successfully.
Could you please help me to solve the issue? Thanks!
h5repack -i mesh.h5 -o mesh_zfp.h5 -f UD=32013,0,4,1,0,0,1074921472 -v
No all objects to modify layout
All objects to apply filter are...
User Defined 32013
Making new file ...
Type Filter (Compression) Name
group /
HDF5-DIAG: Error detected in HDF5 (1.14.3) thread 0:
#000: /glade/derecho/scratch/csgteam/temp/spack/derecho/23.09/builds/spack-stage-hdf5-1.14.3-ten6adliyyajyc2vyv6vu7xg254bnhls/spack-src/src/H5T.c line 2052 in H5Tget_class(): not a datatype
major: Invalid arguments to routine
minor: Inappropriate type
warning: could not create dataset . Applying original settings
dset /DxDy
HDF5-DIAG: Error detected in HDF5 (1.14.3) thread 0:
#000: /glade/derecho/scratch/csgteam/temp/spack/derecho/23.09/builds/spack-stage-hdf5-1.14.3-ten6adliyyajyc2vyv6vu7xg254bnhls/spack-src/src/H5T.c line 2052 in H5Tget_class(): not a datatype
major: Invalid arguments to routine
minor: Inappropriate type
warning: could not create dataset . Applying original settings
dset /DxDyDz
h5repack-error.txt
The text was updated successfully, but these errors were encountered: