Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handling gzip input buffer with zlib compressor #308

Open
kaarrot opened this issue Jul 13, 2020 · 0 comments
Open

Handling gzip input buffer with zlib compressor #308

kaarrot opened this issue Jul 13, 2020 · 0 comments

Comments

@kaarrot
Copy link

kaarrot commented Jul 13, 2020

Hello, I'm having some problems while trying to decompress gzip buffer with zlib compressor. Can anybody give some guidance how to approach this please? I'm aware zlib requires explicit configuration to handle gzip with inflateInit2() but I'm not sure how to pass that. Can this be configured directly from blosc?

Here is a simplified example feeding in base64 input with Python.

import base64
import zlib
data = 'H4sIAAAAAAAAAHWPwQqCQBCGX0Xm7EFtK+smZBEUgXoLCdMhFtKV3akI8d0bLYmibvPPN3wz00CJxmQnTO41whwWQRIctmEcB6sQbFC3CjW3XW8kxpOpP+OC22d1Wml1qZkQGtoMsScxaczKN3plG8zlaHIta5KqWsozoTYw3/djzwhpLwivWFGHGpAFe7DL68JlBUk+l7KSN7tCOEJ4M3/qOI49vMHj+zCKdlFqLaU2ZHV2a4Ct/an0/ivdX8oYc1UVX860fQDQiMdxRQEAAA=='
print (zlib.decompress(base64.b64decode(data), 16 + zlib.MAX_WBITS).decode('utf-8'))

and the c++ version where blosc_decompress() returns 0 size:

#include <iostream>
#include <string>
#include <vector>

#include <blosc.h>
// #include <zlib.h>

using namespace std;

static std::string base64_decode(const std::string &in) {

    std::string out;

    std::vector<int> T(256,-1);
    for (int i=0; i<64; i++) T["ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"[i]] = i; 

    int val=0, valb=-8;
    for (unsigned char c : in) {
        if (T[c] == -1) break;
        val = (val<<6) + T[c];
        valb += 6;
        if (valb>=0) {
            out.push_back(char((val>>valb)&0xFF));
            valb-=8;
        }
    }
    return out;
}

int main(){
        std::string s ="H4sIAAAAAAAAAHWPwQqCQBCGX0Xm7EFtK+smZBEUgXoLCdMhFtKV3akI8d0bLYmibvPPN3wz00CJxmQnTO41whwWQRIctmEcB6sQbFC3CjW3XW8kxpOpP+OC22d1Wml1qZkQGtoMsScxaczKN3plG8zlaHIta5KqWsozoTYw3/djzwhpLwivWFGHGpAFe7DL68JlBUk+l7KSN7tCOEJ4M3/qOI49vMHj+zCKdlFqLaU2ZHV2a4Ct/an0/ivdX8oYc1UVX860fQDQiMdxRQEAAA==";
    std::string dc = base64_decode(s);

    blosc_init();
    blosc_set_nthreads(1);

    int ret = blosc_set_compressor("zlib");
    cout << ret <<endl;

    char * data_dest = new char[dc.size() * 2]; 
    int dsize = blosc_decompress(dc.data(), data_dest, dc.size());
    cout << dsize <<endl;
    return 0;
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant