Data Compression

[daswass]

I am using LZString to significantly compress JSON-ified data meant to be cached. The schema of the data does not match the cast because what is stored is actually just the compressed "string".

Question: does the library provide any compression when setting? If so, it would just make sense to remove my usage of LZString. If not, is there any plan to support schemas validation on compressed data?

[cyrilletuzi]

is there any plan to support schemas validation on compressed data

It wouldn't make sense. If a string is stored (no matter what the string is exactly), the JSON schema for strings must be used:

// Somewhere
const compressedData = someCompressionLib.compress(someData); // string
this.storage.set('key', compressedData).subscribe();

// Elsewhere
this.storage.get('key', { type: 'string' }).subscribe((compressedData) => {
  compressedData; // string
});

What you do with the data after that is no more the concern of the lib. So if you need to decompress with some lib, and you need to validate the result of this other lib, you can do so in any way you want.

You should be able to use the JSON validator from this lib if you want:

import { StorageMap, JsonValidator } from '@ngx-pwa/local-storage';

export class SomeService {

  constructor(
    private storage: StorageMap,
    private jsonValidator: JsonValidator,
  ) {}

  this.storage.get('key', { type: 'string' }).subscribe((compressedData) => {

    const unsafeData = someCompressionLib.decompress(compressedData); // unknown

    if (this.jsonValidator.validate(unsafeData, yourJsonSchema)) {
      const data = unsafeData as YourInterface;
    } else {
      // Error management
    }

  });

}

Note that:

• while JsonValidator is part of the public API of this lib, for special cases like yours, its direct usage is not documented so be careful of potential changes on major releases;
• I've done this in a procedural way to keep the example easy to understand, but it would be much better by doing this in a functional way with RxJS operators.

does the library provide any compression when setting?

Not currently. And not sure it would be very interesting to include that directly in this lib, contrary to validation where I decided to do my own implementation (because I didn't a find a library meeting my requirements and it was not too complicated to do it myself), compression is a more difficult thing to implement so it would be a bad idea to try a custom implementation. So it would mean using an existing library, which would be exactly the same as you already do.

[daswass]

Nice, that I can use the JSON validator on its own, thanks! Im in sync.

  getGlCodes(): Observable<Array<GlCode>> {
    return this.useCachedGlCodes().pipe(mergeMap(useCache => {
      if (useCache) {
        return this.storage.get('GlCodeCache', { type: 'string' }).pipe(map((compressed) => {
          if (!compressed) {
            return [];
          }
          let decompressed = Compression.LZString.decompress(compressed);
          if (this.jsonValidator.validate(decompressed, this.arrayOfGlCodeSchema)) {
            return JSON.parse(decompressed);
          } else {
            return [];
          }
        }));
      } else {
        var query = this.http.get<GlCode[]>(this.GET_GL_CODES_URL);

        return query.pipe(map((response) => {
          var items: Array<GlCode> = response["items"];
          items.forEach(item => item.description = UtilsService.toTitleCase(item.description));

          let json = JSON.stringify(items);
          let compressed = Compression.LZString.compress(json);
          console.log("Compressed " + json.length + " bytes to " + compressed.length + " bytes of GLCode data in StorageMap");

          this.storage.set('GlCodeCache', compressed).subscribe(() => {
            this.setGlCodesLastUpdate();
          });

          return items;
        }));
      }
    }));
  }