[Reporting] CSV report contents are incorrectly encoded to base64

[tsullivan]

Problem: Recently, I noticed that Reporting is encoding CSV file contents to base64. This does not occur in earlier version of Kibana, and should not be happening.

Reporting was designed to encode report contents to base64, but should be doing so only when the content type is binary, such as PDF/PNG.

+ curl -k -u redacted:redacted 'https://redacted/.kibana-reporting*/_search?filter_path=hits.hits._id,hits.hits._source.created_by,hits.hits._source.payload.title,hits.hits._source.output.content&pretty'
{
  "hits" : {
    "hits" : [
      {
        "_id" : "0954be17-9c45-47ed-b315-2b1399f5268c",
        "_source" : {
          "created_by" : "elastic",
          "payload" : {
            "title" : "Untitled discover search"
          },
          "output" : {
            "content" : "bWVzc2FnZSwibWVzc2FnZS5rZXl3b3JkIgpoZWxsbzEsaGVsbG8xCg=="
          }
        }
      }
    ]
  }
}

[elasticmachine]

Pinging @elastic/appex-sharedux (Team:SharedUX)

[Dosant]

This does not occur in earlier version of Kibana, and should not be happening.

Does this mean there could be new csv reports in base64 and old that are not encoded? Do we have a download code path that can handle both in the same time?

[tsullivan]

@paulinashakirova I just noticed this line:

kibana/packages/kbn-reporting/export_types/csv/csv_searchsource.ts

Line 54 in 598d3de

jobContentEncoding = 'base64' as const;

I would see if the problem is fixed by changing that declaration to use 'csv':

-  jobContentEncoding = 'base64' as const;
+  jobContentEncoding = 'csv' as const;

[tsullivan]

@Dosant

Does this mean there could be new csv reports in base64 and old that are not encoded? Do we have a download code path that can handle both in the same time?

Yes, there could be new reports that are encoded in base64 and old reports that are not.

I did some extra testing and found this problem started in 8.10.0. Fortunately, I am able to download a report created in 8.9 (not encoded in base64) when running 8.10 (expects base64). I think there is some magic happening here - maybe Chrome is "un-encoding" the content?

I have found that my above suggestion does fix the issue, but now when we download a report that was created in 8.10-8.16, the raw data sent to the client is base64 text. This gets automatically decoded to plain CSV, at least in my testing in Chrome browser.

The only things I could see that may give us trouble:

1. If some browsers or clients are not decoding the base64 into CSV and users have those clients to downloading old reports.
2. If we have some functional tests that checks whether a report from 8.10-8.16 can be downloaded and the client used for tests does not automatically decode the report data. Fortunately, I don't think we have any tests like this.

To address the first concern, maybe we should make the fix a 9.0-only change, and add documentation to the release notes that there is possibly a known issue? Can you think of any more tests we may want to run first?

[Dosant]

@tsullivan,

Fortunately, I am able to download a report created in 8.9 (not encoded in base64) when running 8.10 (expects base64)

Hm, doesn't seem to be working for me. When I try to download old raw report that is decoded as base64, I see:

but now when we download a report that was created in 8.10-8.16, the raw data sent to the client is base64 text. This gets automatically decoded to plain CSV, at least in my testing in Chrome browser.

I don't see it, when I download such report I get a raw base64 file, this is what I see the base64 when try to open it:

---

This is how I was testing:

• Generate a report with one encoding

  kibana/packages/kbn-reporting/export_types/csv/csv_searchsource.ts

  Line 54 in 598d3de

  jobContentEncoding = 'base64' as const;

• Then change the value and download that report using different encoding

So encoding/decoding doesn't seem to be working when params are different in any direction

---

To make the change safe now, maybe we could change to csv and start saving to the report doc what encoding was used?
When decoding, if it says it is saved as csv we will decode it as raw, when encoding info is missing we fallback to base64 (making sure reports generated from 8.10-8.16 work)

[tsullivan]

@Dosant

To make the change safe now, maybe we could change to csv and start saving to the report doc what encoding was used?
When decoding, if it says it is saved as csv we will decode it as raw, when encoding info is missing we fallback to base64 (making sure reports generated from 8.10-8.16 work)

Something like this will work. Keep in mind we also have a payload.version field in the report documents, which tracks the version that the report was requested under. I think we can make the jobContentExtension field of the ExportType declaration typed to optionally be a function that takes in the report job parameters. Something like this:

1. packages/kbn-reporting/server/export_type.ts:

   -  abstract jobContentEncoding?: 'base64' | 'csv';
   +  abstract jobContentEncoding?: 'base64' | 'csv' | ((payload: TaskPayloadType) => 'base64' | 'csv');

2. packages/kbn-reporting/export_types/csv/csv_searchsource.ts

   -  jobContentEncoding = 'base64' as const;
   +  jobContentEncoding = (payload: TaskPayloadCSV) =>
   +    // example only: we need to check for version from 8.10-8.17 to declare as encoded in base64
   +    payload.version === '8.17' ? 'base64' : 'csv';

[tsullivan]

Reporting is encoding CSV file contents to base64

Out of curiosity, I have tracked down the PR that introduced this. The version was 8.10

• this line: https://github.com/elastic/kibana/pull/162276/files#diff-0cdd5a8e04b7e76544d0b0fa5dcde9248e0772a2873d720fb2070fe7c0467757L29
• became: https://github.com/elastic/kibana/pull/162276/files#diff-7f4b6ea3323749bf50cb1597dd4e0b7a1ae9e612d30fe107e1f4f3711165b6f1R42