Skip to content

Commit

Permalink
Handle HTTP chunking across multi-byte boundaries
Browse files Browse the repository at this point in the history
When a response contains multi-byte UTF-8 characters that cross chunk
boundaries, appending the two halves to the intermediate string results
in two Unicode Replacement Characters being inserted instead of the two
halves being joined.

Use StringDecoder to buffer the partial characters across chunks.

Resolves #22
  • Loading branch information
Freaky committed Mar 8, 2024
1 parent 1cdcf24 commit 7cbcfe4
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion src/utils.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import { version } from "../version.ts";
import https from "node:https";
import qs from "node:querystring";
import { StringDecoder } from "node:string_decoder";
import { RequestTimeoutError } from "./errors.ts";

/**
Expand Down Expand Up @@ -63,15 +64,17 @@ export function execute(
return new Promise((resolve, reject) => {
let timer: number;
const req = https.get(url, (resp) => {
const decoder = new StringDecoder("utf8");
let data = "";

// A chunk of data has been recieved.
resp.on("data", (chunk) => {
data += chunk;
data += decoder.write(chunk);
});

// The whole response has been received. Print out the result.
resp.on("end", () => {
data += decoder.end();
try {
if (resp.statusCode == 200) {
resolve(data);
Expand Down

0 comments on commit 7cbcfe4

Please sign in to comment.