Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

On web, uploading large files does not work #1708

Closed
nicolaspernoud opened this issue Mar 3, 2023 · 9 comments
Closed

On web, uploading large files does not work #1708

nicolaspernoud opened this issue Mar 3, 2023 · 9 comments

Comments

@nicolaspernoud
Copy link

Package

dio

Version

5.0.2

Output of flutter doctor -v

[√] Flutter (Channel stable, 3.7.6, on Microsoft Windows [version
    10.0.22621.1265], locale fr-FR)
    • Flutter version 3.7.6 on channel stable at C:\_apps\_sdk\binaries\flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 12cb4eb7a0 (2 days ago), 2023-03-01 10:29:26 -0800
    • Engine revision ada363ee93
    • Dart version 2.19.3
    • DevTools version 2.20.1

[X] Windows Version (Unable to confirm if installed Windows version is 10 or
    greater)

[X] Android toolchain - develop for Android devices
    X Unable to locate Android SDK.
      Install Android Studio from:
      https://developer.android.com/studio/index.html
      On first launch it will assist you in installing the Android SDK
      components.
      (or visit
      https://flutter.dev/docs/get-started/install/windows#android-setup for   
      detailed instructions).
      If the Android SDK has been installed to a custom location, please use   
      `flutter config --android-sdk` to update to that location.


[√] Chrome - develop for the web
    • CHROME_EXECUTABLE = C:\Program Files
      (x86)\Microsoft\Edge\Application\msedge.exe

[!] Visual Studio - develop for Windows (Visual Studio Community 2022 17.5.0)  
    • Visual Studio at C:\Program Files\Microsoft Visual Studio\2022\Community 
    • Visual Studio Community 2022 version 17.5.33414.496
    • Windows 10 SDK version 10.0.22000.0
    X Visual Studio is missing necessary components. Please re-run the Visual  
      Studio installer for the "Desktop development with C++" workload, and    
      include these components:
        MSVC v142 - VS 2019 C++ x64/x86 build tools
         - If there are multiple build tool versions available, install the    
         latest
        C++ CMake tools for Windows
        Windows 10 SDK

[!] Android Studio (not installed)
    • Android Studio not found; download from
      https://developer.android.com/studio/index.html
      (or visit
      https://flutter.dev/docs/get-started/install/windows#android-setup for   
      detailed instructions).

[√] Connected device (3 available)
    • Windows (desktop) • windows • windows-x64    • Microsoft Windows [version
      10.0.22621.1265]
    • Chrome (web)      • chrome  • web-javascript • unknown
    • Edge (web)        • edge    • web-javascript • Microsoft Edge
      110.0.1587.57

[√] HTTP Host Availability
    • All required HTTP hosts are available

Dart Version

2.19.3

Steps to Reproduce

  1. Select a XFile with https://pub.dev/packages/file_selector
  2. Try to PUT the file with dio using file.openRead() that gives a Stream.

Expected Result

It should upload the file to the server, regardless of its size.

Actual Result

It works with small files (up to approx. 500 MB), but it stalls with large (several GB files).
It seems that Dio consumes and cache the stream instead of streaming it to the server.
Only occurs with web, it is OK with Android.

@nicolaspernoud nicolaspernoud added h: need triage This issue needs to be categorized s: bug Something isn't working labels Mar 3, 2023
@AlexV525
Copy link
Member

Sorry for the late reply, the notification is covered by massive others. Could you provide a reproducible minimal example?

@AlexV525 AlexV525 added the h: need more info Further information is requested label Jun 15, 2023
@nicolaspernoud
Copy link
Author

Yes, here is a minimal golang server :

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
)

func uploadFile(w http.ResponseWriter, r *http.Request) {

	// Enable CORS headers
	w.Header().Set("Access-Control-Allow-Origin", "*")
	w.Header().Set("Access-Control-Allow-Methods", "POST, OPTIONS")
	w.Header().Set("Access-Control-Allow-Headers", "Content-Type")

	if r.Method == "OPTIONS" {
		// Handle preflight request
		w.WriteHeader(http.StatusOK)
		return
	}

	// Open a new file on the server
	dst, err := os.Create("uploaded_file.mkv")
	if err != nil {
		http.Error(w, "Failed to create the file on the server", http.StatusInternalServerError)
		return
	}
	defer dst.Close()

	// Copy the request body to the new file
	_, err = io.Copy(dst, r.Body)
	if err != nil {
		http.Error(w, "Failed to save the file", http.StatusInternalServerError)
		return
	}

	fmt.Fprintf(w, "File uploaded successfully!")
}

func main() {
	http.HandleFunc("/upload", uploadFile)

	fmt.Println("Server listening on http://localhost:8080/upload")
	log.Fatal(http.ListenAndServe(":8080", nil))
}

and the corresponding flutter code :

name: dio_test
description: A new Flutter project.

publish_to: "none" # Remove this line if you wish to publish to pub.dev

environment:
  sdk: ">=3.0.2 <4.0.0"

  flutter:
    sdk: flutter
  dio: ^4.0.0
  file_picker: ^4.2.0

  cupertino_icons: ^1.0.2

dev_dependencies:
  flutter_test:
    sdk: flutter

  flutter_lints: ^2.0.0

flutter:
  uses-material-design: true
import 'package:dio/dio.dart';
import 'package:file_picker/file_picker.dart';
import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Dio Upload Demo',
      theme: ThemeData(
        colorScheme: ColorScheme.fromSeed(seedColor: Colors.deepPurple),
        useMaterial3: true,
      ),
      home: const MyHomePage(title: 'Dio Upload Demo'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  const MyHomePage({super.key, required this.title});

  final String title;

  @override
  State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  Future<void> _upload() async {
    // Create a Dio instance
    var dio = Dio();

    try {
      // Open the file as a stream
      FilePickerResult? result = await FilePicker.platform.pickFiles(
        withData: false,
        withReadStream: true,
      );

      if (result != null) {
        PlatformFile file = result.files.first;
        Stream<List<int>>? fileBytes = file.readStream;

        if (fileBytes != null) {
          var response = await dio.post(
            'http://localhost:8080/upload',
            data: fileBytes,
            options: Options(
              contentType: 'application/octet-stream',
            ),
          );

          print(response.data);
        }
      }
    } catch (e) {
      print('Error: $e');
    }
    setState(() {});
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        backgroundColor: Theme.of(context).colorScheme.inversePrimary,
        title: Text(widget.title),
      ),
      floatingActionButton: FloatingActionButton(
        onPressed: _upload,
        tooltip: 'Upload',
        child: const Icon(Icons.upload),
      ),
    );
  }
}

With a large file, it gives the following error :

RangeError: Array buffer allocation failed
dart-sdk/lib/_internal/js_dev_runtime/private/native_typed_data.dart 1071:7   _create1
dart-sdk/lib/_internal/js_dev_runtime/private/native_typed_data.dart 1041:42  new
dart-sdk/lib/convert/byte_conversion.dart 77:19                               add
dart-sdk/lib/async/zone.dart 1594:10                                          runUnaryGuarded
dart-sdk/lib/async/stream_impl.dart 339:11                                    [_sendData]
dart-sdk/lib/async/stream_impl.dart 271:7                                     [_add]
dart-sdk/lib/async/stream_transformers.dart 63:11                             [_add]
dart-sdk/lib/async/stream_transformers.dart 13:11                             add
packages/dio/src/progress_stream/browser_progress_stream.dart 24:15           <fn>
dart-sdk/lib/async/stream_transformers.dart 209:17                            add
dart-sdk/lib/async/stream_transformers.dart 111:24                            [_handleData]
dart-sdk/lib/async/zone.dart 1594:10                                          runUnaryGuarded
dart-sdk/lib/async/stream_impl.dart 339:11                                    [_sendData]
dart-sdk/lib/async/stream_impl.dart 515:13                                    perform
dart-sdk/lib/async/stream_impl.dart 620:10                                    handleNext
dart-sdk/lib/async/stream_impl.dart 591:7                                     callback
dart-sdk/lib/async/schedule_microtask.dart 40:11                              _microtaskLoop
dart-sdk/lib/async/schedule_microtask.dart 49:5                               _startMicrotaskLoop
dart-sdk/lib/_internal/js_dev_runtime/patch/async_patch.dart 177:15           <fn>

@AlexV525
Copy link
Member

Your first comment said you're using dio 5.0.2, but your provided code wrote dio: ^4.0.0. Could identify this again to see if it occurred on the latest dio?

@AlexV525 AlexV525 added i: outdated and removed h: need triage This issue needs to be categorized s: bug Something isn't working labels Jun 21, 2023
@nicolaspernoud
Copy link
Author

Sorry, and yes that is the same. I updated the comment with the new pubspec.yml .

@AlexV525
Copy link
Member

Can you try to convert the read stream from Stream<List<int>> to Stream<Uint8List>?

@AlexV525 AlexV525 added p: dio Targeting `dio` package platform: web and removed i: outdated labels Aug 21, 2023
@AlexV525
Copy link
Member

I found the exact same issue here: dart-lang/http#854

@AlexV525 AlexV525 added i: out of support p: dependency Targeting packages that's used by dio packages labels Aug 21, 2023
@nicolaspernoud
Copy link
Author

Hello,
I tried converting the stream, but the problem is the same : here is the updated code : https://gist.github.com/nicolaspernoud/236d56c7f28bc25927a4c4f46a6a99b9 .

The issue you found seems to indicate that the upload should be streamed instead of multipart... I thought I was doing that (?!).
Maybe DIO should stream when the content type is 'application/octet-stream' ?

@kuhnroyal
Copy link
Member

From what I gather in the linked issue, streaming on web is not possible with XMLHttpRequest so the file has to be loaded into memory. When the file is to large, the browser can not allocate the memory.

Don't think we have a possible solution here atm.

@AlexV525
Copy link
Member

Closing as above.

@AlexV525 AlexV525 closed this as not planned Won't fix, can't repro, duplicate, stale Sep 21, 2023
@AlexV525 AlexV525 removed h: need more info Further information is requested p: dio Targeting `dio` package p: dependency Targeting packages that's used by dio packages platform: web labels Sep 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants