Thanks for the report and the test case. I haven't taken a look at this yet, but it looks like a serious issue that needs to be solved. Nothing immediately comes to mind but I plan to dig into it within a couple weeks. Feel free to dig in yourself if you want things to happen sooner.

I've been trying to reproduce this issue and I can't seem to get it to happen. Also, looking at the code, I don't understand why it would happen or why your fix would solve it. Do you think you could whip up a .tar.gz with some example code that demonstrates the problem?

@andrewrk Thanks for looking at this issue.

While trying to reproduce this, I realized there was a bit more to the story than I originally thought and described. Basically, the issue is only observed if you are to manually create the directory in an S3 bucket (from the S3 management console), and then upload contents to it. It doesn't happen when the entire directory is uploaded to a bucket.

Let me try to elaborate, with screenshots to avoid any confusion:

Code for the download

The code is very straightforward. It's all in this gist.

Directory structure

I created a directory on my local system, s3-test, containing two subfolders, subdir1 and subdir2. Each of these subfolders contains a very simple text file:

s3-test/
|_ subdir1/
  |_ file1.txt
|_ subdir2/
  |_ file2.txt

Scenario 1: Upload entire directory to S3 bucket (works as expected)

I set up a bucket and uploaded the entire s3-test directory to it.

When I then execute the code to download from the bucket and s3-test directory, I get both subdir1 and subdir2 downloaded into my local assets folder. If I execute the code a second time, I still get the same contents, so no issue here.

Scenario 2: Create a directory in a bucket then upload contents (subdirectory gets deleted on download):

Here, I first create a new directory inside the bucket, s3-test2, using the S3 management console. I then upload just the contents of the local s3-test directory (so just subdir1 and subdir2).

When I execute the code to download from the bucket and s3-test2 directory, the first time, everything works fine and both subdir1 and subdir2 get downloaded into the assets directory. If I re-run the code (without first deleting the assets directory), subdir1 goes missing.

Hope that helps clear things up!

I got the same problem... +1 for finding the working/failing cases... Wasnt easy.
Any solutions yet ? @brzpegasus

I can reproduce the issue. Files on S3 were created manually.

I am also experiencing this error scenario now.. I am noticing that when you create the scenario on s3 by hand s3 creates 'directory' objects for empty directories.. It seems this might be causing an error in the special case scenario in checkDoMoreWork() for handling directory matching. Fixed in merge request below.