Provide a way to sync files outside of the MM sitemap

[fredjean]

This addresses the issue raised at does middleman ignore .txt files by default. The conclusion is that the file was added to the build directory as part of an external build script. As a result, middleman was not aware of it and it was not in the site map.

The potential fixes are:

1. Ensure that all files present in the build directory are accounted for in the sync process
2. Add a way to add files to sync to s3_sync.

[thedanotto]

Hi,

My issue may be related to the issue you described above, that's why I'm posting this here.

I'm new to middleman and I'm running middleman-blog on top of middleman. bundle exec middleman build --clean has generated a multitude of files within my build directory. Some of these files/folders are blog related. For instance, the folder, 2015 has various directories and files categorizing my posts by month and year. There is also the tags folder which categorizes my posts based upon the tags associated with them.

Anyways, when I run the command bundle exec middleman s3_sync, s3_sync ignores the folders and files. If I manually upload the files, s3_sync will delete them. I did chuckle at that when it first happened.

A quick question, what does middleman-s3_sync use to build the sync list? And how do I access the sync list to see if these files are truly being not included.

Thanks

[fredjean]

Can you provide your Gemfile? The answer to your question will depend on which version you are using.

I recently ported s3_sync to use the Middleman site map, which should include all of the files generated by Middleman build. Files that are outside of the site map are not taken into account.

You can turn off deleting files that s3_sync doesn't know about using the config.delete = false option.

[thedanotto]

You're right, I'm using an old version.

source 'http://rubygems.org'

source "https://rails-assets.org" do
gem "rails-assets-jquery"
gem "rails-assets-moment"
end

gem "bitters"
gem "bourbon"
gem "builder", "> 3.0"
gem "fog-core", "1.34.0"
gem "middleman", ">3.4.0"
gem "middleman-blog", "> 3.5.3"
gem "middleman-dotenv", "> 1.0"
gem "middleman-livereload", "> 3.1.0"
gem "middleman-s3_sync", "> 3.3.0"
gem "neat"
gem "redcarpet"
gem "unf"

[fredjean]

Is it working as expected after you updated the version?

[thedanotto]

Well, unfortunately, I cannot really test because I believe I was on the latest version middleman-s3_sync for v 3.x. And when I upgrade I go to version 4.0.0.

Using middleman-s3_sync 4.0.0 (was 3.3.7)

And the other important gem, middleman-blog does not have a release for middleman version 4.x. If I remove middleman-blog, I still run into a couple other deploy errors unrelated to to s3_sync.

Is there a potential for 3.3.8? If not I can simply take your recommendation of config.delete = false and wait until everything settles into v4.0 and reopen an issue if I see it.

[fredjean]

@thedanotto I yanked 4.0.0 from Rubygems because it's gemspec was poorly setup and was causing some issues. You should stick with the 3.3.x stream for now.

I don't have any idea at this point what is causing your issue though. Anything that Middleman put in the build directory should be handled. The only exception are gziped files. The original are ignored and the compressed files are sent up instead.

Could you provide the output from running s3_sync?

[djalmaaraujo]

Where middleman stores this site map?

[fredjean]

It is stored in memory. It's a compilation of all of the pages that are generated by the middleman build process.

[cpence]

I'm 99% sure that Middleman now has the support you need for this without adding it to this extension. The new MM4 import_path functionality adds a directory to the sitemap. The example from the docs is:

import_path File.expand_path('bower_components', app.root)

It's not super clear how this actually would be used, but I think it's going to let you add whatever you want to the sitemap.

[joshukraine]

I yanked 4.0.0 from Rubygems because it's gemspec was poorly setup and was causing some issues. You should stick with the 3.3.x stream for now.

@fredjean So am I understanding correctly that for the moment, middleman-s3_sync does not support MM v4? I'm working on building a few sites with MM v4, and would really like to use your gem. Is there a workaround or maybe an ETA when this will be possible? Thanks!

[fredjean]

@joshukraine Version 4.0.1.rc.3 is available on Rubygems for testing. I haven't converted my own site to MM4 yet so I haven't been able to fully test it.

[fredjean]

@joshukraine v4.0.1 and v4.0.2 provide full support for MM4.

[joshukraine]

@fredjean 😃👍🏻

[fredjean]

@thedanotto Are you still having this issue?

[thedanotto]

Let me do an upgrade tonight and check. If I am I'll get you a more detailed report on what's going on and create you an example repository that can be forked.

[thedanotto]

Just upgraded to latest version and it's ALL GOOD IN THE HOOD! Thanks for all your work on this.

[fredjean]

@thedanotto Excellent! Thank you for following up.

[macandcheese]

@cpence can you elaborate? I move and place a lot of files after build with fileUtil.. and Middleman's sitemap is not aware of these items although they exist in the build folder correctly, so in turn s3_sync can never push them to the bucket. Are you saying that's a method to manually map the build directory ?

Furthermore, and maybe this is more a MM thing, because the files in build dir aren't accounted for, my sitemaps built using resource helpers from middleman's sitemap are incorrect and missing folders.

It's as if MM needs a step in the build where you can "generate MM sitemap from directory structure of the build directory", which then in turn would provide s3_sync an accurate map to upload

[bmedici]

Hi, still having the problem with MM4 and middleman-s3_sync v4.
Building files should add them automatically to the sitemap, or we should have a simple way to add them explicitely.
What about asset files, which are not present in the sitemap, but still uploaded ?