Skip to content
This repository has been archived by the owner on Jul 1, 2024. It is now read-only.

Fails to download with librarian-chef #105

Closed
brian-beckett-hpe opened this issue Sep 16, 2014 · 6 comments
Closed

Fails to download with librarian-chef #105

brian-beckett-hpe opened this issue Sep 16, 2014 · 6 comments

Comments

@brian-beckett-hpe
Copy link

I'm attempting to download the SimpleFinance/chef-zookeeper cookbook with Librarian-Chef 0.0.4. I'm using the following Cheffile:

site 'https://supermarket.getchef.com/api/v1'
cookbook 'zookeeper'

I'm running Librarian-Chef like so:
$ librarian-chef install --verbose

Librarian-Chef fails with the following stacktrack:

[Librarian] Ruby Version: 1.9.3
[Librarian] Ruby Platform: i386-mingw32
[Librarian] Rubygems Version: 1.8.28
[Librarian] Librarian Version: 0.1.2
[Librarian] Librarian Adapter: chef
[Librarian] Librarian Adapter Version: 0.0.4
[Librarian] Project: d:/dev/temp/chef-zookeeper-client
[Librarian] Specfile: Cheffile
[Librarian] Lockfile: Cheffile.lock
[Librarian] Git: C:/Program Files (x86)/Git/bin/git.EXE
[Librarian] Git Version: 1.9.4
[Librarian] Git Environment Variables:
[Librarian]   (empty)
[Librarian] Pre-Cached Sources:
[Librarian] Post-Cached Sources:
[Librarian]   [:site, "https://supermarket.getchef.com/api/v1", {}]
[Librarian] Resolving zookeeper (>= 0) <https://supermarket.getchef.com/api/v1>
[Librarian]   Checking manifests
[Librarian] Caching https://supermarket.getchef.com/api/v1/cookbooks/zookeeper to d:/dev/temp/chef-zookeeper-client/tmp/librarian/cache/source/chef/site/2036ece20da2a6e8/zookeeper/metadata.json
[Librarian] Performing http-get for https://supermarket.getchef.com/api/v1/cookbooks/zookeeper
[Librarian] Responded with success
[Librarian]     Checking zookeeper/2.4.1 <https://supermarket.getchef.com/api/v1>
[Librarian] Unpacking tmp/librarian/cache/source/chef/site/2036ece20da2a6e8/zookeeper/version-uri/ffe81f3a5c974987/package.tar.gz to tmp/librarian/scratch/0b9b7e63e0dcfd26b0adbcdce68c3ec1
d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-chef-0.0.4/lib/librarian/chef/source/site.rb:309:in `unpack_package!': The package archive has too many children! (RuntimeError)
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-chef-0.0.4/lib/librarian/chef/source/site.rb:224:in `cache_version_uri_unpacked!'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-chef-0.0.4/lib/librarian/chef/source/site.rb:93:in `block in version_uri_manifest'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-chef-0.0.4/lib/librarian/chef/source/site.rb:381:in `memo'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-chef-0.0.4/lib/librarian/chef/source/site.rb:92:in `version_uri_manifest'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-chef-0.0.4/lib/librarian/chef/source/site.rb:88:in `version_manifest'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-chef-0.0.4/lib/librarian/chef/source/site.rb:54:in `version_dependencies'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-chef-0.0.4/lib/librarian/chef/source/site.rb:462:in `fetch_dependencies'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/manifest.rb:125:in `fetch_dependencies!'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/manifest.rb:117:in `fetched_dependencies'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/manifest.rb:81:in `dependencies'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:117:in `block in check_manifest_for_cycles'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:117:in `each'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:117:in `map'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:117:in `check_manifest_for_cycles'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:64:in `block in recursive_resolve'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:154:in `block (3 levels) in resolving_dependency_map_find_manifests'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:187:in `block in scope_checking_manifest'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:223:in `scope'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:186:in `scope_checking_manifest'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:153:in `block (2 levels) in resolving_dependency_map_find_manifests'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:211:in `block in map_find'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:210:in `each'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:210:in `map_find'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:152:in `block in resolving_dependency_map_find_manifests'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:165:in `block (2 levels) in scope_resolving_dependency'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:179:in `block in scope_checking_manifests'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:223:in `scope'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:178:in `scope_checking_manifests'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:164:in `block in scope_resolving_dependency'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:223:in `scope'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:163:in `scope_resolving_dependency'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:151:in `resolving_dependency_map_find_manifests'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:62:in `recursive_resolve'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:50:in `resolve'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver.rb:23:in `resolve'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/action/resolve.rb:26:in `run'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/cli.rb:169:in `resolve!'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-chef-0.0.4/lib/librarian/chef/cli.rb:41:in `install'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/cli.rb:26:in `block (2 levels) in bin!'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/cli.rb:31:in `returning_status'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/cli.rb:26:in `block in bin!'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/cli.rb:47:in `with_environment'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/cli.rb:26:in `bin!'
    from d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-chef-0.0.4/bin/librarian-chef:7:in `<top (required)>'
    from d:/dev/tools/Ruby193/bin/librarian-chef:23:in `load'
    from d:/dev/tools/Ruby193/bin/librarian-chef:23:in `<main>'

The important bit from all that was:

[Librarian] Unpacking tmp/librarian/cache/source/chef/site/2036ece20da2a6e8/zookeeper/version-uri/ffe81f3a5c974987/package.tar.gz to tmp/librarian/scratch/0b9b7e63e0dcfd26b0adbcdce68c3ec1
d:/dev/tools/Ruby193/lib/ruby/gems/1.9.1/gems/librarian-chef-0.0.4/lib/librarian/chef/source/site.rb:309:in 'unpack_package!': The package archive has too many children! (RuntimeError)

I've not seen this before with any other Chef cookbooks, and a similar StackOverflow question indicates that the cookbook may be at fault in some way.

So... any thoughts? Are you able to replicate this issue at all?
@brian-beckett-hpe
Copy link
Author

The last release to successfully download was 1.6.1, and the first to fail was 1.7.0.

@brian-beckett-hpe
Copy link
Author

The breaking change should be somewhere in here... v1.6.1...v1.7.0

@jakedavis
Copy link
Contributor

@brian-beckett-hp, unfortunately we moved away from Librarian a long time back. We left the Cheffile in there as kind of a backwards-compatibility thing, but I'll admit we haven't really taken the time to make sure it still works. I can try to circle back sometime soon, but I'm not really equipped off-hand to debug unfortunately :(

We will figure it out though. No ideas off the top of my head.

@brian-beckett-hpe
Copy link
Author

I hadn't even noticed your Cheffile in there! :) I didn't think you needed one for just downloading the package with Librarian...

The issue isn't urgent for me, as I was able to find an alternative Zookeeper cookbook that currently suits my needs. If you are able to resolve it at some point, that would be really great, but at least now there's a record of the issue that might save someone else from tearing their hair out over it ;)

Thanks for the response!

@zuazo
Copy link
Contributor

zuazo commented Oct 20, 2014

These errors seems to be related with the way the cookbook is being uploaded to the Supermarket. Seems to be compressed using POSIX tar format instead of GNU tar.

The problem arises because there are some directories called PaxHeader when the cookbook is extracted.

More details about this problem in Berkshelf issue #1132. I think librarian-chef PR #41 fixes this problem for librarian.

For example, testing this cookbook with test-kitchen 1.2.1, chef-client 11.16.4 and Berkshelf 3.1.5 gives me the following error:

================================================================================       
Recipe Compile Error in /tmp/kitchen/cookbooks/zookeeper/providers/PaxHeader/config.rb       
================================================================================       

SyntaxError       
-----------       
/tmp/kitchen/cookbooks/zookeeper/providers/PaxHeader/config.rb:1: syntax error, unexpected tIDENTIFIER, expecting $end       
17 gid=993687553       
      ^       

Cookbook Trace:       
---------------       
  /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/mixin/from_file.rb:42:in `class_eval'       
  /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/mixin/from_file.rb:42:in `class_from_file'       
  /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/provider/lwrp_base.rb:94:in `build_from_file'       
  /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/run_context/cookbook_compiler.rb:211:in `load_lwrp_provider'       
  /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/run_context/cookbook_compiler.rb:202:in `block in load_lwrps_from_cookbook'       
  /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/run_context/cookbook_compiler.rb:201:in `each'       
  /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/run_context/cookbook_compiler.rb:201:in `load_lwrps_from_cookbook'       
  /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/run_context/cookbook_compiler.rb:121:in `block in compile_lwrps'       
         /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/run_context/cookbook_compiler.rb:120:in `each'
         /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/run_context/cookbook_compiler.rb:120:in `compile_lwrps'

         /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/run_context/cookbook_compiler.rb:73:in `compile'
         /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/run_context.rb:89:in `load'
         /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/policy_builder/expand_node_object.rb:73:in `setup_run_context'
         /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/client.rb:265:in `setup_run_context'
         /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/client.rb:429:in `do_run'
         /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/client.rb:213:in `block in run'
         /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/client.rb:207:in `fork'
         /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/client.rb:207:in `run'
         /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/application.rb:236:in `run_chef_client'
         /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/application/solo.rb:226:in `block in run_application'
         /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/application/solo.rb:218:in `loop'
         /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/application/solo.rb:218:in `run_application'

         /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/application.rb:55:in `run'
         /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/bin/chef-solo:25:in `<top (required)>'
         /usr/bin/chef-solo:23:in `load'
         /usr/bin/chef-solo:23:in `<main>'

       Relevant File Content:
       ----------------------
       /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/mixin/from_file.rb:

        35:  
        36:        # Loads a given ruby file, and runs class_eval against it in the context of the current

        37:        # object.
        38:        #
        39:        # Raises an IOError if the file cannot be found, or is not readable.
        40:        def class_from_file(filename)
        41:          if File.exists?(filename) && File.readable?(filename)
        42>>           self.class_eval(IO.read(filename), filename, 1)
        43:          else
        44:            raise IOError, "Cannot open or read #{filename}!"
        45:          end
        46:        end
        47:  
        48:      end
        49:    end
        50:  end
        51:  


       Running handlers:
       [2014-10-20T08:11:34+00:00] ERROR: Running exception handlers
       Running handlers complete
       [2014-10-20T08:11:34+00:00] ERROR: Exception handlers complete

[2014-10-20T08:11:34+00:00] FATAL: Stacktrace dumped to /tmp/kitchen/cache/chef-stacktrace.out       
Chef Client failed. 0 resources updated in 0.839319278 seconds       
[2014-10-20T08:11:34+00:00] ERROR: /tmp/kitchen/cookbooks/zookeeper/providers/PaxHeader/config.rb:1: syntax error, unexpected tIDENTIFIER, expecting $end       
17 gid=993687553       
      ^       
[2014-10-20T08:11:34+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)       
>>>>>> Converge failed on instance <default-ubuntu-1204>.
>>>>>> Please see .kitchen/logs/default-ubuntu-1204.log for more details
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: SSH exited (1) for command: [sudo -E chef-solo --config /tmp/kitchen/solo.rb --json-attributes /tmp/kitchen/dna.json  --log_level info]
>>>>>> ----------------------

File content example:

root@default-ubuntu-1204:~# cat /tmp/kitchen/cookbooks/zookeeper/providers/PaxHeader/config.rb
17 gid=993687553
18 uid=1377367680
20 ctime=1409321902
20 atime=1409321902
23 SCHILY.dev=16777220
22 SCHILY.ino=5173181
18 SCHILY.nlink=1

Other related errors:

$ tar xvzf zookeeper.tgz 
tar xvzf zookeeper.tgz 
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
zookeeper/
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
zookeeper/.gitignore
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
[...]

@adamb0mb
Copy link
Contributor

@zuazo I can confirm your PR for librarian-chef fixes this, and I pulled it into my fork. Unfortunately, it looks like main librarian-chef may be dead, since he hasn't accepted a PR in over a year.

Seems like fixes for this would be

  • Delete the old (and misleading) Cheffile
  • Change to use GNU tar, not POSIX tar.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@adamb0mb @jakedavis @zuazo @brian-beckett-hpe