Issue with large volume of Profile Metadata

[jkeener1]

I just tried using the force-meta-backup on our salesforce org, and ran into the following error:

C:\force-meta-backup\build\profile-packages-target.xml:11: Failed to process the request successfully. Cause(LIMIT_EXCEEDED): LIMIT_EXCEEDED: The retrieved zip file exceeded the limit of 419430400 bytes.

The likely reason for this is our org has 242 profiles and 451 custom objects. I'm assuming the tool is attempting to pull down all the profiles and custom objects at the same time to get all of the field level security and exceeding the limit.

Is there a way with the tool to set a max items on something like this? If, for example, I were able to specify a max of 100 profiles per retrieve call, then 3 calls could be made.

• retrieve 1 with 100 profiles and 451 custom objects/other necessary related items
• retrieve 2 with 100 profiles and 451 custom objects/other necessary related items
• retrieve 3 with 42 profiles and 451 custom objects/other necessary related items

Profiles and Permission Sets are definitely an area where a could see this being useful, since it's an area where this limit could easily be exceeded.

[jkeener1]

I was digging further into it and found the "CustomObject.xml" that was being used. It is listing all custom objects, * for Profiles and * for Permission Sets.

<?xml version="1.0" encoding="UTF-8"?>
<Package xmlns="http://soap.sforce.com/2006/04/metadata">
  <types>
    <members>Object 1</members>
    ...
    <members>Object 451</members>
    <name>CustomObject</name>
  </types>
  <types>
    <members>*</members>
    <name>Profile</name>
  </types>
  <types>
    <members>*</members>
    <name>PermissionSet</name>
  </types>
  <version>33.0</version>
</Package>

[alan-morey]

There is currently no way to set any kind of limits to the number of items to be retrieved together for Profiles and PermissionSets.

It's currently broken down into a number of retrieval calls. A package retrieve is done for each of the following metadata types with all Profiles and PermissionSets:

• ApexClass
• ApexPage
• CustomApplication
• CustomObject
• CustomObjectTranslation
• CustomPermission
• CustomTab
• ExternalDataSource
• Layout

This is done to get each of the related permissions for the above listed metadata component types, within each of the Profiles and PermissionSets. The result is a bunch of partial Profile and PermissionSet XML files that are then merged back together to give the complete(?) Profile and PermissionSet.

I will look into further breaking down the retrieval, however the side effect of this will be the redundant retrieval of the above listed components multiple times, which will increase overall backup time.

[jkeener1]

I've made the necessary changes to support something like this today, and I am testing now. One side effect of this is that the xmlmerge isn't necessary. I'll share the code once I finish testing and see if you want to incorporate it.