Fixing Profiles

[WillieRuemmele]

Background

The CLI uses the metadata API for retrieves. Profiles and Permission Sets have the following special behavior—unlike normal metadata type, the API will return a subset of a profile that corresponds to other members of the retrieve request.

Example:
Object__c has field Field__c on it.
myProfile grants access to the object and field with the following

...
"fieldPermissions": [
    {
        "editable": false,
        "field": "Object__c.Field__c",
        "readable": false
    }
],
...
"objectPermissions": [
    {
        "allowCreate": true,
        "allowDelete": true,
        "allowEdit": true,
        "allowRead": true,
        "customizeSetup": null,
        "deleteSetup": null,
        "modifyAllRecords": true,
        "object": "Object__c",
        "viewAllRecords": true,
        "viewSetup": null
    }
]

If I retrieve the Profile and also retrieve the Object and Field the Profile will contain the corresponding fieldPermissions and objectPermissions entries

If I retrieve the Profile without retrieving the Object and Field the Profile won’t contain those entries.

That’s on regular orgs and sandboxes that don’t support source tracking.

What happens on scratch orgs and tracking sandboxes? We return the Profile containing objects/field that have SourceMembers associated with them (that is, they’ve changed)

• user pushes project
• user retrieves profile (it has all the obj/field from the original push since that created SourceMembers)

This is usually “pretty close” since it’s tracking modifications and return full-ish profiles from those orgs

Problem

Now, consider what this user is doing #1396 (also #1624)
They update the Description on the Profile in the org, and want to pull that down.

sfdx force:source:pull will return a profile with no object/field permissions (since only the Profile has changed) and then it writes the Profile with all of those removed from their local version of myProfile.

But, it’s also impossible to say that the user didn’t remove that Object/Field from the Profile. Simply “merging” the missing xml would prevent the user from removing that permission intentionally (which is common when transition from Profiles to PermSets)

This isn’t specific to tracking commands, either. You could do sfdx force:source:retrieve -m Profile:myProfile and also see it remove all of those entries.

If you’re retrieving from a non-tracking org, you won’t get full profiles.

---

Current State

We normally recommend that customers ignore Profiles (via.forceignore) and prefer PermissionSet for dealing with permissions—even though those are subject to the same restriction (will omit parts when not retrieved with those items)

Salesforce, however, still focuses users on Profiles (ex: the field creation wizard always asks which profiles, and has some of them checked by default, so users have to manually remove them each time) Customers do workarounds for these—one of the real-world uses of preretrieve hooks for Fabien/Texei was to look through the local copy of any Profile in a retrieve and patch the retrieve manifest with all the field/object/classes mentioned so that the retrieved Profile would be complete. But We Broke deploy/retrieve hooks . They could still have a hook that runs AFTER the main retrieve/pull and does a 2nd retrieve to get the full profile.

---

Location

Inside SDR, divert profile retrieves to the tooling API, which does retrieve the complete profile. You can verify this with

sfdx force:data:soql:query -t -q "select id, name, metadata from Profile where name = 'System Administrator'" -u someOrg --json

1. SDR would remove those from the component set and query via Tooling API
2. In parallel to the normal retrieve (if there are any non-Profiles) SDR would merge the Tooling API results in, so that in all ways (ex: getFileResponses, errors, warnings, component count) it appears that they were retrieved the normal way and are otherwise identical
3. SDR would also need to write the retrieved Profiles, but that’s simple because they’re already JSON in the tooling API response
4. This would need to be optional because people might be depending on the prior behavior (ex: I have a package.xml that I use to retrieve a subset of objects from a production org and retrieving the entire profile produces something un-deployable to a scratch org because it contains references to objects/fields that aren’t in my source)
5. so that’s a flag on each command to tell it how to handle profiles?
6. the advantage of this is that it works everywhere SDR is used (sfdx, sf, VSCode, second-party plugins, other SDR consumers)

How should we enable this ❓

As #4 explains above, this new functionality should be optional. We know you have built processes and tools around current behavior and would hate to break those so we'll be gating this functionality behind a few different options that we'd love your feedback on:

• A sfdx config value/environment variable (default: false): this would allow users to adopt this new behavior as they see fit
• An sfdx flag on retrieve/pull (default:false): this would allow you to have this new behavior on a per command basis, but be able to use the new functionality easily when they want
• A sf config value/environment variable (default: true): this would enable this new and more consistent behavior for the sf cli, so as you transition you'll see more and more improvements over sfdx
• An sf flag sf metadata retrieve (default:true): would allow you to specify --incompleteProfileRetrieve (we would come up with a more succinct and descriptive name) but this would allow you to maintain backwards compatibility between the sf and sfdx cli.
• An entry in the sfdx-project.json (default: false): this would give the same benefits as a sfdx config value, but also allow teams to adopt new behavior at the same time and create a consistent developer experience. We could also enable this behavior for new projects created with force:project:create
• Enabled by default across all tooling with an environment variable, or config, to restore previous behavior

How would you like to see this implemented? Are there any other metadata types that fit into this category that should be included in a parallel retrieve? Is there anything we've missed, would you like to see something else added to this? Let's discuss!

❤️ the CLI team.

[SCWells72]

First off, thank you for giving this some much needed (and LONG overdue) attention! I can't tell you how many times I've had to explain to my IC2 users how authorization metadata retrieval works and how to work around the behavior. And even then it can be extremely cumbersome because adding all of the necessary metadata to the same request can be difficult...or they just punt and perform a full metadata retrieval to cover all the bases.

The proposed solution seems like it would work well. IC2 uses the CLI for all retrievals in source format projects, and I'll be happy to expose the new option (defaulted to enabled!) for configuration if there's someone out there who wants the old behavior. I might even consider implementing this approach in IC2's native metadata retriever for folks using metadata format projects.

Certainly feel free to let me know when you have a working version so I can create a first-pass integration and provide any feedback as a tooling consumer.

[aheber]

I was trying to figure out why I never have this problem on Perm Sets, though I'm very familiar with it on Profiles.

In API version 40.0 and later, when you retrieve or deploy permission set metadata, all content exposed in Metadata API for the permission sets is included. The metadata includes Apex associated with the permission set, CRUD, and so on.
I forgot it used to do the same thing so maybe not a very modern concern specifically for Perm Sets?

The only other note I have is there are other metadata types with this same behavior, Translations being the one that comes to mind. I don't push/pull many profiles via SFDX and have them on .forceignore but I need to move translations all the time and trying to make sure I pull all of the correct metadata every time is a huge pain. Applying these same behaviors to Translations and anything that has this problem would be a huge help for me. (no idea if the tooling API makes life easier for those other types).

[SCWells72]

The issue does in fact still exist for permission sets as well, though it's better than with profiles. I had this exact discussion with @mshanemc in April and discovered that:

It does seem to do the right thing for Apex classes, but not for custom objects/fields (haven't checked VF pages).

I have some screenshots showing the results of retrieving a permission set with and without the custom objects/fields confirming that the corresponding entries are missing. There was one other interesting insight, though:

Interestingly, if the org has source-tracking enabled, these issues don't exist in the same form. It uses the source tracking metadata to decide what additional things to add to retrieved profiles and permission sets based on what the project might be authorizing.

And the conclusion of that conversation was:

So yeah...not at all consistent, but permission sets are affected still.

Neat...

[aheber]

That explains some things, I'm mostly hanging out in source tracked environments these days. Thanks!

[FabienTaillon]

I like the entry inside sfdx-project.json because this can be committed to git, which means that all developers will use it in a consistent way, even without knowing it. It would also allow to have a different setup from one project to another.

For other options, if developer 1 uses the flag whereas developer 2 doesn't, that would not work as profiles could still become invalid at some point.

Moreover, as this seems to be implement as a low level option, this would also work whereas we're using the command line or VS Code command palette. Beside hooks being broken, the reason I don't use them anymore is that they are not firing if using the command palette.

Looking forward to it !

[WillieRuemmele]

I like the idea of having it in the sfdx-project.json as well

[ImJohnMDaniel]

+1 to @FabienTaillon's suggestion. Give the option to set this feature up in the sfdx-project.json file or give us an optional flag on the commands.

I am not excited about the suggestion to turn this new feature on by default with the sf CLI. I believe that would be a mistake because, to Fabian's and other's points, among project teams the profile metadata file would become un-deployable after a while.

Fundamentally, the issue is not the CLI/Metadata API does or does not retrieve all of the metadata references for a profile. The issue is that the user can only ever have one profile. The user can have multiple permission sets and thus it allows for the separation of concerns/segmentation of responsibilities across various packages and projects. Profiles are org-centric in nature. If you are not managing your source code in an org-centric (e.g. all of the code for the entire org in one repo) there is no way that having the profiles retrieve all metadata references would ever work consistently.

[RupertBarrow]

CumulusCI never uses the sfdx-project.json file. This is a configuration file for the project, not for the CLI.

So it's more logical to rely on an environment variable or a command option.

[WillieRuemmele]

I've had a few conversations with @jayree and he's shown me a few issues he's found when using the tooling api and querying Profiles and Permission Sets. He's said that it won't work exactly the way we thought it would :/ it seems like the tooling api will only return the results we need for custom objects in scratch orgs - he confirmed with a few developer edition and production orgs.

There are also problem with getting the correct objectPermission and userPermission entries

He's spent a lot of time working with these two types, and has authored plugins that will help this scenario. He pointed me to a few different methods that he's written to work around these limitations there

• https://github.com/jayree/sfdx-jayree-plugin/blob/1bcf539edf699f6a99e1013a03aa46d34ccd7a31/src/utils/souceUtils.ts#L80
• https://github.com/jayree/sfdx-jayree-plugin/blob/main/.sfdx-jayree-example.json
• https://github.com/jayree/sfdx-jayree-plugin/blob/main/images/jayree_hooks_flow.svg

Cc: @mshanemc

[jayree]

The issues with objectPermissions and userPermisions in Profiles are the reason why we have to use Permission Sets.

If the metadata file of a Permission Set doesn't contain a setting (objectPermissions or userPermisions) because all boolean values are set to false, this setting will be set to 'false' by the deployment. This is the expected result.

If the metadata file of a Profile doesn't contain a setting (objectPermissions and userPermisions) because all boolean values are set to false, this setting will NOT be set by the deployment. E.g., if you enable some settings and permissions in profiles in an org and then deploy the profiles again assuming that the deployment will disable the settings again, you are wrong.

For Profiles the deployment can't ensure that the correct value for a setting or permission is written to the profile in the org if the setting is not available in the metadata file. Even if you do a full org metadtata retrieve based on a full manifest (force:source:manifest:create --fromorg) objectPermissions and userPermisions with all boolean values set to false are not retrieved and written to the metadata file of a profile.

Thats why I came up with a solution wich injects theses 'false' objectPermissions and userPermisions into the profiles based on a local configuration file.

---

Regarding the tooling api issues. I can't tell why the api seems to behave differently in production orgs. But the missing standard objects might be related to the fact that you can't retrieve standard objects with the force:mdapi:retrieve and a manifest file like

    <types>
        <members>*</members>
        <name>CustomObject</name>
    </types>

as well.

FYI: @WillieRuemmele @mshanemc

[FabienTaillon]

That's actually a big issue the first time you deploy a Profile.

Even though some settings are not set to true in the Metadata, you'll get lots of userPermissions set to true in the org once deployed. I think it's because new Profiles are based on the Standard User Profile as a template, and get all the same permissions even if they are not part of the deployed Profile Metadata. I talked about this recently with Cheryl Feldman, might be interesting to talk about this with her.

What we usually do for new profiles (doesn't happen very often) is that we create them manually by cloning the Minimum Access Profile, and then deploy.

[WillieRuemmele]

Hi @FabienTaillon @jayree @ImJohnMDaniel @SCWells72 @aheber

I've got the first pass of enhancing Profile retrieves done in forcedotcom/source-deploy-retrieve#786. But I found a few issues along the way and I want to see if this is still valuable to you even in it's limited form.

Check out the demo below:
https://www.youtube.com/watch?v=poW3V2X2lYI

I think I ended the demo one command too early, if I ran sfdx force:source:retrieve -m Profile after that last :retrieve -m Profile,ApexClass there would be no differences, so it has a consistent behavior

Given the two issues of

1. It only works for Profiles, not PermissionSets
2. It excludes metadata that is part of a managed package

Would you still like to see this optional functionality added?

[FabienTaillon]

Hi Willie, my 2 cents: if this is a specific command, I won't use it, because:

• we already have a command we build ourselves, also not perfect but working
• the point of your 1st proposal is to have something consistent and transparent for everyone for everywhere (command line, VS Code)

Having something that can be use transparently by everyone can't be done anymore as hooks are dead and anyway were not called by VS Code. Also, with the 1st setup you could just pull and it would work, whereas now we're back with one of the initial problem: each member of the team needs not to forget to run the other command (which is already an issue with our own command).

I wanted to make my own command to run automatically but so far I wasn't able to make it work (basically it updates the Profiles, which is tracked by SFDX as a conflict as Profile metadata has changed locally compared to what was retrieved by the pull command).

We'd love to move away from Profile but today it's not possible as there are still stuff where they are needed (Page Layout assignment, some FlexiPage assignments, Default Custom Application, Default Record Types).

I do agree that this feature wouldn't be perfect, but at least it would be consistent.

[FabienTaillon]

Another point for option 1: it "may" work in environments where people don't have access to the CLI, like DevOps Center (depends on whereas DevOps Center is using the flag in the project, not sure how it's build behind the scene.

[azlam-abdulsalam]

@WillieRuemmele If you are looking as a standalone command, we have been maintaining a few commands as a plugin for sometime. Happy to retire this once its part of the core cli. Could be a good testbed as there was lot of nuances that got fixed by the community.

https://github.com/dxatscale/sfpowerkit#sfpowerkitsourceprofileretrieve

https://github.com/dxatscale/sfpowerkit#sfpowerkitsourceprofilereconcile

https://github.com/dxatscale/sfpowerkit#sfpowerkitsourceprofilemerge

[jefersonchaves]

I agree that option 1 is the best with a project-json toggle to keep consistency across the team. If a beta topic needs to be created, that is ok, but the goal should be to integrate it into the standard topic and standard pull/retrieve.

If a separate command is required, I see no benefit other than continue using tools such as sfpowerkit as mentioned above.

[WillieRuemmele]

Thanks for bringing these up @azlam-abdulsalam!! We're currently just planning on writing the Metadata field of Profiles as the profile xml. We know there's a few inconsistencies/caveats with doing that, but want to get this consistent behavior out there in a way that will support every environment. I'll take a deeper look at your commands, and thanks for sharing them with the community 🎉

[infocynic]

I've been dealing with some pain around this for a few months. I'm in the middle of a [uh, I need a color between brown and green....]-field project that's merging 2 orgs into a clean 3rd, and as part of the shift, has moved away from granting permissions in profiles. So while there are nice options for fixing profiles today, the bigger need and the clear direction SF is going is permission sets. I'm not opposed to finding a quick-win for profiles like the various plugins mentioned earlier here, but we absolutely can't stop there and shrug and say good enough.

I've been working out of a full copy and production org, trying to get everything migrated from the FC to Production, so source tracking based options aren't applicable, though we do have it enabled in our production org, the dev sandboxes are currently so old that the feature was enabled in production after their last refresh, so that's not helping.

If we think about "lowering the barrier to entry" or "reducing unpleasant surprises" when using the CLI, it seems like a logical default would be to have the default behavior pull the entire profile / permission set, while giving the option to override. For that, I like both the project json option -- can be git tracked and easy to just set once and never deal with again, and the CLI switch to override for a single command execution (this option may also be helpful to 3rd parties who develop tooling on top of the CLI where you don't want to rewrite a customer's git repo).

[jefersonchaves]

I agree. I would love to stop guessing or having surprises, especially about profiles.

Breaking down profiles & permission sets into smaller chunks (similar to fields on objects) might be the end game, and I would love to see that - even help.

If the first step is to retrieve complete permission sets and profiles that give me consistent code repositories and deployments, I am all for it.

[WillieRuemmele]

Thanks for the input!

Breaking down profiles/perm sets/custom labels... there's a lot of different metadata support that would benefit from better source-format. That's out of scope right now, but I've had a few ideas of how we might achieve that.

[jefersonchaves]

Fair enough @WillieRuemmele and thanks again for the hard work.

One step at a time is a good approach, consistency first. Then breaking it down sounds good.

[pgonzaleznetwork]

How is using the Tooling API different from using the CRUD Metadata API (Read)? There's a project that already solves this problem https://github.com/amtrack/sfdx-plugin-source-read

[ps-carl-bussema]

Which retrieves permissions for standard tabs, which are not deployable, so those have to be removed every time, but otherwise this has been my go-to workaround for now.

[pgonzaleznetwork]

How do you remove them? Manually?

[gitmevg]

@azlam-abdulsalam we have been using your (ACN) plugin (sfpowerkit) and it works like a charm (lifesaver) for full profile retrieve but somehow reconcile and merge aren't that well supported (may be due to some dependencies), i myself am working on a Custom solution.

Many times I feel developers don't use force:source:tracking:reset as a good settler.

There are many plugins that solve the problem which we are trying to address here unless there is a scalability and a vision roadmap because in one way or the other there is no robust one without workarounds

It's essential to write Gherkin and streamline the milestones.

[gitmevg]

But if it's coming anyways, my preference would be --incompleteProfileRetrieve

[ps-carl-bussema]

Yeah a simple find/replace in VSCode takes care of it.

…

On Thu, Dec 22, 2022 at 6:37 PM Pablo Gonzalez ***@***.***> wrote: How do you remove them? Manually? — Reply to this email directly, view it on GitHub <#1785 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3OVKADSSMYLU5MQMPAF2VLWOTQ5BANCNFSM6AAAAAARUPJSUE> . You are receiving this because you commented.Message ID: ***@***.***>

[RupertBarrow]

We need the same approach for CustomLabels and Translations.

Consider both the first retrieve operation and well as subsequent retrieves, whether metadata has been added or deleted in the meantime.
Consider also cases when you have multiple labels files in multiple subfolders of force-app. When you retrieve again, you don't want them bloated with more labels than necessary; I would like an option in the tetrieve/pull command to retrieve all labels of all sibling folders and their subfolders.

When you deploy labels, we need to make sure that multiple label files in the same force-app folder structure do not contain the same label with different values.
I would like the deploy/push command to succeed with a warning if the same label was defined in multiple areas with identical values. It could fail when running in --checkonly mode.
If the label was defined in multiple subfolders with different values, deploy or push should fail.

All this also applies to translations.

[WillieRuemmele]

Hey Everyone 👋

After a discussion with the team we've decided to halt working on this feature.

We came to this conclusion for a few reasons...

• On-going support: The Salesforce platform is moving away from Profiles in general, and moving towards Permission Sets.
• limited functionality: as we discovered, and elaborated on above, this feature wouldn't fulfill our original goal, and the benefits were much reduced. We didn't want to introduce a niche piece of code to partially solve a use case that not everyone was experiencing.
• community plugins: As mentioned in the discussion above, many people have come together and shared different plugins and approaches they have when dealing with this metadata. We loved seeing this and hoped it lead many of you to find a new plugin. We're brainstorming ways we can increase community plugin awareness, and have learned from this discussion.

While we know this news will always be disappointing to hear, we hope you understand.

Thanks for being a part of the CLI Community ❤️ 🚀