force:apex:test:run in JWT context seems to rely on the jwtkeyfile always being present #81
Comments
|
The error message that I get is the following: |
|
Other commands that are affected by this are:
|
|
Is this an issue where the jwt connection works for a while and then the CLI needs to send a refreshtoken request and when that request occurs, it needs the jwtkeyfile to make that refresh request? Is there a way to observe this? Does this suggest that the connected app on Salesforce is not configured correctly? |
|
This is working as designed. Calling Salesforce APIs requires an access token. Access tokens are valid for a certain amount of time before they expire. In the case of JWT auth there are no refresh tokens. When the access token expires, the CLI will use the persisted auth data to get a new, valid access token and update the local auth data so that API calls will succeed. |
|
@shetzel thanks for the update. That may be that it is working as designed, but it breaks the use case that I mentioned above about the "....Jenkins CI processes on SFDX projects" and that Salesforce recommends that the management of the jwtKeyFile on Jenkins is to store it in the "Jenkins Admin Credentials interface." (see documentation here, section 2b). The "Jenkins Admin Credentials interface" stores the file in another directory outside of the Jenkins workspace folder and thus the error. The Also, why would it be that I am able to execute a couple of commands (i.e. |
|
Sounds like there might still be an issue if you were able to run some commands and not others. Also maybe the documentation on Jenkins is wrong the way we are suggesting you set up isn't correct. @shetzel could you shed some light on why the other commands worked but test:run failed? |
clairebianchi
added
the
waiting for internal reply
|
Commands will work as long as the token is valid or if the command doesn't make an API call, such as As for Jenkins, we had a similar issue but it was resolved by wrapping all the steps inside a withCredentials block in order to guarantee that the location Jenkins creates to store the jwt key file remains the same. If multiple withCredential blocks are used then jenkins may actually place the key file in a different spot, and the persisted auth files that the CLI uses will point to an invalid jwtkeyfile location. Try using a single withCredentials block in your Jenkins script. |
|
@shetzel - the I am not certain how wrapping the Thoughts? |
|
As long as everything is wrapped in the same ` } |
|
@shetzel -- a couple of questions:
|
|
@shetzel -- additional question would be:
|
|
re: jenkins - Seems like you're using the newer, declarative method, and it appears that the withCredentials plugin has issues with that method. I don't have any experience with it so doubt I'll be much help there. The CLI doesn't need to connect to the DevHub for each command. The auth for the scratch org uses the same flow as for the devhub, as opposed to the auth code from the scratch org signup request (in which case it would follow the refresh token flow). API requests to the scratch org should be using the access token obtained for the initial scratch org auth. It's possible that the jwtkeyfile is being checked when the access token is still valid, which sounds like what you're describing. Assuming there is no access token expiration happening between |
|
@shetzel -- That is strange. I see now that, if you have a devhubdefault connection that was setup via JWT, then any scratch org created by that connection will also have a JWT File reference -- accessToken and privateKey. If you have a devhubdefault connection that was setup via the Web Flow, the scratch org created has a accessToken/refreshToken setup in its definition file in the Do you know what the reason is to make that kind of distinction between DevHub-WebAuthFlow scratch orgs and DevHub-JWTFlow scratch orgs? Why would you not want to always use the accessToken/refreshToken setup for scratch orgs? |
|
The decision predates me but I believe it was done that way for customer convenience. I reproduced the issue you're seeing and entered a bug for us to track internally. I would try working around the issue from the Jenkins side of things until the fix is released. |
|
@shetzel -- I mentioned to @clairebianchi that I have a Salesforce support ticket active on this issue. It is case number 22563597. |
|
@shetzel -- you mentioned that it would be "....done that way for customer convenience", but you are not certain. I hear ya. Personally, I cannot, at the moment, think of a use case where having the scratch org authenticate via JWT would be helpful at all, but as you point out, there may be a reason. If there is such a reason, I would love to know what that is. My current hunch is that it is closer to a basic design flaw in the approach of how scratch org connections are managed when they are created from a devhub-jwt-type-connnection. I really do expect that we would find that the original CLI Command code was setup to match the devhub connection type for no specific reason. Personally, I am thinking that scratch orgs created from a devhub-jwt-type-connnection should follow the same OAuth connection setup that devhub-webauth-type-connections would setup their scratch orgs with. That would definitely eliminate this issue that I am seeing here on Jenkins and allow SFDX CLI to be used in all ways that Jenkins would consider best practices for CI builds. Any thoughts on this? Could the way that the CLI manages scratch orgs created from a devhub-jwt-type-connnections be switched? Is there any reason that this could not be done? As always, I appreciate the help on these matters!!! Cheers! cc: @clairebianchi |
|
@clairebianchi, @shetzel, and @sfdc-db-gmail -- Any update on this issue? |
|
@ImJohnMDaniel I have this on our backlog and hope to get it picked up by the end of September |
clairebianchi
removed
the
waiting for internal reply
|
@clairebianchi -- Thanks for the update. |
|
The behavior here seems to have changed again - it used to be possible to logout and back into the hub to update the cached path to the jwt file, but it now seems like the path to the file at the time of creating an org is also now pinned - and I'm not aware of a way to update that. This is not every command. Copying the jenkins temp path to the key to a relative path at the start of the jobs requiring it solves the issue. |
|
Hi All, I am trying to use the mdapi deploy command and I am getting the same issue. Org Authorization is success however during mdapi deploy command it is failing saying: Even I am written the script wrapped in the same withCredentials block. Can anyone help on this? |
|
@clairebianchi - Could you please help with the workaround? |
|
@sanpatnaik Here is the known issue that is logged for this https://success.salesforce.com/issues_view?id=a1p3A000001SGxLQAW We were unable to get to this issue last month, it is on our board and I am hoping it will get picked up in the next few weeks. |
|
@sanpatnaik Hello Friend! I was trying to do a similar thing, calling the mdapi deploy command. I gave up after a couple hours of trying to use a jenkinsfile to try this workaround. I prefer using Freestyle Jenkins projects because they are WAY easier to set up.
export SFDX_USE_GENERIC_UNIX_KEYCHAIN=true
cd test-ci
echo y | /usr/local/bin/sfdx force:auth:logout --targetusername my-devhub
/usr/local/bin/sfdx force:auth:jwt:grant --clientid {myclientId} \
--jwtkeyfile $JWT_AUTH_KEY --username {myusername} \
--setdefaultdevhubusername --setalias my-devhub
/usr/local/bin/sfdx force:mdapi:deploy -d mdapi-out -u my-devhub -w -1 -c |
|
Hi @ImJohnMDaniel are you still experiencing this issue? I've tried reproducing it on my local CLI and can't reproduce it. Have you tried running these commands through your own CLI and not Jenkins? |
|
@williamruemmele-sf and @clairebianchi -- I just checked again and this issue is still present. I was able to reproduce the issue locally; outside of Jenkins. Here is the CLI version info that I am currently on... sfdx version sfdx plugins --core FWIW, I am happy to catch up on a call to discuss this and show you what is happening. Just let me know. |
Hello - Could you please provide more details on this as I am still facing the issue as John already mentioned - would love to have a workaround! Are you trying to say, authorize, logout and then re-authorize? |
|
@siddharthmani -- The main issue stems from the fact that when you authenticate to the DevHub with a JWT token, the CLI using that connection will create all scratch orgs also with a JWT token. Best practices in Jenkins and declarative pipelines with Jenkinsfile usage don't really work well with this approach. The best practice assumes that you use the I can confirm that there is movement on this issue. I talked with @clairebianchi and @williamruemmele-sf yesterday about this issue. I gave them my recommendation that scratch orgs should always be created with a standard WebFlow OAuth approach and not a JWT token based approach. Connections to DevHubs in a CI "headless" scenario should definitely still use the JWT token based approach. They were going to discuss with their teams let us know what they plan to do as a next step. In the meantime, if you need a work around, what I did was find where the CI Server is putting the hidden JWT_AUTH_KEY file to expose it to the build. Then copy that file to a location inside the project workspace during the |
|
Thank you so much for the detailed explanation John. Indeed my build also fails with the same error message although I am not creating any scratch orgs as of now. What I am essentially doing consists of 2 stages (both wrapped in "withcredentials") - the first one is the jwt authorisation which succeeds and the second one is the actual force:source:deploy command which fails mentioning that it cannot find the key. |
Hello! My issue in particular was that, once authorized, any additional builds will try to use the same authentication, because SFDX doesn't remove authorized orgs. I am hosting jenkins on an ec2, so not locally. I log out of the org in the beginning of the build so that it is forced to re-authorize the org using the secret file. Because the secret file is only in a particular location for the duration of the build, if you don't logout and then reauthorize, any commands that require authorization will look in the location from the first time you authorized the org to find the authentication file (the secret file). |
|
@jc-torrent -- It sounds like you have a variation of the issue that I am describing. Once the scratch org is created for that build job, it should not require the secret file to be present to interact with the scratch org. Unfortunately, the issue is that OAuth authorization for the scratch org was setup as a JWT based connection and thus is looking for the secret file. If the scratch org OAuth were setup as WebFlow, we would have none of these issues. FYI -- @williamruemmele-sf |
|
@ImJohnMDaniel Thank you for describing this problem, John. I also ran into this problem. Could you send me Jenkinsfile example of how resolve this problem? Thanks a lot. |
|
@singlifyautomationqa -- The work around it pretty straightforward. When you are in the This is a workaround until @williamruemmele-sf, @amphro, and @clairebianchi's team are able to correct this behavior. Please be aware that this does represent a minor security risk to your DevHub. |
|
@clairebianchi -- I wanted to throw my use case in the ring for consideration as it's also impacted by this. Our team has an automated process that uses the JWT auth flow to create scratch orgs for our developers. When someone begins working on a feature, they make a branch and get a pre-built scratch org from our DevHub. Trouble is that they can't do anything with the scratch org because it was created in the JWT auth flow. Anytime they try to |
|
@thebrettbarlow Thank you for adding your use case, this is very helpful. We are currently working on a solution for this. We have a path forward but need to go through a security review before we can proceed, hopefully, I will have good news next week |
|
I believe this is related: #124. I just encountered this yesterday. There was no change to Jenkins and/or SFDX. It just started. I'm leaning for this being a Jenkins issue for the reason that I've deleted projects, added server.key to repo, created an additional jenkins secret file and no matter what I do, the output still points to a project in another directory. I'm kind of freaking out because I have no path forward right now. It was working yesterday morning. It stopped working yesterday afternoon. |
|
@cumulus-robert -- I agree. The two issues are related. Essentially the root issue is that when you have a JWT connection to your DevHub and you create a scratch org from that connection, it too is setup as a JWT connection which requires the server.key to be present for any interaction to that scratch org. @clairebianchi and crew are looking into it! |
|
I'm not working in scratch orgs, technically. I'm working in Production orgs with JWT and in this instance, I'm trying to deploy change to a sandbox. Again, this just started happening around mid-day yesterday. Deployments worked in AM, stopped working in PM. No change to our technology. Nothing I'm doing is working. I've added server.key to repo (terrible practice wouldn't pass smell test) and Jenkins still can't find it. I have to log in to CLI, traverse to workspace and perform SFDX manual commands in order for things to move forward. |
|
@cumulus-robert , this has been my issue for over a month now. I've also tried out different solutions, including the one mentioned by John Daniel but I couldn't get it to work. So at this point awaiting on response from SFDX team as mentioned by Claire. |
|
Hello to everyone on this thread. We've changed the way that scratch orgs are authenticated to the DevHub, so in the next release of the CLI this issue should be resolved. |
|
I appreciate the response. I don’t appreciate my inability to run deployments all of a sudden when nothing changed in my infrastructure having tried and true scripts.
I hope this is resolved soon. I’m dead in the water for everything I do in my practice using Jenkins.
There needs to be better change management in delivery so there is no impact to business. Salesforce is better than this.
The other issue I linked to has a message that something will be released in 1/9. I’m
Hopeful this gets resolved then.
…Sent from my iPhone
On Jan 8, 2020, at 7:26 AM, williamruemmele-sf ***@***.***> wrote:
Hello to everyone on this thread. We've changed the way that scratch orgs are authenticated to the DevHub, so in the next release of the CLI this issue should be resolved.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
Well, What I did was.
|
|
@RobeDevOps This worked for me. Thanks for the information. Linking to #363. |
|
On a similiar note,executing a logout at the end of each ci build solved my problem. its similiar to deleting the .sfdx folder workaround sfdx force:auth:logout --noprompt --targetusername= |
|
@clairebianchi, @shetzel, @amphro -- When you have a moment, can you provide an update on this issue please? |
|
@clairebianchi, @shetzel, @mshanemc -- When you have a moment, can you provide an update on this issue please? |
|
@ImJohnMDaniel, sorry for the delay in the reply. Can you please confirm if you are still looking for a resolution on this. Thanks! |
|
G'day @preddivari -- thanks for reaching out. AFAIK, this issue remains to this day. It is a complicated issue that I have discussed with @mshanemc and others over time. Happy to review it with you if that would help. Cheers! |
cristiand391
added
bug
|
This issue has been linked to a new work item: W-15040090 |
Summary
force:apex:test:run command fails when using JWT connection and the jwtkeyfile file is missing from project folder
Steps To Reproduce:
force:org:create --definitionfile config/project-scratch-def.json --json --setdefaultusername --durationdays 1force:source:push --jsonforce:apex:test:run --testlevel RunLocalTests --outputdir target --resultformat tap --jsonforce:source:push --jsonforce:apex:test:run --testlevel RunLocalTests --outputdir target --resultformat tap --jsonExpected result
Execution of
force:apex:test:runshould succeed regardless of the presence of the jwtKeyFileActual result
Execution of
force:apex:test:runfails if jwtKeyFile is not present in project directory.Additional information
This becomes a blocker in Jenkins CI processes on SFDX projects. The recommended way to manage the jwtKeyFile on Jenkins is store it in the "Jenkins Admin Credentials interface." During build job execution, Jenkins will checkout all code to the build job's "workspace" folder. It will download the the jetKeyFile and other secret files to the adjacent folder "workspace@tmp" and inject that file during commands that explicitly utilize it (like the
force:auth:jwt:grantcommand).As it stands now, I am unable to utilize
force:apex:test:runas part of my CI process.SFDX CLI Version(to find the version of the CLI engine run sfdx --version):
SFDX plugin Version(to find the version of the CLI plugin run sfdx plugins --core)
OS and version:
The text was updated successfully, but these errors were encountered: