Fixes for minor defects with Jenkins flavor of the pipeline (#148)

* Improve support for Git SSH key with local Jenkins

The jenkins/start.sh now recognizes option -key <path_to_private_key>
and uses it to create the SSH Git credentials when starting new local
Jenkins instance for the first time.

The #137 is fixed by adding missed argument to the jenkins/Dockerfile.

Also the documentation now reflects the additional parameter to start.sh

* Fix #134 by using OS-specific file path separator

* Fix #138 by encapsulating SCM config in one place

Added a unit test to reproduce the issue and refactored logic in the
jenkins_pipeline_sample.groovy to use the new extension function to
configure SCM block for all jobs.

* Fix #141

Fixed typos in the marker of the end of the K8S block for removal.

Fixed pipeline tests to not assume existence of either K8S or CF
scripts.

Fixed JobScripts tests to only verify availability of the CF/K8S jobs in
case of them not being excluded during the customization.

* Convert the code in fixes to be type-safe

Author: oiavorskyi

README.adoc

@@ -1553,6 +1553,9 @@ any manual work on the Jenkins side. In the above parameters, the third paramete
 could be yourForkedGithubOrg or yourGithubUsername. Also the `REPOS` env variable will
 contain your GitHub org in which you have the forked repos.
 
+Instead of the Git username and password parameters you could pass `-key <path_to_private_key>`
+if you prefer to use the key-based authentication with your Git repositories.
+
 [[deploy-infra-cf]]
 ===== Deploy the infra JARs to Artifactory
 
@@ -1879,6 +1882,9 @@ any manual work on the Jenkins side. In the above parameters, the third paramete
 could be yourForkedGithubOrg or yourGithubUsername. Also the `REPOS` env variable will
 contain your GitHub org in which you have the forked repos.
 
+Instead of the Git username and password parameters you could pass `-key <path_to_private_key>`
+if you prefer to use the key-based authentication with your Git repositories.
+
 You need to pass the credentials for the Docker organization (by default we will
 search for the Docker images at Docker Hub) so that the pipeline will be able
 to push images to your org.
@@ -5366,4 +5372,4 @@ version of the release.
 docker login
 docker build -t springcloud/spring-cloud-pipeline-jenkins:<version> ./jenkins
 docker push springcloud/spring-cloud-pipeline-jenkins:<version>
-----
+----

buildSrc/src/test/groovy/org/springframework/cloud/pipelines/ProjectCustomizerSpec.groovy

@@ -9,6 +9,8 @@ import spock.lang.Specification
 
 import org.springframework.boot.test.rule.OutputCapture
 
+import java.util.regex.Matcher
+
 /**
  * @author Marcin Grzejszczak
  */
@@ -59,15 +61,15 @@ class ProjectCustomizerSpec extends Specification {
 		then:
 			String logs = capture.toString()
 			//
-			!logs.contains("buildSrc/foo.bash")
+			!logs.contains(filePath("buildSrc/foo.bash"))
 			// we want to remove only Concourse stuff
-			!logs.contains("/jenkins,")
-			logs.contains("images/concourse")
-			logs.contains("CONCOURSE.adoc")
-			logs.contains("CF_DEMO.adoc")
-			logs.contains("CF_JENKINS")
-			logs.contains("tools/cf-helper")
-			logs.contains("bash/pipeline-cf")
+			!logs.contains(filePath("/jenkins,"))
+			logs.contains(filePath("images/concourse"))
+			logs.contains(filePath("CONCOURSE.adoc"))
+			logs.contains(filePath("CF_DEMO.adoc"))
+			logs.contains(filePath("CF_JENKINS"))
+			logs.contains(filePath("tools/cf-helper"))
+			logs.contains(filePath("bash/pipeline-cf"))
 			logs.contains("Removing lines")
 			logs.contains("Removing files")
 		and:
@@ -90,15 +92,15 @@ class ProjectCustomizerSpec extends Specification {
 		then:
 			String logs = capture.toString()
 			//
-			!logs.contains("buildSrc/foo.bash")
+			!logs.contains(filePath("buildSrc/foo.bash"))
 			// we want to remove only Concourse stuff
-			!logs.contains("/concourse,")
-			logs.contains("images/jenkins")
-			logs.contains("JENKINS.adoc")
-			logs.contains("K8S_DEMO.adoc")
-			logs.contains("K8S_JENKINS")
-			logs.contains("tools/k8s-helper")
-			logs.contains("bash/pipeline-k8s")
+			!logs.contains(filePath("/concourse,"))
+			logs.contains(filePath("images/jenkins"))
+			logs.contains(filePath("JENKINS.adoc"))
+			logs.contains(filePath("K8S_DEMO.adoc"))
+			logs.contains(filePath("K8S_JENKINS"))
+			logs.contains(filePath("tools/k8s-helper"))
+			logs.contains(filePath("bash/pipeline-k8s"))
 			logs.contains("Removing lines")
 			logs.contains("Removing files")
 		and:
@@ -109,4 +111,8 @@ class ProjectCustomizerSpec extends Specification {
 			}
 	}
 
+	String filePath(String canonicalPath) {
+		return canonicalPath.replaceAll("/", Matcher.quoteReplacement(File.separator))
+	}
+
 }

buildSrc/src/test/resources/project_customizer/docs-sources/src/main/asciidoc/CF_JENKINS.adoc

@@ -61,6 +61,9 @@ any manual work on the Jenkins side. In the above parameters, the third paramete
 could be yourForkedGithubOrg or yourGithubUsername. Also the `REPOS` env variable will
 contain your GitHub org in which you have the forked repos.
 
+Instead of the Git username and password parameters you could pass `-key <path_to_private_key>`
+if you prefer to use the key-based authentication with your Git repositories.
+
 [[deploy-infra-cf]]
 ===== Deploy the infra JARs to Artifactory
 

buildSrc/src/test/resources/project_customizer/docs-sources/src/main/asciidoc/K8S_JENKINS.adoc

@@ -61,6 +61,9 @@ any manual work on the Jenkins side. In the above parameters, the third paramete
 could be yourForkedGithubOrg or yourGithubUsername. Also the `REPOS` env variable will
 contain your GitHub org in which you have the forked repos.
 
+Instead of the Git username and password parameters you could pass `-key <path_to_private_key>`
+if you prefer to use the key-based authentication with your Git repositories.
+
 You need to pass the credentials for the Docker organization (by default we will
 search for the Docker images at Docker Hub) so that the pipeline will be able
 to push images to your org.

common/src/test/bats/fixtures/sc-pipelines-generic.yml

@@ -0,0 +1,30 @@
+test:
+  services:
+    - type: broker
+      name: rabbitmq-github-webhook
+      broker: foo
+      plan: bar
+    - type: broker
+      name: mysql-github-webhook
+      broker: foo
+      plan: bar
+    - type: app
+      name: eureka-github-webhook
+      coordinates: com.example.eureka:github-eureka:0.0.1.M1
+    - type: stubrunner
+      name: stubrunner
+      coordinates: com.example.github:github-analytics-stub-runner-boot-classpath-stubs:0.0.1.M1
+      pathToManifest: foo/manifest.yml
+stage:
+  services:
+    - type: broker
+      name: github-rabbitmq
+      broker: foo
+      plan: bar
+    - type: broker
+      name: mysql-github
+      broker: foo
+      plan: bar
+    - type: app
+      name: github-eureka
+      coordinates: com.example.eureka:github-eureka:0.0.1.M1

common/src/test/bats/pipeline.bats

@@ -28,7 +28,7 @@ teardown() {
 @test "yaml2json should convert valid YAML file to JSON" {
 	source "${SOURCE_DIR}/pipeline.sh"
 
-	run yaml2json "${FIXTURES_DIR}/sc-pipelines-cf.yml"
+	run yaml2json "${FIXTURES_DIR}/sc-pipelines-generic.yml"
 
 	assert_success
 }
@@ -42,24 +42,24 @@ teardown() {
 }
 
 @test "deployServices should deploy services from pipeline descriptor" {
-	export PIPELINE_DESCRIPTOR="sc-pipelines-k8s.yml"
+	export PIPELINE_DESCRIPTOR="sc-pipelines-generic.yml"
 	source "${SOURCE_DIR}/pipeline.sh"
 
 	cd "${FIXTURES_DIR}"
 
 	run deployServices
 
 	assert_success
-	assert_line --index 0 'rabbitmq-github-webhook rabbitmq'
-	assert_line --index 1 'mysql-github-webhook mysql'
-	assert_line --index 2 'eureka-github-webhook eureka'
-	assert_line --index 3 'stubrunner-github-webhook stubrunner'
+	assert_line --index 0 'rabbitmq-github-webhook broker'
+	assert_line --index 1 'mysql-github-webhook broker'
+	assert_line --index 2 'eureka-github-webhook app'
+	assert_line --index 3 'stubrunner stubrunner'
 }
 
 @test "parsePipelineDescriptor should export an env var with parsed YAML" {
 	source "${SOURCE_DIR}/pipeline.sh"
 
-	PIPELINE_DESCRIPTOR="${FIXTURES_DIR}/sc-pipelines-cf.yml"
+	PIPELINE_DESCRIPTOR="${FIXTURES_DIR}/sc-pipelines-generic.yml"
 
 	assert_equal "${PARSED_YAML}" ""
 

docs-sources/src/main/asciidoc/CF_JENKINS.adoc

@@ -61,6 +61,9 @@ any manual work on the Jenkins side. In the above parameters, the third paramete
 could be yourForkedGithubOrg or yourGithubUsername. Also the `REPOS` env variable will
 contain your GitHub org in which you have the forked repos.
 
+Instead of the Git username and password parameters you could pass `-key <path_to_private_key>`
+if you prefer to use the key-based authentication with your Git repositories.
+
 [[deploy-infra-cf]]
 ===== Deploy the infra JARs to Artifactory
 

docs-sources/src/main/asciidoc/K8S_JENKINS.adoc

@@ -61,6 +61,9 @@ any manual work on the Jenkins side. In the above parameters, the third paramete
 could be yourForkedGithubOrg or yourGithubUsername. Also the `REPOS` env variable will
 contain your GitHub org in which you have the forked repos.
 
+Instead of the Git username and password parameters you could pass `-key <path_to_private_key>`
+if you prefer to use the key-based authentication with your Git repositories.
+
 You need to pass the credentials for the Docker organization (by default we will
 search for the Docker images at Docker Hub) so that the pipeline will be able
 to push images to your org.

docs-sources/src/main/jekyll/index.html

@@ -126,7 +126,8 @@
 * Clone the `spring-cloud-pipelines` project
 * Go to `jenkins` folder
 * Run `./start.sh yourGithubUser yourGithubPass yourGithubOrg` to start Jenkins and Artifactory (`yourGithubUser` and `yourGithubPass`
-are your Github credentials required by the pipeline to tag the forked repo that lays in `yourGithubOrg`)
+are your Github credentials required by the pipeline to tag the forked repo that lays in `yourGithubOrg`;
+you could also pass `-key path_to_private_key` if you prefer to use the key-based authentication with your Git repositories)
 * Run `./deploy_infra.sh` to upload to your Artifactory the infrastructure apps (`eureka` and `stub runner`)
 * Go to Jenkins and click the `jenkins-pipeline-seed` in order to generate the pipeline jobs
 * Go to the `github-webhook` pipeline view and click the `play` button to run the pipeline

docs/index.html

@@ -126,7 +126,8 @@
 * Clone the `spring-cloud-pipelines` project
 * Go to `jenkins` folder
 * Run `./start.sh yourGithubUser yourGithubPass yourGithubOrg` to start Jenkins and Artifactory (`yourGithubUser` and `yourGithubPass`
-are your Github credentials required by the pipeline to tag the forked repo that lays in `yourGithubOrg`)
+are your Github credentials required by the pipeline to tag the forked repo that lays in `yourGithubOrg`;
+you could also pass `-key path_to_private_key` if you prefer to use the key-based authentication with your Git repositories)
 * Run `./deploy_infra.sh` to upload to your Artifactory the infrastructure apps (`eureka` and `stub runner`)
 * Go to Jenkins and click the `jenkins-pipeline-seed` in order to generate the pipeline jobs
 * Go to the `github-webhook` pipeline view and click the `play` button to run the pipeline

jenkins/Dockerfile

@@ -6,6 +6,7 @@
 FROM jenkins:2.19.1
 ARG gituser=changeme
 ARG gitpass=changeme
+ARG gitsshkey=changeme
 ARG dockerRegistryOrg=changeme
 ARG dockerRegistryUser=changeme
 ARG dockerRegistryPass=changeme

jenkins/README.adoc

@@ -298,6 +298,9 @@ any manual work on the Jenkins side. In the above parameters, the third paramete
 could be yourForkedGithubOrg or yourGithubUsername. Also the `REPOS` env variable will
 contain your GitHub org in which you have the forked repos.
 
+Instead of the Git username and password parameters you could pass `-key <path_to_private_key>`
+if you prefer to use the key-based authentication with your Git repositories.
+
 [[deploy-infra-cf]]
 ===== Deploy the infra JARs to Artifactory
 
@@ -624,6 +627,9 @@ any manual work on the Jenkins side. In the above parameters, the third paramete
 could be yourForkedGithubOrg or yourGithubUsername. Also the `REPOS` env variable will
 contain your GitHub org in which you have the forked repos.
 
+Instead of the Git username and password parameters you could pass `-key <path_to_private_key>`
+if you prefer to use the key-based authentication with your Git repositories.
+
 You need to pass the credentials for the Docker organization (by default we will
 search for the Docker images at Docker Hub) so that the pipeline will be able
 to push images to your org.
@@ -1268,4 +1274,4 @@ REUSE_CF_LOGIN=true ./tools/cf-helper.sh setup-prod-infra
 ```
 
 This script will create the mysql db, rabbit mq service, download and deploy Eureka
-to the space and organization you're logged into.
+to the space and organization you're logged into.