The Security Assertion Markup Language (SAML) Conformance Test Kit (CTK)

The SAML Conformance Test Kit is intended to be a set of blackbox tests that verify the conformance of an Identity Provider (IdP) to the SAML V2.0 Standard Specification.

Note the following before running the tests:

• This test kit only supports SAML Version 2.0. All other versions are not supported.
• This test kit does not support proxying.
• Currently, the CTK only tests POST and Redirect bindings.
• Single Sign-On and Single Logout are the only two protocols tested.
• Only statements containing the keywords MUST, MUST NOT and REQUIRED are tested.
• Some MUSTs are hard to test. For a full list, visit the Not Tested List.
• This test kit includes built-in support for Keycloak and Distributed Data Framework (DDF) Identity Providers.

Building

To build the project execute gradlew build. After the build, the distribution zip will be located at deployment/distribution/build/distributions. The unzipped distribution can be found at deployment/distribution/build/install/samlconf.

Running

Connect to an Identity Provider

The SAML-CTK includes built-in support for Keycloak, OpenAM, and Distributed Data Framework (DDF). If the IdP being tested is not Keycloak, OpenAM, or DDF see Developing a plugin to test your IdP.

In general, to connect an IdP with the CTK, the IdP needs the CTK SP metadata and vice-versa. If you are testing Keycloak, OpenAM, or DDF, see their corresponding documentation on how to connect them to the test kit:

• Keycloak Documentation
• OpenAM Documentation
• DDF documentation

Run the tests

After a successful gradle build, tests can be run with the generated samlconf scripts under deployment/distribution/build/install/samlconf/bin. Run the executable samlconf (*NIX) or samlconf.bat (Windows).

The samlconf script may take the following parameters:

NAME
       SamlConf - Runs the SAML Conformance Tests against an IdP

SYNOPSIS
       samlconf [arguments ...]

DESCRIPTION
       Runs the SAML Conformance Tests which test the compliance of an IdP with the SAML Specifications.
       If a compliance issue is identified, a SAMLComplianceException will be thrown with an explanation
       of the error and a direct quote from the specification. Tests will not run if the corresponding
       endpoints do not exist in the IdP's metadata. All of the parameters are optional and if they are
       not provided, the default values will use Distributed Data Framework (DDF)'s parameters.

OPTIONS
       -ddf, --ddf
            Runs the DDF profile. If provided runs the optional SAML V2.0 Standard
            Specification rules required by DDF.

       -debug, --debug
            Enables debug mode which enables more logging. This mode is off by default.

       -h, --help
	        Displays the possible arguments.

       -i path, --implementation=path
            The path to the directory containing the implementation's plugin and metadata.
            The default value is /implementations/ddf.

       -l, --lenient
            When an error occurs, the SAML V2.0 Standard Specification requires an IdP to
            respond with a 200 HTTP status code and a valid SAML response containing an
            error <StatusCode>.
            If the -l flag is given, this test kit will allow HTTP error status codes as
            a valid error response (i.e. 400's and 500's).
            If it is not given, this test kit will only verify that a valid SAML error
            response is returned.

       -q, --quiet
            If provided, only displays whether a test or a section passed or failed.
            Errors will not be printed.

       -u username:password, --userLogin=username:password
            The username and password to use when logging in.
            The default value is admin:admin.

Developing

Developing a plugin

Before testing an IdP, it has to be connected with the SAML-CTK. To do so, a custom plugin is required.

Why is a plugin required?

When logging in to an IdP using the Single Sign-On protocol, there are 6 steps:

1. HTTP Request to Service Provider
2. Service Provider Determines Identity Provider
3. <AuthnRequest> issued by Service Provider to Identity Provider
4. Identity Provider identifies Principal
5. Identity Provider issues <Response> to Service Provider
6. Service Provider grants or denies access to Principal

The specification mentions that in step 4, "the principal is identified by the identity provider by some means outside the scope of [the Single Sign-On] profile". This means that the CTK doesn't know how to give the IdP the principal's credentials, since the authentication step is implementation specific.
Therefore each IdP must write a plugin which will process the authentication of a principal. See 4.1 Web Browser SSO Profile for more information on the different steps for the SSO profile.

How to implement a plugin jar?

1. Implement a plugin jar for the IdP's authentication implementation.
   Write a Java or Kotlin class that implements the IdpSSOResponder interface and has "org.codice.samlconf.implementations" as the groupID.
   DDFIdpSSOResponderProvider, OpenAMIdPSSOResponderProvider, and KeycloakIdpSSOResponderProvider can be used as examples (see Examining the DDF plugin jar).
   NOTE: The implementation class is implementing a Java service therefore either
   • the @MetaInfServices annotation should be used (which uses the kapt plugin) or
   • a hand-crafted services reference file META-INF/services/org.codice.compliance.saml.plugin.IdpSSOResponder will need to be added to the jar (see below)
2. Package that file into a jar.
3. Place the jar from step 1 and the IdP's metadata into a directory.
4. Setup your IdP.
5. Configure your IdP with the test kit's SP metadata.
6. The directory from step 3 should be referred to with the -i option when running tests. (See Run the tests)

Examining the DDF plugin jar

In the case of DDF's IdP plugin:

1. The plugin implements the IdpSSOResponder interface.
2. The plugin receives the login page from DDF's IdP as an HTTP response in the interface methods.
3. The plugin parses everything it needs from the login page and gets ready to send a request back to DDF's IdP.
4. The plugin sends an HTTP request to the proper location, with the principal's credentials, in the format that DDF's IdP expects the resulting request.

Formatting

The project uses Spotless to ensure consistent style. Any style violations noted by Spotless can easily be resolved by running ./gradlew spotlessApply.

Project Structure

This section will briefly talk about the project structure.

.
├── ctk - test related modules
│   │
│   ├── common - classes relating to utility and verification of the test classes
│   │               NOTE: Schema verification is run before tests and verifications.
│   │
│   └── idp - tests being written against a SAML IdP. The `src` directory of the module is organized
│             by the SAML specification as follows:
│              * Package: Based on Profile (i.e. WebSSO, Single Logout)
│                * Class: Based on Binding (i.e. POST, REDIRECT, ARTIFACT)
│
├── deployment - the project's full package deployment
│   │
│   ├── distribution - runtime elements including scripts, jars, and configurations
│   │
│   └── docker - logic for building a Docker image. Docker is used exclusively for our Jenkins builds.
│                To build a docker image, execute `gradlew build docker`. The docker image can also
│                be found on Docker Hub at https://hub.docker.com/r/codice/samlconf/.
│
├── external - files related to a specific SAML implementing product
│   │
│   ├── samlconf-plugins-api -  API that must be implemented for a SAML product in order to run this
│   │                           test kit against that product
│   │
│   └── implementations - implementations of the API for specific SAML products
│       │
│       ├── samlconf-ddf-impl - plugin and IdP metadata XML for the DDF implementation of IdP
│       │
│       └── samlconf-keycloak-impl - plugin and IdP metadata XML for the Keycloak implementation of IdP
│
└── library - Java classes copied from DDF to support operations outside the scope of the test code,
              e.g. signature validation using x509 certificates.

References

• SAML-CTK Forum: https://groups.google.com/forum/#!forum/saml-ctk
• FICAM: https://www.idmanagement.gov/wp-content/uploads/sites/1171/uploads/SAML2_1.0.2_Functional_Reqs.pdf
• SAML: https://wiki.oasis-open.org/security/FrontPage

Copyright / License

Copyright (c) Codice Foundation

This is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. A copy of the GNU Lesser General Public License is distributed along with this program and can be found at http://www.gnu.org/licenses/lgpl.html.