-
Notifications
You must be signed in to change notification settings - Fork 7
How to...
This is a checklist to verify the connectivity and proper state of a new CSP local node installation. These checks are carried out between the local, central, and debug nodes. They cannot be carried out on just the local node.
First, verify that the network/server/application configuration is correct.
- Run a docker stats to verify that all the services are up
docker stats --all --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}" --no-stream | sort
- Check if the integration layers can communicate with each other. Send the following curl from debug to local, from central to local, and from local to debug.
cd /opt/csp/apache2/ssl/server
curl --cacert ../ca/common-external-ca.crt --key csp-external.key --cert csp-external.crt https://integration.debug.env.domain:5443/apiversion/
NB: Remember to use the correct target URL.
- Log in to central.env, add new local CSP to CSP_ALL and nuke
- After adding the local CSP to CSP_ALL, click the nuke button
- Verify that the local CSP received the new team (debug) and that CSP_ALL and CSP_SHARING central trust circles are synced with central.env
- Set up LTC::CSP_SHARING for this run
- Edit LTC::CSP_SHARING in the local CSP and add only the new team and debug
- Do the same in the debug.env CMM(TC)
- When the two teams exist in each others LTC::CSP_SHARING, then all automatic sharing will be exclusively between those two CSPs without spamming other CSP nodes
- Configure sharing policies to allow sharing of contacts
- In the local CSP go to integration-ui and put “share as is” in the “contact” datatype (this is needed for the CMM test cases)
- The datatypes “event”, “threat”, “incident” should also be “share as is”
- Check if SMTP config is working
- Check if the local CSP received any regular reports in the mailbox of the user that was specified as sender in the CSP SMTP configuration. _This is an indication that the services can send/receive emails_
- Check MISP configuration
- Make sure that the local CSP configured MSIP according to the latest installation manual version. This includes checking configuration values such as MISP.org
- Verify that the integration user is present
- Ensure that SSO works and at least one oAM user has a corresponding MISP user
Having passed the above checks, the test cases can be carried out.
Tips
- Do the first negative test case in IntelMQ at the very end
- The test case breaks IntelMQ and you want to avoid trying to fix it before you run the rest of the test cases
- Remember to fix the broken IntelMQ configuration
- When you have to verify that an event/message arrived successfully at your end, then you can share your own screen with the other party for validation.
- If both parties are using Windows, then you can join the conference that was created during the VCB test cases without having to exit the current video conference.
- In Chrome, if the browser has not been set up to access the MeliCERTes CA certificate, then:
- The other party will join the videoconference and the main screen (iframe) will show an error message that https://vc.cert.preprod... is down. The error is due to the unknown certificate warning and fails due to the iframe.
- The other party should copy the link that is showing in the middle of the screen (the one that starts with https://vc....) and paste it on a new tab. Then they can bypass the warning and it should take them to a bare Jitsi. They can close this tab and refresh the previous one (the normal videoconference one). The iframe will work now.
| WARNING: This procedure will reset your local CSP and erase all data |
|---|
| The procedure documented below will reset the CSP VM to its initial clean state. There will be a total loss of all data. Please be very careful if and when to execute these instructions. |
The following prerequisites are required for the procedure to be completed
Prerequisites
| Required infrastructure/feature | Yes/No |
|---|---|
| Internet connection | Yes |
| Command line (terminal) access | Yes |
Required inputs Input from supported local CSP:
- /tmp/spring.log
The following commands should be executed to reset:
| Execute at CSP VM prompt |
|---|
docker stop `docker ps -q` && docker kill `docker ps -q` && docker rm `docker ps -q -a` docker volume rm `docker volume ls -q` docker rmi `docker images -q` && docker system prune -f && docker volume prune -f && docker network prune -f rm -fr /opt/cspinst/.clientconf* /opt/csp/* /root/*conf /root/*env reboot.
|
Prerequisites
| Required infrastructure/feature | Yes/No |
|---|---|
| Internet connection | Yes |
| Command line (terminal) access | Yes |
| New certificates from PKI team for local and central CSP | Yes |
| CA chain as received from the PKI service | Yes |
| New IP addresses (optionally) for location change | Yes |
Required inputs Input from central CSP:
- content of volumes DATA and ROOT (for the installer)
- content of volume CONFIG (for the configuration service) Input from local CSP:
- content of /opt/cspinst/.clientconf* (2 hidden files, for the installer)
Overview
The following steps must be carried out sequentially to change the domain of the local CSP
Step 1 - correcting the CSP domain and IP addresses
- in CENTRAL, CSP node information should be modified with respect to domain name and IP addresses recorded
- in CENTRAL, CSP installer information should be modified with respect to domain name and IP addresses recorded.
- in CSP Node(s), CSP installer information should be modified with respect to domain name and IP addresses recorded.
Step 2 - Update the DNS with the new DNS names and entries for the new domain
Step 3 - updating the received databases to the various locations
Step 4 - recreating internal and external certificates and distributing them
Step 5 - reinstalling modules
Step 6 - registering vhosts and agents for OpenAM
Step 7 - starting CSP
For every step, there is an 'Output' section that contains information for verification that the step has been carried without any errors.
In this step, the support team requests the 3 different databases for local update and inspection. This step verifies that configuration stored in databases is updated.
Output: 3 databases updated with correct information, produced by support team and emailed to the CENTRAL admins
In this step, the support team and CENTRAL administration create the DNS entries for the new domain. CENTRAL and CSP Node names should be created/updated with the new IP addresses.
Network configuration should be correct at this point; the connectivity via firewall whitelisting must be verified.
Output: no output
The CENTRAL team should update the databases found in the various locations with the new updated DBs as received by the support team.
To replace the databases, installer and configuration service must be stopped.
- For the CSP Node, rename 'cspinstaller.jar' to 'cspinstaller.jar.old' and kill the installer via kill -15 and copy the files over. Rename back to .jar, and reboot for the installer to start again.
- For the Central, stop the installer via the docker-compose.yml: docker-compose stop and copy the database. When complete, restart the central config and installer using the opposite operation as per the manual.
Output: list of volume directories with changed files should be provided - installer on VM should be running and installer and configuration service on CENTRAL should be running
To recreate the certificates on CENTRAL and CSP Node, the following steps must be done:
CSP Node For the CSP node, the Installer should be running. Please verify that the installer used is latest version (currently 4.0.7 commitId 3427f27). You can verify the installer version by hovering over the main page, top-left, on the installer logo icon. A tooltip should appear with the installer version. The commitId appears after the SNAPSHOT/.
The following files should be copied to the /opt/csp/downloads directory:
- ca-bundle.crt <-- CA file, you should obtain by downloading https://pki.dfn-cert.de/melicertes-ca/pub/cacert/chain.txt and removing the 1st line(important step)
- sslprivate.key <-- private key (should have this)
- sslpublic.crt <-- public key (should have received this, signed by the PKI service)
The files should be placed in /opt/csp/downloads. Verify the names and file content.
Note: these files should be for the new domain, not the old one!
When the files are in place, a system call to the CSP Installer, from the command line, should be placed:
curl -v http://localhost:18080/regenerateEnv
This step will generate 3 background jobs, which will read the updated information and:
- Setup new internal certificates for the new domain
- Setup new external certificates
- Recreate the common environment variables on the /root folder. The jobs should complete within the next 1-2 minutes, you should be monitoring logs for completion.
Output: the content of /tmp/spring.log and file /root/common.0.env should be sent to the central team for verification of certificate generation.
CENTRAL setup For the CENTRAL node, the Installer should be running. Please verify that the installer used is latest version. You can verify the installer version by hovering over the main page, top-left, on the installer logo icon. A tooltip should appear with the installer version. The commitId appears after the SNAPSHOT/.
In CENTRAL, the installer is running in a docker container. For the process to work, the certificates should be copied in the /opt/csp/downloads folder within the docker container, not the /opt/csp/downloads of the VM!
To do so, you will need the following sub-steps:
- copy the files to the ROOT volume,
- exec in the container by executing the command
docker exec -ti csp-sa_cfg_client bash - copy the files from
/rootto the/opt/csp/downloadsdirectory. (rest of the guide for CENTRAL is within the container, not on the outside VM)
For the process, we need **curl **and openssl. These tools are not installed by default in the container, in order to install them execute the following command:
apk update && apk add curl && apk del libressl libressl-dev && apk add openssl
The following files should be copied to the /opt/csp/downloads directory (from the /root folder):
- ca-bundle.crt <-- CA file, you should obtain by downloading https://pki.dfn-cert.de/melicertes-ca/pub/cacert/chain.txt and removing the 1st line(important step)
- sslprivate.key <-- private key (should have this)
-
sslpublic.crt <-- public key (should have received this, signed by the PKI service)
The files should be placed in
/opt/csp/downloads. Verify the names and file content. Note, these files should be for the new domain - not the old one!
When the files are in place, a system call to the CSP Installer, from the command line, should be placed:
curl -v http://localhost:18080/regenerateEnvThis step will generate 3 background jobs, which will read the updated information and:
- Setup new internal certificates for the new domain
- Setup new external certificates
- Recreate the common environment variables on the /root folder. The jobs should complete within the next 1-2 minutes, you should be monitoring logs for completion.
Output: the content of /tmp/spring.log and file /root/common.0.env should be sent to the central team for verification of certificate generation.
In this step, all modules should be reinstalled by the installer. This is a sequential process, should be executed for both CENTRAL and CSP Node from the 'Updates' screen of the installer.
Prerequisites:
- Installer is running
- Services are stopped
- Updates screen shows modules and actions. In particular it should not show the "stop" action. Use the 'reinstall' button to perform a module delete and install, for each module, EXCEPT the base module (first on the list). This process may take up to 5 minutes per module, but to expedite, you may do CENTRAL and CSP Node in parallel.
Important: OpenAM is the module that has the security configuration of identities. This module needs to be properly initialised.
Removing the OpenAM volumes
Delete the OpenAM volumes:
cd /opt/csp/modules/oam<**hashid**>
docker volume list |grep OAM# 3 volumes should appear
docker volume rm <volume name from above># this should happen for all 3 volumes
docker volume create <volume name from above># this should happen for all 3 volumes
Removing the Apache volumes
Delete the Apache volume for web agents:
cd /opt/csp/modules/apache<**hashid**>
docker volume list | grep Web
docker volume rm <volume name from above>
docker volume create <volume name from above>
Output: the content of files *env and *conf in the /root folder. Note that /root on the CSP Node is on the VM itself, but on the CENTRAL, it is on the ROOT volume. Both outputs should be sent over to the central team for verification.
This step applies to both CENTRAL and CSP Node instances
To register again the OpenAM and VHost entries, execute the following command:
curl -v http://localhost:18080/recreateOAMVH
You should receive a HTTP 200 response. Note that in the CENTRAL, this has to be within the docker container not on the outside.
For CENTRAL, follow the CENTRAL administration guide to start the CENTRAL CSP.
For the CSP Node, start the CSP instances via "START" button. When the start process is complete, send the central team the logs from /tmp/spring.log.
Verify that the log contains entries with "ACTIONS COMPLETED" (example: fgrep "ACTIONS COMPLETED" /tmp/spring.log)
All entries found should mention OAM : true - APC : true for service <A SERVICE NAME HERE, e.g. misp / rt / intelmq etc>
When this is verified, the CSP should have started and the services should be running and be accessible from their web interfaces.
There are cases where incorrect certificates and/or wrong files with correct extension were provided.
Prerequisites
| Required infrastructure/feature | Yes/No |
|---|---|
| Internet connection | Yes |
| Command line (terminal) access | Yes |
| Original certificate private key | Yes |
| Original certificate public key signed by CA | Yes |
| CA chain as received by the PKI service | Yes |
Required inputs
Requested input from supported local CSP:
-
docker ps | grep apache: shows if apache docker containers are running -
docker logs csp-apache | tail -1000: shows last 1000 lines of logs of apache -
find /opt/csp/apache2 -exec ls -lrt {} \;: shows files in directories, plus sizes and date/time of generation
Checks after input is received
- Check that
csp-apacheis running. If the rest of the services are running andcsp-apacheis not, it is in most cases a certificate problem. Use the first requested output to check ifcsp-apacheis listed as a container that is running. It should be. - Check
csp-apachelogs by checking output 2.
| Sample log that indicates that there are issues with the certificates |
|---|
| Executing apache with -f /etc/apache2/httpd.conf AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.19.0.2. Set the 'ServerName' directive globally to suppress this message [Thu May 09 08:06:53.058798 2019] [ssl:emerg] [pid 8:tid 140608194799488] AH02562: Failed to configure certificate viper-ui.cert-ro.prod.melicertes.eu:443:0 (with chain), check /etc/apache2/ssl/server/csp-external.crt [Thu May 09 08:06:53.058851 2019] [ssl:emerg] [pid 8:tid 140608194799488] SSL Library Error: error:0906D06C:PEM routines:PEM_read_bio:no start line (Expecting: TRUSTED CERTIFICATE) -- Bad file contents or format - or even just a forgotten SSLCertificateKeyFile? [Thu May 09 08:06:53.058865 2019] [ssl:emerg] [pid 8:tid 140608194799488] SSL Library Error: error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib AH00016: Configuration Failed |
Error is "(Expecting: TRUSTED CERTIFICATE) -- Bad file contents or format - or even just a forgotten SSLCertificateKeyFile?"
The checks 1 and 2 should indicate that the certificates are not properly generated.
- Check output of 3. Make sure there are no 0-size files, all files should have content and the certificate files should be between 1.8K to 2.9K size.
The following processes need to be followed correctly by the local CSP team for the process to succeed. Essentially, the steps recreate the process that is performed by the installer at the initial phase of installation.
For this process to succeed, CSP services should be stopped. Please perform this task via the "Services" tab, "STOP" button. Wait until all are stopped.
Verify that services are stopped, by executing docker ps - this command should return no entries.
STEP 1: copying files to CSP VM
-
ca-bundle.crt<-- CA file, you should obtain by downloading https://pki.dfn-cert.de/melicertes-ca/pub/cacert/chain.txt and removing the 1st line (important step) -
sslprivate.key<-- private key (CSIRT should have this) -
sslpublic.crt<-- public key (CSIRT should have received this, signed by the PKI service) The files should be placed in/tmp. Verify the names and file content.
STEP 2: setup the environment for the re-generation
a) stop CSP services by pressing the "STOP" Button on the Services page, wait until all services are stopped.
b) login to the terminal and do the following environement variables:
-
CSPNAME variable:
export CSPNAME=cert-xxNote: the name should be adjusted to a CSP. Please modify appropriately for the given CSIRT -
CSPDOMAIN variable:
export CSPDOMAIN=<env>.<domain> -
TMPDIR variable:
export TMPDIR=/tmp
c) verify you have the correct Docker image already: docker images | grep jdk
This should find a result similar to thanosa75/alpine-jdk8. If the grep does not find the jdk, then it will be pulled at time of script execution which requires internet access.
STEP 3: execute the certificate regeneration
In the terminal, execute the following script: /opt/csp/modules/externalCerts.sh
| Expected output |
|---|
| Files copied correctly ...(other output) Process completed successfully, external certificates generated for JKS |
If the wrong names have been used or the files are not found in the folder, or the TMPDIR variable is not the directory where the files should be, then following error message will be shown:
| Error output due to incorrect names or misplacement of files |
|---|
| Files not copied correctly or not found in /tmp folder with expected names |
If the certificates are not in the correct name and/or location, then a generic error message will be shown:
| Error output due to incorrect certificates |
|---|
| Process has failed. Oups |
In this case get the full output of the command execution. Most likely, certificates are not in correct name and/or location. Start again with STEP 1.
STEP 1: Stop CSP Stop CSP again (STOP button by the Services tab).
STEP 2: Repopulate volume
Manually execute the following command to repopulate the certificate storage. Check the output of the executed command for errors: docker run -d --rm -v SSLDatavolume:/ssl_data -v /opt/csp/apache2/ssl/:/mnt thanosa75/alpine-jdk8:slim sh -c "cp -r /mnt/server /mnt/ca /ssl_data"
STEP 3: Restart CSP to re-register OAM and VHOST entries
Execute the following command: curl -v http://localhost:18080/recreateOAMVH
There will be HTTP 200 response if the command is successful.
STEP 4: Start CSP Start the CSP instances with the "START" button. When the start process is complete, check /tmp/spring.log.
| Expected output |
|---|
ACTIONS COMPLETED" (example: fgrep "ACTIONS COMPLETED" /tmp/spring.log) |
All entries should mention:
OAM : true - APC : true for service <A SERVICE NAME HERE, e.g. misp / rt / intelmq etc>
When this is verified, CSP should have started and the services should be running and be accessible from their web interfaces.
The MISP adapter will not start if the authentication key used to connect the various services together is not properly set or is missing. Follow these steps to fix the problem.
The following steps fix the problem.
Grab authkey from MISP UI:
- Log in MISP
- Event Actions > Automation
- Copy the highlighted key
Create the file and place the key within it:
cd /opt/csp/modules/misp<MODULEHASH>
docker exec -ti csp-misp /bin/bash
echo {KEY} > /run/secrets/authkey# {KEY} is the copied key, without the brackets.
Restart MISP, VIPER, and Apache:
cd /opt/csp/modules/misp<MODULEHASH>
docker-compose stop && docker-compose up -d
cd /opt/csp/modules/viper<MODULEHASH>
docker-compose stop && docker-compose up -d
cd /opt/csp/modules/apache<MODULEHASH>
docker-compose stop && docker-compose up -d