+ +
+

Welcome to DataLab Self-Service Node deployment script documentation!

+
+
+

Self-Service Node deployment script

+
+

deploy_datalab - CLI interface

+

DataLab Self-Service Node deployment

+
deploy_datalab [-h] [--conf_network_type {public,private}] [--conf_vpc_cidr CONF_VPC_CIDR]
+               [--conf_vpc2_cidr CONF_VPC2_CIDR]
+               [--conf_allowed_ip_cidr CONF_ALLOWED_IP_CIDR]
+               [--conf_user_subnets_range CONF_USER_SUBNETS_RANGE]
+               [--conf_private_subnet_prefix CONF_PRIVATE_SUBNET_PREFIX]
+               [--conf_additional_tags CONF_ADDITIONAL_TAGS]
+               [--conf_image_enabled CONF_IMAGE_ENABLED] [--conf_os_family {debian,redhat}]
+               [--ssn_hosted_zone_name SSN_HOSTED_ZONE_NAME]
+               [--ssn_hosted_zone_id SSN_HOSTED_ZONE_ID] [--ssn_subdomain SSN_SUBDOMAIN]
+               [--ssl_cert_path SSL_CERT_PATH] [--ssl_key_path SSL_KEY_PATH]
+               [--workspace_path WORKSPACE_PATH]
+               [--conf_tag_resource_id CONF_TAG_RESOURCE_ID]
+               [--conf_billing_tag CONF_BILLING_TAG] [--datalab_id DATALAB_ID]
+               [--usage_date USAGE_DATE] [--product PRODUCT] [--usage_type USAGE_TYPE]
+               [--usage USAGE] [--cost COST] [--resource_id RESOURCE_ID]
+               [--keycloak_realm_name KEYCLOAK_REALM_NAME] [--tags TAGS]
+               [--conf_stepcerts_enabled {true,false}]
+               [--conf_stepcerts_root_ca CONF_STEPCERTS_ROOT_CA]
+               [--conf_stepcerts_kid CONF_STEPCERTS_KID]
+               [--conf_stepcerts_kid_password CONF_STEPCERTS_KID_PASSWORD]
+               [--conf_stepcerts_ca_url CONF_STEPCERTS_CA_URL]
+               [--conf_letsencrypt_enabled {true,false}]
+               [--conf_letsencrypt_domain_name CONF_LETSENCRYPT_DOMAIN_NAME]
+               [--conf_letsencrypt_email CONF_LETSENCRYPT_EMAIL] --conf_service_base_name
+               CONF_SERVICE_BASE_NAME --action {build,deploy,create,terminate} --key_path
+               KEY_PATH --conf_key_name CONF_KEY_NAME --keycloak_auth_server_url
+               KEYCLOAK_AUTH_SERVER_URL --keycloak_client_name KEYCLOAK_CLIENT_NAME
+               --keycloak_client_secret KEYCLOAK_CLIENT_SECRET --keycloak_user KEYCLOAK_USER
+               --keycloak_user_password KEYCLOAK_USER_PASSWORD --default_endpoint_name
+               DEFAULT_ENDPOINT_NAME [--aws_report_path AWS_REPORT_PATH]
+               {aws,azure,gcp} ...
+
+
+
+

deploy_datalab optional arguments

+
    +
  • -h, --help - show this help message and exit

  • +
  • --conf_network_type CONF_NETWORK_TYPE -

    Type of network. Define in which network DataLab will be deployed.

    (valid choices: ['public', 'private'])

    +
    + (default: public)

  • +
  • --conf_vpc_cidr CONF_VPC_CIDR - CIDR of VPC (default: 172.31.0.0/16)

  • +
  • --conf_vpc2_cidr CONF_VPC2_CIDR - CIDR of secondary VPC (default: None)

  • +
  • --conf_allowed_ip_cidr CONF_ALLOWED_IP_CIDR - Comma-separated CIDR of IPs which will have access to SSN (default: 0.0.0.0/0)

  • +
  • --conf_user_subnets_range CONF_USER_SUBNETS_RANGE -

    Range of subnets which will be using for users environments.

    For example: 10.10.0.0/24 - 10.10.10.0/24

    +
    + (default: None)

  • +
  • --conf_private_subnet_prefix CONF_PRIVATE_SUBNET_PREFIX - Private subnet prefix (default: 24)

  • +
  • --conf_additional_tags CONF_ADDITIONAL_TAGS - Additional tags in format "Key1:Value1;Key2:Value2" (default: None)

  • +
  • --conf_image_enabled CONF_IMAGE_ENABLED - Enable or Disable creating image at first time (default: None)

  • +
  • --conf_os_family CONF_OS_FAMILY - Operating system distribution. (valid choices: ['debian', 'redhat']) (default: debian)

  • +
  • --ssn_hosted_zone_name SSN_HOSTED_ZONE_NAME - Name of hosted zone (default: None)

  • +
  • --ssn_hosted_zone_id SSN_HOSTED_ZONE_ID - ID of hosted zone (default: None)

  • +
  • --ssn_subdomain SSN_SUBDOMAIN - Subdomain name (default: None)

  • +
  • --ssl_cert_path SSL_CERT_PATH - Full path to SSL certificate (default: None)

  • +
  • --ssl_key_path SSL_KEY_PATH - Full path to key for SSL certificate (default: None)

  • +
  • --workspace_path WORKSPACE_PATH - Docker workspace path (default: )

  • +
  • --conf_tag_resource_id CONF_TAG_RESOURCE_ID - The name of user tag (default: datalab)

  • +
  • --conf_billing_tag CONF_BILLING_TAG - Billing tag (default: datalab)

  • +
  • --datalab_id DATALAB_ID - Column name in report file that contains datalab id tag (default: resource_tags_user_user_tag)

  • +
  • --usage_date USAGE_DATE - Column name in report file that contains usage date tag (default: line_item_usage_start_date)

  • +
  • --product PRODUCT - Column name in report file that contains product name tag (default: product_product_name)

  • +
  • --usage_type USAGE_TYPE - Column name in report file that contains usage type tag (default: line_item_usage_type)

  • +
  • --usage USAGE - Column name in report file that contains usage tag (default: line_item_usage_amount)

  • +
  • --cost COST - Column name in report file that contains cost tag (default: line_item_blended_cost)

  • +
  • --resource_id RESOURCE_ID - Column name in report file that contains datalab resource id tag (default: line_item_resource_id)

  • +
  • --keycloak_realm_name KEYCLOAK_REALM_NAME - Keycloak Realm name (default: datalab)

  • +
  • --tags TAGS - Column name in report file that contains tags (default: line_item_operation,line_item_line_item_description)

  • +
  • --conf_stepcerts_enabled CONF_STEPCERTS_ENABLED - Enable or disable step certificates. (valid choices: ['true', 'false']) (default: false)

  • +
  • --conf_stepcerts_root_ca CONF_STEPCERTS_ROOT_CA - Step root CA (default: None)

  • +
  • --conf_stepcerts_kid CONF_STEPCERTS_KID - Step KID (default: None)

  • +
  • --conf_stepcerts_kid_password CONF_STEPCERTS_KID_PASSWORD - Step KID password (default: None)

  • +
  • --conf_stepcerts_ca_url CONF_STEPCERTS_CA_URL - Step CA URL (default: None)

  • +
  • --conf_letsencrypt_enabled CONF_LETSENCRYPT_ENABLED - Enable or disable Let`s Encrypt certificates. (valid choices: ['true', 'false']) (default: false)

  • +
  • --conf_letsencrypt_domain_name CONF_LETSENCRYPT_DOMAIN_NAME -

    Domain names to apply. For multiple domains enter a comma separated list of domains

    as a parameter. ssn.domain_name will be used for ssn_node,DNS A record have to exist during deployment

    +
    + (default: None)

  • +
  • --conf_letsencrypt_email CONF_LETSENCRYPT_EMAIL -

    Email that will be entered during

    certificate obtaining and can be user for urgent renewal and security notices. Use comma to register +multiple emails, e.g. u1@example.com,u2@example.com.

    +
    + (default: None)

  • +
  • --aws_report_path AWS_REPORT_PATH - The path to billing reports directory in S3 bucket (default: None)

  • +
+
+
+

deploy_datalab Required arguments

+ +
+
+

deploy_datalab aws

+
deploy_datalab aws [-h] [--aws_user_predefined_s3_policies AWS_USER_PREDEFINED_S3_POLICIES]
+                   [--aws_access_key AWS_ACCESS_KEY]
+                   [--aws_secret_access_key AWS_SECRET_ACCESS_KEY]
+                   [--ssn_assume_role_arn SSN_ASSUME_ROLE_ARN] [--aws_vpc_id AWS_VPC_ID]
+                   [--conf_duo_vpc_enable {true,false}] [--aws_vpc2_id AWS_VPC2_ID]
+                   [--aws_peering_id AWS_PEERING_ID] [--aws_subnet_id AWS_SUBNET_ID]
+                   [--aws_security_groups_ids AWS_SECURITY_GROUPS_IDS]
+                   [--aws_billing_bucket AWS_BILLING_BUCKET] [--aws_job_enabled {true,false}]
+                   --aws_region AWS_REGION --aws_zone AWS_ZONE --aws_ssn_instance_size
+                   AWS_SSN_INSTANCE_SIZE --aws_account_id AWS_ACCOUNT_ID
+
+
+
+
deploy_datalab aws optional arguments
+
    +
  • -h, --help - show this help message and exit

  • +
  • --aws_user_predefined_s3_policies AWS_USER_PREDEFINED_S3_POLICIES - Predefined policies for users instances (default: None)

  • +
  • --aws_access_key AWS_ACCESS_KEY -

    AWS Access Key ID. reuqired in case of deployment with IAM user DataLab

    deployment script is executed on local machine and uses +IAM user permissions to create resources in AWS.

    +
    + (default: None)

  • +
  • --aws_secret_access_key AWS_SECRET_ACCESS_KEY - AWS Secret Access Key (default: None)

  • +
  • --ssn_assume_role_arn SSN_ASSUME_ROLE_ARN - Role ARN for creating Route53 record in different AWS account (default: None)

  • +
  • --aws_vpc_id AWS_VPC_ID - AWS VPC ID (default: None)

  • +
  • --conf_duo_vpc_enable CONF_DUO_VPC_ENABLE - Duo VPC scheme enable. (valid choices: ['true', 'false']) (default: false)

  • +
  • --aws_vpc2_id AWS_VPC2_ID - Secondary AWS VPC ID (default: None)

  • +
  • --aws_peering_id AWS_PEERING_ID - Amazon peering connection id (default: None)

  • +
  • --aws_subnet_id AWS_SUBNET_ID - AWS Subnet ID (default: None)

  • +
  • --aws_security_groups_ids AWS_SECURITY_GROUPS_IDS - One of more comma-separated Security groups IDs for SSN (default: None)

  • +
  • --aws_billing_bucket AWS_BILLING_BUCKET - The name of S3 bucket where billing reports will be placed. (default: None)

  • +
  • --aws_job_enabled AWS_JOB_ENABLED - Billing format. (valid choices: ['true', 'false']) (default: false)

  • +
+
+
+
deploy_datalab aws Required arguments
+ +
+
+
+

deploy_datalab azure

+
deploy_datalab azure [-h] [--azure_vpc_name AZURE_VPC_NAME]
+                     [--azure_subnet_name AZURE_SUBNET_NAME]
+                     [--azure_security_group_name AZURE_SECURITY_GROUP_NAME]
+                     [--azure_edge_security_group_name AZURE_EDGE_SECURITY_GROUP_NAME]
+                     [--azure_resource_group_name AZURE_RESOURCE_GROUP_NAME]
+                     [--azure_datalake_enable {true,false}]
+                     [--azure_ad_group_id AZURE_AD_GROUP_ID]
+                     [--azure_offer_number AZURE_OFFER_NUMBER]
+                     [--azure_currency AZURE_CURRENCY] [--azure_locale AZURE_LOCALE]
+                     [--azure_application_id AZURE_APPLICATION_ID]
+                     [--azure_validate_permission_scope {true,false}]
+                     [--azure_oauth2_enabled {true,false}]
+                     [--azure_region_info AZURE_REGION_INFO]
+                     [--azure_source_vpc_name AZURE_SOURCE_VPC_NAME]
+                     [--azure_source_resource_group_name AZURE_SOURCE_RESOURCE_GROUP_NAME]
+                     --azure_region AZURE_REGION --azure_ssn_instance_size
+                     AZURE_SSN_INSTANCE_SIZE --azure_auth_path AZURE_AUTH_PATH
+
+
+
+
deploy_datalab azure optional arguments
+ +
+
+
deploy_datalab azure Required arguments
+ +
+
+
+

deploy_datalab gcp

+
deploy_datalab gcp [-h] [--billing_dataset_name BILLING_DATASET_NAME]
+                   [--gcp_subnet_name GCP_SUBNET_NAME] [--gcp_vpc_name GCP_VPC_NAME]
+                   [--gcp_firewall_name GCP_FIREWALL_NAME] --gcp_region GCP_REGION --gcp_zone
+                   GCP_ZONE --gcp_ssn_instance_size GCP_SSN_INSTANCE_SIZE --gcp_project_id
+                   GCP_PROJECT_ID --gcp_service_account_path GCP_SERVICE_ACCOUNT_PATH
+
+
+
+
deploy_datalab gcp optional arguments
+
    +
  • -h, --help - show this help message and exit

  • +
  • --billing_dataset_name BILLING_DATASET_NAME - Name of GCP dataset (BigQuery service) for billing (default: None)

  • +
  • --gcp_subnet_name GCP_SUBNET_NAME - GCP Subnet Name (default: None)

  • +
  • --gcp_vpc_name GCP_VPC_NAME - GCP VPC Name (default: None)

  • +
  • --gcp_firewall_name GCP_FIREWALL_NAME - One of more comma-separated GCP Firewall rules for SSN (default: None)

  • +
+
+
+
deploy_datalab gcp Required arguments
+ +
+
+
+
+
+

Examples

+

Examples How to deploy DataLab for different cloud providers.

+

GCP example:

+
$ infrastructure-provisioning/scripts/deploy_datalab.py         --conf_service_base_name <SERVICE_NAME>         --conf_os_family debian         --action create         --key_path /home/ubuntu/.ssh         --conf_key_name gcp         --billing_dataset_name billing         gcp         --gcp_ssn_instance_size n1-standard-2         --gcp_project_id <PROJECT_ID>        --gcp_service_account_path /home/ubuntu/secret.json        --gcp_region us-west1        --gcp_zone us-west1-a
+
+
+

AWS example:

+
$ infrastructure-provisioning/scripts/deploy_datalab.py        --conf_service_base_name datalab-test        --conf_os_family debian        --action create         --key_path /path/to/key/        --conf_key_name key_name        --conf_tag_resource_id datalab        aws        --aws_vpc_id vpc-xxxxx        --aws_subnet_id subnet-xxxxx        --aws_security_groups_ids sg-xxxxx,sg-xxxx        --aws_access_key XXXXXXX        --aws_secret_access_key XXXXXXXXXX        --aws_region xx-xxxxx-x        --aws_account_id xxxxxxxx        --aws_billing_bucket billing_bucket        --aws_report_path /billing/directory/
+
+
+
+
+
+ + +