release-0.5.0

SDK 0.5.0 Changes:
- Remove certificate pinning for java sdk
- Removing certificate pinning from docker sdk
- Provide and option for custom groupid suffix in mirrormaker
- Updated unittest stack
- Updated ReadME for certificate pinning

docker/Dockerfile

@@ -18,10 +18,6 @@ COPY --from=builder /ncdssdk-client/target/ncdssdk-client.jar /home/kafka/app.ja
 
 WORKDIR /home/kafka
 
-RUN mkdir truststore
-
-ENV JAVAX_NET_SSL_TRUSTSTORE=truststore/ncdsTrustStore.p12
-
 ENTRYPOINT ["bash","docker/run-sdk-app.sh"]
 
 CMD ["-opt", "TOPICS"]

docker/README.md

@@ -6,21 +6,21 @@ Replace example `bootstrap.servers` property in the file kafka.properties (https
 Replace example `oauth.token.endpoint.uri` property in the file auth.properties (https://github.com/Nasdaq/CloudDataService/blob/master/docker/auth.properties) with provided values during on-boarding.
 
 ## Building
-Run docker build in project home directory 
-    
+Run `docker build` in project home directory
+
 ```
 docker build -f docker/Dockerfile . -t sdk-app --no-cache
 ```
- 
+
 ## Running Locally Built Images
 
-Replace client id(`{clinet-id-value}`) and client secret(`{client-secret-value}`) provided during on-boarding from Nasdaq team. Also, provide the password (`{trsustore-pass}`) for java truststore.
+Replace client id(`{clinet-id-value}`) and client secret(`{client-secret-value}`) provided during on-boarding from Nasdaq team.
 
 ```
-docker run -e "OAUTH_CLIENT_ID={clinet-id-value}" -e "OAUTH_CLIENT_SECRET={client-secret-value} -e "JAVAX_NET_SSL_TRUSTSTOREPASSWORD={trsustore-pass}" sdk-app:latest
+docker run -e "OAUTH_CLIENT_ID={clinet-id-value}" -e "OAUTH_CLIENT_SECRET={client-secret-value} sdk-app:latest
 ```
- 
- User can pass arguments to run the application with specific commands
+
+User can pass arguments to run the application with specific commands
  ```
  -opt -- Provide the operation you want to perform \n" +
    "        * TOP - View the top nnn records in the Topic/Stream\n"+
@@ -34,11 +34,11 @@ docker run -e "OAUTH_CLIENT_ID={clinet-id-value}" -e "OAUTH_CLIENT_SECRET={clien
  "-n -- Provide number of messages to retrieve        --- REQUIRED for TOP \n"+
  "-msgName -- Provide name of message based on schema --- REQUIRED for GETMSG \n"+
  ```
- 
- Example to get `TOP 10` messages from GIDS stream
- 
+
+Example to get `TOP 10` messages from GIDS stream
+
  ```
- docker run -e "OAUTH_CLIENT_ID={clinet-id-value}" -e "OAUTH_CLIENT_SECRET={client-secret-value} -e "JAVAX_NET_SSL_TRUSTSTOREPASSWORD={trsustore-pass}" sdk-app:latest -opt TOP -n 10 -topic GIDS
+ docker run -e "OAUTH_CLIENT_ID={clinet-id-value}" -e "OAUTH_CLIENT_SECRET={client-secret-value} sdk-app:latest -opt TOP -n 10 -topic GIDS
 ```
 
 ## Nasdaq Cloud Data Service - Kafka mirroring with MirrorMaker

docker/auth.properties

@@ -1,3 +1,2 @@
 oauth.token.endpoint.uri=https://{auth_endpoint_url}/auth/realms/pro-realm/protocol/openid-connect/token
-oauth.username.claim=preferred_username
-javax.net.ssl.trustStoreType=PKCS12
+oauth.username.claim=preferred_username

docker/mirrormaker/README.md

@@ -2,43 +2,53 @@
 This tool uses a Kafka consumer to consume messages from the source cluster, and re-publishes those messages to the local (target) cluster using an embedded Kafka producer. (https://kafka.apache.org/documentation.html#basic_ops_mirror_maker)
 
 ## Running Mirror Maker on docker
-This example shows how to setup standalone Mirror Maker instance application. 
+This example shows how to setup standalone Mirror Maker instance application.
 
 #### Consumer Configuration (NCDS cluster)
-- Replace example `bootstrap.servers` property in the file kafka.properties (https://github.com/Nasdaq/CloudDataService/blob/master/docker/mirrormaker/consumer.properties) with provided values during on-boarding. 
+- Replace example `bootstrap.servers` property in the kafka.properties file (https://github.com/Nasdaq/CloudDataService/blob/master/docker/mirrormaker/consumer.properties) with provided values during on-boarding.
 
 #### Producer Configuration (Target Cluster)
 - The producer is the part of Mirror Maker that uses the data read by the and replicates it to the destination cluster.
 - Update the producer.properties based target cluster. (https://github.com/Nasdaq/CloudDataService/tree/master/docker/mirrormaker/producer.properties)
 - Make sure the bootstrap.server IPs, truststore location if using SSL, and password are correct.
 
+#### Group Id Suffix
+- User can customize the group id suffix (property name : `groupidsuffix`) 
+  - If custom suffix is reused then consumer will start where it left off on restart
+  - If you don't pass a custom suffix, the SDK will create a new group id and start from the earliest message available on topic.
+
 #### Creating docker build
-- Run docker build in the project home directory. (https://github.com/Nasdaq/CloudDataService) 
+- Run `docker build` in the project home directory. (https://github.com/Nasdaq/CloudDataService)
 
 ```
 docker build -f docker/Dockerfile . -t sdk-app --no-cache
 ```
 
 #### Running mirror maker
 - Run mirror maker for given topics list.
-- Replace client id(`{client-id-value}`) and client secret(`{client-secret-value}`) provided during on-boarding from Nasdaq team. Also, provide the password (`{truststore-pass}`) for java truststore.
+- Replace client id(`{client-id-value}`) and client secret(`{client-secret-value}`) provided during on-boarding from Nasdaq team.
 
 ```
-docker run -e "OAUTH_CLIENT_ID={client-id-value}" -e "OAUTH_CLIENT_SECRET={client-secret-value} -e "JAVAX_NET_SSL_TRUSTSTOREPASSWORD={truststore-pass}" sdk-app:latest -opt mirrormaker -topics NLSUTP.stream
+docker run -e "OAUTH_CLIENT_ID={client-id-value}" -e "OAUTH_CLIENT_SECRET={client-secret-value}" sdk-app:latest -opt mirrormaker -topics NLSUTP.stream
+```
+
+example with custom groupid suffix
+
+```
+docker run -e "OAUTH_CLIENT_ID={client-id-value}" -e "OAUTH_CLIENT_SECRET={client-secret-value}" sdk-app:latest -opt mirrormaker -topics NLSUTP.stream -groupidsuffix mycustomsuffix
 ```
 
 ## Deploying Kafka Mirror Maker on Strimzi kafka cluster
 Strimzi is an open source project that provides container images and operators for running Apache Kafka on Kubernetes.(https://github.com/strimzi/strimzi-kafka-operator)
-The Cluster Operator deploys one or more Kafka Mirror Maker replicas to replicate data between Kafka clusters. 
+The Cluster Operator deploys one or more Kafka Mirror Maker replicas to replicate data between Kafka clusters.
 
-### Prerequisites 
+### Prerequisites
 - Before deploying Kafka Mirror Maker, the Cluster Operator must be deployed.
 
 ### Deploying mirror maker
-- Download kafka bootstrap server certificate from NCDS endpoint and add that to Kubernetes secret.
 - Create Kubernetes secret for Oauth Client Secret.
 - Update Oauth Client Id in kafka-mirror-maker.yaml.
 - Create a Kafka Mirror Maker cluster from the command-line:
-    ```kubectl apply -f mirrormaker/template/kafka-mirror-maker.yaml```
+  ```kubectl apply -f mirrormaker/template/kafka-mirror-maker.yaml```
 
-Provided example script `install_mirror_maker.sh` to deploy the mirror maker in your cluster.
+Provided example script `install_mirror_maker.sh` to deploy the mirror maker in your cluster.

docker/mirrormaker/consumer.properties

@@ -1,8 +1,7 @@
 bootstrap.servers={streams_endpoint_url}:9094
-ssl.endpoint.identification.algorithm=
 group.id=
 security.protocol=SASL_SSL
 sasl.mechanism=OAUTHBEARER
 sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;
 sasl.login.callback.handler.class=io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler
-auto.offset.reset=earliest
+auto.offset.reset=earliest

docker/mirrormaker/run-mirror-maker.sh

@@ -4,15 +4,14 @@ export OAUTH_TOKEN_ENDPOINT_URI="https://clouddataservice.auth.nasdaq.com/auth/r
 export OAUTH_CLIENT_ID="$OAUTH_CLIENT_ID"
 export OAUTH_CLIENT_SECRET=$OAUTH_CLIENT_SECRET
 export OAUTH_USERNAME_CLAIM=preferred_username
-export KAFKA_OPTS=" \
-  -Djavax.net.ssl.trustStore=/home/kafka/truststore/ncdsTrustStore.p12 \
-  -Djavax.net.ssl.trustStorePassword=$JAVAX_NET_SSL_TRUSTSTOREPASSWORD  \
-  -Djavax.net.ssl.trustStoreType=PKCS12"
 
-uuid=$(uuidgen)
-groupid="$OAUTH_CLIENT_ID$uuid"
+topics=$1
+group_id_suffix=$2
+
+groupid="$OAUTH_CLIENT_ID$group_id_suffix"
+echo $groupid
 
 #update the client Id in consumer
 sed -i "s/group.id=/group.id=$groupid/" /home/kafka/docker/mirrormaker/consumer.properties
 
-/opt/kafka/bin/kafka-mirror-maker.sh --consumer.config /home/kafka/docker/mirrormaker/consumer.properties --producer.config /home/kafka/docker/mirrormaker/producer.properties --num.streams 3 --whitelist $@
+/opt/kafka/bin/kafka-mirror-maker.sh --consumer.config /home/kafka/docker/mirrormaker/consumer.properties --producer.config /home/kafka/docker/mirrormaker/producer.properties --num.streams 3 --whitelist $topics

docker/run-sdk-app.sh

@@ -1,30 +1,28 @@
 #!/usr/bin/env sh
 
-file="/home/kafka/truststore/ncdsTrustStore.p12"
-
-# Remove the existing truststore if exists
-if [ -f $file ] ; then
-   rm $file
-fi
-
 OPT=$2
-
 #Function to get Topics
 function get_topics {
-        if [ $3 == "-topics" ]; then
+        if [[ $3 == "-topics" ]]; then
                 TOPICS=$4
         else
                 TOPICS='.*'
         fi
 }
 
-# Install Trust Store
-java -jar app.jar -opt INSTALLCERTS -path /home/kafka/truststore -pass $JAVAX_NET_SSL_TRUSTSTOREPASSWORD &&
+function get_groupids_suffix {
+        if [[ $5 == "-groupidsuffix" ]]; then
+                group_id_suffix=$6
+        else
+                group_id_suffix=$(uuidgen)
+        fi
+}
 
-if [ $OPT == "mirrormaker" ]; then
+if [[ $OPT == "mirrormaker" ]]; then
   get_topics $@
-  bash /home/kafka/docker/mirrormaker/run-mirror-maker.sh $TOPICS
+  get_groupids_suffix $@
+  bash /home/kafka/docker/mirrormaker/run-mirror-maker.sh $TOPICS $group_id_suffix
 else
   # Run the user command
-  java -Djavax.net.ssl.trustStore="/home/kafka/truststore/ncdsTrustStore.p12" -Djavax.net.ssl.trustStorePassword=$JAVAX_NET_SSL_TRUSTSTOREPASSWORD -Doauth.client.id=$OAUTH_CLIENT_ID -Doauth.client.secret=$OAUTH_CLIENT_SECRET -jar app.jar -kafkaprops /home/kafka/docker/kafka.properties -authprops /home/kafka/docker/auth.properties $@
-fi
+  java -Doauth.client.id=$OAUTH_CLIENT_ID -Doauth.client.secret=$OAUTH_CLIENT_SECRET -jar app.jar -kafkaprops /home/kafka/docker/kafka.properties -authprops /home/kafka/docker/auth.properties $@
+fi

ncds-sdk/pom.xml

@@ -7,8 +7,7 @@
     <parent>
         <groupId>com.nasdaq.ncds</groupId>
         <artifactId>ncds</artifactId>
-        <version>0.4.0</version>
-        <relativePath>../pom.xml</relativePath>
+        <version>0.5.0</version>
     </parent>
 
     <artifactId>ncds-sdk</artifactId>

ncds-sdk/src/main/java/com/nasdaq/ncdsclient/NCDSClient.java

@@ -34,7 +34,7 @@ public class NCDSClient {
      */
     public NCDSClient(Properties securityCfg,Properties kafkaCfg) throws Exception {
         try {
-            if (securityCfg != null && AuthenticationConfigLoader.validateSecurityConfig(securityCfg)) {
+            if (securityCfg != null && AuthenticationConfigLoader.validateSecurityConfig(securityCfg, kafkaCfg)) {
                 nasdaqKafkaAvroConsumer = new NasdaqKafkaAvroConsumer(securityCfg,kafkaCfg);
             }
             else if (IsItJunit.isJUnitTest()) {

ncds-sdk/src/main/java/com/nasdaq/ncdsclient/internal/utils/AuthenticationConfigLoader.java

@@ -14,9 +14,6 @@ public class AuthenticationConfigLoader {
     public static String OAUTH_CLIENT_ID ="oauth.client.id";
     public static String OAUTH_CLIENT_SECRET="oauth.client.secret";
     public static String OAUTH_USERNAME_CLAIM="oauth.username.claim";
-    public static String JAVAX_NET_SSL_TRUSTSTORE="javax.net.ssl.trustStore";
-    public static String JAVAX_NET_SSL_TRUSTSTOREPASSWORD="javax.net.ssl.trustStorePassword";
-    public static String JAVAX_NET_SSL_TRUSTSTORETYPE="javax.net.ssl.trustStoreType";
 
     public static String getClientID(){
         String clientID;
@@ -59,7 +56,7 @@ public static String getClientID(Properties cfg){
         }
     }
 
-    public static boolean validateSecurityConfig(Properties cfg) throws Exception {
+    public static boolean validateSecurityConfig(Properties cfg, Properties kafkaCfg) throws Exception {
 
         addNasdaqSpecificAuthProperties(cfg);
         if (cfg.getProperty(OAUTH_TOKEN_ENDPOINT_URI) == null) {
@@ -74,15 +71,6 @@ public static boolean validateSecurityConfig(Properties cfg) throws Exception {
         if (cfg.getProperty(OAUTH_USERNAME_CLAIM) == null) {
             throw new Exception("Authentication Setting :" + OAUTH_USERNAME_CLAIM  + " Missing" );
         }
-        if (cfg.getProperty(JAVAX_NET_SSL_TRUSTSTORE) == null && System.getenv("JAVAX_NET_SSL_TRUSTSTORE") == null) {
-            throw new Exception("Authentication Setting :" + JAVAX_NET_SSL_TRUSTSTORE  + " Missing" );
-        }
-        if (cfg.getProperty(JAVAX_NET_SSL_TRUSTSTOREPASSWORD) == null && System.getenv("JAVAX_NET_SSL_TRUSTSTOREPASSWORD") == null) {
-            throw new Exception ("Authentication Setting :" + JAVAX_NET_SSL_TRUSTSTOREPASSWORD  + " Missing" );
-        }
-        if (cfg.getProperty(JAVAX_NET_SSL_TRUSTSTORETYPE) == null) {
-            throw new Exception ("Authentication Setting :" + JAVAX_NET_SSL_TRUSTSTORETYPE  + " Missing" );
-        }
 
         return true;
     }