Skip to content

Commit

Permalink
[ED-3649] merge release-6.0.0 changes to 7.0.0 (#3998)
Browse files Browse the repository at this point in the history
* Fix syntax (#3894)

* Updated vars for cron schedule (#3895)

* Updated vars for cron schedule

* Update main.yml

* [ED-2793] Fix: release-6.0.0 installation issues  (#3900)

* Update Jenkins version

* Fix job parameter

* Fix error: You need to install jmespath prior to running json_query filter

* Update main.yml with cassandra (#3845) (#3901)

Co-authored-by: gohilamariappan <[email protected]>

* ED-2838: ##Release-6.0.0 new variables (#3899)

* ED-2838 Release-6.0.0 new variables

* Added with comments

* ED-2838 jenkins josb update and new variables

* added storage endpoint detials

* [ED-2793] Fix installation issues (#3905)

* Update maven repo url

* Add missing jobs

* Fix docker creds secrets issue when having specialcharacters

* [ED-2793] Add OCI specific vars (#3912)

* Add OCI specific vars

* Add mobile deeplink url var

* Add ml_core_internal_access_token for private repo template

Add ml_core_internal_access_token for private repo template

* Update cassandra jenkins job config (#3927)

* saving changes (#3935)

Co-authored-by: ankitshahu <[email protected]>

* #OB-553 fix: Analytics service issue fixes for OCI

* #OB-553 fix: Analytics service issue fixes for OCI

* ED-3534:- Adoption Changes in ML service (#3963)

* saving changes

* added Upload api endpoint in public kong

---------

Co-authored-by: ankitshahu <[email protected]>

* ED-3079: deploy nginx to support cors for s3 compatible object storage (#3915)

* nginx-cors

Signed-off-by: Deepak Devadathan <[email protected]>

* hostname changed

Signed-off-by: Deepak Devadathan <[email protected]>

* test

Signed-off-by: Deepak Devadathan <[email protected]>

* test

Signed-off-by: Deepak Devadathan <[email protected]>

* added nginx-cors-public chart

Signed-off-by: Deepak Devadathan <[email protected]>

* test

Signed-off-by: Deepak Devadathan <[email protected]>

* corrected proxy conf

Signed-off-by: Deepak Devadathan <[email protected]>

* corrected syntax

Signed-off-by: Deepak Devadathan <[email protected]>

* changed nodeport

Signed-off-by: Deepak Devadathan <[email protected]>

* removed duplicate

Signed-off-by: Deepak Devadathan <[email protected]>

* nodeport change

Signed-off-by: Deepak Devadathan <[email protected]>

* nodeport change

Signed-off-by: Deepak Devadathan <[email protected]>

* changed public ip

Signed-off-by: Deepak Devadathan <[email protected]>

* test

Signed-off-by: Deepak Devadathan <[email protected]>

* removed nginx-cors ansible roles

Signed-off-by: Deepak Devadathan <[email protected]>

* added jenkins job for nginx-cors-public deployment

Signed-off-by: Deepak Devadathan <[email protected]>

* updated variable in values.j2

Signed-off-by: Deepak Devadathan <[email protected]>

* testing public ingress along with s3 cors

Signed-off-by: Deepak Devadathan <[email protected]>

* testing with condition for csp

Signed-off-by: Deepak Devadathan <[email protected]>

* removed nginx-cors-public

Signed-off-by: Deepak Devadathan <[email protected]>

---------

Signed-off-by: Deepak Devadathan <[email protected]>

* Added new kafka topics in main.yml and config.j2 (#3970)

* Issue #CO-824: dynamic redirect URL

* Issue #CO-824: dynamic redirect URL

* Issue #CO-824: SSO Isse condition update

* Issue ED-0000 report service env update

* Issue ED-0000 report service env update (#3977)

* Issue ED-0000 report service env update

* Issue ED-0000 report service env update (#3981)

* ED-3534:- Adoption Changes in ML service (#3983)

* helm chart for ml-analytics

* Update and rename values.yaml to values.j2

* Update main.yml

* Update ml-analytics-service.conf

* Update ml-analytics-service.conf

* Delete kubernetes/helm_charts/core/ml-analytics-service/templates/NOTES.txt

* Update deployment.yaml

* Update deployment.yaml

* Update deployment.yaml

* Update values.j2

* Update deployment.yaml

* Update deployment.yaml

* Update ml-analytics-service.conf

* Update deployment.yaml

* Update ml-analytics-service.conf

* Update ml-analytics-service.conf

* Update ml-analytics-service.conf

* Update ml-analytics-service.conf

* Updated ml-analytics-service.conf

* added new kafka topics for observations

* Update ml-analytics-service.conf

---------

Co-authored-by: raghupathi <[email protected]>
Co-authored-by: Raghupathi Guduri <[email protected]>

* Issue #IQ-750 fix: added script for inquiry schema upload

* Issue #IQ-750 fix: updated jenkins file

* added the faust related changes

* Issue #IQ-750 fix: updated jenkins script

* added the value to ml_analytics_faust_applications_list

* Update secrets.yml

* ED-3534:- Adoption Changes in ML service (#3993)

* updated druid ports

* updated druid ports

* key changes

* key changes

---------

Co-authored-by: vivek <[email protected]>

* Add release-6.0.0 missing vars

* Add release-6.0.0 missing vars (#3995)

* Add inquiry-schema-upload Jenkinsfile

---------

Signed-off-by: Deepak Devadathan <[email protected]>
Co-authored-by: Raghupathi Guduri <[email protected]>
Co-authored-by: gohilamariappan <[email protected]>
Co-authored-by: Prasath Sivasubramaniyan <[email protected]>
Co-authored-by: Akash Shah <[email protected]>
Co-authored-by: Ankit Shahu <[email protected]>
Co-authored-by: ankitshahu <[email protected]>
Co-authored-by: sowmya-dixit <[email protected]>
Co-authored-by: Deepak Devadathan <[email protected]>
Co-authored-by: Prashant Gunjeti <[email protected]>
Co-authored-by: Shubham Bansal <[email protected]>
Co-authored-by: Rajeev Sathish <[email protected]>
Co-authored-by: Vivek M <[email protected]>
Co-authored-by: raghupathi <[email protected]>
Co-authored-by: Kumar Gauraw <[email protected]>
Co-authored-by: vivek <[email protected]>
  • Loading branch information
16 people authored May 6, 2024
1 parent 788217e commit b2c4f1d
Show file tree
Hide file tree
Showing 26 changed files with 781 additions and 39 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -3159,6 +3159,7 @@ var handleGoogleAuthEvent = () => {
redirectToLib();
}
};

var redirectToPortal = (redirectUrlPath) => { // redirectUrlPath for sso and self signUp
const curUrlObj = window.location;
var redirect_uri = getValueFromSession('redirect_uri');
Expand All @@ -3171,9 +3172,7 @@ var redirectToPortal = (redirectUrlPath) => { // redirectUrlPath for sso and sel
const redirect_uriLocation = new URL(redirect_uri);
if (client_id === 'android' || client_id === 'desktop') {
window.location.href = sessionUrlObj.protocol + '//' + sessionUrlObj.host + redirectUrlPath + updatedQuery;
} else if(client_id === 'portal' &&
redirectUrlPath === '/sign-in/sso/select-org' &&
(redirect_uri.includes('dock.sunbirded.org') || redirect_uri.includes('dockstaging.sunbirded.org'))) {
} else if(client_id === 'portal' && redirectUrlPath === '/sign-in/sso/select-org' && (sessionUrlObj.host !== redirect_uriLocation.host)) {
window.location.href = sessionUrlObj.protocol + '//' + sessionUrlObj.host + redirectUrlPath + updatedQuery;
} else {
window.location.href = redirect_uriLocation.protocol + '//' + redirect_uriLocation.host +
Expand Down
21 changes: 21 additions & 0 deletions ansible/roles/kong-api/defaults/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8300,6 +8300,27 @@ kong_apis:
config.required: false
config.enabled: false

- name: uploadFileToCloud
uris: "{{ cloud_service_prefix }}/mlcore/v1/files/upload"
upstream_url: "{{ ml_core_service_url }}/v1/cloud-services/files/upload"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- cloudUrlsCreate
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ small_request_size_limit }}"
- name: opa-checks
config.required: false
config.enabled: false

- name: listPlatformPrograms
uris: "{{ user_extension_prefix }}/mlcore/v1/programsByPlatformRoles"
upstream_url: "{{ ml_core_service_url }}/v1/user-extension/programsByPlatformRoles"
Expand Down
1 change: 1 addition & 0 deletions ansible/roles/ml-analytics-service/defaults/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -138,3 +138,4 @@ ml_analytics_batch_cron_hour: "{{ ml_batch_cron_hour | default('18') }}"
ml_analytics_nvsk_cron_minute: "{{ ml_nvsk_cron_minute | default('30') }}"
ml_analytics_nvsk_cron_hour: "{{ ml_nvsk_cron_hour | default('7') }}"
ml_analytics_nvsk_cron_weekday: "{{ ml_nvsk_cron_weekday | default('4') }}"
ml_analytics_faust_applications_list: "'observations/py_observation_streaming observations/', 'observations/py_observation_evidence_streaming observations/', 'survey/py_survey_streaming survey/', 'survey/py_survey_evidence_streaming survey/'"
5 changes: 5 additions & 0 deletions ansible/roles/ml-analytics-service/templates/config.j2
Original file line number Diff line number Diff line change
Expand Up @@ -263,3 +263,8 @@ store = {{ ml_analytics_reports_store }}
[ML_CORE_SERVICE_URL]

url = {{ ml_analytics_core_service }}


[SERVICES]

faust_applications_list = {{ ml_analytics_faust_applications_list }}
1 change: 1 addition & 0 deletions ansible/roles/stack-sunbird/defaults/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,7 @@ service_env:
ml-reports-service: ../../../../ansible/roles/stack-sunbird/templates/ml-reports-service.env
ml-core-service: ../../../../ansible/roles/stack-sunbird/templates/ml-core-service.env
ml-projects-service: ../../../../ansible/roles/stack-sunbird/templates/ml-projects-service.env
ml-analytics-service: ../../../../ansible/roles/stack-sunbird/templates/ml-analytics-service.conf
inbound: ../../../../ansible/roles/stack-sunbird/templates/inbound.env
outbound:
- ../../../../ansible/roles/stack-sunbird/templates/outbound.env
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -201,10 +201,10 @@ public {
}
}
cloud_storage_type="{{cloud_service_provider}}"
storage.key.config="{{cloud_public_storage_accountname}}"
storage.secret.config="{{cloud_public_storage_secret}}"
public.storage.key.config="{{cloud_public_storage_accountname}}"
public.storage.secret.config="{{cloud_public_storage_secret}}"
storage.key.config="cloud_storage_key"
storage.secret.config="cloud_storage_secret"
public.storage.key.config="public_cloud_storage_key"
public.storage.secret.config="public_cloud_storage_secret"
cloud_storage_endpoint_with_protocol="{{cloud_storage_url}}"
metrics.time.interval.min=30
cache.refresh.time.interval.min=1440
Expand Down
294 changes: 294 additions & 0 deletions ansible/roles/stack-sunbird/templates/ml-analytics-service.conf

Large diffs are not rendered by default.

6 changes: 5 additions & 1 deletion ansible/roles/stack-sunbird/templates/ml-core-service.env
Original file line number Diff line number Diff line change
Expand Up @@ -87,4 +87,8 @@ ID={{ml_core_service_id | default(env_name+"."+sunbird_instance+".ml.core.servic
TELEMETRY_ON_OFF={{manage_learn_telemetry_on_off | default("ON")}}

#Topic name for telemetry
TELEMETRY_TOPIC={{telemetry_raw_topic_name | default(env_name+".telemetry.raw")}}
TELEMETRY_TOPIC={{telemetry_raw_topic_name | default(env_name+".telemetry.raw")}}

#Public base url of sunbird
PUBLIC_BASE_URL={{ml_core_public_base_url | default("")}}

Original file line number Diff line number Diff line change
Expand Up @@ -86,3 +86,4 @@ TELEMETRY_ON_OFF={{manage_learn_telemetry_on_off | default("ON")}}

#Topic name for telemetry
TELEMETRY_TOPIC={{telemetry_raw_topic_name | default(env_name+".telemetry.raw")}}

10 changes: 10 additions & 0 deletions ansible/roles/stack-sunbird/templates/sunbird_report-service.env
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,13 @@ sunbird_cache_ttl={{sunbird_cache_ttl}}
sunbird_report_sas_expiry_in_minutes=3600
sunbird_dataservice_url={{sunbird_dataservice_url}}

# 6.0.0

sunbird_cloud_storage_provider={{cloud_service_provider}}
cloud_private_storage_accountname={{cloud_private_storage_accountname | default("")}}
cloud_private_storage_secret={{cloud_private_storage_secret | default("")}}
cloud_storage_privatereports_bucketname={{cloud_storage_privatereports_bucketname | default("reports")}}
cloud_storage_resourceBundle_bucketname={{cloud_storage_label_bucketname | default("label")}}
cloud_private_storage_region={{cloud_private_storage_region | default("ap-south-1")}}
cloud_private_storage_project={{cloud_private_storage_project | default("")}}
cloud_private_storage_endpoint={{cloud_private_storage_endpoint | default("")}}
16 changes: 8 additions & 8 deletions kubernetes/helm_charts/core/analytics/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,14 @@ spec:
value: {{ .Values.env.min_heap | quote }}
- name: MAX_HEAP
value: {{ .Values.env.max_heap | quote }}
- name: azure_storage_secret
value: {{ .Values.env.azure_private_account_secret | quote }}
- name: azure_storage_key
value: {{ .Values.env.azure_private_account_name | quote }}
- name: public_azure_storage_secret
value: {{ .Values.env.azure_public_account_secret | quote }}
- name: public_azure_storage_key
value: {{ .Values.env.azure_public_account_name | quote }}
- name: cloud_storage_secret
value: {{ .Values.env.cloud_private_account_secret | quote }}
- name: cloud_storage_key
value: {{ .Values.env.cloud_private_account_name | quote }}
- name: public_cloud_storage_secret
value: {{ .Values.env.cloud_public_account_secret | quote }}
- name: public_cloud_storage_key
value: {{ .Values.env.cloud_public_account_name | quote }}
- name: _JAVA_OPTIONS
value: -Dlog4j2.formatMsgNoLookups=true
envFrom:
Expand Down
8 changes: 4 additions & 4 deletions kubernetes/helm_charts/core/analytics/values.j2
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ env:
javaoptions: {{analytics_java_mem_limit|default('-Xmx600m')}}
min_heap: {{analytics_min_heap_limit|default('-Xms1g')}}
max_heap: {{analytics_max_heap_limit|default('-Xmx2g')}}
azure_private_account_secret: {{ cloud_private_storage_secret }}
azure_private_account_name: {{ cloud_private_storage_accountname }}
azure_public_account_secret: {{ cloud_public_storage_secret }}
azure_public_account_name: {{ cloud_public_storage_accountname }}
cloud_private_account_secret: {{ cloud_private_storage_secret }}
cloud_private_account_name: {{ cloud_private_storage_accountname }}
cloud_public_account_secret: {{ cloud_public_storage_secret }}
cloud_public_account_name: {{ cloud_public_storage_accountname }}
replicaCount: {{analytics_replicacount|default(1)}}
repository: {{analytics_repository|default('sunbird-analytics-service')}}
image_tag: {{ image_tag }}
Expand Down
23 changes: 23 additions & 0 deletions kubernetes/helm_charts/core/ml-analytics-service/.helmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/
24 changes: 24 additions & 0 deletions kubernetes/helm_charts/core/ml-analytics-service/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
apiVersion: v2
name: ml-analytics-service
description: A Helm chart for Kubernetes

# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.0

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: "1.16.0"
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "ml-analytics-service.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "ml-analytics-service.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "ml-analytics-service.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Common labels
*/}}
{{- define "ml-analytics-service.labels" -}}
helm.sh/chart: {{ include "ml-analytics-service.chart" . }}
{{ include "ml-analytics-service.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

{{/*
Selector labels
*/}}
{{- define "ml-analytics-service.selectorLabels" -}}
app.kubernetes.io/name: {{ include "ml-analytics-service.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

{{/*
Create the name of the service account to use
*/}}
{{- define "ml-analytics-service.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "ml-analytics-service.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
{{- $keys := .Files.Glob "keys/*" }}
{{ if $keys }}
apiVersion: v1
kind: Secret
metadata:
name: ml-analytics-access-keys
type: Opaque
data:
{{ (.Files.Glob "keys/*").AsSecrets | indent 2 }}
{{ end }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Chart.Name }}
namespace: {{ .Values.namespace }}
annotations:
reloader.stakater.com/auto: "true"
spec:
replicas: {{ .Values.replicaCount }}
strategy:
rollingUpdate:
maxSurge: {{ .Values.strategy.maxsurge }}
maxUnavailable: {{ .Values.strategy.maxunavailable }}
selector:
matchLabels:
app: {{ .Chart.Name }}
template:
metadata:
labels:
app: {{ .Chart.Name }}
spec:
{{- if .Values.imagepullsecrets }}
imagePullSecrets:
- name: {{ .Values.imagepullsecrets }}
{{- end }}
volumes:
- name: {{ .Chart.Name }}-config
configMap:
name: {{ .Chart.Name }}-config
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.dockerhub }}/{{ .Values.repository }}:{{ .Values.image_tag }}"
imagePullPolicy: Always
resources:
{{ toYaml .Values.resources | indent 10 }}
ports:
- containerPort: {{ .Values.network.port }}
volumeMounts:
- name: {{ .Chart.Name }}-config
mountPath: /opt/sparkjobs/ml-analytics-service/config.ini
subPath: ml-analytics-service.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
apiVersion: v1
kind: Service
metadata:
name: {{ .Chart.Name }}
namespace: {{ .Values.namespace }}
labels:
app: {{ .Chart.Name }}
spec:
ports:
- name: http-{{ .Chart.Name }}
protocol: TCP
port: {{ .Values.network.targetport }}
selector:
app: {{ .Chart.Name }}
21 changes: 21 additions & 0 deletions kubernetes/helm_charts/core/ml-analytics-service/values.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
namespace: {{ namespace }}
imagepullsecrets: {{ imagepullsecrets }}
dockerhub: {{ dockerhub }}

replicaCount: {{ml_analytics_replicacount|default(1)}}
repository: {{ml_analytics_repository|default('ml-analytics-service')}}
image_tag: {{ image_tag }}
resources:
requests:
cpu: {{ml_analytics_cpu_req|default('1000m')}}
memory: {{ml_analytics_mem_req|default('1000Mi')}}
limits:
cpu: {{ml_analytics_cpu_limit|default('2')}}
memory: {{ml_analytics_mem_limit|default('2048Mi')}}
network:
port: 7000
targetport: 7000
strategy:
type: RollingUpdate
maxsurge: {{ ml_core_maxsurge|default('25%') }}
maxunavailable: {{ ml_core_maxunavailable|default('25%') }}
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ metadata:
data:
proxy-default.conf: |
{{ .Values.proxyconfig | indent 4 }}
{{- if eq .Values.csp "oci" }}
cors-proxy-default.conf: |
{{ .Values.corsproxyconfig | indent 4 }}
{{- end }}
compression.conf: |
{{ .Values.compressionConfig | indent 4 }}

Expand Down
Loading

0 comments on commit b2c4f1d

Please sign in to comment.