Releases: Sanketika-Obsrv/obsrv-automation
v1.4.0
What's Changed
- private and public key & keycloak helm chart automation by @divyagovindaiah in #207
- Velero upgrade 7.1.1 by @divyagovindaiah in #221
- #OBS-I398 : added default_task_duration to env by @yashashkumar in #224
- configured the redis-exporter helmchart by @divyagovindaiah in #222
- Hudi metrics by @divyagovindaiah in #225
- #OBS-I320: updated oauth_users table by @Rakshitha-D in #229
- I392 fix:removed the schedule chnages in velero valyes.yaml file by @divyagovindaiah in #231
- fix: #OBS-I406 storage type support envs by @JeraldJF in #232
- #OBS-I416: alert names changes by @SurabhiAngadi in #235
- Lakehouse metrics by @divyagovindaiah in #234
- Release 1.4.0 by @manjudr in #236
New Contributors
- @yashashkumar made their first contribution in #224
- @Rakshitha-D made their first contribution in #229
Full Changelog: 1.3.1...1.4.0
v1.3.1
What's Changed
- Connector Instances API DDL changes by @SanthoshVasabhaktula in #200
- Automation fix by @divyagovindaiah in #197
- #OBS-I23: Service to send the metrics to prometheus by @SurabhiAngadi in #201
- #OBS-I338 spark worker fixes by @ravismula in #204
- 00: fix:automation fix by @divyagovindaiah in #202
- Open Telemetry Collector by @manjudr in #203
- Open telemetry loki exporter configurations by @manjudr in #205
- #OBS-I316 #OBS-I315 connector bundling scripts by @ravismula in #206
- Configure MiniIO for druid deep storage by @SurabhiAngadi in #208
- Updating of open telemetry configurations by @manjudr in #209
- #OBS-I380 - add connector information to the system events by @SanthoshVasabhaktula in #210
- Commented out logging and monitoring service variables by @SurabhiAngadi in #211
- Fix: indentation issue in global-cloud-values-azure.yaml by @divyagovindaiah in #214
- Updated image tags by @SurabhiAngadi in #215
- Configure MinIO for local datacenter installation. by @SurabhiAngadi in #212
- update flink connector namespace by @ravismula in #217
- #OBS-I345 feat: Add Dockerfile for Superset by @anandp504 in #220
- Release 1.3.1 by @ravismula in #226
- #OBS-I436 velero and loki fixes by @ravismula in #227
Full Changelog: 1.2.0-RC...1.3.1
v1.2.0-RC
🎉 Obsrv 1.2.0 RC Release
📅 Release Date: November 8, 2024
🌟 What’s New in Obsrv 1.2.0 RC?
Obsrv 1.2.0 RC is here, and it’s all about stability, performance, and introducing the powerful new Obsrv Management Console. Get ready for a redefined data management experience. 🚀
🚀 Core Innovations
🛠️ Transaction Data Lake Platform
- Major stabilization improvements for seamless integration with Obsrv.
- Effortlessly ingest and manage data in the transaction data lake platform.
🔗 Data Connector Framework
- Enhanced stability with rigorous testing and fixes for key issues.
- Now featuring open-source connectors for:
- Kafka
- JDBC
- Object Store
💡 Note: Sink connectors are not available in this release.
🌐 API Enhancements
🔧 Dataset Management APIs
Experience robust and reliable API performance with:
- Dataset CRUD operations
- Export/Import workflows
- Status transitions
- Schema generation
- Template query APIs
- Data ingestion and querying
These API updates ensure seamless interaction with your datasets and improve overall efficiency.
🔧 Obsrv Management Console
Say hello to the all-new Obsrv Management Console—your one-stop solution for managing datasets and connectors with ease and precision.
🌟 Key Features:
-
Effortless Dataset Management
- Easily manage datasets across their lifecycle: Draft → Ready for Publish → Published → Retired.
- Intuitive controls simplify even the most complex workflows.
-
Source Connector Configuration
- Configure Kafka, JDBC, and Object Store connectors directly from the console.
- Monitor and manage connectors with built-in tools.
-
Seamless Export & Import
- Transfer datasets between environments with robust Export and Import functionalities.
-
Automatic Schema Generation & Indexing
- Instantly generate schemas tailored to your datasets.
- Effortlessly index data from sources like Redis, Druid, and Lakehouse.
-
Proactive Alerts & Monitoring
- Set up intelligent alerts to monitor dataset health and ensure smooth operations.
-
Resource Monitoring
- Keep tabs on system performance with integrated resource monitoring tools.
Obsrv Management Console puts complete control at your fingertips, enabling your team to work smarter and faster. 🌟
🔄 Automation Enhancements
⚙️ Refined Automation Framework
- Stabilized scripts for greater reliability and ease of use.
- Broadened support for:
- Cloud platforms: AWS, GCP, Azure
- On-premises environments
- Streamlined installation for smooth deployments across any setup.
⚠️ Deprecation Notice
- Version 1.x APIs:
- APIs from version 1.0.x are now deprecated and unsupported.
- Transition to V2 APIs to access the latest features and ensure continued support.
📖 Migration Guide
A comprehensive guide to help you migrate from v1.0.x to v1.2.0 RC will be available soon. Stay tuned for step-by-step instructions.
🎯 Why Upgrade to Obsrv 1.2.0 RC?
This release offers unparalleled stability, performance improvements, and the all-new Obsrv Management Console. Upgrade now to unlock a new era of data management efficiency.
🚀 Obsrv 1.2.0 RC: Stability. Performance. Innovation.
before-refactor
What's Changed
- Azure terraform changes by @ManojKrishnaChintaluri in #150
- Local Installation Scripts by @ravismula in #151
- terraform destroy deletes cloudwatch log group by @divyagovindaiah in #152
- Add connector info into system events ingestion spec. by @sowmya-dixit in #153
- #8: Update Documentation for AWS and Azure by @ManojKrishnaChintaluri in #154
- Correct the redis ip address in the superset helm chart by @divyagovindaiah in #157
- Issue #304 postgres changes in datasource tables to support Hudi schema by @GayathriSrividya in #156
- Revert "Issue #304 postgres changes in datasource tables to support Hudi schema" by @GayathriSrividya in #159
- Release 1.0.5-GA by @ravismula in #162
- oss unified helm fix by @ravismula in #163
- Release 1.0.5-GA by @ravismula in #164
- Release v1.0.5-GA by @ravismula in #165
- Datasource DB schema changes to include type. by @sowmya-dixit in #169
- Terraform changes in obsrv to support lakehouse changes by @GayathriSrividya in #170
- Open sourcing transformer job by @SanthoshVasabhaktula in #173
- Merging Develop into Main by @GayathriSrividya in #171
New Contributors
- @SanthoshVasabhaktula made their first contribution in #173
Full Changelog: 1.0.4-GA...before-refactor
v1.0.4-GA
Release 1.0.4-GA (#155) * #0000 feat azure terraform changes (#150) * Local Installation Scripts (#151) * Sanketika-Obsrv/issue-tracker#26: Helm Changes for Local Setup * Sanketika-Obsrv/issue-tracker#26: CRD's for Local Setup * delete cloudwatch loggroups when using terraform destroy * Sanketika-Obsrv/issue-tracker#65: fix:Bug fix terraform destroy deletes cloudwatch log group * Sanketika-Obsrv/issue-tracker#65: fix:Bug fix terraform destroy deletes cloudwatch log group * Sanketika-Obsrv/issue-tracker#65: fix:Bug fix terraform destroy deletes cloudwatch log group * Add connector info into system events ingestion spec. (#153) * #8: Update Documentation for AWS and Azure (#154) * Sanketika-Obsrv/issue-tracker#8: doc: update shell script documentation for infra setup * Sanketika-Obsrv/issue-tracker#8: doc: update azure documentation * Sanketika-Obsrv/issue-tracker#8: doc: update AWS documentation for velero backup * Sanketika-Obsrv/issue-tracker#8: fix: typo in installation doc --------- Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: Ravi Mula <[email protected]> Co-authored-by: divyagovindaiah <[email protected]> Co-authored-by: Sowmya N Dixit <[email protected]>
v1.0.3-GA
Release 1.0.3-GA (#148) * Fix: removed docker secrets * updated the grafana_secrete * Feat: updated image tags according to release 1.1.0 * Issue #171 feat: update job to deployment for jobmanagers * #90 fix: Add default value for published_date in tables (#94) * fix: remove restart policy from flink deployment manifest (#96) * #196 Fix:Remove default region. (#95) * fix: #196 remove default value for region * update: #196 missing variable in variables.tf file * #196 update: example on how provide region * #196 update: comments on how to provide region * Separation of redis into dedup & meta/denorm. (#92) * feat: Separation of redis into dedup & meta/denorm. * feat: Separation of redis into dedup & meta/denorm. * feat: Separation of redis into dedup & meta/denorm. * feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * develop into release 1.3.0 (#97) * #90 fix: Add default value for published_date in tables (#94) * fix: remove restart policy from flink deployment manifest (#96) * #196 Fix:Remove default region. (#95) * fix: #196 remove default value for region * update: #196 missing variable in variables.tf file * #196 update: example on how provide region * #196 update: comments on how to provide region * Separation of redis into dedup & meta/denorm. (#92) * feat: Separation of redis into dedup & meta/denorm. * feat: Separation of redis into dedup & meta/denorm. * feat: Separation of redis into dedup & meta/denorm. * feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. --------- Co-authored-by: Praveen Veleneni <[email protected]> Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: Sowmya N Dixit <[email protected]> * #126 feat: add new property to datasources table (#99) * #126 feat: add new property to datasources table * #126 fix: allow property to be nullable * #0000 - feat: Fix masterdata stats ingestion spec (#100) * #0000 - feat: Fix masterdata stats ingestion spec * #0000 - feat: Add latency and processing times * #0000 - feat: Flink helm configuration changes (#103) * #0000 - feat: Flink helm configuration changes * #0000 - feat: Ingestion spec changes for system-events * #0000 - feat: Remove outdated ingestion specs * #0000 - feat: Secor changes for unified failed topic (#104) * feat: azure terraform changes to provision cluster (#106) * feat: azure terraform changes to provision cluster * cleanup azure terraform code * update node count and instance size for azure (#107) * Issue #305 feat: exclude system-events for the datasource validation (#108) * Issue #0000 fix: system events ingestion spec typo fix (#105) * Issue #0000 fix: system events ingestion spec typo fix * Issue #0000 fix: system-events ingestion spec update to add dataset_type * #0 fix: add unique relation for source config table (#110) * feat new image tags (#112) * * #0000 - feat: Fix masterdata stats ingestion spec * #0000 - feat: Add latency and processing times * * #0000 - feat: Flink helm configuration changes * #0000 - feat: Ingestion spec changes for system-events * #0000 - feat: Remove outdated ingestion specs * #000: Rebased the develop with release-1.0.0-GA * #0000 - feat: Import error dashboard (#119) * [ Secor Timzone ] - Infra automation changes to take the timezone property (#120) * #000: Configuring the helm to handle the instance level timezone * Removed the unwanted timezone properties * #203: Removed the commented code * #226 Feat: Update postgres migrations to add system settings and default values (#121) * #226 fix: add system_settings table * #226 fix: Add data encryption key to PostgreSQL migration * #226 fix: move system settings new migration script * update tags for ga release and fix undeclared variable in secor (#122) * System Telemetry Events Ingestion Spec and Backup Configurations (#118) * #223: Configured the auto submission of telemetry events ingestion spec and configured the secor backup for system telemetry events * #223: Updated the secor jobs variabl by including system-telemetry-events process * #203: typo error fix * #203: Configured the flag to submit the ingestion spec * #203: Configured the flag to submit the ingestion spec * #67 fix: add grant permissions for created table (#123) * #172 feat: add outputs for kafka service (#130) * #172 feat: add outputs for kafka service (#132) * #245 feat: Update Druid, PostgreSQL, and Superset versions (#133) * #0 feat: Update Druid, PostgreSQL, and Superset versions * #245 fix: Grant all privileges on schema to superset and druid_raw * #245 fix: Update database ownership in migration scripts * #241 Enhance Infrastructure Flexibility, Deployment Reliability, and Observability (#131) * #241:feat: Infra automation script * #241 feat: Toggle vpc creation, Toggle velero user creation * update: #241 main.tf for velero user creation * remove: #241 revert automation script changes * update: #241 Toggle to change service type and dataset api cm bug fix * #241:feat: Updated the infra setup shell script * #241:doc: Updated the setup doc file * #241:doc: Updated the setup doc file and removed commented code * #241:doc: Added the Prerequisites section * feat: #241 Toggle to enable cloudwatch logs for EKS * feat: #241 Deployment changes to provision infra * feat: # 241 Deployment script for provisioning infra * clean: #241 remove unwanted code. * update: #241 update helm version in shell script and remove overrides.tfvars file * enhance: #241 Enhance shell script create a tfvars from setup.conf file * fix: #241 typo in displayed message, fix terraform installation command * update: #241 eks cluster version * update: #241 kuberentes service type * update: #241 kubernetes service type to loadbalancer * Update helm version --------- Co-authored-by: Manjunath Davanam <[email protected]> * update: AKS cluster version (#134) * bug fix: #241 region typo and modify shell script (#135) * bug fix: #241 region typo and modify shell script * Bug fix: #241 modify velero module * Installation fixes (#138) * fix: installation-fixes and enhancements * update: [1/2] Enabling cluster logs by default. [2/2] Creating vpc, kong ingress, velero user by default * feat: Enable kafka connector and fix class name for kafka connector * update: add decrypt command * upgrade: eks version * dummy commit to reflect changes * dummy commit to reflect changes * modify desired node capacity * revert node capacity changes * #0000 upgrade eks add ons (#144) * #0000 update granularity for system events and system telemetry events (#145) * #0000 update granularity for system events and system telemetry events * code refactoring Signed-off-by: SurabhiAngadi <[email protected]> * code refactoring Signed-off-by: SurabhiAngadi <[email protected]> * feat: added cluster-overrides.tfvars Signed-off-by: SurabhiAngadi <[email protected]> --------- Signed-off-by: SurabhiAngadi <[email protected]> Co-authored-by: SurabhiAngadi <[email protected]> * fix: Update default max.message.bytes configuration of kafka topic to 10mb. (#147) * Installation fixes (#146) * #000:feat: Updated the obsrv automation script * #000:feat: Updated the obsrv automation script * #000:feat: Updated the configurations * #00 fix: update gcp automation scripts * #00 fix: update gcp installation instructions * feat: azure os helm chart * #0000 update aks node count. --------- Co-authored-by: Ravi Mula <[email protected]> Co-authored-by: ManojKrishnaChintaluri <[email protected]> --------- Signed-off-by: SurabhiAngadi <[email protected]> Co-authored-by: GayathriSrividya <[email protected]> Co-authored-by: divyagovindaiah <[email protected]> Co-authored-by: Manjunath Davanam <[email protected]> Co-authored-by: Praveen <[email protected]> Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: Sowmya N Dixit <[email protected]> Co-authored-by: Anand Parthasarathy <[email protected]> Co-authored-by: Ravinder Kumar <[email protected]> Co-authored-by: SurabhiAngadi <[email protected]> Co-authored-by: ManojKrishnaChintaluri <[email protected]>
v1.0.2-GA
What's Changed
- Release-1.0.2-GA to main by @manjudr in #141
- #0000 update release tags by @ManojKrishnaChintaluri in #142
- #0000 update image tags for 1.0.2-GA release by @ManojKrishnaChintaluri in #143
Full Changelog: 1.0.1-GA...1.0.2-GA
v1.0.1-GA
v1.3.0
What's Changed
- feat: adding AWS terraform code by @beepdot in #3
- Fix druid pvc and upgrade to 25.0.0 by @anandp504 in #5
- feat: updated terraform code for azure cloud by @beepdot in #6
- Flink helm changes for obsrv core by @anandp504 in #8
- Loki changes to api-service branch by @manjudr in #12
- Fix flink deployment configurations by @anandp504 in #14
- feat: adding loki, minio, promtail charts by @beepdot in #9
- Feat: Secor provision by @ManojKrishnaChintaluri in #15
- fix: adding PV for prom by @beepdot in #13
- issue: Feat: Deploying api service by @ManojKrishnaChintaluri in #11
- Infra automation by @ManojKrishnaChintaluri in #7
- Release 5.1.0 by @manjudr in #1
- issue: Feat: Add symbolic link for exporters in aws and azure by @ManojKrishnaChintaluri in #17
- Symlink exporters by @ManojKrishnaChintaluri in #20
- feat: rewrite terraform code as modules by @beepdot in #18
- fix: reordered vars by @beepdot in #21
- feat: adding oidc and sa iam role by @beepdot in #22
- feat: adding velero chart by @beepdot in #23
- fix: added service monitor label by @beepdot in #24
- fix: adding loki retention by @beepdot in #26
- fix: renamed vars by @beepdot in #27
- Enable flink checkpointing and configure kafka persistence by @anandp504 in #25
- Fix postgres user details for flink and dataset api by @anandp504 in #28
- Redis provisioning changes by @anandp504 in #29
- Use service accounts for flink and druid cloud storage access by @anandp504 in #30
- fix: iam sa changes for secor, flink, druid by @beepdot in #31
- Druid indexer default memory changes by @anandp504 in #35
- Obsrv Secor Helm chart changes by @manjudr in #34
- Initdb script to create tables by @GayathriSrividya in #32
- feat: ingestion spec for masterdata system stats by @GayathriSrividya in #36
- fix: issues when running multiple clusters by @beepdot in #39
- Deploy flink service account and secor fixes by @ManojKrishnaChintaluri in #37
- feat: added tags column for datasets by @GayathriSrividya in #38
- fix: dataset api variables, postgres indentation issue by @ManojKrishnaChintaluri in #40
- feat: increase node by @ManojKrishnaChintaluri in #43
- fix: adding vpc id to output by @beepdot in #44
- Issue #81 added masterdata system stats in submit_ingestion module by @GayathriSrividya in #46
- Issue #82 fix: update readme with instructions for AWS, GCP and Azure… by @AniketSaki in #48
- Flink release deploy by @ManojKrishnaChintaluri in #47
- #87 Web Console Deployment by @ManojKrishnaChintaluri in #49
- feat: Obsrv installation document by @GayathriSrividya in #51
- GCP automation scripts by @ravismula in #41
- Added installation steps for GCP and Azure by @GayathriSrividya in #53
- Issue #0 fix: refactor installation guide. by @AniketSaki in #56
- Update obsrv core by @SadanandGowda in #54
- Remove postgresql variables by @SadanandGowda in #57
- update: secor image repository by @ManojKrishnaChintaluri in #59
- #00 feat: Fix superset db variables by @anandp504 in #58
- #00 Feat: variabilise env in flink by @ManojKrishnaChintaluri in #60
- fix: install without failing first time by @beepdot in #63
- fix: adding s3 cors policy by @beepdot in #64
- #35 feat: Update dataset-api service by @pveleneni in #62
- Fix: configuration vars update for deployments by @ManojKrishnaChintaluri in #61
- #115 feat: changed dataset api image tag by @GayathriSrividya in #66
- Update dataset imagepullsecret by @SadanandGowda in #67
- feat: update helm values for all the services by @ManojKrishnaChintaluri in #70
- #0 fix: superset env variables fix by @pveleneni in #71
- update helm values, add limits for resources by @ManojKrishnaChintaluri in #69
- Add labels to all the services. by @ManojKrishnaChintaluri in #73
- Add limits to services and update helm charts of secor,druid,flink for multiple deepstorage configurations by @ManojKrishnaChintaluri in #74
- fix: issue fixes by @beepdot in #75
- Update eks addon version by @beepdot in #80
- #Issue144 Feat: deployment changes for flink command service in open … by @GayathriSrividya in #79
- #Issue: 140 Kafka connector deployment by @GayathriSrividya in #76
- #Issue142 Feat: removed unique constraint in source config table table by @GayathriSrividya in #77
- #Issue 142 Fix: removed extra comma from create table syntax by @GayathriSrividya in #82
- #0 feat: release 1.0.0 by @pveleneni in #84
- dataset-api-helmchart changes by @divyagovindaiah in #78
- Removal of docker secrets by @GayathriSrividya in #85
- dataset-api deployment indentation issue fix by @divyagovindaiah in #86
- Updated image tags according to release 1.1.0 by @GayathriSrividya in #87
- Issue #171 feat: update job to deployment for job managers by @pveleneni in #91
- develop into release 1.3.0 by @ravismula in #97
New Contributors
- @beepdot made their first contribution in #3
- @AniketSaki made their first contribution in #48
- @SadanandGowda made their first contribution in #54
Full Changelog: https://github.com/Sanketika-Obsrv/obsrv-automation/commits/1.3.0