Skip to content

Commit

Permalink
Upgrade azure-eventhub to input v2 API (elastic#39511)
Browse files Browse the repository at this point in the history 
* Rename client to pipelineClient for clarity

* Rename azure to eventHubMetadata for clarity

* Move remaining code out of eph.go

* Remove erroneous error check

* Update tests

Switch from the v1 Outlet to the v2 PipelineClient

* Simplify run()

It seems we only need to wait for the context to be done here.

* Update NOTICE

* Clarify handler tradeoffs

The existing input version does not handle publishing acks from the
Beats pipeline. The input API v1 does not seem to offer this feature.

With the transition to the input API v2, we have acks management.
However, the legacy event hub SDK internally updates the checkpoint
info after a successful handler call, and does not seem to offer hooks
for acks management.

Since the new modern SDK offers better checkpoint management, we keep
the current behavior intact, and we'll implement ACKs in the
event hub input v2.

---------

Co-authored-by: Tiago Queiroz <[email protected]>
  • Loading branch information
@zmoog @belimawr
zmoog and belimawr authored Jun 28, 2024
1 parent 8bbcfcc commit 68e2b8b
Show file tree
Hide file tree
Showing 13 changed files with 565 additions and 576 deletions.
104 changes: 50 additions & 54 deletions NOTICE.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1742,11 +1742,11 @@ SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/azcore
Version: v1.9.0
Version: v1.11.1
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.0/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.11.1/LICENSE.txt:

MIT License

Expand All @@ -1773,11 +1773,11 @@ SOFTWARE

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/azidentity
Version: v1.4.0
Version: v1.5.2
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/azidentity@v1.4.0/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/azidentity@v1.5.2/LICENSE.txt:

MIT License

Expand Down Expand Up @@ -1954,11 +1954,11 @@ SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Version: v1.0.0
Version: v1.3.2
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.0.0/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.3.2/LICENSE.txt:

MIT License

Expand Down Expand Up @@ -17511,11 +17511,11 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------------------------------
Dependency : github.com/golang-jwt/jwt/v5
Version: v5.0.0
Version: v5.2.1
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/golang-jwt/jwt/v5@v5.0.0/LICENSE:
Contents of probable licence file $GOMODCACHE/github.com/golang-jwt/jwt/v5@v5.2.1/LICENSE:

Copyright (c) 2012 Dave Grijalva
Copyright (c) 2021 golang-jwt maintainers
Expand Down Expand Up @@ -29001,11 +29001,11 @@ Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-pipeline-g

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/internal
Version: v1.5.0
Version: v1.7.0
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/internal@v1.5.0/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/internal@v1.7.0/LICENSE.txt:

MIT License

Expand Down Expand Up @@ -29122,13 +29122,43 @@ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage
Version: v1.5.0
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/resourcemanager/storage/[email protected]/LICENSE.txt:

MIT License

Copyright (c) Microsoft Corporation. All rights reserved.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/Azure/go-amqp
Version: v1.0.0
Version: v1.0.5
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/[email protected].0/LICENSE:
Contents of probable licence file $GOMODCACHE/github.com/!azure/[email protected].5/LICENSE:

MIT License

Expand Down Expand Up @@ -30826,11 +30856,11 @@ SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/AzureAD/microsoft-authentication-library-for-go
Version: v1.1.1
Version: v1.2.2
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure!a!d/microsoft-authentication-library-for-go@v1.1.1/LICENSE:
Contents of probable licence file $GOMODCACHE/github.com/!azure!a!d/microsoft-authentication-library-for-go@v1.2.2/LICENSE:

MIT License

Expand Down Expand Up @@ -36984,40 +37014,6 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


--------------------------------------------------------------------------------
Dependency : github.com/dnaeon/go-vcr
Version: v1.2.0
Licence type (autodetected): BSD-2-Clause
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/dnaeon/[email protected]/LICENSE:

Copyright (c) 2015-2016 Marin Atanasov Nikolov <[email protected]>
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:

1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer
in this position and unchanged.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


--------------------------------------------------------------------------------
Dependency : github.com/dnephin/pflag
Version: v1.0.7
Expand Down Expand Up @@ -45608,11 +45604,11 @@ SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/joho/godotenv
Version: v1.3.0
Version: v1.5.1
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/joho/godotenv@v1.3.0/LICENCE:
Contents of probable licence file $GOMODCACHE/github.com/joho/godotenv@v1.5.1/LICENCE:

Copyright (c) 2013 John Barton

Expand Down Expand Up @@ -49585,11 +49581,11 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------------------------------
Dependency : github.com/pkg/browser
Version: v0.0.0-20210911075715-681adbf594b8
Version: v0.0.0-20240102092130-5ac0b6a4141c
Licence type (autodetected): BSD-2-Clause
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/pkg/[email protected]20210911075715-681adbf594b8/LICENSE:
Contents of probable licence file $GOMODCACHE/github.com/pkg/[email protected]20240102092130-5ac0b6a4141c/LICENSE:

Copyright (c) 2014, Dave Cheney <[email protected]>
All rights reserved.
Expand Down Expand Up @@ -49899,11 +49895,11 @@ Contents of probable licence file $GOMODCACHE/github.com/prometheus/client_golan

--------------------------------------------------------------------------------
Dependency : github.com/rogpeppe/go-internal
Version: v1.11.0
Version: v1.12.0
Licence type (autodetected): BSD-3-Clause
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/rogpeppe/go-internal@v1.11.0/LICENSE:
Contents of probable licence file $GOMODCACHE/github.com/rogpeppe/go-internal@v1.12.0/LICENSE:

Copyright (c) 2018 The Go Authors. All rights reserved.

Expand Down
17 changes: 9 additions & 8 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -180,14 +180,14 @@ require (
cloud.google.com/go v0.110.8
cloud.google.com/go/compute v1.23.0
cloud.google.com/go/redis v1.13.1
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.0
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.4.0
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.11.1
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.2
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/consumption/armconsumption v1.1.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v4 v4.6.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/costmanagement/armcostmanagement v1.1.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/monitor/armmonitor v0.8.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.1.1
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.3.2
github.com/Azure/go-autorest/autorest/adal v0.9.21
github.com/apache/arrow/go/v14 v14.0.2
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.1
Expand All @@ -209,7 +209,7 @@ require (
github.com/elastic/toutoumomoma v0.0.0-20221026030040-594ef30cb640
github.com/foxcpp/go-mockdns v0.0.0-20201212160233-ede2f9158d15
github.com/go-ldap/ldap/v3 v3.4.6
github.com/golang-jwt/jwt/v5 v5.0.0
github.com/golang-jwt/jwt/v5 v5.2.1
github.com/google/cel-go v0.19.0
github.com/googleapis/gax-go/v2 v2.12.0
github.com/gorilla/handlers v1.5.1
Expand Down Expand Up @@ -240,15 +240,15 @@ require (
github.com/AdaLogics/go-fuzz-headers v0.0.0-20230811130428-ced1acdcaa24 // indirect
github.com/Azure/azure-amqp-common-go/v4 v4.2.0 // indirect
github.com/Azure/azure-pipeline-go v0.2.3 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.0 // indirect
github.com/Azure/go-amqp v1.0.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.7.0 // indirect
github.com/Azure/go-amqp v1.0.5 // indirect
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
github.com/Azure/go-autorest/autorest/to v0.4.0 // indirect
github.com/Azure/go-autorest/autorest/validation v0.3.1 // indirect
github.com/Azure/go-autorest/logger v0.2.1 // indirect
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.1.1 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 // indirect
github.com/JohnCGriffin/overflow v0.0.0-20211019200055-46fa312c352c // indirect
github.com/andybalholm/brotli v1.0.5 // indirect
github.com/antlr4-go/antlr/v4 v4.13.0 // indirect
Expand Down Expand Up @@ -320,6 +320,7 @@ require (
github.com/jcmturner/gokrb5/v8 v8.4.2 // indirect
github.com/jcmturner/rpc/v2 v2.0.3 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/joho/godotenv v1.5.1 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/karrick/godirwalk v1.17.0 // indirect
Expand Down Expand Up @@ -348,7 +349,7 @@ require (
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.0.2 // indirect
github.com/pierrec/lz4 v2.6.0+incompatible // indirect
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
github.com/prometheus/client_golang v1.11.1 // indirect
github.com/rootless-containers/rootlesskit v1.1.0 // indirect
Expand Down
Loading

0 comments on commit 68e2b8b

Please sign in to comment.