Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

security: TestRotateCerts failed #143561

Open
cockroach-teamcity opened this issue Mar 27, 2025 · 4 comments
Open

security: TestRotateCerts failed #143561

cockroach-teamcity opened this issue Mar 27, 2025 · 4 comments
Assignees
@angles-n-daemons
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-db-server T-product-security T-security-engineering

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 27, 2025 •
edited by cockroach-jira-scripts
Loading

security.TestRotateCerts failed on master @ d6f6132e2970741a16692c306b52331a7dedf019:

GOROOT/src/time/time.go:187 time.(*Time).unixSec ???
pkg/sql/pgwire/auth_methods.go:459 pgwire.authCert.func1 ???
pkg/sql/pgwire/auth_behaviors.go:53 pgwire.(*AuthBehaviors).Authenticate ???
pkg/sql/pgwire/auth.go:184 pgwire.(*conn).handleAuthentication ???
pkg/sql/pgwire/conn.go:203 pgwire.(*conn).processCommands ???
pkg/sql/pgwire/server.go:1205 pgwire.(*Server).serveImpl.func4 ???

happened after
pkg/util/metric/aggmetric/agg_metric.go:135 aggmetric.(*childSet).add ??? <<<<<
pkg/util/metric/aggmetric/agg_metric.go:134 aggmetric.(*childSet).add ???
pkg/security/clientcert/cert_expiry_cache.go:167 clientcert.(*ClientCertExpirationCache).MaybeUpsert ???
pkg/security/clientcert/cert_expiry_cache.go:166 clientcert.(*ClientCertExpirationCache).MaybeUpsert ???
pkg/security/auth.go:362 security.UserAuthCertHook.func1 ???
GOROOT/src/time/time.go:187 time.(*Time).unixSec ???
pkg/sql/pgwire/auth_methods.go:459 pgwire.authCert.func1 ???
pkg/sql/pgwire/auth_behaviors.go:53 pgwire.(*AuthBehaviors).Authenticate ???
pkg/sql/pgwire/auth.go:184 pgwire.(*conn).handleAuthentication ???
pkg/sql/pgwire/conn.go:203 pgwire.(*conn).processCommands ???
pkg/sql/pgwire/server.go:1205 pgwire.(*Server).serveImpl.func4 ???

in another goroutine: happened before
pkg/util/metric/aggmetric/agg_metric.go:99 aggmetric.(*childSet).Each ??? <<<<<
pkg/util/metric/aggmetric/agg_metric.go:98 aggmetric.(*childSet).Each ???
pkg/security/certs_rotation_test.go:115 security_test.TestRotateCerts.func3 ???
pkg/security/certs_rotation_test.go:150 security_test.TestRotateCerts ???

happened after
pkg/security/clientcert/cert_expiry_cache.go:128 clientcert.(*ClientCertExpirationCache).GetExpiration ??? <<<<<
pkg/security/clientcert/cert_expiry_cache.go:127 clientcert.(*ClientCertExpirationCache).GetExpiration ???
pkg/security/clientcert/cert_expiry_cache.go:166 clientcert.(*ClientCertExpirationCache).MaybeUpsert.func1 ???
pkg/util/metric/aggmetric/gauge.go:187 aggmetric.(*Gauge).ToPrometheusMetric ???
pkg/util/metric/aggmetric/gauge.go:187 aggmetric.(*Gauge).ToPrometheusMetric ???
pkg/util/metric/aggmetric/agg_metric.go:102 aggmetric.(*childSet).Each.func1 ???
pkg/util/metric/aggmetric/agg_metric.go:257 aggmetric.BtreeWrapper.Do.func1 ???
external/com_github_google_btree/btree.go:523 btree.(*node).iterate ???
pkg/util/metric/aggmetric/agg_metric.go:259 aggmetric.BtreeWrapper.Do ???
pkg/util/metric/aggmetric/agg_metric.go:255 aggmetric.BtreeWrapper.Do ???
pkg/util/metric/aggmetric/agg_metric.go:116 aggmetric.(*childSet).Each ???
pkg/security/certs_rotation_test.go:115 security_test.TestRotateCerts.func3 ???
pkg/security/certs_rotation_test.go:150 security_test.TestRotateCerts ???

Other goroutines holding locks:
goroutine 16443 lock 0xc00291f278
pkg/util/metric/aggmetric/agg_metric.go:99 aggmetric.(*childSet).Each ??? <<<<<
pkg/util/metric/aggmetric/agg_metric.go:98 aggmetric.(*childSet).Each ???
pkg/security/certs_rotation_test.go:115 security_test.TestRotateCerts.func3 ???
pkg/security/certs_rotation_test.go:150 security_test.TestRotateCerts ???

Parameters:

  • attempt=1
  • deadlock=true
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/security-engineering @cockroachdb/server @cockroachdb/product-security

This test on roachdash | Improve this report!

Jira issue: CRDB-48915
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-db-server T-product-security T-security-engineering labels Mar 27, 2025
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Mar 27, 2025

This issue has multiple T-eam labels. Please make sure it only has one, or else issue synchronization will not work correctly.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

2 similar comments
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Mar 27, 2025

This issue has multiple T-eam labels. Please make sure it only has one, or else issue synchronization will not work correctly.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Mar 27, 2025

This issue has multiple T-eam labels. Please make sure it only has one, or else issue synchronization will not work correctly.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@cockroach-teamcity
Copy link
Member Author

security.TestRotateCerts failed on master @ e2e7260bff93d06dcd057c5c3f6018625e8a8d05:

GOROOT/src/time/time.go:187 time.(*Time).unixSec ???
pkg/sql/pgwire/auth_methods.go:459 pgwire.authCert.func1 ???
pkg/sql/pgwire/auth_behaviors.go:53 pgwire.(*AuthBehaviors).Authenticate ???
pkg/sql/pgwire/auth.go:184 pgwire.(*conn).handleAuthentication ???
pkg/sql/pgwire/conn.go:203 pgwire.(*conn).processCommands ???
pkg/sql/pgwire/server.go:1205 pgwire.(*Server).serveImpl.func4 ???

happened after
pkg/util/metric/aggmetric/agg_metric.go:135 aggmetric.(*childSet).add ??? <<<<<
pkg/util/metric/aggmetric/agg_metric.go:134 aggmetric.(*childSet).add ???
pkg/security/clientcert/cert_expiry_cache.go:167 clientcert.(*ClientCertExpirationCache).MaybeUpsert ???
pkg/security/clientcert/cert_expiry_cache.go:166 clientcert.(*ClientCertExpirationCache).MaybeUpsert ???
pkg/security/auth.go:362 security.UserAuthCertHook.func1 ???
GOROOT/src/time/time.go:187 time.(*Time).unixSec ???
pkg/sql/pgwire/auth_methods.go:459 pgwire.authCert.func1 ???
pkg/sql/pgwire/auth_behaviors.go:53 pgwire.(*AuthBehaviors).Authenticate ???
pkg/sql/pgwire/auth.go:184 pgwire.(*conn).handleAuthentication ???
pkg/sql/pgwire/conn.go:203 pgwire.(*conn).processCommands ???
pkg/sql/pgwire/server.go:1205 pgwire.(*Server).serveImpl.func4 ???

in another goroutine: happened before
pkg/util/metric/aggmetric/agg_metric.go:99 aggmetric.(*childSet).Each ??? <<<<<
pkg/util/metric/aggmetric/agg_metric.go:98 aggmetric.(*childSet).Each ???
pkg/security/certs_rotation_test.go:115 security_test.TestRotateCerts.func3 ???
pkg/security/certs_rotation_test.go:150 security_test.TestRotateCerts ???

happened after
pkg/security/clientcert/cert_expiry_cache.go:128 clientcert.(*ClientCertExpirationCache).GetExpiration ??? <<<<<
pkg/security/clientcert/cert_expiry_cache.go:127 clientcert.(*ClientCertExpirationCache).GetExpiration ???
pkg/security/clientcert/cert_expiry_cache.go:166 clientcert.(*ClientCertExpirationCache).MaybeUpsert.func1 ???
pkg/util/metric/aggmetric/gauge.go:187 aggmetric.(*Gauge).ToPrometheusMetric ???
pkg/util/metric/aggmetric/gauge.go:187 aggmetric.(*Gauge).ToPrometheusMetric ???
pkg/util/metric/aggmetric/agg_metric.go:102 aggmetric.(*childSet).Each.func1 ???
pkg/util/metric/aggmetric/agg_metric.go:257 aggmetric.BtreeWrapper.Do.func1 ???
external/com_github_google_btree/btree.go:523 btree.(*node).iterate ???
pkg/util/metric/aggmetric/agg_metric.go:259 aggmetric.BtreeWrapper.Do ???
pkg/util/metric/aggmetric/agg_metric.go:255 aggmetric.BtreeWrapper.Do ???
pkg/util/metric/aggmetric/agg_metric.go:116 aggmetric.(*childSet).Each ???
pkg/security/certs_rotation_test.go:115 security_test.TestRotateCerts.func3 ???
pkg/security/certs_rotation_test.go:150 security_test.TestRotateCerts ???

Other goroutines holding locks:
goroutine 16293 lock 0xc001269038
pkg/util/metric/aggmetric/agg_metric.go:99 aggmetric.(*childSet).Each ??? <<<<<
pkg/util/metric/aggmetric/agg_metric.go:98 aggmetric.(*childSet).Each ???
pkg/security/certs_rotation_test.go:115 security_test.TestRotateCerts.func3 ???
pkg/security/certs_rotation_test.go:150 security_test.TestRotateCerts ???

Parameters:

  • attempt=1
  • deadlock=true
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

dhartunian added a commit to dhartunian/cockroach that referenced this issue Mar 28, 2025
@dhartunian
security: fix bug with double lock acquisition
8e370e9 
Resolves: cockroachdb#143561
Resolves: cockroachdb#143560

Release note: None
@dhartunian dhartunian mentioned this issue Mar 28, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@angles-n-daemons angles-n-daemons

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-db-server T-product-security T-security-engineering
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

security: fix bug with double lock acquisition dhartunian/cockroach
2 participants
@angles-n-daemons @cockroach-teamcity