Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Twilio Credentials #619

Merged
merged 12 commits into from
Nov 11, 2024
Merged

Twilio Credentials #619

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
e2a48c5
Tencent WeChat API App ID
babenek Nov 6, 2024
b46cc87
twilio credentials
babenek Nov 6, 2024
a2a3857
[skip actions] [twilio] 2024-11-06T15:37:13+02:00
babenek Nov 6, 2024
45d1b22
TokenPattern
babenek Nov 6, 2024
ec8a20f
custom BM
babenek Nov 6, 2024
7e5c447
fix
babenek Nov 6, 2024
5ab69b5
Merge branch 'main' into wechat
babenek Nov 8, 2024
9a7b5fd
[no ci] Merge branch 'wechat' into twilio
babenek Nov 8, 2024
c7fe01c
[no ci] Update credsweeper/rules/config.yaml
babenek Nov 8, 2024
91fe4ca
Apply suggestions from code review
babenek Nov 11, 2024
2610d7d
BM scores fix
babenek Nov 11, 2024
ead8a1e
test data fix
babenek Nov 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions .github/workflows/benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@ jobs:
- name: Checkout CredData
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
repository: Samsung/CredData
repository: babenek/CredData
ref: twillio
babenek marked this conversation as resolved.
Show resolved Hide resolved

- name: Markup hashing
run: |
Expand Down Expand Up @@ -85,7 +86,8 @@ jobs:
- name: Checkout CredData
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
repository: Samsung/CredData
repository: babenek/CredData
ref: twillio
babenek marked this conversation as resolved.
Show resolved Hide resolved

- name: Markup hashing
run: |
Expand Down Expand Up @@ -187,7 +189,8 @@ jobs:
- name: Checkout CredData
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
repository: Samsung/CredData
repository: babenek/CredData
ref: twillio
babenek marked this conversation as resolved.
Show resolved Hide resolved

- name: Markup hashing
run: |
Expand Down Expand Up @@ -374,7 +377,8 @@ jobs:
- name: Checkout CredData
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
repository: Samsung/CredData
repository: babenek/CredData
ref: twillio
babenek marked this conversation as resolved.
Show resolved Hide resolved

- name: Markup hashing
run: |
Expand Down
1 change: 1 addition & 0 deletions credsweeper/common/morpheme_checklist.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,7 @@ batt
beac
beans
beat
beef
begin
behav
behind
Expand Down
30 changes: 27 additions & 3 deletions credsweeper/rules/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -644,15 +644,39 @@
target:
- code

- name: Twilio API Key
- name: Twilio Credentials
severity: high
confidence: moderate
type: pattern
values:
- (?<![0-9A-Za-z_-])(?P<value>SK[0-9A-Fa-f]{32})(?![0-9A-Za-z_-])
filter_type: GeneralPattern
- (?<![0-9A-Za-z_-])(?P<value>(AC|AD|AL|CA|CF|CL|CN|CR|FW|IP|KS|MM|NO|PK|PN|QU|RE|SC|SD|SK|SM|TR|UT|XE|XR)[0-9A-Fa-f]{32})(?![0-9A-Za-z_-])
filter_type: TokenPattern
required_substrings:
- AC
- AD
- AL
- CA
- CF
- CL
- CN
- CR
- FW
- IP
- KS
- MM
- "NO"
- PK
- PN
- QU
- RE
- SC
- SD
- SK
- SM
- TR
- UT
- XE
- XR
min_line_len: 34
target:
- code
Expand Down
10 changes: 5 additions & 5 deletions tests/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
from pathlib import Path

# total number of files in test samples
SAMPLES_FILES_COUNT: int = 139
SAMPLES_FILES_COUNT: int = 140

# the lowest value of ML threshold is used to display possible lowest values
NEGLIGIBLE_ML_THRESHOLD = 0.0001

# credentials count after scan
SAMPLES_CRED_COUNT: int = 410
SAMPLES_CRED_LINE_COUNT: int = 428
SAMPLES_CRED_COUNT: int = 411
SAMPLES_CRED_LINE_COUNT: int = 429

# credentials count after post-processing
SAMPLES_POST_CRED_COUNT: int = 367
SAMPLES_POST_CRED_COUNT: int = 368

# with option --doc
SAMPLES_IN_DOC = 446
SAMPLES_IN_DOC = 447

# archived credentials that are not found without --depth
SAMPLES_IN_DEEP_1 = SAMPLES_POST_CRED_COUNT + 30
Expand Down
41 changes: 34 additions & 7 deletions tests/data/depth_3.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12955,18 +12955,18 @@
"api_validation": "NOT_AVAILABLE",
"ml_validation": "NOT_AVAILABLE",
"ml_probability": null,
"rule": "Twilio API Key",
"severity": "high",
"rule": "Twilio Credentials",
"severity": "critical",
"confidence": "moderate",
"line_data_list": [
{
"line": " \"mk\" : \"SK4D2F64E2A108CD72F648B1984C3B5A13\",",
"line": "SK4D2F64E2A108CD72F648B1984C3B5A13",
"line_num": 1,
"path": "./tests/samples/twilio_key.template",
"info": "./tests/samples/twilio_key.template|RAW",
"path": "./tests/samples/twilio_credentials_p",
"info": "./tests/samples/twilio_credentials_p|RAW",
"value": "SK4D2F64E2A108CD72F648B1984C3B5A13",
"value_start": 12,
"value_end": 46,
"value_start": 0,
"value_end": 34,
"variable": null,
"variable_start": -2,
"variable_end": -2,
Expand All @@ -12978,6 +12978,33 @@
}
]
},
{
"api_validation": "NOT_AVAILABLE",
"ml_validation": "NOT_AVAILABLE",
"ml_probability": null,
"rule": "Twilio Credentials",
"severity": "critical",
"confidence": "moderate",
"line_data_list": [
{
"line": "AC4d2f64e2a108cd72f648b1984c3b5a13",
"line_num": 2,
"path": "./tests/samples/twilio_credentials_p",
"info": "./tests/samples/twilio_credentials_p|RAW",
"value": "AC4d2f64e2a108cd72f648b1984c3b5a13",
"value_start": 0,
"value_end": 34,
"variable": null,
"variable_start": -2,
"variable_end": -2,
"entropy_validation": {
"iterator": "BASE36_CHARS",
"entropy": 3.7215926003976607,
"valid": true
}
}
]
},
{
"api_validation": "NOT_AVAILABLE",
"ml_validation": "VALIDATED_KEY",
Expand Down
41 changes: 34 additions & 7 deletions tests/data/doc.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13794,18 +13794,18 @@
"api_validation": "NOT_AVAILABLE",
"ml_validation": "NOT_AVAILABLE",
"ml_probability": null,
"rule": "Twilio API Key",
"severity": "high",
"rule": "Twilio Credentials",
"severity": "critical",
"confidence": "moderate",
"line_data_list": [
{
"line": "\"mk\" : \"SK4D2F64E2A108CD72F648B1984C3B5A13\",",
"line": "SK4D2F64E2A108CD72F648B1984C3B5A13",
"line_num": 1,
"path": "./tests/samples/twilio_key.template",
"info": "./tests/samples/twilio_key.template|RAW",
"path": "./tests/samples/twilio_credentials_p",
"info": "./tests/samples/twilio_credentials_p|RAW",
"value": "SK4D2F64E2A108CD72F648B1984C3B5A13",
"value_start": 12,
"value_end": 46,
"value_start": 0,
"value_end": 34,
"variable": null,
"variable_start": -2,
"variable_end": -2,
Expand All @@ -13817,6 +13817,33 @@
}
]
},
{
"api_validation": "NOT_AVAILABLE",
"ml_validation": "NOT_AVAILABLE",
"ml_probability": null,
"rule": "Twilio Credentials",
"severity": "critical",
"confidence": "moderate",
"line_data_list": [
{
"line": "AC4d2f64e2a108cd72f648b1984c3b5a13",
"line_num": 2,
"path": "./tests/samples/twilio_credentials_p",
"info": "./tests/samples/twilio_credentials_p|RAW",
"value": "AC4d2f64e2a108cd72f648b1984c3b5a13",
"value_start": 0,
"value_end": 34,
"variable": null,
"variable_start": -2,
"variable_end": -2,
"entropy_validation": {
"iterator": "BASE36_CHARS",
"entropy": 3.7215926003976607,
"valid": true
}
}
]
},
{
"api_validation": "NOT_AVAILABLE",
"ml_validation": "NOT_AVAILABLE",
Expand Down
39 changes: 33 additions & 6 deletions tests/data/ml_threshold.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10920,18 +10920,18 @@
"api_validation": "NOT_AVAILABLE",
"ml_validation": "NOT_AVAILABLE",
"ml_probability": null,
"rule": "Twilio API Key",
"severity": "high",
"rule": "Twilio Credentials",
"severity": "critical",
"confidence": "moderate",
"line_data_list": [
{
"line": "f2e6bca350950cdcde32e308cfe8364d4191dd2eec8c71a8aef673bdb2fd9158",
"line": "579d9ef4c0aae3cbc1384ca9cd7800e4b0b686607b84121f3584f00e08950f49",
"line_num": 1,
"path": "./tests/samples/twilio_key.template",
"path": "./tests/samples/twilio_credentials_p",
"info": "",
"value": "579d9ef4c0aae3cbc1384ca9cd7800e4b0b686607b84121f3584f00e08950f49",
"value_start": 12,
"value_end": 46,
"value_start": 0,
"value_end": 34,
"variable": null,
"variable_start": -2,
"variable_end": -2,
Expand All @@ -10943,6 +10943,33 @@
}
]
},
{
"api_validation": "NOT_AVAILABLE",
"ml_validation": "NOT_AVAILABLE",
"ml_probability": null,
"rule": "Twilio Credentials",
"severity": "critical",
"confidence": "moderate",
"line_data_list": [
{
"line": "40f36c54d27048b59e0a4f5cd72565bfa8b370b1544ebfd403b8768eb4f4063f",
"line_num": 2,
"path": "./tests/samples/twilio_credentials_p",
"info": "",
"value": "40f36c54d27048b59e0a4f5cd72565bfa8b370b1544ebfd403b8768eb4f4063f",
"value_start": 0,
"value_end": 34,
"variable": null,
"variable_start": -2,
"variable_end": -2,
"entropy_validation": {
"iterator": "BASE36_CHARS",
"entropy": 3.7215926003976607,
"valid": true
}
}
]
},
{
"api_validation": "NOT_AVAILABLE",
"ml_validation": "VALIDATED_KEY",
Expand Down
39 changes: 33 additions & 6 deletions tests/data/output.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9786,18 +9786,18 @@
"api_validation": "NOT_AVAILABLE",
"ml_validation": "NOT_AVAILABLE",
"ml_probability": null,
"rule": "Twilio API Key",
"severity": "high",
"rule": "Twilio Credentials",
"severity": "critical",
"confidence": "moderate",
"line_data_list": [
{
"line": " \"mk\" : \"SK4D2F64E2A108CD72F648B1984C3B5A13\",",
"line": "SK4D2F64E2A108CD72F648B1984C3B5A13",
"line_num": 1,
"path": "./tests/samples/twilio_key.template",
"path": "./tests/samples/twilio_credentials_p",
"info": "",
"value": "SK4D2F64E2A108CD72F648B1984C3B5A13",
"value_start": 12,
"value_end": 46,
"value_start": 0,
"value_end": 34,
"variable": null,
"variable_start": -2,
"variable_end": -2,
Expand All @@ -9809,6 +9809,33 @@
}
]
},
{
"api_validation": "NOT_AVAILABLE",
"ml_validation": "NOT_AVAILABLE",
"ml_probability": null,
"rule": "Twilio Credentials",
"severity": "critical",
"confidence": "moderate",
"line_data_list": [
{
"line": "AC4d2f64e2a108cd72f648b1984c3b5a13",
"line_num": 2,
"path": "./tests/samples/twilio_credentials_p",
"info": "",
"value": "AC4d2f64e2a108cd72f648b1984c3b5a13",
"value_start": 0,
"value_end": 34,
"variable": null,
"variable_start": -2,
"variable_end": -2,
"entropy_validation": {
"iterator": "BASE36_CHARS",
"entropy": 3.7215926003976607,
"valid": true
}
}
]
},
{
"api_validation": "NOT_AVAILABLE",
"ml_validation": "VALIDATED_KEY",
Expand Down
16 changes: 0 additions & 16 deletions tests/rules/test_twilio_key.py
View file
Open in desktop

This file was deleted.

2 changes: 2 additions & 0 deletions tests/samples/twilio_credentials_n
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
QpQcm9jLVR5cGU6IDQsRU5DU/SK4D2F64E2A108CD72F648Bx984C3B5A13+FSy1JbmZvOiBBRVMtMTI
AC123456E2A108CD72F648B1984C3B5A13
2 changes: 2 additions & 0 deletions tests/samples/twilio_credentials_p
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
SK4D2F64E2A108CD72F648B1984C3B5A13
AC4d2f64e2a108cd72f648b1984c3b5a13
1 change: 0 additions & 1 deletion tests/samples/twilio_key.template
View file
Open in desktop

This file was deleted.