Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add possibility to configure key prefix and expiration time from env (Redis, MemCached) #2094

Merged
merged 5 commits into from
Feb 20, 2024
Merged

feat: Add possibility to configure key prefix and expiration time from env (Redis, MemCached) #2094

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
508dbd9
Unify sccache _KEY_PREFIX docs & processing
AJIOB Feb 12, 2024
5b4b49c
Add expiration time parsing for the Redis
AJIOB Feb 12, 2024
adaa9b3
Simplify one match
AJIOB Feb 12, 2024
05035bf
New memcached env vars were implemented
AJIOB Feb 13, 2024
b0332ca
Add different key prefix to the test
AJIOB Feb 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions docs/Configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@ configuration variables
* `SCCACHE_ENDPOINT` s3 endpoint
* `SCCACHE_REGION` s3 region, required if using AWS S3
* `SCCACHE_S3_USE_SSL` s3 endpoint requires TLS, set this to `true`
* `SCCACHE_S3_KEY_PREFIX` s3 key prefix (optional)

The endpoint used then becomes `${SCCACHE_BUCKET}.s3-{SCCACHE_REGION}.amazonaws.com`.
If you are not using the default endpoint and `SCCACHE_REGION` is undefined, it
Expand All @@ -141,17 +142,21 @@ will default to `us-east-1`.
* `SCCACHE_BUCKET` is the name of your R2 bucket.
* `SCCACHE_ENDPOINT` must follow the format of `https://<ACCOUNT_ID>.r2.cloudflarestorage.com`. Note that the `https://` must be included. Your account ID can be found [here](https://developers.cloudflare.com/fundamentals/get-started/basic-tasks/find-account-and-zone-ids/).
* `SCCACHE_REGION` should be set to `auto`.
* `SCCACHE_S3_KEY_PREFIX` s3 key prefix (optional).

#### redis

* `SCCACHE_REDIS` full redis url, including auth and access token/passwd
* `SCCACHE_REDIS_TTL` ttl for redis cache, don't set for default behavior.
* `SCCACHE_REDIS` full redis url, including auth and access token/passwd.
* `SCCACHE_REDIS_EXPIRATION` / `SCCACHE_REDIS_TTL` ttl for redis cache, don't set for default behavior.
* `SCCACHE_REDIS_KEY_PREFIX` key prefix (optional).

The full url appears then as `redis://user:[email protected]:6379/?db=1`.

#### memcached

* `SCCACHE_MEMCACHED` memcached url
* `SCCACHE_MEMCACHED` memcached url.
* `SCCACHE_MEMCACHED_EXPIRATION` ttl for memcached cache, don't set for default behavior.
* `SCCACHE_MEMCACHED_KEY_PREFIX` key prefix (optional).

#### gcs

Expand Down
3 changes: 3 additions & 0 deletions docs/Memcached.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,6 @@
Set `SCCACHE_MEMCACHED` to a [Memcached](https://memcached.org/) url in format `tcp://<hostname>:<port> ...` to store the cache in a Memcached instance.

Set `SCCACHE_MEMCACHED_EXPIRATION` to the default expiration seconds of memcached. The default value is `86400` (1 day) and can up to `2592000` (30 days). Set this value to `0` will disable the expiration. memcached will purge the cache entry while it exceed 30 days or meets LRU rules.

Set `SCCACHE_MEMCACHED_KEY_PREFIX` if you want to prefix all cache keys. This can be
useful when sharing a Memcached instance with another application or cache.
7 changes: 6 additions & 1 deletion docs/Redis.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,12 @@ Set `SCCACHE_REDIS` to a [Redis](https://redis.io/) url in format `redis://[[<us

Redis over TLS is supported. Use the [`rediss://`](https://www.iana.org/assignments/uri-schemes/prov/rediss) url scheme (note `rediss` vs `redis`). Append `#insecure` the the url to disable hostname verification and accept self-signed certificates (dangerous!). Note that this also disables [SNI](https://en.wikipedia.org/wiki/Server_Name_Indication).

Set `SCCACHE_REDIS_TTL` in seconds if you don't want your cache to live forever. This will override the default behavior of redis.
Set `SCCACHE_REDIS_EXPIRATION` in seconds if you don't want your cache to live forever. This will override the default behavior of redis.

`SCCACHE_REDIS_TTL` is a deprecated synonym for `SCCACHE_REDIS_EXPIRATION`.

Set `SCCACHE_REDIS_KEY_PREFIX` if you want to prefix all cache keys. This can be
useful when sharing a Redis instance with another application or cache.

## Examples

Expand Down
124 changes: 63 additions & 61 deletions src/config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -567,6 +567,28 @@ pub struct EnvConfig {
cache: CacheConfigs,
}

fn key_prefix_from_env_var(env_var_name: &str) -> String {
env::var(env_var_name)
.ok()
.as_ref()
.map(|s| s.trim_end_matches('/'))
.filter(|s| !s.is_empty())
.unwrap_or_default()
.to_owned()
}

fn number_from_env_var<A: std::str::FromStr>(env_var_name: &str) -> Option<Result<A>>
where
<A as FromStr>::Err: std::fmt::Debug,
{
let value = env::var(env_var_name).ok()?;

value
.parse::<A>()
.map_err(|err| anyhow!("{env_var_name} value is invalid: {err:?}"))
.into()
}

fn config_from_env() -> Result<EnvConfig> {
// ======= AWS =======
let s3 = if let Ok(bucket) = env::var("SCCACHE_BUCKET") {
Expand All @@ -591,13 +613,7 @@ fn config_from_env() -> Result<EnvConfig> {
),
})?;
let endpoint = env::var("SCCACHE_ENDPOINT").ok();
let key_prefix = env::var("SCCACHE_S3_KEY_PREFIX")
.ok()
.as_ref()
.map(|s| s.trim_end_matches('/'))
.filter(|s| !s.is_empty())
.map(|s| s.to_owned() + "/")
.unwrap_or_default();
let key_prefix = key_prefix_from_env_var("SCCACHE_S3_KEY_PREFIX");

Some(S3CacheConfig {
bucket,
Expand All @@ -620,33 +636,45 @@ fn config_from_env() -> Result<EnvConfig> {
}

// ======= redis =======
let ttl = match env::var("SCCACHE_REDIS_TTL") {
Ok(v) => v
.parse()
.map_err(|err| anyhow!("SCCACHE_REDIS_TTL value is invalid: {err:?}"))?,
Err(_) => DEFAULT_REDIS_CACHE_TTL,
let redis = if let Ok(url) = env::var("SCCACHE_REDIS") {
let ttl = number_from_env_var("SCCACHE_REDIS_EXPIRATION")
.or_else(|| number_from_env_var("SCCACHE_REDIS_TTL"))
.transpose()?
.unwrap_or(DEFAULT_REDIS_CACHE_TTL);

let key_prefix = key_prefix_from_env_var("SCCACHE_REDIS_KEY_PREFIX");

Some(RedisCacheConfig {
url,
ttl,
key_prefix,
})
} else {
None
};
let redis = env::var("SCCACHE_REDIS").ok().map(|url| RedisCacheConfig {
url,
ttl,
key_prefix: String::new(),
});

if env::var_os("SCCACHE_REDIS_EXPIRATION").is_some()
&& env::var_os("SCCACHE_REDIS_TTL").is_some()
{
bail!("You mustn't set both SCCACHE_REDIS_EXPIRATION and SCCACHE_REDIS_TTL. Use only one.");
}

// ======= memcached =======
let expiration = match env::var("SCCACHE_MEMCACHED_EXPIRATION").ok() {
None => DEFAULT_MEMCACHED_CACHE_EXPIRATION,
Some(v) => v
.parse()
.map_err(|err| anyhow!("SCCACHE_MEMCACHED_EXPIRATION value is invalid: {err:?}"))?,
};
let memcached = if let Ok(url) = env::var("SCCACHE_MEMCACHED") {
let expiration = number_from_env_var("SCCACHE_MEMCACHED_EXPIRATION")
.transpose()?
.unwrap_or(DEFAULT_MEMCACHED_CACHE_EXPIRATION);

let memcached = env::var("SCCACHE_MEMCACHED")
.ok()
.map(|url| MemcachedCacheConfig {
let key_prefix = key_prefix_from_env_var("SCCACHE_MEMCACHED_KEY_PREFIX");

Some(MemcachedCacheConfig {
url,
expiration,
key_prefix: String::new(),
});
key_prefix,
})
} else {
None
};

// ======= GCP/GCS =======
if (env::var("SCCACHE_GCS_CREDENTIALS_URL").is_ok()
Expand All @@ -660,15 +688,7 @@ fn config_from_env() -> Result<EnvConfig> {
}

let gcs = env::var("SCCACHE_GCS_BUCKET").ok().map(|bucket| {
let key_prefix = env::var("SCCACHE_GCS_KEY_PREFIX")
.ok()
.as_ref()
.map(|s| s.trim_end_matches('/'))
.filter(|s| !s.is_empty())
.unwrap_or_default()
.to_owned();


let key_prefix = key_prefix_from_env_var("SCCACHE_GCS_KEY_PREFIX");

if env::var("SCCACHE_GCS_OAUTH_URL").is_ok() {
eprintln!("SCCACHE_GCS_OAUTH_URL has been deprecated");
Expand Down Expand Up @@ -733,13 +753,7 @@ fn config_from_env() -> Result<EnvConfig> {
env::var("SCCACHE_AZURE_CONNECTION_STRING"),
env::var("SCCACHE_AZURE_BLOB_CONTAINER"),
) {
let key_prefix = env::var("SCCACHE_AZURE_KEY_PREFIX")
.ok()
.as_ref()
.map(|s| s.trim_end_matches('/'))
.filter(|s| !s.is_empty())
.unwrap_or_default()
.to_owned();
let key_prefix = key_prefix_from_env_var("SCCACHE_AZURE_KEY_PREFIX");
Some(AzureCacheConfig {
connection_string,
container,
Expand All @@ -751,13 +765,7 @@ fn config_from_env() -> Result<EnvConfig> {

// ======= WebDAV =======
let webdav = if let Ok(endpoint) = env::var("SCCACHE_WEBDAV_ENDPOINT") {
let key_prefix = env::var("SCCACHE_WEBDAV_KEY_PREFIX")
.ok()
.as_ref()
.map(|s| s.trim_end_matches('/'))
.filter(|s| !s.is_empty())
.unwrap_or_default()
.to_owned();
let key_prefix = key_prefix_from_env_var("SCCACHE_WEBDAV_KEY_PREFIX");
let username = env::var("SCCACHE_WEBDAV_USERNAME").ok();
let password = env::var("SCCACHE_WEBDAV_PASSWORD").ok();
let token = env::var("SCCACHE_WEBDAV_TOKEN").ok();
Expand All @@ -776,13 +784,7 @@ fn config_from_env() -> Result<EnvConfig> {
// ======= OSS =======
let oss = if let Ok(bucket) = env::var("SCCACHE_OSS_BUCKET") {
let endpoint = env::var("SCCACHE_OSS_ENDPOINT").ok();
let key_prefix = env::var("SCCACHE_OSS_KEY_PREFIX")
.ok()
.as_ref()
.map(|s| s.trim_end_matches('/'))
.filter(|s| !s.is_empty())
.unwrap_or_default()
.to_owned();
let key_prefix = key_prefix_from_env_var("SCCACHE_OSS_KEY_PREFIX");

let no_credentials =
env::var("SCCACHE_OSS_NO_CREDENTIALS").map_or(Ok(false), |val| match val.as_str() {
Expand Down Expand Up @@ -1202,7 +1204,7 @@ fn config_overrides() {
redis: Some(RedisCacheConfig {
url: "myotherredisurl".to_owned(),
ttl: 24 * 3600,
key_prefix: String::new(),
key_prefix: "/redis/prefix".into(),
}),
..Default::default()
},
Expand All @@ -1223,7 +1225,7 @@ fn config_overrides() {
}),
redis: Some(RedisCacheConfig {
url: "myredisurl".to_owned(),
ttl: 24 * 3600,
ttl: 25 * 3600,
key_prefix: String::new(),
}),
..Default::default()
Expand All @@ -1238,7 +1240,7 @@ fn config_overrides() {
cache: Some(CacheType::Redis(RedisCacheConfig {
url: "myotherredisurl".to_owned(),
ttl: 24 * 3600,
key_prefix: String::new(),
key_prefix: "/redis/prefix".into(),
}),),
fallback_cache: DiskCacheConfig {
dir: "/env-cache".into(),
Expand Down
Loading