Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core: data_source block, replace variables in filter.tags #2613

Merged
merged 3 commits into from
Nov 21, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 31 additions & 6 deletions core/src/blocks/data_source.rs
Original file line number Diff line number Diff line change
Expand Up @@ -55,12 +55,8 @@ impl DataSource {
})
}

fn replace_query_variables(text: &str, env: &Env) -> Result<String> {
replace_variables_in_string(text, "query", env)
}

fn query(&self, env: &Env) -> Result<String> {
let mut query = Self::replace_query_variables(&self.query, env)?;
let mut query = replace_variables_in_string(&self.query, "query", env)?;

// replace <DUST_TRIPLE_BACKTICKS> with ```
query = query.replace("<DUST_TRIPLE_BACKTICKS>", "```");
Expand Down Expand Up @@ -236,7 +232,36 @@ impl Block for DataSource {

let filter = match config {
Some(v) => match v.get("filter") {
Some(v) => Some(SearchFilter::from_json_str(v.to_string().as_str())?),
Some(v) => {
let mut filter = SearchFilter::from_json_str(v.to_string().as_str())?;

match filter.tags.as_mut() {
Some(tags) => {
let replace_tags = |tags: &mut Vec<String>, field: &str| {
for t in tags.iter_mut() {
*t = replace_variables_in_string(t, field, env)?;
}
Ok::<_, anyhow::Error>(())
};

match tags.is_in.as_mut() {
Some(is_in) => {
replace_tags(is_in, "filter.tags.is_in")?;
}
None => (),
};
match tags.is_not.as_mut() {
Some(is_not) => {
replace_tags(is_not, "filter.tags.is_not")?;
}
None => (),
};
}
None => (),
};

Some(filter)
}
_ => None,
},
_ => None,
Expand Down
3 changes: 3 additions & 0 deletions docs/src/pages/core-blocks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -454,5 +454,8 @@ for more details about how documents are chunked to enable semantic search.
"timestamp": {"gt": 1675215950729, "lt": 1680012404017}
}
```
The `tags.in` and `tags.not` values can be templated using the
[Tera](https://keats.github.io/tera/docs/#templates) templating language (see the [LLM
block](/core-blocks#llm-block) documentation for more details on templating).
</Property>
</Properties>