hallucination with log probs #281
Conversation
| tuple: A tuple containing two lists - filtered tokens and their corresponding probabilities. | ||
| """ | ||
| # Use regex to identify tokens without special characters | ||
| special_tokens = ["\\n", '{"', '":', ' "', '",', ' {"', '"}}\\n', " ", '"}}\n'] |
There was a problem hiding this comment.
is that exhaustive list, are there more to add? and how did you come up with this list?
| filtered_tokens = [token for token in tokens if token not in special_tokens] | ||
| filtered_probs = [ | ||
| prob for token, prob in zip(tokens, probs) if token not in special_tokens | ||
| ] |
There was a problem hiding this comment.
you could zip them in the beginning and then use zip(*) to unzip
| combined_token = tokens[i] # Start a new combination | ||
|
|
||
| # Check if the combined token matches any parameter name | ||
| for func, params in parameter_names.items(): |
| if found_param: | ||
| break # Exit the outer loop if parameter was matched | ||
| i += 1 # Move to the next token if no match was found yet |
There was a problem hiding this comment.
so even if one parameter is matched we will exit the loop?
| combined_token += tokens[ | ||
| i | ||
| ] # Append next token to the current combination |
There was a problem hiding this comment.
first param match will contain everything token from start to the matched token?
| return property_name in parameter_info | ||
|
|
||
|
|
||
| def calculate_entropy(log_probs): |
| - entropy (float): The calculated entropy. | ||
| - varentropy (float): The calculated variance of entropy. | ||
| """ | ||
| log_probs = torch.tensor(log_probs) |
| entropy_thd: float = 0.7, | ||
| varentropy_thd: float = 4.0, | ||
| ) -> bool: |
There was a problem hiding this comment.
take default values from consts like ENTRYPOY_DEFAULT_THRESHOLD = 0.7
| def hallucination_detect( | ||
| token: str, | ||
| log_probs: List[float], | ||
| current_state: Dict[str, Any], |
There was a problem hiding this comment.
does the caller need to see current_state, if not then maybe make this func stack param
| return log_probs.tolist(), entropy.item(), varentropy.item() | ||
|
|
||
|
|
||
| def hallucination_detect( |
There was a problem hiding this comment.
I think this current_state could be captured better if you declare rewrite it as class and move the state as class variables. I see that you didn't want to use global vars here and hence used state dict. But I think using class would be better here.
No description provided.