"can't adapt type 'CombinedExpression'" in admin filter #89
Comments
|
@Datamance can you check whether this is fixed in the 1.9.5 release that went out today? |
|
@timabbott I just encountered the same issue in the same situation (trying to filter by flag in the admin) using Django 2.0.5 and django-bitfield 1.9.6. |
|
OK great. I don't have a lot of time for debugging this (I don't use the Django admin) but I imagine this isn't a deep issue, and I'd happily merge a patch for it. |
ortholeviator
added a commit
to ortholeviator/django-bitfield
that referenced
this issue
Apr 6, 2020
`Field.get_db_prep_lookup` merely "prepares" the right hand side operand, and returns a (sql, params) pair. For a plain value `params` will simply be the value itself; compound expressions with `as_sql` method, however, are never designed to work with `get_db_prep_lookup` and instead needs their `as_sql` method called directly to retrieve the pair. The error message has been changed from time to time across Django versions. Related issues include: * disqus#89 "can't adapt type 'CombinedExpression'" in admin filter Since Django 1.10 removed `Field.get_db_prep_lookup`, `Lookup.get_db_prep_lookup` implementation is simply as follows. ```python def get_db_prep_lookup(self, value, connection): return ('%s', [value]) ``` `CombinedExpression` is an expression type returned by combining exprssions with operators, such as `__add__` or `bitor`. They are never meant to be passed to the SQL engine directly; instead, it has an `as_sql` method that returns what we want. `process_rhs` first applies any bilateral transforms on both sides, and if it finds either `as_sql` or `_as_sql` method on the RHS, calls it and wraps the SQL in the pair with parentheses; otherwise, it diverts to the usual `get_db_prep_lookup` as above. * disqus#64 'CombinedExpression' object is not iterable when using an admin filter on Django 1.8 Same as above, except `Lookup.get_db_prep_lookup` is as follows. ```python def get_db_prep_lookup(self, value, connection): return ( '%s', self.lhs.output_field.get_db_prep_lookup( self.lookup_name, value, connection, prepared=True)) ``` Following is the relevant part of `Field.get_db_prep_lookup`. ```python def get_db_prep_lookup(self, lookup_type, value, connection, prepared=False): """ Returns field's value prepared for database lookup. """ if not prepared: value = self.get_prep_lookup(lookup_type, value) prepared = True if hasattr(value, 'get_compiler'): value = value.get_compiler(connection=connection) if hasattr(value, 'as_sql') or hasattr(value, '_as_sql'): # If the value has a relabeled_clone method it means the # value will be handled later on. if hasattr(value, 'relabeled_clone'): return value if hasattr(value, 'as_sql'): sql, params = value.as_sql() else: sql, params = value._as_sql(connection=connection) return QueryWrapper(('(%s)' % sql), params) if lookup_type in ('month', 'day', 'week_day', 'hour', 'minute', 'second', 'search', 'regex', 'iregex', 'contains', 'icontains', 'iexact', 'startswith', 'endswith', 'istartswith', 'iendswith'): return [value] ``` Normally it is supposed to return the parameters for the SQL "prepared statement" as a `list`; however, if the value has either `as_sql` or `_as_sql`, it either returns the value directly if it also has `relabeled_clone` (which is the case for expressions) or wraps it in `QueryWrapper`. This is not a desired behavior, and using them would result in `TypeError` saying the object was not iterable since they were not lists in the first place. * disqus#61 'SQLEvaluator' object is not iterable when using admin filter on Django 1.7 Same as above, except `SQLEvaluator` emerges from the SQL compiler wrapping expressions for internal use. - Update expressions: django.db.models.sql.compiler.SQLUpdateCompiler.as_sql: ... if hasattr(val, 'evaluate'): val = SQLEvaluator(val, self.query, allow_joins=False) ... - filter() resolution: django.db.models.sql.query.Query.add_filter: ... elif isinstance(value, ExpressionNode): # If value is a query expression, evaluate it value = SQLEvaluator(value, self, reuse=can_reuse) having_clause = value.contains_aggregate ... This commit also causes "regression" for the following issue; its legitimacy, however, is ambiguous considering that the lookup essentially "hijacks" the original definition of 'exact'. disqus#26 admin BitFieldListFilter not working as expected It should be noted, however, that the original behavior is still possible to achieve by wrapping the `int` in a `BitHandler`. Fixing this correctly would involve treating `int` as `Bit` as before while showing a deprecation warning. Meanwhile `filter(flags=Value(4))` will find a record of which `flags` is exactly `4` across all versions.
ortholeviator
added a commit
to ortholeviator/django-bitfield
that referenced
this issue
Apr 6, 2020
This was referenced Apr 6, 2020
ortholeviator
added a commit
to ortholeviator/django-bitfield
that referenced
this issue
Apr 6, 2020
`FieldListFilter.__init__` populates `used_parameters` as follows.
for p in self.expected_parameters():
if p in params:
value = params.pop(p)
self.used_parameters[p] = prepare_lookup_value(p, value)
`FieldListFilter.expected_parameters` simply returns a singleton list
containing `self.lookup_kwarg`, which is simply a copy of `field_path`.
`used_parameters` therefore has at most one entry with key set to
`field_path` and value from the parameters, since `prepare_lookup_value`
returns the value as-is unless the key ends with either `'__in'` or
`'__isnull'`.
`BitHandler` "represents an array of bits, each as a `Bit` object."
It is one of two classes (the other being `Bit`) that is accepted by the
specialized `exact` lookup. Here, we would only like to filter out
records which has all of the bits in the given integer parameter set.
The lookup uses the value in the given `BitHandler` as a mask, in the
form `field & mask = mask`.
The keys are not used here, since they are simply used for named bit
enumeration and manipulation. This is justified for the following
reasons:
1. The user may supply an integer with multiple bits set which `Bit`
cannot handle.
2. The class is unlikely to change to adopt a strict behavior to filter out anything not in `_keys`
3. This commit only aims to fix the bug and it would be out of the scope.
Later version may make the `keys` parameter of `BitHandler optional
or introduce some new wrapper object or a custom lookup for achieving
this.
This is a bit more efficient implementation compared to the intended one
since instead of doing `field = field | mask`, it instead chooses the
straightforward approach of `field & mask = mask`. SQL databases are
more likely to recognize the last pattern for optimization, although they
should in all way be equivalent.
Plus, empty keyword arguments for `filter` essentially means a no-op, so
in this case we just return `queryset` directly since they are meant to
be immutable anyway.
Fixes disqus#61, disqus#64, and disqus#89
ortholeviator
added a commit
to ortholeviator/django-bitfield
that referenced
this issue
Apr 6, 2020
`Field.get_db_prep_lookup` merely "prepares" the right hand side operand, and returns a (sql, params) pair. For a plain value `params` will simply be the value itself; compound expressions with `as_sql` method, however, are never designed to work with `get_db_prep_lookup` and instead needs their `as_sql` method called directly to retrieve the pair. The error message has been changed from time to time across Django versions. Related issues include: * disqus#89 "can't adapt type 'CombinedExpression'" in admin filter Since Django 1.10 removed `Field.get_db_prep_lookup`, `Lookup.get_db_prep_lookup` implementation is simply as follows. ```python def get_db_prep_lookup(self, value, connection): return ('%s', [value]) ``` `CombinedExpression` is an expression type returned by combining exprssions with operators, such as `__add__` or `bitor`. They are never meant to be passed to the SQL engine directly; instead, it has an `as_sql` method that returns what we want. `process_rhs` first applies any bilateral transforms on both sides, and if it finds either `as_sql` or `_as_sql` method on the RHS, calls it and wraps the SQL in the pair with parentheses; otherwise, it diverts to the usual `get_db_prep_lookup` as above. * disqus#64 'CombinedExpression' object is not iterable when using an admin filter on Django 1.8 Same as above, except `Lookup.get_db_prep_lookup` is as follows. ```python def get_db_prep_lookup(self, value, connection): return ( '%s', self.lhs.output_field.get_db_prep_lookup( self.lookup_name, value, connection, prepared=True)) ``` Following is the relevant part of `Field.get_db_prep_lookup`. ```python def get_db_prep_lookup(self, lookup_type, value, connection, prepared=False): """ Returns field's value prepared for database lookup. """ if not prepared: value = self.get_prep_lookup(lookup_type, value) prepared = True if hasattr(value, 'get_compiler'): value = value.get_compiler(connection=connection) if hasattr(value, 'as_sql') or hasattr(value, '_as_sql'): # If the value has a relabeled_clone method it means the # value will be handled later on. if hasattr(value, 'relabeled_clone'): return value if hasattr(value, 'as_sql'): sql, params = value.as_sql() else: sql, params = value._as_sql(connection=connection) return QueryWrapper(('(%s)' % sql), params) if lookup_type in ('month', 'day', 'week_day', 'hour', 'minute', 'second', 'search', 'regex', 'iregex', 'contains', 'icontains', 'iexact', 'startswith', 'endswith', 'istartswith', 'iendswith'): return [value] ``` Normally it is supposed to return the parameters for the SQL "prepared statement" as a `list`; however, if the value has either `as_sql` or `_as_sql`, it either returns the value directly if it also has `relabeled_clone` (which is the case for expressions) or wraps it in `QueryWrapper`. This is not a desired behavior, and using them would result in `TypeError` saying the object was not iterable since they were not lists in the first place. * disqus#61 'SQLEvaluator' object is not iterable when using admin filter on Django 1.7 Same as above, except `SQLEvaluator` emerges from the SQL compiler wrapping expressions for internal use. - Update expressions: django.db.models.sql.compiler.SQLUpdateCompiler.as_sql: ... if hasattr(val, 'evaluate'): val = SQLEvaluator(val, self.query, allow_joins=False) ... - filter() resolution: django.db.models.sql.query.Query.add_filter: ... elif isinstance(value, ExpressionNode): # If value is a query expression, evaluate it value = SQLEvaluator(value, self, reuse=can_reuse) having_clause = value.contains_aggregate ... This commit also causes "regression" for the following issue; its legitimacy, however, is ambiguous considering that the lookup essentially "hijacks" the original definition of 'exact'. disqus#26 admin BitFieldListFilter not working as expected It should be noted, however, that the original behavior is still possible to achieve by wrapping the `int` in a `BitHandler`. Fixing this correctly would involve treating `int` as `Bit` as before while showing a deprecation warning. Meanwhile `filter(flags=Value(4))` will find a record of which `flags` is exactly `4` across all versions.
ortholeviator
added a commit
to ortholeviator/django-bitfield
that referenced
this issue
Apr 6, 2020
`Field.get_db_prep_lookup` merely "prepares" the right hand side operand, and returns a (sql, params) pair. For a plain value `params` will simply be the value itself; compound expressions with `as_sql` method, however, are never designed to work with `get_db_prep_lookup` and instead needs their `as_sql` method called directly to retrieve the pair. The error message has been changed from time to time across Django versions. Related issues include: * disqus#89 "can't adapt type 'CombinedExpression'" in admin filter Since Django 1.10 removed `Field.get_db_prep_lookup`, `Lookup.get_db_prep_lookup` implementation is simply as follows. ```python def get_db_prep_lookup(self, value, connection): return ('%s', [value]) ``` `CombinedExpression` is an expression type returned by combining exprssions with operators, such as `__add__` or `bitor`. They are never meant to be passed to the SQL engine directly; instead, it has an `as_sql` method that returns what we want. `process_rhs` first applies any bilateral transforms on both sides, and if it finds either `as_sql` or `_as_sql` method on the RHS, calls it and wraps the SQL in the pair with parentheses; otherwise, it diverts to the usual `get_db_prep_lookup` as above. * disqus#64 'CombinedExpression' object is not iterable when using an admin filter on Django 1.8 Same as above, except `Lookup.get_db_prep_lookup` is as follows. ```python def get_db_prep_lookup(self, value, connection): return ( '%s', self.lhs.output_field.get_db_prep_lookup( self.lookup_name, value, connection, prepared=True)) ``` Following is the relevant part of `Field.get_db_prep_lookup`. ```python def get_db_prep_lookup(self, lookup_type, value, connection, prepared=False): """ Returns field's value prepared for database lookup. """ if not prepared: value = self.get_prep_lookup(lookup_type, value) prepared = True if hasattr(value, 'get_compiler'): value = value.get_compiler(connection=connection) if hasattr(value, 'as_sql') or hasattr(value, '_as_sql'): # If the value has a relabeled_clone method it means the # value will be handled later on. if hasattr(value, 'relabeled_clone'): return value if hasattr(value, 'as_sql'): sql, params = value.as_sql() else: sql, params = value._as_sql(connection=connection) return QueryWrapper(('(%s)' % sql), params) if lookup_type in ('month', 'day', 'week_day', 'hour', 'minute', 'second', 'search', 'regex', 'iregex', 'contains', 'icontains', 'iexact', 'startswith', 'endswith', 'istartswith', 'iendswith'): return [value] ``` Normally it is supposed to return the parameters for the SQL "prepared statement" as a `list`; however, if the value has either `as_sql` or `_as_sql`, it either returns the value directly if it also has `relabeled_clone` (which is the case for expressions) or wraps it in `QueryWrapper`. This is not a desired behavior, and using them would result in `TypeError` saying the object was not iterable since they were not lists in the first place. * disqus#61 'SQLEvaluator' object is not iterable when using admin filter on Django 1.7 Same as above, except `SQLEvaluator` emerges from the SQL compiler wrapping expressions for internal use. - Update expressions: django.db.models.sql.compiler.SQLUpdateCompiler.as_sql: ... if hasattr(val, 'evaluate'): val = SQLEvaluator(val, self.query, allow_joins=False) ... - filter() resolution: django.db.models.sql.query.Query.add_filter: ... elif isinstance(value, ExpressionNode): # If value is a query expression, evaluate it value = SQLEvaluator(value, self, reuse=can_reuse) having_clause = value.contains_aggregate ... This commit also causes "regression" for the following issue; its legitimacy, however, is ambiguous considering that the lookup essentially "hijacks" the original definition of 'exact'. disqus#26 admin BitFieldListFilter not working as expected It should be noted, however, that the original behavior is still possible to achieve by wrapping the `int` in a `BitHandler`. Fixing this correctly would involve treating `int` as `Bit` as before while showing a deprecation warning. Meanwhile `filter(flags=Value(4))` will find a record of which `flags` is exactly `4` across all versions.
timabbott
pushed a commit
that referenced
this issue
Apr 6, 2020
`FieldListFilter.__init__` populates `used_parameters` as follows.
for p in self.expected_parameters():
if p in params:
value = params.pop(p)
self.used_parameters[p] = prepare_lookup_value(p, value)
`FieldListFilter.expected_parameters` simply returns a singleton list
containing `self.lookup_kwarg`, which is simply a copy of `field_path`.
`used_parameters` therefore has at most one entry with key set to
`field_path` and value from the parameters, since `prepare_lookup_value`
returns the value as-is unless the key ends with either `'__in'` or
`'__isnull'`.
`BitHandler` "represents an array of bits, each as a `Bit` object."
It is one of two classes (the other being `Bit`) that is accepted by the
specialized `exact` lookup. Here, we would only like to filter out
records which has all of the bits in the given integer parameter set.
The lookup uses the value in the given `BitHandler` as a mask, in the
form `field & mask = mask`.
The keys are not used here, since they are simply used for named bit
enumeration and manipulation. This is justified for the following
reasons:
1. The user may supply an integer with multiple bits set which `Bit`
cannot handle.
2. The class is unlikely to change to adopt a strict behavior to filter out anything not in `_keys`
3. This commit only aims to fix the bug and it would be out of the scope.
Later version may make the `keys` parameter of `BitHandler optional
or introduce some new wrapper object or a custom lookup for achieving
this.
This is a bit more efficient implementation compared to the intended one
since instead of doing `field = field | mask`, it instead chooses the
straightforward approach of `field & mask = mask`. SQL databases are
more likely to recognize the last pattern for optimization, although they
should in all way be equivalent.
Plus, empty keyword arguments for `filter` essentially means a no-op, so
in this case we just return `queryset` directly since they are meant to
be immutable anyway.
Fixes #61, #64, and #89
|
I believe this is fixed in #103. |
timabbott
pushed a commit
that referenced
this issue
Apr 6, 2020
`Field.get_db_prep_lookup` merely "prepares" the right hand side operand, and returns a (sql, params) pair. For a plain value `params` will simply be the value itself; compound expressions with `as_sql` method, however, are never designed to work with `get_db_prep_lookup` and instead needs their `as_sql` method called directly to retrieve the pair. The error message has been changed from time to time across Django versions. Related issues include: * #89 "can't adapt type 'CombinedExpression'" in admin filter Since Django 1.10 removed `Field.get_db_prep_lookup`, `Lookup.get_db_prep_lookup` implementation is simply as follows. ```python def get_db_prep_lookup(self, value, connection): return ('%s', [value]) ``` `CombinedExpression` is an expression type returned by combining exprssions with operators, such as `__add__` or `bitor`. They are never meant to be passed to the SQL engine directly; instead, it has an `as_sql` method that returns what we want. `process_rhs` first applies any bilateral transforms on both sides, and if it finds either `as_sql` or `_as_sql` method on the RHS, calls it and wraps the SQL in the pair with parentheses; otherwise, it diverts to the usual `get_db_prep_lookup` as above. * #64 'CombinedExpression' object is not iterable when using an admin filter on Django 1.8 Same as above, except `Lookup.get_db_prep_lookup` is as follows. ```python def get_db_prep_lookup(self, value, connection): return ( '%s', self.lhs.output_field.get_db_prep_lookup( self.lookup_name, value, connection, prepared=True)) ``` Following is the relevant part of `Field.get_db_prep_lookup`. ```python def get_db_prep_lookup(self, lookup_type, value, connection, prepared=False): """ Returns field's value prepared for database lookup. """ if not prepared: value = self.get_prep_lookup(lookup_type, value) prepared = True if hasattr(value, 'get_compiler'): value = value.get_compiler(connection=connection) if hasattr(value, 'as_sql') or hasattr(value, '_as_sql'): # If the value has a relabeled_clone method it means the # value will be handled later on. if hasattr(value, 'relabeled_clone'): return value if hasattr(value, 'as_sql'): sql, params = value.as_sql() else: sql, params = value._as_sql(connection=connection) return QueryWrapper(('(%s)' % sql), params) if lookup_type in ('month', 'day', 'week_day', 'hour', 'minute', 'second', 'search', 'regex', 'iregex', 'contains', 'icontains', 'iexact', 'startswith', 'endswith', 'istartswith', 'iendswith'): return [value] ``` Normally it is supposed to return the parameters for the SQL "prepared statement" as a `list`; however, if the value has either `as_sql` or `_as_sql`, it either returns the value directly if it also has `relabeled_clone` (which is the case for expressions) or wraps it in `QueryWrapper`. This is not a desired behavior, and using them would result in `TypeError` saying the object was not iterable since they were not lists in the first place. * #61 'SQLEvaluator' object is not iterable when using admin filter on Django 1.7 Same as above, except `SQLEvaluator` emerges from the SQL compiler wrapping expressions for internal use. - Update expressions: django.db.models.sql.compiler.SQLUpdateCompiler.as_sql: ... if hasattr(val, 'evaluate'): val = SQLEvaluator(val, self.query, allow_joins=False) ... - filter() resolution: django.db.models.sql.query.Query.add_filter: ... elif isinstance(value, ExpressionNode): # If value is a query expression, evaluate it value = SQLEvaluator(value, self, reuse=can_reuse) having_clause = value.contains_aggregate ... This commit also causes "regression" for the following issue; its legitimacy, however, is ambiguous considering that the lookup essentially "hijacks" the original definition of 'exact'. #26 admin BitFieldListFilter not working as expected It should be noted, however, that the original behavior is still possible to achieve by wrapping the `int` in a `BitHandler`. Fixing this correctly would involve treating `int` as `Bit` as before while showing a deprecation warning. Meanwhile `filter(flags=Value(4))` will find a record of which `flags` is exactly `4` across all versions.
|
If this was indeed fixed in #103, it has been broken again in the mean time. It doesn't work for me on Django 2.2.15 |
|
Essentially no development has happened since #103, so it seems more likely that it didn't fix your situation. I'd encourage you to put together a reproducer and/or PR; I mostly only have time to review PRs and publish releases for this project. |
Open
|
I got this error too. Add quick fix at #118 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Getting an error with Django 1.11.21 and the latest PyPI release of this library (1.9.3) -
django.db.utils.ProgrammingError: can't adapt type 'CombinedExpression'When I try to filter by a flag in admin.
The text was updated successfully, but these errors were encountered: