Reverse lookup

[sinisaos]

Related to #378. Prototype for reverse foreign key lookup. Any suggestions and help are welcome. If I missed the point or it's not a good api design feel free to close PR.
Usage of reverse FK lookup:

from piccolo.table import Table
from piccolo.columns import Varchar, ForeignKey
from piccolo.columns.reference import LazyTableReference
from piccolo.columns.reverse_lookup import ReverseLookup

class Manager(Table):
    name = Varchar()
    artists = ReverseLookup(
        LazyTableReference("Artist", module_path=__name__), reverse_fk="manager"
    )

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(Manager)

rows = Manager.select(
        Manager.name, Manager.artists() # or as list Manager.artists(Artist.name, as_list=True)
    ).run_sync()

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 88.79310% with 13 lines in your changes missing coverage. Please review.

Project coverage is 91.33%. Comparing base (ff26a88) to head (a7405b7).
Report is 157 commits behind head on master.

Files	Patch %	Lines
piccolo/columns/reverse_lookup.py	88.73%	8 Missing ⚠️
piccolo/query/methods/select.py	87.80%	5 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #599      +/-   ##
==========================================
- Coverage   91.35%   91.33%   -0.02%     
==========================================
  Files         108      109       +1     
  Lines        7517     7629     +112     
==========================================
+ Hits         6867     6968     +101     
- Misses        650      661      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dantownsend]

Thanks for this! It looks very promising.

[sinisaos]

Thanks. I used a lot of code (with some modifications) from M2M fields. I tried to use the best names of variables and methods. The main obstacle is that the user would have to enter reverse_lookup_name. I will try to make that optional (if table has only one FK default index value for foreign_key_columns will be 0), so that the user has to specify reverse_lookup_name only if the table has multiple FK so ReverseLookup knows which column to use. If you have any ideas, any help is welcome.
UPDATE:
I removed the reverse_lookup_name and added a table parameter to the ReverseLookupSelect class. With that, Piccolo can identify which table to use for reverse lookup if the table has multiple FKs. I think that is a better option.

[powellnorma]

Just wanted to note that there is interest in this functionality. Is there anything specific holding this back (from being merged)?

Also I like how in peewee one can just set the backref parameter on the ForeignKeyField, see here
Could we have a similar API in piccolo?

[powellnorma]

It looks like we compare column names with table names here. They don't have to match. I think instead of

Manager.artists(table=Manager)

it would be better to be able to specify:

Manager.artists(column="manager")

Since the column that is used for the reverse lookup doesn't change(?), I think it would be best if we could also specify it in the ReverseLookup constructor.

[sinisaos]

@powellnorma Thanks for your interest and review. It's been a while since I wrote this code, but if I remeber it has to be a table because we refer to the foreign key of parent table later in the code. We need this function to find the foreign key column index if there are multiple foreign keys per single table.

[dantownsend]

Just wanted to note that there is interest in this functionality. Is there anything specific holding this back (from being merged)?

I haven't had chance to review it yet - but I agree that it's a feature we need.

[powellnorma]

I think instead of _meta.name it should be _meta.table._meta.tablename. Otherwise reverse_lookup_fk_table_name is a column name

[powellnorma]

I think we should use AS "{original_column_name}s". The table must give us that name. See m2m.py:

piccolo/piccolo/columns/m2m.py

Line 93 in 14012e5

) AS "{m2m_relationship_name}"

[sinisaos]

@dantownsend It would be great if you could find the time to review this. @powellnorma This code works and passes all tests. If you think that things should be changed, it would be best after Daniel's review. Thank you both.

[powellnorma]

Why do we need this JOIN here? Isn't the following enough?

        reverse_select = f"""
            "{reverse_lookup_table_name}"
            WHERE "{reverse_lookup_table_name}"."{reverse_lookup_fk}"
                = "{reverse_lookup_fk_table_name}"."{reverse_lookup_pk}"
        """

[powellnorma]

This code works and passes all tests

I haven't looked, but I guess the tests always use a ForeignKey named the same as the referenced Table. As e.g. here:

class Manager(Table):
    name = Varchar()

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(Manager)

But if this is not true, as e.g. here:

class MainManager(Table):
    name = Varchar()

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(MainManager)

the sections I pointed lead to errors / wrong queries

[powellnorma]

When I query an M2M field of the referenced table, I get:

AttributeError: type object 'M2MSelect' has no attribute 'value_type'

[powellnorma]

I have fixed the parts that I reviewed on my branch reverse_lookup, see: https://github.com/powellnorma/piccolo/commits/reverse_lookup

However querying an M2M field (of the referenced table) seems to be rather slow at the moment. When I set self.serialisation_safe = False in the case of the above mentioned AttributeError, it seems that all m2m joining keys are being used as arguments to a second query. This is overall (in my case) slower than using a for loop in python (which is located in a transaction) instead of the ReverseLookup.
@dantownsend what could we do to make this (querying m2m of reverse-referenced table) faster?

[sinisaos]

But if this is not true, as e.g. here:

class MainManager(Table):
    name = Varchar()

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(Manager)

the sections I pointed lead to errors / wrong queries

I'm sorry, but I don't understand. How can we refer to a table that doesn't exist. Shouldn't it be like this

class MainManager(Table):
    name = Varchar()

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(MainManager)

I want to try your branch. In my reverse lookup attempt I use LazyTableReference like this

class Manager(Table):
    name = Varchar()
    bands = ReverseLookup(
        LazyTableReference("Band", module_path=__name__),
    )


class Band(Table):
    name = Varchar()
    manager = ForeignKey(Manager)

# query for all managers bands
query = await Manager.select(
        Manager.name, Manager.bands(Band.name, as_list=True, table=Manager)
    )

Can you show me how to write a table schema and an example query because right now I get TypeError: __init__() missing 1 required positional argument: 'reverse_fk'. Thanks in advance

[powellnorma]

I'm sorry, but I don't understand. How can we refer to a table that doesn't exist.

The example I gave was wrong, I corrected it. What I meant was a situation in which the column name (of the foreign key) has a different name than the table that it refers to

Can you show me how to write a table schema and an example query because right now I get TypeError: init() missing 1 required positional argument: 'reverse_fk'.

Like this:

class MainManager(Table):
    name = Varchar()
    artists = ReverseLookup(LazyTableReference("Artist", module_path=__name__), "manager")

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(MainManager)

# select:
await MainManager.select(MainManager.artists(Artist.name))

[sinisaos]

The example I gave was wrong, I corrected it. What I meant was a situation in which the column name (of the foreign key) has a different name than the table that it refers to

You are right, it doesn't work if the column is different from the table name. I tried to use foreign_key_columns_index so that if the table has multiple fk columns we know the index of those columns, but it obviously doesn't work well.

class MainManager(Table):
    name = Varchar()
    artists = ReverseLookup(LazyTableReference("Artist", module_path=__name__), "manager")

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(MainManager)

# select:
await MainManager.select(MainManager.artists(Artist.name))

This also doesn't work. Raise ValueError: _name isn't defined - the Table Metaclass should set it. on line 148 name property.

[powellnorma]

This also doesn't work. Raise ValueError: _name isn't defined - the Table Metaclass should set it. on line 148 name property.

Hm, for me that example works. I (already) made changes so that _name should (indeed) be set from the Table Metaclass here.
Have you made sure that you checked out my branch correctly? Maybe only parts of my changes got applied?

[sinisaos]

Sorry, that was my mistake. I tried everything from scratch because I probably messed something up and you are right, the tests from my PR pass with your code.
The only thing I've noticed is that if the column name is plural the tests fail, but singular name pass. I tried to strip column name if it ends with s and it works in Sqlite but not in Postgres.
I have to go now and will be gone for 2 days, but you should continue with this because your version is much better and easier to understand and it works. I'll be happy to delete my PR so you can make a new one. I'll try to help as much as I can.

[powellnorma]

@sinisaos I created a PR on your fork/branch. If you accept it, my changes should be included in this PR.

The only thing I've noticed is that if the column name is plural the tests fail

Do you have an example that demonstrates this problem? How can I reproduce this? Thank you

[powellnorma]

The only thing I've noticed is that if the column name is plural the tests fail

Does it still occur after this commit? sinisaos@d6c6453

[sinisaos]

@powellnorma I have added your changes. Thanks again for your help.

[lqmanh]

Hi, just wonder what's the status quo of this PR, as it's been around for a long time. Are there any blockers left? I'd really love to see this feature getting merged.

Thanks you guys in advance.

[dantownsend]

@lqmanh Will try and get this merged in soon. It just needs reviewing in detail to make sure we're happy with the API 👍

[powellnorma]

One thing this is currently missing is the ability to use .order_by on the ReverseLookupSelect. Similarly with .where etc. @dantownsend Do you think this should be supported? And if so, do you see a way in which we could reuse the existing implementations for these methods?

[erhuabushuo]

Wow, I'm waiting for this PR.

[py-radicz]

it would be cool to have reverse lookup in the api, cant wait for it

[dantownsend]

OK, seems like a lot of people want this. Getting Piccolo v1 out has taken most of my attention. Will take another look at this once I've released Python 3.12 support. Bear with me, I know this PR has been around for a while 🙏

[kayprogrammer]

You guys are doing well.
Still waiting for this PR, including composite unique constraints and check constraints.