dumpdata doesn't dump image polymorphic types from filer_file

[vasdee]

When trying to migrate my media from filer (within a djangocms 3.2 and 3.3 environment if that helps)

manage.py dumpdata --natural-foreign --indent=4 filer

The resulting json file has no Image records within filer_file, however it does produce the filer_image records - which then causes a foreign key constraint violation if i try to load the data in another environment.

I've also tried dumpdata across the single model table, filer.file - with the same result.

For a more in depth example. If i have 20 files manged by filer in my application, 5 pdfs and 15 images. Running dumpdata across filer will result in 5 filer.file JSON objects and 15 filer.image JSON objects in the resulting JSON fixture. Whereas i would expect 20 filer.file JSON objects as the two tables are linked.

This is the same result running against sqlite with both django 1.8.13 and 1.9.7

[vasdee]

I believe the error is definitely related to django-polymorphic (my project uses 0.8.1) I can now get around the issue by writing my own command/script and manually changing the queryset_class to the generic QuerySet. My guess is polymorphic may introduce it's own queryset class here.

the tip off was found from this random gist.
https://gist.github.com/czpython/b94c346e4b6cac473bff

app = apps.get_app_config('filer')
model_list = app.get_models()
objects = []
for model in model_list:
    # Change the queryset model to django default. 
    # Removing this causes only 'documents' to be serialized from filer_file, not image types.
    model.objects.queryset_class = QuerySet
    for obj in model.objects.iterator():
        objects.append(obj)

with open(output, "w") as fp:
    serializers.serialize('json', objects, indent=2, use_natural_foreign_keys=True, stream=fp)

I'm not sure where a fix is required, there does seem to be some history around dumpdata and polymorphic: jazzband/django-polymorphic#175

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This will now be closed due to inactivity, but feel free to reopen it.

[greyhare]

I'm running into this problem, too. Filer's part of the dumpdata process emits the filer_image table when it should emit the filer_file table, leading to FK violations on load, as well as a duplicate set of filer_image records in the fixture.

You can even see this happening when running the dumpscript command in the djangop_extensions package:

/app $ python3 manage.py dumpscript filer
Processing model: filer.models.foldermodels.Folder
Processing model: filer.models.foldermodels.FolderPermission
Processing model: filer.models.filemodels.File
Processing model: filer.models.clipboardmodels.Clipboard
Processing model: filer.models.clipboardmodels.ClipboardItem
Processing model: filer.models.imagemodels.Image
Processing model: filer.models.thumbnailoptionmodels.ThumbnailOption
#!/usr/bin/env python


# This file has been automatically generated.
# Instead of changing it, create a file called import_helper.py
# and put there a class called ImportHelper(object) in it.

# [... boilerplate omitted ...]

def import_data():
    # Initial Imports
    from django.contrib.auth.models import User

    # Processing model: filer.models.foldermodels.Folder

    from filer.models.foldermodels import Folder


    # Processing model: filer.models.foldermodels.FolderPermission

    from filer.models.foldermodels import FolderPermission


    # Processing model: filer.models.filemodels.File

    from filer.models.filemodels import File

    filer_image_1 = Image()
    filer_image_1.polymorphic_ctype = ContentType.objects.get(app_label="filer", model="image")
[... the rest of the fields ...]
    filer_image_1 = importer.save_or_locate(filer_image_1)

[... further images ...]

    # Processing model: filer.models.clipboardmodels.Clipboard

    from filer.models.clipboardmodels import Clipboard

    filer_clipboard_1 = Clipboard()
[... fields omitted ...]
    filer_clipboard_1 = importer.save_or_locate(filer_clipboard_1)

    # Processing model: filer.models.clipboardmodels.ClipboardItem

    from filer.models.clipboardmodels import ClipboardItem


    # Processing model: filer.models.imagemodels.Image

    from filer.models.imagemodels import Image

    filer_image_1 = Image()
[... duplicate of above ...]

    # Processing model: filer.models.thumbnailoptionmodels.ThumbnailOption

    from filer.models.thumbnailoptionmodels import ThumbnailOption

(Trying to run this script fails because it's trying to use filer.models.imagemodels.Image before it's imported.)

The snippet above doesn't work for me, because apps isn't defined, and I'm not sure where to import it.

[greyhare]

I have a total of two images in filer right now, so dumping just the filer app is pretty short:

[
{
    "model": "filer.image",
    "pk": 1,
    "fields": {
        "_height": 824.0,
        "_width": 1600.0,
        "_transparent": false,
        "default_alt_text": null,
        "default_caption": null,
        "subject_location": "",
        "file_ptr": 1,
        "date_taken": "2024-01-11T05:38:34.404Z",
        "author": null,
        "must_always_publish_author_credit": false,
        "must_always_publish_copyright": false
    }
},
{
    "model": "filer.image",
    "pk": 2,
    "fields": {
        "_height": 420.0,
        "_width": 600.0,
        "_transparent": true,
        "default_alt_text": null,
        "default_caption": null,
        "subject_location": "",
        "file_ptr": 2,
        "date_taken": "2024-01-11T05:39:42.557Z",
        "author": null,
        "must_always_publish_author_credit": false,
        "must_always_publish_copyright": false
    }
},
{
    "model": "filer.clipboard",
    "pk": 1,
    "fields": {
        "user": [
            "desimone"
        ]
    }
},
{
    "model": "filer.image",
    "pk": 1,
    "fields": {
        "_height": 824.0,
        "_width": 1600.0,
        "_transparent": false,
        "default_alt_text": null,
        "default_caption": null,
        "subject_location": "",
        "file_ptr": 1,
        "date_taken": "2024-01-11T05:38:34.404Z",
        "author": null,
        "must_always_publish_author_credit": false,
        "must_always_publish_copyright": false
    }
},
{
    "model": "filer.image",
    "pk": 2,
    "fields": {
        "_height": 420.0,
        "_width": 600.0,
        "_transparent": true,
        "default_alt_text": null,
        "default_caption": null,
        "subject_location": "",
        "file_ptr": 2,
        "date_taken": "2024-01-11T05:39:42.557Z",
        "author": null,
        "must_always_publish_author_credit": false,
        "must_always_publish_copyright": false
    }
}
]

You can see the images are duplicated and there are no file objects, so the file_ptr FKs point nowhere.

[greyhare]

Also this is my pip freeze, with some annotation:

# Django
Django==4.2.9
django-environ==0.11.2
django-extensions==3.2.3
django-crispy-forms==2.1
django-sekizai==4.1.0
django-downloadview==2.3.0
django-bootstrap5==23.4
requests==2.31.0

# SCSS processing
django-compressor==4.4
django-sass-processor==1.4
libsass==0.23.0
# Django CMS
django-cms==4.1.0
django-entangled==0.5.4
django-filer==3.1.1
django-select2==8.1.2
djangocms-alias==2.0.0
djangocms-attributes-field==3.0.0
djangocms-file==3.0.1
djangocms-frontend==1.2.1
djangocms-googlemap==2.2.0
djangocms-icon==2.1.0
djangocms-picture==4.1.1
djangocms-style==3.1.0
djangocms-text-ckeditor==5.1.5
djangocms-versioning==2.0.0
djangocms-video==3.1.0
# Database interface
psycopg==3.1.17
# Debugging
django-debug-toolbar==4.2.0
Werkzeug==3.0.1
ipdb==0.13.13
Sphinx==7.2.6
tzlocal==5.2

It's dated, as I'm resurrecting an old project and trying to update the database (psql 15 -> 17) before I start updating Python packages. This is also done in containers, so I can't get both versions of psql in the db container so I can do an in-place upgrade.

[fsbraun]

That output is surprising, especially since dumpdata works for me. Have you tried vasdee's script? I do not see a quick fix other than manually dumping the data. :-(

[greyhare]

It's surprising to me, too.

Have you tried vasdee's script?

Yeah. It didn't work, because it assumes some setup not in the snippet.

The snippet above doesn't work for me, because apps isn't defined, and I'm not sure where to import it.

[fsbraun]

Probably from django.apps import apps.

[greyhare]

The problem was the use of the -a flag with dumpdata, apparently. Dropping the flag makes it work right.

  -a, --all             Use Django's base manager to dump all models stored in the database, including those that would otherwise
                        be filtered or modified by a custom manager.

In hindsight, the problem seems obvious.