Adam's list of Data Liberation wishes and ideas

[adamziel]

Related to #1894. Here's my private note I'm maintaining as a public issue for transparency and to enable discussion.

Importing

• Reuse prior art on WXR imports
• Jekyll import – see this Jekyll template for generating a WXR file
• Use wp_cron to process the import bit by bit. Split large image downloads, like 2GB image, over multiple cron calls to avoid timeouts derailing the import.

Resource quotas

• Memory quota
• Disk space quota
• Max attachment size
• Max requests per domain per minute
• Max download retries
• wp_suspend_cache_invalidation() before the import
• Cleanup stale resources, like partial downloads

Security

• Disable KSES to prevent messing up post contents
• Check remote IP addresses and domains before requesting
• Pin IPs to prevent dns rebinding attack

Extensibility

• Custom frontmatter filter
• Pre-filter any entity before importing it
• Pre-filter any asset URL before requesting it
• Post-filter downloaded assets
• Filter the local filename of any created attachment

Streams

• Let's lower the default chunk size in WP_File_Byte_Stream to 100 bytes to overexpose ourselves to streaming issues. Then, once it matures, let's increase it to a decently sized number. I'm not sure how to choose one – let's read up on PHP and linux mailing lists to figure out how these projects chose their defaults.