Make it damn cheap to refresh staging

[ibes]

The original needs

• x deactivate CDN
• x deactivate Backups
• x change SMTP settings

---

• x maybe there are plugins to selectively activate or deactivate eg query monitor or caching
• x find a way to keep in sync with images at best without the need to migrate them

---

• x add anonymization in this step? -> WP Migrate DB Annonymisation #46
• x is there DB overhead to reduce? -> reduce DB overhead #45
• x is there a meaningful sub portion of the DB to speed up sync? Eg. leaving out users, comments, newsletters, tags. So there is an express refresh that can be done quickly and full refresh for bigger features

---

the plan

• optimal: switch to WordPlate now (for plugin sync and .env)
• optimal: add sub portion pull setup to staging (just posts, post_meta, options)
• minimal: add WP Migrate DB filter to not sync active_plugins (so locally deactivate WP Mail, CDN, backup)
• optimal: automatically pull DB after deployment to staging through release branches
• optimal: use WordPlate/Mail for SMTP
• minimal: Images via https://wordpress.org/plugins/be-media-from-production/
• check if popular post table (1mio lines) can always be excluded

[ibes]

@michael-voit
please check if there are more tasks that are needed for cloning
and or needed after WPMigrate

[ibes]

There might be a cheap way to selectively use filters for that job depending on environment.
With WordPlate we have .env-Files with knowledge about the current environment.

[michael-voit]

I think you got everything.... If I do refresh the staging next time, I can write down every step I make. But I'm quite sure that the list is a wonderful startingpoint.

[ibes]

change SMTP settings

This could be accomplished by https://github.com/wordplate/mail via .env settings that are not set or set so specific data on staging

Would require to introduce an .env file that will anyway come with WordPlate

[ibes]

deactivate CDN

what needs to be done here?
Could it be enough to simple deactivate the CDN plugin?

Could CDN be a solution for syncing images to staging without the need to download them?
I guess we talked about that option at some point for a minute but I don't remember the conclusion

alternatively there could be a function to trigger after-db-pull option overwrite.
So a function that is triggered after WP Migrate DB did it's job that runs an option_update() function against the database to overwrite settings for eg. the CDN plugin.
In the plugin we use, I have not found good filters or a way to overwrite the settings by configuration.

[ibes]

deactivate Backups

that could also be as cheap as deactivating the plugin, right?

[ibes]

maybe there are plugins to selectively activate or deactivate eg query monitor or caching

that is most likely worth to build.

Could be a MU plugin that filters the 'active_plugins'-Setting like in the shy-ajax-handler:
https://github.com/ibes/shy-ajax-handler/blob/master/shy-ajax-handler.php

If switching to WordPlate these plugins would need to stay out of the mu-activation.

The configuration for that most likely should go in the .env-file so this can be configurated by instance.

Maybe I want to have some other plugins auto activated/deactivated on my local installation.

Live could be the main source and we add/substract what we don't want on staging / local
or there could be a set of must use plugins that definitely need to be on everywhere
and than each setup can define "also_active" plugins.

Hm - that would ultimately overpower the database. Which would be fine most of the time. But I feel like that could cause troubles.

Could it be a thing to deactivate the possibility to activate/deactivate plugins in live all together?

I guess this should be an action that runs after refresh -> so after pulling db change and after deployment
After that there is the possibility to change settings/try out things.

[ibes]

add anonymization of DB

https://github.com/deliciousbrains/wp-migrate-db-anonymization

Install on live site - have deactivated on others

[ibes]

is there DB overhead to reduce?

Already trying to reduce overhead but will open a separate issue for that.

[ibes]

minimal meaningful sub-portion to pull

The goal is to reduce wait time for DB pull to encourage us to keep the database fresh
It should be easy to have more than one pull profile in WP Migrate DB Pro.

What is the MVsP (minimal valuable sub portion)

• options table (including menu settings, plugin settings, active plugins, image sizes etc)
• posts + post_meta (for all posts and custom post types - including attachment data)

Rarely important:

• taxonomy stuff (tags and categories rarely change or make a big design difference)
• user stuff (core members are special but rarely change - other members will rarely make a difference)
• comments (will rarely make a design difference)
• mailpoet stuff (will rarely make a difference)
• mathrank (most of that doesn't even affect the visible parts)
• contact form 7 (rarely change)
• other plugins - nothing thing fancy right now

Yes, Options, Posts and Post_Meta should be enough for a quick update

[ibes]

WP Migrate DB Pro CLI

WP Migrate DB Pro has a CLI - so there would be the option to run a pull on each deployment to staging:
https://deliciousbrains.com/wp-migrate-db-pro/doc/cli-push-pull-subcommand/

negative consequences:
DB would be reset on each code deployment and could overwrite the current test

There could be an automatic pull on deploying branch "develop" only.
As experiments are not supposed to happen by deploying "develop" this could even enforce good practice.
Also checking develop on staging would be a good step before a release and this should definitely happen with a "as close as possible" DB to live

[ibes]

WP Migrate DB Pro Hooks

there is the possibility to preserve individual options (eg 'active_plugins')
https://github.com/deliciousbrains/wp-migrate-db-pro-tweaks/blob/master/wp-migrate-db-pro-tweaks.php#L55
(https://snippets.webaware.com.au/snippets/use-wp-migrate-db-pro-and-keep-your-development-environment-settings/)

Run something after the migration
https://github.com/deliciousbrains/wp-migrate-db-pro-tweaks/blob/master/wp-migrate-db-pro-tweaks.php#L99

this might be handy:

global $wp_current_filter;
$is_remote = strstr($wp_current_filter[0], 'nopriv') ? true : false;

[ibes]

speed up migration a bit

up the size of rows per segment to lower the number of requests needed
increase work load for server but live should be in great shape to at least double or quadruple that

add_filter( 'wpmdb_rows_per_segment', function () {
return 200; // default is 100
} );

[ibes]

This will be maintained in the original issue (all the way up) now

so here is the plan

• optimal: switch to WordPlate now (for plugin sync and .env)
• optimal: add sub portion pull setup to staging (just posts, post_meta, options)
• minimal: add WP Migrate DB filter to not sync active_plugins (so locally deactivate WP Mail, CDN, backup)
• optimal: automatically pull DB after deployment to staging through develop branch
• optimal: use WordPlate/Mail for SMTP

OPEN - image handling

[michael-voit]

change SMTP settings

This could be accomplished by https://github.com/wordplate/mail via .env settings that are not set or set so specific data on staging

Would require to introduce an .env file that will anyway come with WordPlate

Traumhaft. Vor der Umstellung auf production halt auf staging gut testen - aber das ist eh klar.... ;)

[michael-voit]

Could CDN be a solution for syncing images to staging without the need to download them?
I guess we talked about that option at some point for a minute but I don't remember the conclusion

I think it could work for all images you also have on production. But if you test something with new images on staging, the CDN plugin rewrites the url and the CDN than looks on production for this file - and will not find it there....

So it should work in the most cases - but not always...

to be tested...

[michael-voit]

so here is the plan

* [ ]  optimal: switch to WordPlate now (for plugin sync and .env)

* [ ]  optimal: add sub portion pull setup to staging (just posts, post_meta, options)

* [ ]  minimal: add WP Migrate DB filter to not sync active_plugins (so locally deactivate WP Mail, CDN, backup)

OPEN - image handling

Sounds good to me (the optimal version).

The second point includes the option to run a pull on each deployment to staging, right? So we would include the CLI commands to deployHQ. Pull via CLI after the deployment, after that flush the cache via CLI.

Like it!

[ibes]

Sounds good to me (the optimal version).

The second point includes the option to run a pull on each deployment to staging, right? So we would include the CLI commands to deployHQ. Pull via CLI after the deployment, after that flush the cache via CLI.

Like it!

I forgot about the automatic db pull after deploying staging, good catch.
I like the idea but I am worried that this could be over-automized as this might overwrite text database entries.
Eg. tweaking around with new menu settings, then noticing that there needs to be a code update, deploying it, and needing to redo the temporary database setting.

But just limiting it on the develop branch feels like a great idea. That way feature branches can live in their own world for as long as that is needed (but can manually synchronize any time) but develop will always be (forced) on the newest database state.

Cache should be flushed on every deployment (all branches, all version), so should be the rewrite rules and transients.

[ibes]

For the images/attachments, I want to test Plugins/Solutions that "fall back" to server files.
Oh, actually they would need to "fall back" on CDN files, right?

This is how it should work:

• Client: I need a file
• Server: Ok, I am searching for the file
• Server is searching for the file locally - found it? great - delivering it!
  if not
• Server is searching on remote (-> CDN) - found it? great - delivering it!
  if not
• responding 404

I started testing such tools but was not yet convinced/sucessful

---

possible solutions

• browser extension: https://chrome.google.com/webstore/detail/resource-override/pkoacgokdfckfpndoffpifphamojphii?hl=en (already not my favorite)
• https://rzen.net/serve-missing-media-production-apache-nginx/
• https://wordpress.org/plugins/uploads-by-proxy/
• https://wordpress.org/plugins/be-media-from-production/ (current untested favorite)

[michael-voit]

Sounds good to me (the optimal version).
The second point includes the option to run a pull on each deployment to staging, right? So we would include the CLI commands to deployHQ. Pull via CLI after the deployment, after that flush the cache via CLI.
Like it!

I forgot about the automatic db pull after deploying staging, good catch.
I like the idea but I am worried that this could be over-automized as this might overwrite text database entries.
Eg. tweaking around with new menu settings, then noticing that there needs to be a code update, deploying it, and needing to redo the temporary database setting.

But just limiting it on the develop branch feels like a great idea. That way feature branches can live in their own world for as long as that is needed (but can manually synchronize any time) but develop will always be (forced) on the newest database state.

If we can limit it to only the develop branch, would be good. Even better could be to limit it to release-branches, if possible? In that way we would get a fresh clone for the final tests (I always do them in a release-branch) but can work as we are used to in develop.

Cache should be flushed on every deployment (all branches, all version), so should be the rewrite rules and transients.

absolutely. deployHQ should have the cli command to flush the cache after every deployment. I can check that once we implement this.

[michael-voit]

For the images/attachments, I want to test Plugins/Solutions that "fall back" to server files.
Oh, actually they would need to "fall back" on CDN files, right?

This is how it should work:

* Client: I need a file

* Server: Ok, I am searching for the file

* Server is searching for the file locally - found it? great - delivering it!
  if not

* Server is searching on remote (-> CDN) - found it? great - delivering it!
  if not

* responding 404

I started testing such tools but was not yet convinced/sucessful

Exactly. I searched for such solutions, too, back than - but not found anything working.

[ibes]

Ich aktualisiere die initiale Issue-Karte um dort den Zwischenstand des Plans festzuhalten.
Fühlt sich nach und nach immer runder an.

[ibes]

First update - Plugins

as WordPlate is already initiated in the "next" repository this task was checked from the beginning.

I have now configurated the plugin loading mechanism.

This plugins are not yet solved:

• backwpup-pro
• mailpoet-premium
• YARPP Experiments

These are three "premium" plugins that are not open to being grabbed via composer and not popular enough to have workarounds (in contrast to ACF Pro)

I contacted the support of BackWpUp -> they use composer internally but not yet externally and will ask the support. I have not too much hopes right now but could be a thing to check their update mechanism.
This plugin was in question before.

I contacted MailPoet but have not yet heard from them. There is nothing in their docs + nothing on google so I have not big hopes.
I know this is a very essential plugin.

YARPP Experiments seems to be an old experiment thing of YARPPs. Do you know what that adds to the mix?
Is it good? Is it still needed (sounds like the experiments might have reached the core in the meantime?!)

For these three there would be the possibility to just throw them in version control and update them the old fashion way. Not thrilled by that, but at least all the other plugins are in the new loop.
There might be a way to figure out how to require the premium plugins as a zip - same way it is done for ACF Pro.

Any thoughts @michael-voit

[michael-voit]

Hey @ibes - thank you very much....

let's try it the other way round: is there a backup-plugin (just database is enough) and a newsletter plugin (that can at least send a weekly automated newsletter with the new articles) that would work with composer and ideally an env-file (as we have sensible access data for both of them)?

YARPP Experiments will be checked by me. I think, we will remove it...

[michael-voit]

YARPP Experiments is removed as it was not doing anything anymore with the newer versions of YARPP.

FYI: A new plugin we are using is "Lightweight Subscribe To Comments"

[ibes]

MailPoet is quite an advanced plugin for that task. I am not sure if there will be a replacement we actually want.

For backup: What does the pro-Version add?
Oh, seems like playing back backups is a pro feature (which is super strange if that is not part of the free deal)

https://wordpress.org/plugins/updraftplus/ could be an alternative. I have used it in a couple of websites but never had shit hitting the fan - but as it is one of the popular ones I imagine it would be fine. It allows to store backups in the cloud.
Which cloud service would you like to store the backup to?
Oh - we just need backup of "/upload" and the database.

Cool - YARPP Experiments is removed.

Lightweight Subscribe To Comments is already in my list :)

[michael-voit]

Newsletter
I also do not see any equivalent replacement for MailPoet.

Backup
updraftplus could really be a solution. I will have a look soon. It's about database backups all 4 hours during the day and also a weekly update of the uploads folder (incremental would be fine - but with updraftplus this is also premium). The whole website is backuped every night via the server.
I also thought about using mariabackup and automate it via a script. So we wouldn't need a plugin.

[ibes]

BackWpUp Pro
could you give me login data to the site where BackWpUp is managed?
Maybe it is possible to make it composer compatible similar to how ACF Pro is handled.

[ibes]

Newsletter
The same for MailPoet Pro would be great.
Maybe I can make it work with Composer