Lake location

[philrz]

Summary

zed/4758 improved how the Zed tooling locates the lake such that when zed is invoked there's now a sensible default fallback path that will be used in absence of any flag or environment variable override. This makes way for Zed-dependent tools like Zui to converge on the same default path. However, Zui has traditionally used its own separate lake path when it invokes zed serve. This issue lays out a proposal to migrate Zui users to this converged future.

Goals

1. Default behaviors in absence of overrides via flags, environment variables, or configurable settings should enable seamless access to the same lake from both Zui and the Zed CLI tools.
2. Flags, environment variables, and/or Settings in Zui should allow users to access separate lakes from Zui and the Zed CLI tools if they want.

Definitions

To simplify the examples, this proposal will use macOS as a reference. The following shorthand will be used in this doc to refer to different lake paths.

• Traditional Zui Path (TZP) - This has always been hard-coded as $HOME/Library/Application Support/Zui/lake.
• New Default Path (NDP) - Since the merge of zed/4758, this may resolve to multiple locations depending on if zed flags or environment variable overrides are present, but otherwise resolves to $HOME/.local/share/zed.
• Configurable Zui Path (CZP) - A path that may be set by a user via a new configurable Zui Setting proposed below.

GA Zui Proposal

The following enhancements are proposed for an upcoming GA Zui release.

1. Add a new Setting to Zui where users can see/change the CZP.

   In addition to being flexibility desired by users (see zed/3956 comments) this setting can be used as a fallback if an automatic move of the lake is impossible (see below).

   When a value for CZP is present, Zui should invoke zed serve -lake with this path and hence override Zed defaults and any configured environment variables.

   Whenever the CZP is changed, the zed serve launched by Zui should be stopped and restarted with the new path.

2. Change Zui such that it now launches zed serve with the-lake flag only if the CZP is set.

   This means that by default (when CZP is blank) Zui will now rely on zed serve to follow its new steps for locating the lake and hence (in absence of environment variables) use the NDP.

3. At startup, have Zui check for an existing lake at the TZP and, if found, attempt to move it to the NDP.

   a. If an existing lake is found to already exist at the NDP (unlikely, but possible), leave Zui's lake at the TZP, populate the CZP setting with the TZP, and notify the user that their lake would have been moved but intentionally wasn't.

   b. If the lake move is successful, leave the CZP setting blank, notify the user that the lake was moved to the NDP (show the new lake path to the user), and offer a link to the Zed locating the lake docs if they want to learn more.

   It seems that implementing this would either require Zui to replicate Zed's NDP-determining logic or have Zed provide a way it can be queried to ask what it would use as the NDP.

4. Update the Zui Filesystem Paths doc to reflect this new reality.

The "happy path" is that most users will now have their lake at $HOME/.local/share/zed. If they drop to the shell and run zed commands, that same lake at $HOME/.local/share/zed will be accessed even if Zui is not running (i.e., when there's no zed serve process).

Zui Insiders Proposal

Since we've always positioned Zui Insiders as a beta-style "run side-by-side with Zui" option that would not affect their more stable GA Zui experience, it seems like we should not try to bring Zui Insiders into this converged future. This would allow for things like testing a Zui Insiders that ships with a Zed version that's non-backward-compatible with current GA Zed releases. Therefore, with the Zui Insiders release that comes out as part of this effort:

1. Have the new Zui Insiders CZP Setting initialized to the traditional Zui Insiders lake path (i.e., $HOME/Library/Application Support/Zui - Insiders/lake).

2. Do not have Zui Insiders execute any of the "lake move" scripting that's proposed above for GA Zui.

Dev Mode Zui

In a zed/3956 comment @mccanne indicated that at one time he started Zui in Dev mode with the hope it would find & use his existing lake he'd been managing from the Zed CLI (i.e., at the NDP). It's not clear if he'd been making use of an environment variable like ZED_LAKE with the Zed CLI.

Zui in Dev mode has traditionally always had its own lake storage below a run/lake/ directory in the source code checkout. It does this for what seem like defensible reasons: Testing often requires specific clean/preconfigured storage tailored to the needs of the test exercise. To have Dev Mode Zui start converging by default on the NDP location instead (which is presumed to be treated as production storage) without some action on the part of the user seems like it would require a significant rethink of how tests are run in the Zui repo and how developers at Brim think about using Zui in Dev mode during active development.

It seems a reasonable compromise would be to have Dev Mode Zui populate the CZP with the run/lake/ path and by default invoke this with the -lake flag. This would allow a user like @mccanne to point their Dev Mode Zui at their production lake in a couple different ways:

1. They could jump into Settings and point the CZP at the explicit path of their production lake, or if they have their production lake at the NDP they could just delete the CZP setting and let zed serve follow its steps to find it.
2. They could create a symbolic link from run/lake/ in their code checkout to the directory for their production lake. FWIW I did precisely this with current tip-of-main Dev Mode Zui and it seemed to work fine.

Notes

• If necessary to avoid complexity of implementation, "notifying the user" as mentioned above could take the form of entries in Zui's main.log combined with a Release Note that explains the change and advises the user to check the Zui log and Settings to confirm the final state.

[jameskerr]

Awesome, this is really thought out and makes sense to me Phil. I will begin work on this next.

[jameskerr]

Questions

• What about the log files?
• Does the directory need to exist before we start zed serve?