Docsy as a Go module?

[LisaFC]

Looking for more discussion/thoughts around #520! It's a breaking change, so we would need to consider the potential overhead of maintaining two approaches if not everyone (particularly our community's larger, more established websites) wanted to switch.

[narrenfrei]

I'm not really a GO specialist, but I think that would be a important, big step forward for Docsy. Although it will cause surely some problems on the way.

Have we/you a possibility to ask our "larger, more established websites" about their opinion?

[LisaFC]

Hmm, we could ask via the Docsy Users group (or if anyone came here....). K8s.io is by far our biggest site, if there's anyone from there about.....

[dinotoa]

If I understood correctly, this change would require the installation of the go compiler tool chain in addition to Hugo, which today is sufficient to get started.

The biggest advantage of that change, again, if I understood correctly, would be faster git fetch time and less space occupied on disk.

Personally, I find none of those advantages particularly interesting, especially if the price to pay is installing one more dependency (go), unless I had it installed already for other reasons.

Hence, I would vote against the change, sorry 😞

[sftim]

Hi.
Tech lead for Kubernetes SIG Docs here. I've no objection to being able to build Hugo as a module, but it's very important to new contributors that they can get started with local development as easily as possible (the current process is already more complex than I'd like).

Many docs contributors to projects, even projects implemented in Go, aren't Go coders themselves.

[nathlaroche]

We would definitely be interested in having Docsy as a Hugo module, especially if there's a way to use a specific version of the module.

We have a large team of writers working on different projects and we are implementing some of our common Hugo content (archetypes, layouts, etc.) and reusable content as Hugo modules. We would like to have the same approach for the Docsy theme. This way, writers only see the doc content and do not have to worry about the framework.

[deining]

Anyone interested in having Docsy as a Hugo module can now give it a try:

git clone -b hugo-module https://github.com/deining/docsy-example
cd docsy-example
hugo server

That's all, no fiddling around with submodules any more!

You may read about the new way of installing docsy as theme, about the conversion of your existing site and about the update of docsy as module already. These are preview chapters of the user guide explaining docsy as hugo module, as submitted by me via #802.

Looking forward to get community feedback on this soon!

[chalin]

Looking forward to trying it out in the new year. Happy holidays y'all!

[LisaFC]

Should have time today/next week to work through this, and also have a look at what other Hugo themes are doing - the last time we looked at Hugo modules almost nobody was using them for themes so it was hard to see what the community thought, but I suspect they're more common now. Also interested to hear how people feel now about whether the prereqs feel onerous compared to the "submodule approach" prereqs.

I think regardless of what we present as the default approach for installing/updating Docsy, we should present it was an approach. From what @deining was saying before the holidays it seems like it would be backward/forward compatible for users who decided to switch.

[deining]

Should have time today/next week to work through this

Looking forward to your comments!

the last time we looked at Hugo modules almost nobody was using them for themes so it was hard to see what the community thought, but I suspect they're more common now.

Out of interest, I had a look at the most popular themes out there and assembled the table below:

Theme	GitHub stars	go.mod present	go.mod added	Theme installation instructions	Comments
PaperMod	2554	yes	21.10.20	clone, submodule, zip
Academic	1778	yes	02.09.20	website builder	modules are integral part of theme, theme has common base components as module depencenies
coder	1740	yes	25.05.21	submodule only
Even	1646	no	-	clone only
Book	1593	yes	07.09.21	submodule, module
LoveIt	1590	no	-	clone only
Stack	1385	no	-
Learn	1331	no	-
Terminal	1134	no	-	clone, download
Hello-friend-ng	1090	no	-	clone, submodule
Ananke	674	yes	19.10.20	Module, submodule	Use as Hugo module is recommended way of installation
DocuApi	553	yes	25.06.19	module	modules are integral part of theme, theme has AlpineJs, lunr and slate as module dependencies

I hope this table can serve as a goood starting point for your own research!

[LisaFC]

Thanks, this is great!

[willwade]

Sounds neat. We’d migrate for sure.

[LisaFC]

Just as an update this is now merged into the main branch, albeit with the modules feature turned off by default.

[chalin]

... regardless of what we present as the default approach for installing/updating Docsy, we should present it was an approach.

Agreed that we should:

• First document it as one approach to use Docsy
• Give the community time to try it out
• Gather feedback

If feedback is positive, we can then consider documenting it as the recommended way of using Docsy (if that's what we decide in the end).

... this is now merged into the main branch, albeit with the modules feature turned off by default.

The feature is in but the docs haven't been updated, is that correct?

[LisaFC]

Correct - feature is in but disabled by default, docs have not been updated yet. Anyone who wants to try it can use the preview for the docs PR for more info. The docs update is sizeable and I want to take some time to do some organization work on it.

[deining]

Correct - feature is in but disabled by default

Please note that this is not strictly true. Modules are not a feature that can be enabled/disabled. It's a fundamentally different approach of handling a theme component (=docsy) and it's dependencies (= bootstrap + Font-Awesome). That's also the reason why we need to make decision which route we want to go in the future (IMHO).
And yes it's in. However, keeping both ways open is a real challenge and really tough. Let's have a look at where we are right now:

• modules: we have a first testimony from a user who converted his site successfully to hugo modules. Others weren't successful for reasons I cannot tell. I really like the way modules are implemented, it's clean and a fresh breath of air concerning the installation of docsy.
• example site: users can successfully install docsy using this site since it's docsy submodule is behind the current master. I envision to bring it into a form that users can decide whether they want use modules or submodules. At the end of this refactoring, config.toml will look like:

  # choose what you want:
  theme = ["docsy"]     # git submodule
  #theme = ["github.com/google/docsy"]   # hugo module
  

  Users can switch back and forth between hugo modules and git submodules easily. I'm willing to contribute this, but it may take a bit of time.
• Fresh install: here we can have the same setup as for the example site, allowing users to switch back and forth between hugo modules and git submodules, too. Needs a bit of rework of the current installation instructions, nothing serious. I will come up with a PR with these changes soon.
• Update of existing sites: That's for sure the most critical part. We do have two testimonies that users can upgrade to HEAD of repo and use their site successfully. In it's current form, there is some manual work needed. @LisaFC @chalin Is that acceptable? Yes or no? We need to make a decision here soon.
  That's what currently is needed to make the site work again:
  • navigate to your themes folder
  • create two new directory structures (empty directories):

  mkdir -p github.com/twbs/bootstrap
  mkdir -p github.com/FortAwesome/Font-Awesome
  

  No code change, simply create two empty directories and your site is up again. Is this acceptable? I don't see any good way to make it a seamless update without any intervention. I asked on the hugo discourse list, haven't received any answer yet. I doubt whether anyone will still come up with a good answer.
  Disclaimer: Yes, there actually is a way to make it a seamless update without any intervention. But it's not clean, and I don't like it.
  It's up to you to decide: can we keep going the way we started? Then minor manual intervention is needed when updating existing sites, and we have to document this at a very prominent place (README.md or so). Or we can modify the solution with which we started already again. But that modification will be not very clean and doesn't cleanly implement the module philosophy. I would like to keep the way going we started. Please let me know what you think, we need a decision soon.

[LisaFC]

One thing I'd like to look at is how some other themes have dealt with this - I see, eg, in the Ananke site from your table that they have modules as a recommended approach, I wonder do/how they still let users use submodules.

If the manual intervention is fairly minor then that might be a good solution - "this update will break your site unless you a)migrate to modules or b)make this small change, then your site will continue working with submodules as usual". It at least gives users three choices:

• Stay with old version
• Update to new version with modules
• Update to new version with submodules (and when you might want to do this, eg if you cannot install Go)

Thoughts from others?

[LisaFC]

Ah, Ananke doesn't have submodules of its own, so that makes things a lot simpler....

[chalin]

@deining - will using docsy as a Hugo module require users to have Go installed? (I don't know anything about Hugo modules, or much about Go modules at this point and the answer to that question isn't immediately obvious to me.) Thanks!

[deining]

@deining - will using docsy as a Hugo module require users to have Go installed?

Short form: Yes. With current HEAD of repo, users can set up a new site based on modules or convert their site to modules. In both cases go needs to be installed. If you are a site owner and want full control over your module installation, you will certainly need go installed.

Long form: yes and no.
Hugo modules / theme components can be vendored by issueing hugo mod vendor. This will copy all dependencies (docsy + bootstrap + Font-Awesome) inside a folder _vendor in the root of the site directory. We can zip this folder and distribute as installation source for docsy theme + all dependencies. If we go this way, a fresh installation for novices and/or less experienced users will be as simple as

hugo new site my-site
cd mysite
wget https://github.com/gohugoio/hugo/releases/download/v1.0.0/docsy_1.0.0.zip
unzip docsy_1.0.0.zip
hugo serve

That's super easy, isnt it? And this kind of installation comes with extra bonus: neither go nor git needs to be installed in this case.
That's somehow comparable to how hugo itself gets distributed (a single zip file with one single executable in it).
That's one great feature that we can get when switching over to modules, I have more in mind. That's why I'm pushing the module approach and that's why I think we should make it a first class citizen inside docsy theme. I would even get further and say it should be the recommended way of setting up docsy. But admittedly, I'm biased here.

[sftim]

I'm assuming that nothing about this change stops people who are currently using Hugo as a (Git) submodule from continuing to do so?

[LisaFC]

@deining can speak to this better, but right now, no - in the currently implemented version you need to make a small manual change and then the submodule version keeps working. Correct, @deining?

[chalin]

I'd also like to ensure that moving forward, we don't force existing projects to have Go installed.

To be clear, and add to Tim's question: that current changes re. modules won't prevent existing projects from continuing to use docsy via submodules without being required to install Go. Is that possible?

[deining]

@deining can speak to this better, but right now, no - in the currently implemented version you need to make a small manual change and then the submodule version keeps working. Correct, @deining?

Yes, correct.

[deining]

I'd also like to ensure that moving forward, we don't force existing projects to have Go installed.

I fully agree.

To be clear, and add to Tim's question: that current changes re. modules won't prevent existing projects from continuing to use docsy via submodules without being required to install Go. Is that possible?

That is possible: modules won't prevent existing projects from continuing to use docsy via submodules. And they don't need go installed.

[chalin]

Wonderful! Thank you for that confirmation.

[chalin]

@deining - thank you for your detailed answers. In a nutshell, I like where this is heading and the options that it gives docsy users (for new and existing sites). The need to create two subdirectories as a requirement when upgrading is acceptable to me (and I/we can always create an NPM script that makes the adjustment easier to do).

I very much like the new go-and-git-less option of a .zip release.

My only hesitancy is in terms of what we recommend / choose as the default. At the moment, IMHO, it shouldn't be the modules option. I don't think this is a decision we need to finalize before we move forward with module-ifying docsy -- which makes me wonder if there is anything left to do other than fully document our decisions?

Btw, this change might be a good first candidate change for the upcoming CHANGLOG. /cc @emckean

[deining]

@deining - thank you for your detailed answers.

You are welcome.

In a nutshell, I like where this is heading.

Me too 😉 . I will complete the documentation tomorrow.

Adding it to the CHANGELOG is a great idea, let me know if you need any inut from me here.

[LisaFC]

Excellent. @deining, let me know when you think the docs are up-to-date with the current theme/recommendations and then I'll do a big edit - might need a little bit of reorganization, especially if we're not immediately recommending modules as the default.

[nathlaroche]

We just tested using Docsy as a Hugo module and it worked like a charm! Took me about 5 minutes. I used the following procedure:
Convert an existing site

We are already using Hugo modules, so I didn’t have to do some of these steps (my repo was already a Hugo module). When adding Docsy as a Hugo module (that is, running this command):

hugo mod get github.com/google/docsy

I got the following error message:

$ hugo mod get github.com/google/docsy
hugo: downloading modules …
hugo: collected modules in 4658 ms
Error: from config: failed to resolve output format "print" from site config

We also have that error when adding other Hugo modules (this is not specific to Docsy); the solution is to comment out this section in config.toml:

# Comment out if you don't want the "print entire section" link enabled.
#[outputs]
#section = ["HTML", "RSS", "print"]

Add the module, and then uncomment again.

This approach will be very useful for us, since installing and upgrading to a new version of Docsy will be much easier.

Hope this helps! 😊

[deining]

We just tested using Docsy as a Hugo module and it worked like a charm!

Glad to hear that! Enjoy modules!

$ hugo mod get github.com/google/docsy
hugo: downloading modules …
hugo: collected modules in 4658 ms
Error: from config: failed to resolve output format "print" from site config

I've seen this error, too. Thanks for the workaround.
I just realized that this was an issue for others in a different context, too (see #451, #805).
Does anyone have an idea why hugo is complaining about the print statement here?

[LisaFC]

@gwatts any idea why it's complaining?

[gwatts]

Hugo is complaining as the print format is defined by Docsy, but at that moment docsy isn't yet downloaded.. chicken and egg..

For my own site conversion, as long as I start from the point of a functioning site (ie. the docsy submodule is cloned), I can follow the steps for hugo mod init, hugo mod get and all works well

My final step is then git rm themes/docsy to remove the submodule from the repo

[deining]

When adding Docsy as a Hugo module (that is, running this command):

hugo mod get github.com/google/docsy

I got the following error message:

$ hugo mod get github.com/google/docsy
hugo: downloading modules …
hugo: collected modules in 4658 ms
Error: from config: failed to resolve output format "print" from site config

This error was reported to the hugo discourse list just yesterday, see this post.
Hopefully this can be fixed with the next release.

[chalin]

All: I've created

• Ensure a smooth transition for existing docsy-as-submodule users #848

so that we can move the subtopic discussion there.

I've also created #849 so that we can track subtasks (of which #848 is one).

[tshort]

My main issue with integrating Docsy is that I overwrite a lot of Docsy's theme by placing files under /themes. Then, as Docsy is updated, it's challenging to keep my overwrites in sync. I got the impression that use of modules might make this harder to do. Am I wrong on that? Do modules provide easier ways to integrate theme modifications?

[deining]

My main issue with integrating Docsy is that I overwrite a lot of Docsy's theme by placing files under /themes.

Does that mean that you are overwriting files inside /themes/docsy? If so, I consider this bad practice.

Then, as Docsy is updated, it's challenging to keep my overwrites in sync.

Exactly for that reason.

I cite from docsy repo, userguide/config,toml, line 6 -7:

# Hugo allows theme composition (and inheritance). The precedence is from left to right.
theme = ["docsy"]

For me, best practice would be to have a structure like:

hugo-site
│
└───themes
    │
    └───docsy
    │
    └───docsy_custom  (theme component, optional)

Then you can place the files you want to overwrite either directly into hugo-site or into docsy_custom. In case you are using docsy_custom folder, you should use this line inside your config.toml:

theme = ["docsy", "docsy_custom"]

Both ways, your local overwrites will remain in place once you update docsy theme.

I got the impression that use of modules might make this harder to do.

No. Use of module is total agnostic here. It does it make neither harder nor easier.

Am I wrong on that?

Yes.

Do modules provide easier ways to integrate theme modifications?

Docsy modules don't adress theme modifications, therefore no, they don't.

What Docsy modules do provide: they offer an easier way to integrate modules. And modules can be theme components, but they are not restricted to that, you can pull in various other dependencides into your project as well.

This allows for better composition of themes. An example: some day, we might see something like:

hugo-site
│
└───themes
    │
    └───docsy_base
    │
    └───docsy_search_algolia
    │
    └───docsy_whatever_module

Corresponding module entry inside `config.toml´ would look like:

[module]
[[module.imports]]
path = "github.com/google/docsy/base"
[[module.imports]]
path = "github.com/google/docsy/search/algolia"
[[module.imports]]
path = "github.com/google/docsy/whatever/module"

This is only possible with modules. Currently, all search options (lunr, google, algolia) are part of docsy. In the future users might be able to pull on of the modules docsy_search_algolia, docsy_search_google, docsy_search_lunr, ... This way, we can add further search options by providing another module like docsy_search_flexsearch, docsy_search_stork, ... This allows to keep docsy lean and will prevent us from bloating the theme. That's why I'm advocating and actively working on modules.

Does this clarify things?

[tshort]

That helps a lot. That does makes sense for future flexibility with Docsy.

I wrote my original post wrong. I don't overwrite things in /themes. Although, I've thought of switching to that approach to try to manage my theme changes with git.

I think I'm doing "local overwrites". I add files under /layouts (and a few under /assets). That started small, but it grew. Some are new shortcodes that don't overlap with Docsy. However, a lot are files copied from Docsy with small changes. Those are the changes that are hard to keep updated.

[LisaFC]

That's a really good question, and thanks @deining for the thorough answer! @tshort, my understanding is the same as @deining's (I think) - Hugo's lookup order for layouts etc. (ie look in your project layouts directory first, then the theme one) shouldn't change if you're using modules, though it also means that modules won't make it notably harder or easier to keep in sync with Docsy if you're using heavily customized versions of our layout files. One of the topics/goals we discussed in our recent community meeting was making more style features configurable through variables or config to avoid users having to override entire layouts to get the behaviour they want.

[chalin]

For those who are only tracking this discussion thread, I've captured my latest thoughts on this in #858, Fix the build for docsy-as-submodule.

[chalin]

@LisaFC @emckean @deining et all. Until we're quite sure that we have a solution that works for all, I suggest that all module-related commits be merged into the module-feature branch, not the default branch.

(FYI, I'll be OOO and/or mostly AFK until Tuesday.)

[deining]

@LisaFC @emckean @deining et all. Until we're quite sure that we have a solution that works for all, I suggest that all module-related commits be merged into the module-feature branch, not the default branch.

I agree.

Could make real good progress this weekend:

• Found a way to work around the bug in go's module management completely behind the scenes (actually this bug is really nasty and also caused the intial implementation of my workaround to fail, so I had to implement a workaround for the workaround 😄).
• Had another idea that IMHO is superior to the multiple config files/dirs approach: for module installations, we simply pull in another hugo module github.com/google/docsy/module. The config.toml of this module is merged into the config.toml of the docsy core module. By doing so, we can pull in the dependencies that git-submodule installations choke on. This way we use modules to solve the problems we hadn't without them 😉.
• Found a bug in hugo on the way.

With #801 reverted, I take my time to to get everything in right place, incl. documentation. Stay tuned, we will bring this to a good end soon!

NB: Had some other ideas on the way, expect to see installation related tickets during the next week! Good night again!