-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathprint.html
2960 lines (2807 loc) · 273 KB
/
print.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE HTML>
<html lang="en" class="sidebar-visible no-js rust">
<head>
<!-- Book generated using mdBook -->
<meta charset="UTF-8">
<title>Golocron – Software Development With Go</title>
<meta name="robots" content="noindex" />
<!-- Custom HTML head -->
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<meta name="description" content="Design software for change">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#ffffff" />
<link rel="icon" href="favicon.svg">
<link rel="shortcut icon" href="favicon.png">
<link rel="stylesheet" href="css/variables.css">
<link rel="stylesheet" href="css/general.css">
<link rel="stylesheet" href="css/chrome.css">
<link rel="stylesheet" href="css/print.css" media="print">
<!-- Fonts -->
<link rel="stylesheet" href="FontAwesome/css/font-awesome.css">
<link rel="stylesheet" href="fonts/fonts.css">
<!-- Highlight.js Stylesheets -->
<link rel="stylesheet" href="highlight.css">
<link rel="stylesheet" href="tomorrow-night.css">
<link rel="stylesheet" href="ayu-highlight.css">
<!-- Custom theme stylesheets -->
</head>
<body>
<!-- Provide site root to javascript -->
<script type="text/javascript">
var path_to_root = "";
var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "coal" : "rust";
</script>
<!-- Work around some values being stored in localStorage wrapped in quotes -->
<script type="text/javascript">
try {
var theme = localStorage.getItem('mdbook-theme');
var sidebar = localStorage.getItem('mdbook-sidebar');
if (theme.startsWith('"') && theme.endsWith('"')) {
localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));
}
if (sidebar.startsWith('"') && sidebar.endsWith('"')) {
localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));
}
} catch (e) { }
</script>
<!-- Set the theme before any content is loaded, prevents flash -->
<script type="text/javascript">
var theme;
try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }
if (theme === null || theme === undefined) { theme = default_theme; }
var html = document.querySelector('html');
html.classList.remove('no-js')
html.classList.remove('rust')
html.classList.add(theme);
html.classList.add('js');
</script>
<!-- Hide / unhide sidebar before it is displayed -->
<script type="text/javascript">
var html = document.querySelector('html');
var sidebar = 'hidden';
if (document.body.clientWidth >= 1080) {
try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }
sidebar = sidebar || 'visible';
}
html.classList.remove('sidebar-visible');
html.classList.add("sidebar-" + sidebar);
</script>
<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<div class="sidebar-scrollbox">
<ol class="chapter"><li class="chapter-item expanded affix "><a href="u00-intro.html">Golocron</a></li><li class="chapter-item expanded affix "><li class="part-title">The Style Guide</li><li class="chapter-item expanded "><a href="u01-00-introduction.html"><strong aria-hidden="true">1.</strong> Project Layout</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="u01-01-good-and-bad-layout.html"><strong aria-hidden="true">1.1.</strong> Good and Bad Layout</a></li><li class="chapter-item expanded "><a href="u01-02-types-of-layouts.html"><strong aria-hidden="true">1.2.</strong> Types of Layouts</a></li><li class="chapter-item expanded "><a href="u01-03-common.html"><strong aria-hidden="true">1.3.</strong> Common</a></li><li class="chapter-item expanded "><a href="u01-04-library.html"><strong aria-hidden="true">1.4.</strong> Library</a></li><li class="chapter-item expanded "><a href="u01-05-single-application.html"><strong aria-hidden="true">1.5.</strong> Single Application</a></li><li class="chapter-item expanded "><a href="u01-06-monorepo.html"><strong aria-hidden="true">1.6.</strong> Monolithic Repository</a></li><li class="chapter-item expanded "><a href="u01-07-monorepo-extra.html"><strong aria-hidden="true">1.7.</strong> Monorepo: Additional Chapters</a></li><li class="chapter-item expanded "><a href="u01-08-versioning-and-go.html"><strong aria-hidden="true">1.8.</strong> Versioning and Go</a></li><li class="chapter-item expanded "><a href="u01-09-notes-on-release-notes.html"><strong aria-hidden="true">1.9.</strong> Notes on Release Notes</a></li><li class="chapter-item expanded "><a href="u01-10-summary.html"><strong aria-hidden="true">1.10.</strong> Summary</a></li></ol></li><li class="chapter-item expanded "><a href="u02-00-introduction.html"><strong aria-hidden="true">2.</strong> Package Layout</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="u02-01-what-is-a-package.html"><strong aria-hidden="true">2.1.</strong> What is a Package</a></li><li class="chapter-item expanded "><a href="u02-02-when-to-create-a-package.html"><strong aria-hidden="true">2.2.</strong> When to Create a Package</a></li><li class="chapter-item expanded "><a href="u02-03-keep-public-api-narrow.html"><strong aria-hidden="true">2.3.</strong> Keep Public API as Narrow as Possible</a></li><li class="chapter-item expanded "><a href="u02-04-the-main-package.html"><strong aria-hidden="true">2.4.</strong> The Main Package</a></li><li class="chapter-item expanded "><a href="u02-05-package-provides-something.html"><strong aria-hidden="true">2.5.</strong> Package Provides Something</a></li><li class="chapter-item expanded "><a href="u02-06-naming-a-package.html"><strong aria-hidden="true">2.6.</strong> Naming a Package</a></li><li class="chapter-item expanded "><a href="u02-07-structure.html"><strong aria-hidden="true">2.7.</strong> Structure</a></li><li class="chapter-item expanded "><a href="u02-08-files-in-a-package.html"><strong aria-hidden="true">2.8.</strong> Files in a Package</a></li><li class="chapter-item expanded "><a href="u02-09-cross-platform-code.html"><strong aria-hidden="true">2.9.</strong> Cross-Platform Code</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="u02-10-basic-principles-cp.html"><strong aria-hidden="true">2.9.1.</strong> Basic Principles of Writing Cross-Platform Code</a></li><li class="chapter-item expanded "><a href="u02-11-cp-options-in-go.html"><strong aria-hidden="true">2.9.2.</strong> Cross-Platform Options in Go</a></li></ol></li><li class="chapter-item expanded "><a href="u02-12-cp-package-and-file-organisation.html"><strong aria-hidden="true">2.10.</strong> Package and File Organisation</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="u02-13-cp-keep-code-at-minimum.html"><strong aria-hidden="true">2.10.1.</strong> Keep Platform-Dependent Code at Minimum</a></li><li class="chapter-item expanded "><a href="u02-14-cp-simple-branching.html"><strong aria-hidden="true">2.10.2.</strong> Use Simple Branching in Trivial Cases</a></li><li class="chapter-item expanded "><a href="u02-15-cp-no-platform-code-in-main.html"><strong aria-hidden="true">2.10.3.</strong> No Platform-Specific Code in Main</a></li><li class="chapter-item expanded "><a href="u02-16-cp-file-suffix-by-default.html"><strong aria-hidden="true">2.10.4.</strong> Use File Suffix by Default</a></li><li class="chapter-item expanded "><a href="u02-17-cp-build-tags-in-mixed-cases.html"><strong aria-hidden="true">2.10.5.</strong> Use Build Tags in Mixed Cases</a></li><li class="chapter-item expanded "><a href="u02-18-cp-advanced-example.html"><strong aria-hidden="true">2.10.6.</strong> An Advanced Example</a></li><li class="chapter-item expanded "><a href="u02-19-cp-platform-independent-tests.html"><strong aria-hidden="true">2.10.7.</strong> Strive for Platform-Independent Tests</a></li><li class="chapter-item expanded "><a href="u02-20-cp-cross-platform-tests.html"><strong aria-hidden="true">2.10.8.</strong> Provide Cross-Platform Tests Only When You Must</a></li></ol></li><li class="chapter-item expanded "><a href="u02-21-summary.html"><strong aria-hidden="true">2.11.</strong> Summary</a></li></ol></li><li class="chapter-item expanded "><div><strong aria-hidden="true">3.</strong> File Layout</div></li><li class="chapter-item expanded affix "><li class="part-title">Foundation</li><li class="chapter-item expanded affix "><li class="part-title">Application Design</li></ol>
</div>
<div id="sidebar-resize-handle" class="sidebar-resize-handle"></div>
</nav>
<div id="page-wrapper" class="page-wrapper">
<div class="page">
<div id="menu-bar-hover-placeholder"></div>
<div id="menu-bar" class="menu-bar sticky bordered">
<div class="left-buttons">
<button id="sidebar-toggle" class="icon-button" type="button" title="Toggle Table of Contents" aria-label="Toggle Table of Contents" aria-controls="sidebar">
<i class="fa fa-bars"></i>
</button>
<button id="theme-toggle" class="icon-button" type="button" title="Change theme" aria-label="Change theme" aria-haspopup="true" aria-expanded="false" aria-controls="theme-list">
<i class="fa fa-paint-brush"></i>
</button>
<ul id="theme-list" class="theme-popup" aria-label="Themes" role="menu">
<li role="none"><button role="menuitem" class="theme" id="light">Light</button></li>
<li role="none"><button role="menuitem" class="theme" id="rust">Rust (default)</button></li>
<li role="none"><button role="menuitem" class="theme" id="coal">Coal</button></li>
<li role="none"><button role="menuitem" class="theme" id="navy">Navy</button></li>
<li role="none"><button role="menuitem" class="theme" id="ayu">Ayu</button></li>
</ul>
<button id="search-toggle" class="icon-button" type="button" title="Search. (Shortkey: s)" aria-label="Toggle Searchbar" aria-expanded="false" aria-keyshortcuts="S" aria-controls="searchbar">
<i class="fa fa-search"></i>
</button>
</div>
<h1 class="menu-title">Golocron – Software Development With Go</h1>
<div class="right-buttons">
<a href="print.html" title="Print this book" aria-label="Print this book">
<i id="print-button" class="fa fa-print"></i>
</a>
</div>
</div>
<div id="search-wrapper" class="hidden">
<form id="searchbar-outer" class="searchbar-outer">
<input type="search" name="search" id="searchbar" name="searchbar" placeholder="Search this book ..." aria-controls="searchresults-outer" aria-describedby="searchresults-header">
</form>
<div id="searchresults-outer" class="searchresults-outer hidden">
<div id="searchresults-header" class="searchresults-header"></div>
<ul id="searchresults">
</ul>
</div>
</div>
<!-- Apply ARIA attributes after the sidebar and the sidebar toggle button are added to the DOM -->
<script type="text/javascript">
document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');
document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');
Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {
link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);
});
</script>
<div id="content" class="content">
<main>
<h1 id="golocron--software-development-with-go"><a class="header" href="#golocron--software-development-with-go">Golocron – Software Development With Go</a></h1>
<p><em>by Pavel Burmistrov</em></p>
<p>TBD.</p>
<h1 id="project-layout"><a class="header" href="#project-layout">Project Layout</a></h1>
<p>Starting from a clean slate is always a pleasure when it comes to a new software project. There is no legacy <em>yet</em>, no technical debt and any other sources of anxiety. The pleasant experience can be maintained at a high level throughout the project's lifecycle. One of the key contributors to the experience of developing and maintaining a project is its layout, the file and folder structure and organisation. In the world of Go it's extremely important because a structure can be both helping and limiting, depending on how it's implemented.</p>
<p>This unit focuses on the project layout design. A layout design is how the folders and files are organised in a codebase, and thus in the repository containing the codebase. A layout sets the stage for all further design and architecture decisions for a project, such as package layout, logical and functional grouping. It also affects maintenance routines like producing build artefacts, CI and CD, rolling out and shipping to the user.</p>
<p>Depending on whether a layout is good or not, it may either significantly help and simplify everyday tasks, or it can make them quite complicated and poor. A good project layout helps you leverage and benefit from technology, whereas with a bad design any change to the structure is more of a fight with technology.</p>
<p>The importance of creating and following a good layout is clear. You create a project once, and this operation is technically quick and easy. But you have to work with the project until it's sunsetted, and this may and should last for years. The ease of creating a new folder hides the importance of putting a thought into how to set a project in a way that's both helping, convenient and makes sense. It's better to take some time to think about how this is going to look like in a week, a month, half a year, and two years from now. If this is done <em>before</em> anything is created, then working on the project will be much easier.</p>
<p>The rest of the unit covers the following:</p>
<ul>
<li>Good vs. Bad project layout</li>
<li>Layout for a library</li>
<li>Single application layout</li>
<li>Layout for a monolithic, multi-service repository</li>
<li>Versioning and Go</li>
<li>Creating a Changelog.</li>
</ul>
<p>Now, let's figure out what makes a project layout good and bad.</p>
<p><em>Disclaimer:</em> Terms "good" and "bad" are used as simple words to describe something that feels right, positive, helps and supports, as opposed to something that is less right, inconvenient, slows down and distracts. They're used not as judgements or labels, but as synonyms for respective broader terms.</p>
<h2 id="good-and-bad-layout"><a class="header" href="#good-and-bad-layout">Good and Bad Layout</a></h2>
<p>You can't know if something is good until you learn what a similar bad thing looks like. A bad thing might be perceived as good <em>until</em> you experience a truly good thing. Take, for instance, coffee. For someone who has never had it, almost any coffee can be considered as coffee, and it may even be unquestioned. But when after trying various kinds you take a sip of Supreme coffee made well, you no longer think that coffee from a Nespresso machine is tasty.</p>
<p>So, what makes a project layout bad? Can it even be bad? Why should we bother about it?</p>
<h3 id="bad-layout"><a class="header" href="#bad-layout">Bad Layout</a></h3>
<p>Have you ever been supporting a project with so long list of files scattered in the root directory, so it takes you several scrolls to get to the README file on the project's repository homepage on GitHub?</p>
<p>Have you ever been maintaining a project where half of the application is contained in the <code>main</code> package, and the entirety of the <code>main</code> package resides at the root level of the repository?</p>
<p>How often do you find unnecessary artefacts in a repository after building an application or running tests?</p>
<p>How often did you need to replace or move parts of a project and then fix the imports to fix an import cycle, or when the structure no longer makes sense?</p>
<p>How often do you find a folder that exists only to contain another folder?</p>
<p>Positive answers to any of these and other similar questions are signs of that you've encountered a bad project layout. When working with such a project, even as simple operation as building a binary can turn into trouble since you don't know whether the resulting binary will be accidentally included with the next commit or not.</p>
<p>So what are the characteristics of a suboptimal layout anyway? It can be nearly anything that leads to any of the following symptoms:</p>
<ul>
<li>a long list of files scattered at the root or any other level of a project</li>
<li>a long list directories scattered at the root or any other level of a project</li>
<li>files of different purposes are mixed at the root or any other level
<ul>
<li>code and configuration files (the app's configuration files)</li>
<li>code and CI scripts</li>
<li>code and tool's configurations</li>
<li>images or any other content</li>
</ul>
</li>
<li>there is no definite place for outputting build artefacts</li>
<li><code>.gitignore</code> does not exclude anything that should not be committed</li>
<li>the structure does not support or reflect architecture of a project
<ul>
<li>having both <code>app</code> and <code>server</code> directories at the same level - does this project provide an app? Or a server? Or it's a poor project layout design?</li>
</ul>
</li>
<li>the project structure leads to leakage of implementation details to the public API surface level
<ul>
<li>users can use parts of the implementation that are supposed to be internal</li>
<li>thus maintenance of the project is now complicated and slowed down since the public API is much broader than it should be</li>
</ul>
</li>
<li>the project structure is unnecessarily complicated and contains a multi-level hierarchy that implements a taxonomy of packages</li>
</ul>
<p>A mixture of even two of the symptoms listed above can significantly complicate project development and maintenance in long-term. Even worse may things become if a project is large, and is being developed by more than one person. Suffering can be endless, if that's an open-source project, and it's a popular one.</p>
<p>So you really want to invest in a good project structure that will help and support you as the application evolves. What does make a good layout?</p>
<h3 id="good-layout"><a class="header" href="#good-layout">Good Layout</a></h3>
<p>Now as we have learned some of symptoms of a bad project layout, it's time to know how a good one looks and feels like?</p>
<p>Overall, a good structure should be unnoticeable in everyday work. It exists, but as if it were invisible. Once defined, described and set, it just guides you. It limits the number of mistakes, helps to onboard new developers, supports maintenance routines, and does help in many other different ways. Let's look at some of those.</p>
<p>A good design for layout suits well for a project:</p>
<ul>
<li>if a project is a library, the layout should reflect and <em>support</em> that</li>
<li>if a project is an application, then the layout should make development and maintenance easier</li>
<li>if a project uses the monolithic repository approach, it's clear where different services, libraries and other parts reside, and where to add new things.</li>
</ul>
<p>A good layout limits the chances of misplacing something:</p>
<ul>
<li>when files and directories are logically and functionally well organised, it's harder to misplace a new file</li>
<li>the structure helps in deciding whether a new file should be a part of the existing tree, or it deserves a new level.</li>
</ul>
<p>A good layout reduces the mental load required to work with a project day by day:</p>
<ul>
<li>it's clear where binaries should go</li>
<li>test coverage reports do not pollute commits</li>
<li>changing CI configuration is straightforward</li>
<li>files of different purposes are grouped in a way that makes logical and functional sense</li>
<li>it's hard to accidentally commit and push garbage files.</li>
</ul>
<p>A good layout simplifies navigation through a project:</p>
<ul>
<li>it's convenient and easy to navigate in an editor/IDE</li>
<li>the homepage looks nice and well organised</li>
<li>in a terminal window, it's easy to work with files using standard and simple console tools.</li>
</ul>
<p>A good layout helps you to maintain a project in a healthy state:</p>
<ul>
<li>implementation details are kept private, yet still available as exported within a project</li>
<li>the possibility of creating an import cycle is reduced by design</li>
<li>the structure and hierarchy represent the business logic and relations between different parts of a project.</li>
</ul>
<p>As you can see, a good layout is more than just the opposite of the symptoms of a bad one. A good layout is the foundation for a development process focused on producing great software. If you have it, then you've got more mental energy to focus on creative solutions for hard problems. If you don't, then part of the energy is wasted.</p>
<p>The layout of a project depends on many factors. The most important one is the kind of a project. The layout will be different for a library, a single application or service, and for a monolithic repository, while there are some commonalities that make better any kind of a project.</p>
<p>As we now have learned what makes a good layout, it's time to know how to structure projects of different types.</p>
<h2 id="types-of-layouts"><a class="header" href="#types-of-layouts">Types of Layouts</a></h2>
<p>The layout of a project depends on its type and purpose. The layout should suit the project well to help supporting it.</p>
<p>A good way to describe the importance of using a good layout is an analogy with clothes. We all have things that suit us well; in this case the clothes fit us naturally so we don't even notice its presence. When a thing doesn't suit well, we might not notice it too, but by the middle of a day we've experienced more stress; when the garment is taken off at the end of a day, it feels much better.</p>
<p>The same is true for a project. Its structure may be unnoticed because it's natural and convenient, whereas with other projects we sometimes can't recognise the source of anxiety. That anxiety, in fact, is a sum of many factors, and one of them might be an inconvenient structure of a project.</p>
<p>A project is created once, and is maintained until its end of life. This means that you really want to invest in a good structure so interactions with codebase are on the positive side, and facilitate productivity.</p>
<p>All projects are different, and there is no a hard set of rules one can follow without thinking about a particular use case. Yet we can confidently say that the majority of projects fall into one of the following three categories:</p>
<ul>
<li>libraries (packages, modules, frameworks, you name it, depending on the language and environment)</li>
<li>single services/applications (web services, CLI tools, GUI applications, etc)</li>
<li>all-in-one, or monolithic repositories (multiple services/applications and libraries).</li>
</ul>
<p>Some recommendations for different types of projects can be applied to all of them, and some are specific to a particular use case. For example, a library layout will be slightly different from a monolithic one.</p>
<p>By a project we mean a few related and similar things at the same time:</p>
<ul>
<li>the directory containing a project</li>
<li>the repository containing a project.</li>
</ul>
<p>In the modern world it's hard to imagine a project that is not published on some service for collaboration and synchronisation with other people. Even if a developer is working on a project alone, they still need to keep it in sync between different devices, and to make sure the code is not lost, if something happens to a computer.</p>
<p>Given that, the section assumes that some service is used to store code of a project. While there are many of them, for simplicity we assume it's GitHub.</p>
<p>The rest of the section focuses on describing guidelines; we start with the commonalities, and then consider aspects that are different. Because it's not hard rules, but rather suggestions, they're flexible, and will differ from project to project. There are also some exceptions, which is natural for any healthy-sense driven guidelines.</p>
<p><em>Disclaimer: Neither the author, nor this work are affiliated or benefit in any way from GitHub.</em></p>
<h2 id="common"><a class="header" href="#common">Common</a></h2>
<p>The ultimate goal of choosing an appropriate structure is to keep things nice and tidy which simplifies interactions with a codebase.</p>
<p>This includes:</p>
<ul>
<li>providing required files for a project</li>
<li>keeping the number of elements at the root level at a reasonable minimum</li>
<li>keeping the root (and any other) level clean and up to date</li>
<li>grouping non-code files by their purpose</li>
<li>refraining from creating unnecessary hierarchy of folders</li>
<li>having a single entry point for routine tasks</li>
<li>storing credentials somewhere else</li>
</ul>
<p>Let's talk about these in more detail.</p>
<h3 id="provide-files-required-for-a-project"><a class="header" href="#provide-files-required-for-a-project">Provide Files Required for a Project</a></h3>
<p>A project should contain files that describe its purpose, provide required information for the programming language and tools, and simplify its maintenance.</p>
<p>The following files are considered as required:</p>
<table><thead><tr><th>Name</th><th>Description</th></tr></thead><tbody>
<tr><td><code>.gitignore</code></td><td>Provides rules that govern what's allowed to be committed. It helps keeping a repository free from garbage files.</td></tr>
<tr><td><code>go.mod</code>, <code>go.sum</code></td><td>Identify a Go module.</td></tr>
<tr><td><code>README.md</code></td><td>Provides the description and any useful information and documentation for a project.</td></tr>
<tr><td><code>LICENCE</code></td><td>Provides clarity on licensing for a project, and helps to avoid potential legal inconveniences.</td></tr>
<tr><td><code>Makefile</code></td><td>Provides a single entry point to all maintenance and routine tasks.</td></tr>
</tbody></table>
<p>The files listed above are a bare minimum, but required for a modern project. You do this once, but if it's done right, it helps you all the way further.</p>
<p>One thing to note is that even with no code at all, a repository is likely to contain about 6 files. This is why the next suggestion is important.</p>
<h3 id="keep-the-number-of-root-elements-at-minimum"><a class="header" href="#keep-the-number-of-root-elements-at-minimum">Keep the Number of Root Elements at Minimum</a></h3>
<p>The number of folders and files at the root level should be as minimal as it's possible and makes sense.</p>
<p>It's important because it allows you to stay focused on the task you intend to perform, and not be distracted by the need of wading through the files, scripts and configs.</p>
<p>To continue our clothes analogy, it's like with a wardrobe - in the morning it's better to take a nicely folded thing from a shelf. But this convenience costs you the responsibility. To get it, you have to fold clothes neatly after the use, and it's also important to carefully choose what to put into the wardrobe.</p>
<p>While this suggestion sounds vague, and it is, it can help you. Each time when you're about to add something at the root level, by default question the need. Usually, things are set at the root level at the beginning, and then added here rarely. When you justify the presence of something at the top level, think twice about a better place for a file/folder, or if it's needed at all.</p>
<h3 id="keep-any-level-clean-and-up-to-date"><a class="header" href="#keep-any-level-clean-and-up-to-date">Keep Any Level Clean and Up to Date</a></h3>
<p>Similarly to the root level, any level of the file tree in a project should be clean, with reasonably minimal number of files, free from garbage, and up to date.</p>
<p>This is important because:</p>
<ul>
<li>long file listings make navigation harder</li>
<li>poor organisation complicates maintenance of the mental model of a project in the developer's head</li>
<li>out-of-date files take up extra mental and physical space</li>
<li>accidentally committed files are misleading and confusing.</li>
</ul>
<p>When a new developer joins a project, how much time do they need to get a firm understanding of what's going on in a project? Does the structure help and guide your colleagues through the project? Does it reduce the effort to keep the important details in their heads? Or do they have to stop for a moment each time when stumbled upon a file which no one knows about, whether it's current or not?</p>
<p>A well-organised and maintained project itself helps people in their everyday jobs. Each time when adding a new file, think twice what <em>else</em> can be added in it further. When creating a directory, consider different scenarios a few months in advance. You don't want to end up with a mess one year after, and tidying up a larger and live project is much more of an effort than keeping it clean everyday. If the structure is supported on a daily basis, it will pay off in long-term.</p>
<p>While it's important to have a neat structure, sometimes it's easy to get too serious about a hierarchy. This is what the next suggestion is about.</p>
<h3 id="dont-create-artificial-hierarchy"><a class="header" href="#dont-create-artificial-hierarchy">Don't Create Artificial Hierarchy</a></h3>
<p>Keep the directory tree in a project as flat as it possible and makes sense. In other words, don't create artificial hierarchy just for the sake of hiding something.</p>
<p>This applies to both, files containing actual code (and we will discuss it in Unit 2), and other files like scripts, configurations, documentation, etc.</p>
<p>A general piece of advice here is that any directory that contains only a single or multiple directories, or a single file, should be questioned for its purpose and existence. Does it belong in here, or somewhere else? Sometimes it might not be clear at the beginning, and this is why it's important to think about evolution of a project some time further.</p>
<p>Consider a situation with scripts. One might have a root-level directory called <code>scripts</code>, and the directory contains several files providing tools for CI/CD process, maybe tests, some pre- and post-install steps. That's a good example.</p>
<p>On the opposite, consider the same number of files but located under separate directories like <code>scripts/build/build.sh</code>, <code>scripts/install/install.sh</code>, and then <code>scripts/ci.sh</code> that sources those files. Don't do this.</p>
<p>Here's another example. Say we have an <code>http</code> directory. From here, we've got two options. If we know <em>for sure</em> that there will be no other networking stuff in the project any time soon (means a couple of years), then it makes sense to keep it simply as <code>http</code>. On the other hand, if there is a non-zero chance of implementing things for <code>smtp</code> and <code>dns</code>, then it does make sense to put it under <code>net</code>. Later, everyone will be happier with the structure because it reflects the nature of things and supports the purpose of the files.</p>
<p>At this point, it becomes clear that grouping for files and folders should be done based on its purpose. Even for non-code files.</p>
<h3 id="group-non-code-files-by-their-purpose"><a class="header" href="#group-non-code-files-by-their-purpose">Group Non-Code Files by Their Purpose</a></h3>
<p>Group and keep maintenance and routine files together by their purpose, in one or a few directories.</p>
<p>This helps in keeping the root level free from clatter, as well as reduces the time required to find a file when looking at the listing. You simply know where all your scripts reside, so no time and energy is wasted. When a new developer joins the project, they need minimal time to get an idea of what is where. Everyone is a bit happier.</p>
<p>As it was shown previously, group all scripts in the <code>scripts</code> directory. If there is plenty of them, and they're specific to the project itself (like startup scripts, init system scripts, cron files, etc), then it might make sense to keep the CI/CD stuff separately. Stay lean though, having <code>ci/scripts</code> and <code>scripts</code> is not a good sign. What else do you have for CI but scripts? If that's a couple of yaml files, let them stay under the same <code>ci</code> directory (see the previous point).</p>
<p>Nowadays many projects use Docker. If a project uses only one <code>Dockerfile</code>, and maybe one file for Docker Compose, then it's fine to keep them at the root. If you have multiple Docker files, and also have some scripts for building containers, then keep them grouped in a directory.</p>
<p>At this point, a valid concern may arise: if all those tools are in directories, it's less convenient to type commands in the terminal. This is what the next recommendation helps you with.</p>
<h3 id="provide-the-makefile"><a class="header" href="#provide-the-makefile">Provide the Makefile</a></h3>
<p>Provide the Makefile for a project to automate and simplify routine and maintenance tasks.</p>
<p>The <code>make</code> tool is available on almost any platform one can imagine, even on Windows without WSL. A minimal knowledge of the syntax of a Makefile is enough to produce a handy set of targets that simplify everyday workflows. The tool have existed for more than 44 years now, and it's likely to continue existing for another 40 years. And in the course of its life, the syntax hasn't changed very much. So it makes a lot of sense investing in learning it once, and benefit from it for the rest of your career.</p>
<p>Just to give you an idea how it can be helpful, consider several, rather basic, examples below.</p>
<ol>
<li>Building a binary.</li>
</ol>
<p>Without <code>make</code> and Makefile:</p>
<pre><code class="language-bash">GOOS=linux GOARCH=amd64 go build -ldflags "-X main.revision=`git rev-parse --short HEAD`" -o bin/my-app ./cmd/my-app
</code></pre>
<p>With <code>make</code>:</p>
<pre><code class="language-bash">make my-app
</code></pre>
<ol start="2">
<li>Building a container and running an app in it.</li>
</ol>
<p>Without <code>make</code>:</p>
<pre><code class="language-bash">docker-compose build --no-cache my-app
docker-compose up -d my-app
</code></pre>
<p>With <code>make</code>:</p>
<pre><code class="language-bash">make run-my-app
</code></pre>
<p>And so forth. While being basic, these examples make a huge difference in everyday experience.</p>
<p>To make your and colleagues' lives easier, define all routine tasks as targets in the Makefile. It can be then used not only by developers directly, but also in your CI/CD scenarios. This makes the configurations clean, straightforward, and easy to understand and maintain.</p>
<p>It's important to give targets in the Makefile meaningful names. A meaningful name reflects the purpose and meaning of the operation it describes. When this is done well, instead of cryptic bash incantations your config for CI may look something like the one below. Pretty nice, isn't it?</p>
<pre><code class="language-yaml"># skipped
script:
- make test
- make cover
- make build
- make deploy
# skipped
</code></pre>
<p>We're almost done with common suggestions. Last in the list, but not the least in importance, is a piece of advice about storing credentials.</p>
<h3 id="store-credentials-somewhere-else"><a class="header" href="#store-credentials-somewhere-else">Store Credentials Somewhere Else</a></h3>
<p>Under no circumstance should you store any sensitive data in a repository, even if it's a private one. Just don't do that.</p>
<p>Don't add ssh keys, files with passwords, private ssl keys, API or any other credentials. It's alright if a repository contains those for development purpose. Anything else is a taboo. Good sleep at night is worth more than deceptive convenience of having production credentials committed to a repository.</p>
<p>What can be used instead? A private bucket in a block storage that contains an encrypted file with credentials might do a good job. A specialised service from your cloud provider of choice might help too. A special service running in your infrastructure that is responsible for storing and providing other services with sensitive data may be even a better option. But once again, do not store any credentials in a repository along with code.</p>
<hr />
<p>All these suggestions are supposed to help and support a team working with a project throughout its lifecycle. Considered and implemented carefully, they form a strong foundation for the project's success.</p>
<h2 id="library"><a class="header" href="#library">Library</a></h2>
<p>We create libraries for re-using in different places, projects, and by other people. The library's code is read more often than it's modified, simply because it's maintained by a limited number of people, whereas the number of users can be much higher. This is especially true for an open-source library. The library's code is consumed by a computer, but a person is the user and who interacts with it.</p>
<p>The layout of a library project should take these factors into account. This means that the layout needs to be optimised for interaction with the user, a person. We seek for answers in documentation, and read the code provided by a library to understand how it works. If a library is well-organised and neat, the process of using it smooth and unobtrusive.</p>
<p>So how can we make this interaction more positive from the layout perspective? If the <a href="u01-03-common.html#common">Common guidelines</a> are taken into account and implemented, that's mostly enough. They cover the most important aspects, and can and should be applied to a library project. Additionally, the following will help you in providing a better experience for the users of a library:</p>
<ul>
<li>providing a good README file</li>
<li>keeping the number of files at the top level of a manageable size</li>
<li>providing examples of using the library</li>
<li>providing benchmarking information, if applicable</li>
</ul>
<p>The rest of the section describes these suggestions in detail.</p>
<h3 id="provide-a-good-readme-file"><a class="header" href="#provide-a-good-readme-file">Provide a Good README File</a></h3>
<p>Provide a well-organised README file with the most important information about the library.</p>
<p>The README file should give answers to the following questions:</p>
<ul>
<li>What is it?
Provide a short description of the library. Ideally, one-two sentences.</li>
<li>What does it do?
Describe the problem the library is solving.</li>
<li>Why does it exist?
It should be clear why this library exists, especially if it's not the only library solving a particular problem. What does it do differently from others in its area?</li>
<li>How does it solve the problem?
Give the users an idea on what's going on under the hood, and what to expect from it.</li>
</ul>
<p>In addition to the above, the README should contain links to related resources like:</p>
<ul>
<li>the GoDoc page</li>
<li>any resources that describe details of algorithms or approaches taken in the library</li>
<li>articles or other resources that might be related to the topic.</li>
</ul>
<p>Your goal with the README file is to help your users in answering the most important question about the library: "Does it help me in solving my problem?". Even if a library does solve a hard problem in a great way, if it isn't reflected in the documentation, that might be unnoticed, thus leading to less people using the library. A well-prepared README file helps your users to answer their questions.</p>
<h3 id="keep-the-top-level-list-of-a-manageable-size"><a class="header" href="#keep-the-top-level-list-of-a-manageable-size">Keep the Top-Level List of a Manageable Size</a></h3>
<p>Keep the root level list of files short and focused. While this sounds very similar to <a href="u01-03-common.html#keep-the-number-of-root-elements-at-minimum">this guideline</a>, it's worth a mention here.</p>
<p>Strive to keep the number of root elements such that it takes less than a full page height, so the user doesn't need to scroll to get to the README. It's also important when the user interacts with a library from their IDE. An endless list of files visually complicates navigation, and gives an impression of a poor organisation.</p>
<p>Additional recommendations will be given further in the unit about designing a good layout for a package.</p>
<h3 id="provide-examples"><a class="header" href="#provide-examples">Provide Examples</a></h3>
<p>In the README, provide clear examples of using your library.</p>
<p>A user should not be forced to go to the GoDoc page, or consult with the code on what using the library looks like. Think about possible use cases, and provide appropriate examples that show it.</p>
<p>Pay special attention to <em>how</em> you're showing the use of the library. Write high-quality code as if it was production code. A half, if not more, of the users will copy and paste code from the examples as it is given. So make sure the examples are not misleading, and do not contain things that you wouldn't want to find in your own production codebase.</p>
<p>Follow these suggestions in order to provide your users with good and quality examples:</p>
<ul>
<li>handle errors properly instead of omitting them</li>
<li>avoid globals as much as possible</li>
<li>avoid using the <code>init</code> function</li>
<li>use regular imports instead of dot-imports</li>
<li>handle errors properly instead of instead of <code>panic</code>-ing, <code>log.Fatal</code>-ing.</li>
</ul>
<p>This is not an exhaustive list, and more detail will be given in the unit about writing good code. But this short list is enough to make examples better, as well as the code that your users write.</p>
<h3 id="provide-benchmarking-results"><a class="header" href="#provide-benchmarking-results">Provide Benchmarking Results</a></h3>
<p>Provide benchmarks whenever possible.</p>
<p>When someone uses your library, it becomes part of their application. For those who do care about performance, it's important. Those who aren't interested in performance, will skip it. But those who are, will appreciate that. Oftentimes, performance is what differentiates one library from another, and the interested users would love to know about it.</p>
<p>Go provides us with all we need to write benchmarks, and to share the results with our users. So help your users in making the decision about the library, write and run benchmarks, and add the results to the README. Then, keep it up to date.</p>
<hr />
<p>The suggestions given above are simple, yet not always easy to follow. Take the time, make sure the project is prepared and organised. Eventually, the users of the library will find it helpful, and they will thank you for having done this for them.</p>
<h3 id="examples"><a class="header" href="#examples">Examples</a></h3>
<p>Below you can find some examples of well-prepared libraries, and a few that could have been organised better.</p>
<h4 id="good-examples"><a class="header" href="#good-examples">Good Examples</a></h4>
<ul>
<li><a href="https://github.com/julienschmidt/httprouter">julienschmidt/httprouter</a></li>
<li><a href="https://github.com/dimfeld/httptreemux">dimfeld/httptreemux</a></li>
<li><a href="https://github.com/jmoiron/sqlx">jmoiron/sqlx</a></li>
<li><a href="https://github.com/rs/xid">rs/xid</a></li>
<li><a href="https://github.com/cryptowatch/cw-sdk-go">cryptowatch/cw-sdk-go</a></li>
</ul>
<h4 id="can-be-improved"><a class="header" href="#can-be-improved">Can Be Improved</a></h4>
<ul>
<li><a href="https://github.com/lib/pq">lib/pq</a>
While the README lists some features, it doesn't answer many questions, nor it gives us examples of using it, benchmarks, or any other information about why should we use it. The file organisation could also be improved.</li>
<li><a href="https://github.com/go-redis/redis/tree/v7">go-redis/redis</a>
As the number of Redis clients is relatively high, it would be valuable to provide users with comparisons and benchmarks that show the difference from other packages. Also, the structure of the repository can be better as not everything need to live at the root level of the package.</li>
</ul>
<h2 id="single-application"><a class="header" href="#single-application">Single Application</a></h2>
<p>A project (and its repository) for a single application is a further development of the guidelines given so far. Almost, if not everything, from the <a href="u01-03-common.html#common">Common</a> and <a href="u01-04-library.html#library">Library</a> sections can, and most likely, should be applied when working on a single service from the layout perspective. There are a couple of recommendations that make the experience better.</p>
<p>One distinction between a service and a library is that there is always at least one artefact, a binary. It also means that a project has at least one <code>main.go</code> file, which is not needed for a library. The experience of working on a project depends, to some extent, on where those files go.</p>
<p>The following will make interaction with a single-application project more convenient and easier:</p>
<ul>
<li>use a separate directory for entry points to the service and satellite tools/applications</li>
<li>use a special, excluded from Git, directory for outputting build artefacts.</li>
</ul>
<p>Next couple of sections go in expand the suggestions.</p>
<h3 id="use-cmd-directory-for-entry-points"><a class="header" href="#use-cmd-directory-for-entry-points">Use <code>cmd</code> Directory for Entry Points</a></h3>
<p>Use <code>cmd</code> directory for entry points to your service. Inside the directory, create a directory per each entry point, named after the application. The inner directories should mainly contain only one file, <code>main.go</code>.</p>
<p>Even when a project is a single application, oftentimes it has a number of satellite CLI and development tools, or different modes. Instead of a long list of <code>if</code> statements, this approach allows for small and focused, yet different entry points.</p>
<p>Create a directory named <code>cmd</code> at the top level of the file tree, and move any entry points, i.e. <code>main.go</code> files in to appropriate directories inside <code>cmd</code>. These second-level directories should be named as you want the binaries to be. Each of those directories is an isolated <code>main</code> package, hence a separate entry point to the app, defined in it.</p>
<h3 id="use-bin-directory-for-binaries"><a class="header" href="#use-bin-directory-for-binaries">Use <code>bin</code> Directory for Binaries</a></h3>
<p>Use <code>bin</code> directory for build artefacts. The directory must be excluded from the source control system. The directory is used in both development and CI processes.</p>
<p>As you remember, there should be no garbage in a repository. A binary that is built during development process should not appear among committed content.</p>
<p>Usually, a binary is excluded by listing its name in the <code>.gitignore</code> file. While this gives an immediate result and may work, it doesn't mean there is no a better option. This approach is not scalable. If someone builds the service with a custom name for the binary, the output can be accidentally committed. The same with CI, the process will just put files right at the root level.</p>
<p>A way to improve the situation is to have a separate directory in the project which is:</p>
<ul>
<li>excluded from Git</li>
<li>managed (created and cleaned) automatically</li>
<li>and used by both developers and build tools.</li>
</ul>
<p>The special directory is named <code>bin</code>, and its place is at the root level. The entire directory is excluded from Git.</p>
<p>Then, since the <a href="u01-03-common.html#common">Common</a> recommendations are in effect, you've got the Makefile. Given the previous tip, the targets work like this:</p>
<ul>
<li>inputs are taken from the <code>cmd</code> directory, e.g. <code>cmd/my-app</code></li>
<li>binaries are put to the <code>bin</code> directory, e.g. <code>bin/my-app</code></li>
</ul>
<p>Combined together, we get:</p>
<pre><code class="language-bash">go build -ldflags "-X main.revision=`git rev-parse --short HEAD`" -o bin/my-app ./cmd/my-app
</code></pre>
<p>If the directory doest not exist, make sure it's created as needed. When you do <code>make clean</code>, the target should remove everything from that directory.</p>
<p>All targets in the build or testing pipelines also use this path for outputting and accessing binaries. When running the service locally, you run it from the directory. Nice and tidy: <code>bin/my-app</code>. If typing in extra characters is a concern, define a target the Makefile, and use <code>make run-my-app</code>. Having this as a target is a scalable way since everyone in the team can use it. If need, you can also create a shell alias as <code>alias run-my-app='make run-my-app</code>.</p>
<p>The targets used in CI process also use the <code>bin</code> directory for artefacts, and then packaging and deploy steps seek for the artefacts in there. The process is structured and organised.</p>
<p>Having a separate directory for binaries is important because chances of multiple binaries are very high. One popular scenario is when a developer works under macOS, and uses Docker. While local version is going to be built for the Mac, the Docker version is built for Linux, hence there are two different binaries already.</p>
<p>It's likely that sometimes there will be a need to build binaries for many systems locally. Thus, two entries should be present in the <code>.gitignore</code> file. Then, if a new binary is introduced later, it will be another two entries. That's already a lot to keep track of, and too many files are at the root level.</p>
<p>Instead, define the <code>bin</code> directory once, add it to the <code>.gitignore</code> file, set up all appropriate targets in the Makefile to use the directory, and don't worry about it for the rest of the project's life.</p>
<hr />
<p>Here is how a single-application repository may look like with the two recent suggestions applied:</p>
<pre><code class="language-bash">├── bin
│ ├── example-agent
│ ├── example-cli
│ ├── example-devsrv
│ └── example-plugins
└── cmd
├── example-agent
│ └── main.go
├── example-cli
│ └── main.go
├── example-devsrv
│ └── main.go
└── example-plugins
└── main.go
</code></pre>
<p>As you see, the structure is organised in a good order, and easy to work with.</p>
<p>Even more important the <code>bin</code> and <code>cmd</code> directories are when a project contains code for multiple <em>different</em> services. In this case, it's a monolithic repository, and many people, potentially several teams, are working on it. All previously given recommendations work especially well with a monorepo. That's what we will talk about in the next section.</p>
<h2 id="monolithic-repository"><a class="header" href="#monolithic-repository">Monolithic Repository</a></h2>
<p><em>Warning:</em> Do not confuse a monolithic repository with a monolithic application. They're absolutely different things.</p>
<p>Unless you're in one of the <a href="https://en.wikipedia.org/wiki/Big_Tech">FAANG</a> or similarly-sized companies, or just don't like the idea, there are probably few technical reasons, if any, against using the <a href="https://en.wikipedia.org/wiki/Monorepo">monorepo</a> approach to organising a project that has multiple moving parts. And Go projects can benefit quite well from it.</p>
<p>Moreover, teams are likely to benefit from this technique as it has several strong supporting arguments like:</p>
<ul>
<li>simplified workflows for CI/CD and testing</li>
<li>reduced burden with managing internal dependencies</li>
<li>significantly less chores with permissions/access</li>
<li>unified tooling</li>
<li>easier to package/ship and so forth.</li>
</ul>
<p>Of course, there are some limitations and downsides, but they are not in effect until after at least a couple of hundreds of thousands SLOCs have been committed in a couple of millions of commits.</p>
<p>A valid concern may be access control, but that's only partially true. If the security policies at a company do make sense and implemented and respected well, there is little to worry about. Experience of companies for which security <strong>is</strong> the biggest asset, suggests that the attitude and mindset are what matter. When everyone is accountable for what they're doing, there is no need in introducing artificial barriers.</p>
<p>The decision on whether to use a monorepo or not is upon you, the reader. The goal of this section is not to make you use this approach, although it makes sense rather often than not. This work covers various aspects of software development, is based on experience, and provides practical advice and guidelines. Several services will be introduced in the third module (it's quite far from this point), and the materials <em>do use</em> a monolithic repository.</p>
<p>With that in mind, further we talk about how to structure a monorepo in a way that makes the experience of using it better. As there are several ways to approach organising a monorepo, and to understand what makes a good one, it's worth getting familiar with some of them. Finally, one reasonable approach will be introduced, and we'll learn more about it.</p>
<p>There is no official classification for monolithic repositories. Yet some of common traits can be derived after investigating and working with different projects. The experience shows that most of the time a monorepo is of one of the following types:</p>
<ul>
<li>naive</li>
<li>based on services</li>
<li>focused on entities.</li>
</ul>
<p>Let's have a quick look at these and consider their pros and cons. Then we introduce an alternative, the structured approach.</p>
<h3 id="naive-monorepo"><a class="header" href="#naive-monorepo">Naive Monorepo</a></h3>
<p>A naive monorepo is exactly what its name says. It's what usually happens when a team decides to use a monorepo, without taking the time to learn and putting a good thought into how to make it work:</p>
<ul>
<li>a new repository is created</li>
<li>then each of the existing services is copied over to the new repo, as is</li>
<li>done</li>
<li>alternatively, everything can be moved into one of the existing repositories, but this doesn't do much of a difference.</li>
</ul>
<p>As a result, the team got just a rather large directory with lots of semi-independent parts, lots of duplicated scripts, files, words, whatnot. One of the immediate benefits <em>seems to be</em> that now something from service A can be imported by service B, but this will quickly become troublesome, especially if the original repositories did not follow the suggestions (such as) listed above, which is highly likely.</p>
<p>This approach has many downsides. Besides being messy and containing many duplications, it is the shortest path to creating dependency cycles. This can often lead to either a poor hierarchy and structure that is created just for the sake of breaking a cycle, or even to concluding that the monorepo was a mistake, and should be avoided. Neither of the those leads to productive and efficient collaboration.</p>
<p>It's hard to justify following this way. Instead, plan the migration, and then execute the plan iteratively. You need to come up with a good structure that suits well and will serve your needs for years. Don't rush, think, plan, try, and only then execute. Not the other way around.</p>
<p>After having realised that this way is not better than it was before the move, the team may decide to organise the repository somehow. One of obvious ways is to use a service as a boundary and splitting criterion. This is how a naive monorepo may evolve to a service-based one.</p>
<h3 id="service-based-monorepo"><a class="header" href="#service-based-monorepo">Service-Based Monorepo</a></h3>
<p>The service-based approach is a slight improvement of naive. The main difference is that some of the duplicated components among services are unified, e.g. CI and building routines, but the codebase continues using services as boundaries for packages. Put simply, each folder at the root level contains a service along with everything that's in the service's scope - data types, business and transport logic, etc. When a service needs something that's already implemented somewhere, it just imports that. New functionality is being developed within the boundaries of a service.</p>
<p>While it might work for some time, such a repository still has exactly the same major downside as naive - it's too easy to end up with a dependency cycle, more so when you try to re-use some code with business logic. Also, there isn't much of an order, since data, logic and utility code are spread across the entire codebase.</p>
<p>A few other serious downsides enter the stage at this point, caused by importing different parts of various services:</p>
<ul>
<li>increased sizes of binaries</li>
<li>increased compilation times</li>
<li>not always clear what to do with tests.</li>
</ul>
<p>As the project evolves, it might seem natural to think of grouping code based on entities it belongs to. Here is how a monorepo may transform into entity-focused.</p>
<h3 id="entity-focused-monorepo"><a class="header" href="#entity-focused-monorepo">Entity-Focused Monorepo</a></h3>
<p>The entity-focused technique is organising code around a particular entity. Packages are often created for different units of the business domain of a project, such as <code>user</code>, <code>photo</code>, <code>library</code> and so forth. Developers add logic into appropriate packages, and then use it in services.</p>
<p>This approach is a bit better than the previous two. It allows for working on services' part separately from the business logic, if implemented correctly.</p>
<p>Still, there are two potential problems:</p>
<ul>
<li>different levels of representation and responsibility could be mixed together, such as data types, methods for accessing storage, business logic and transport details</li>
<li>a major risk in creating cycles, specifically at three levels:
<ul>
<li>entity - when several entities depend on each other</li>
<li>business logic - when a business process depends on the logic of several entities</li>
<li>transport and representation - when representations of several entities/processes depend on each other.</li>
</ul>
</li>
</ul>
<p>The second issue comes from a fact that it's rare for entities, business logic and their representations to be independent from each other, i.e.:</p>
<ul>
<li>entities often aggregate or are parts of other entities</li>
<li>business logic for one process depends or is included into another process</li>
<li>representation for one area of the domain requires other parts.</li>
</ul>
<p>Is there a solution to address these and the problems described above? How to organise a repository in a way which reduces the dependency cycle risk to a minimum (or better to zero)? How to organise a repository in a way when it's easy to re-use entities and logic in different services? How to make developers happier, and services better organised?</p>
<p>The first problem is to be addressed by making better package and architectural design decisions. The former is the subject for Unit 2 in this module, the latter is the topic for Module 3. Some suggestions will be given shortly, in the very next section.</p>
<p>There is no ultimate solution, of course. But there is an option which, if implemented carefully and everyone respects the process, can help to achieve better efficiency and maintainability.</p>
<h3 id="the-structured-monorepo"><a class="header" href="#the-structured-monorepo">The Structured Monorepo</a></h3>
<p>The structured approach is based on grouping code by responsibilities, and levels where objects play their roles. In other words, things are put together by what they do and where they belong:</p>
<ul>
<li>data layer (models)</li>
<li>database layer (repositories)</li>
<li>business processes (controllers)</li>
<li>transport representations (handlers) and so on.</li>
</ul>
<p>By doing this way, we avoid problems described above, and get some additional benefits:</p>
<ul>
<li>at the model level, any model can safely relate to another</li>
<li>at the business process level, any process is free to do the following, without the risk of introducing a cycle:
<ul>
<li>use any model or combine multiple</li>
<li>include any other business process, or be included in other business process, as a step</li>
<li>interact with database representations for the models it works with</li>
</ul>
</li>
<li>similarly, at the transport level, any service can use or combine various business processes, and more.</li>
</ul>
<p>The insightful reader might have already noticed that this has many similarities with properly designing and structuring a good single service. Moreover, if a single service has followed this way, adding another service wouldn't even require to do anything, since the project, hence the repo, is already prepared to accommodate as many applications as needed.</p>
<h3 id="the-layout-of-a-structured-monorepo"><a class="header" href="#the-layout-of-a-structured-monorepo">The Layout of a Structured Monorepo</a></h3>
<p>Everything that has been discussed about different layouts so far comes together and applies to a monolithic repository. Taken into account, implemented and followed carefully, the practices establish the foundation for a good monorepo.</p>
<p>This is what a project may look like at this point:</p>
<ul>
<li>the documentation is provided and up to date</li>
<li>the list of elements at any level is of a reasonable size</li>
<li>all maintenance scripts and other non-code files are organised</li>
<li>the entry points to services are located in the <code>cmd</code> directory</li>
<li>the binaries automatically go and picked up from the <code>bin</code> directory</li>
<li>code that implements the project's data and logic is grouped by responsibilities and roles.</li>
</ul>
<p>A few questions inevitably arise while working on a reasonably large project within one or a couple of teams:</p>
<ul>
<li>Where do we put code that is meant to be used by many services?</li>
<li>Where should utility code go?</li>
<li>How to gradually and safely introduce something new or a breaking change?</li>
</ul>
<p>There is no simple and direct answer. It's also where good planning and thinking should be done, as well as some exceptions. With that in mind, let's consider the following suggestions that help to keep things at the right places:</p>
<ul>
<li>use appropriate position at the file tree to reflect the importance of a package</li>
<li>organise utility code as own small standard library</li>
<li>keep breaking changes in sandbox.</li>
</ul>
<h4 id="use-appropriate-position-at-the-file-tree"><a class="header" href="#use-appropriate-position-at-the-file-tree">Use Appropriate Position at the File Tree</a></h4>
<p>Place packages appropriately in the file tree of a monolithic repository to reflect the importance and nature of a package.</p>
<p>What does it mean and what properties can we use to determine a right placement of a package? There's no hard set of rules. The following heuristics can help in understanding where a package should be placed:</p>
<ul>
<li>An approximate position of a package in the dependency graph.
A package that is imported by many other different packages should be located closer to the root of the hierarchy. A rarely imported package is most likely an implementation detail, and should go deeper in the tree.</li>
<li>The frequency of use.
The position of a package is in direct proportion to of how frequently the package is used.</li>
<li>The importance of a package.
Something that is unique, and provides and implements an important piece of functionality should be placed closer to the top.</li>
<li>The level of abstraction and role.
The higher the abstraction, the higher the level at which a package should be placed.</li>
</ul>
<p>For example, a package with code for converting internal errors from their internal representation to the external, and which is used by the most of the packages that implement application functionality, should be placed at the top level of the structure.</p>
<p>Another example is the packages that <em>are</em> the definitions of the business logic - packages with models, controllers, and the database layer.</p>
<p>On the other hand, a set of middleware handlers for an API should be located deeper in the tree, as it's used only in the context of API, and only by an instance of API. Similarly, routines for data validation, migrations, etc are better to be placed at the second or third level of the tree.</p>
<p>More on this to come in later units covering package design and architecture of a service.</p>
<h4 id="organise-utility-code-as-own-standard-library"><a class="header" href="#organise-utility-code-as-own-standard-library">Organise Utility Code as Own Standard Library</a></h4>
<p>Organise, treat, and maintain all utility code as a small private extension to the standard library. If possible, consider releasing it open source.</p>
<p>This recommendation sounds controversial, and needs further explanations. To understand it better, we first need to clarify what differentiates utility code from other, and then go deeper into how to apply it in real life.</p>
<h5 id="utility-code"><a class="header" href="#utility-code">Utility Code</a></h5>
<p>Utility code is code that implements technical details of a process, and is <strong>independent</strong> from the business logic of a project. The independence from the business logic is the crucial part that distinguishes utility code from any other.</p>
<p>The following traits are common for utility code:</p>
<ul>
<li>it's independent from any other code besides the standard library and/or itself</li>
<li>most of the methods operate on built-in types, or on the types from the standard library</li>
<li>provides common and often used routines</li>
<li>it can be extracted as a separate library</li>
<li>it can be open sourced</li>
<li>provides methods that do not exist in the standard library.</li>
</ul>
<p>Here are some examples of code that can be part of a private extension to the standard library:</p>
<ul>
<li>managing the lifecycle of a process, including proper signal handling, graceful and forced shutdown</li>
<li>archiving/compressing directories</li>
<li>extended configuration and methods for the <code>http.Client</code>, such as custom transports, file downloading, etc</li>
<li>handling multipart uploads</li>
<li>advanced strings manipulation</li>
<li>some standard and generic data structures and algorithms, such as queues, graphs, lists, and so forth</li>
<li>concurrency patterns</li>
<li>file tree traversing utilities and so forth.</li>
</ul>
<p>Having clarified what utility code is and what is not, we can discuss what to do with it.</p>
<h5 id="details-and-discussion"><a class="header" href="#details-and-discussion">Details and Discussion</a></h5>
<p>To begin with, it's worth reminding one of the most often repeated mantras in the Go community:</p>
<blockquote>
<p>Prefer duplication over the wrong abstraction.</p>
<p>– Sandi Metz, and many gophers.</p>
</blockquote>
<p>This is true, and this section should not be considered as something opposite. The author is one of those who respects and follows this advice.</p>
<p>Nonetheless, many rules do have exceptions. It's more about finding what works. What is good for a small project/service, can be a poor choice when applied to multiple services. What's good at a small scale, can significantly complicate things at a larger, and vice-versa.</p>
<p>The definition of utility code given above implicitly prohibits building additional abstractions on top of the standard library code. It can be only considered as an extension.</p>
<p>Duplication works well for a small service, when a small team maintains a couple of medium-sized services. In other words, duplication suits well when it's used moderately and rarely, and in isolation.</p>
<p>Things are different in a project that is supported by one large or several teams, when it's a monorepo, when the number of services grows. The duplication approach is not scalable. Employing it at a larger scale leads to <em>unnecessary</em> duplications, mess in codebase, lack of guarantees, and makes the project prone to errors. The quality of testing decreases.</p>
<p>One of the biggest strengths of Go as a programming language, is that there is mostly one way for accomplishing a task. It becomes almost a requirement when working with many services. There should be <em>only</em> one way to download a file, zip or unzip a directory, parse an authentication header and so forth. Failing to acknowledge this can turn out to be a major source of obscure and nasty bugs at later stages of a project.</p>
<p>So when a project is a monorepo, and has more than one service, the following is true about utility code:</p>
<ul>
<li>it will inevitably occur</li>
<li>it should be reliable and trustworthy</li>
<li>it should be tested</li>
<li>it should be maintained</li>
<li>it should be standardised.</li>
</ul>
<p>One of the ways to provide these guarantees is to have such code in one place, and being conscious about and responsible for it.</p>
<h5 id="in-practice"><a class="header" href="#in-practice">In Practice</a></h5>
<p>The guideline can be simply employed like this:</p>
<ul>
<li>at the root level, have the <code>lib</code> directory as home for utility code</li>
<li>put packages inside <code>lib</code></li>
<li>for packages those names clash with ones from the standard library, add a prefix, for example, <code>x</code>, <code>xstrings</code></li>
<li>alternatively, keep names for packages as is, but have an agreement on how a custom package should be named when imported along with a package from the standard library with the same name. Do not use custom names for the standard packages, ever.</li>
</ul>
<p>This approach also works especially well when implementations of some routines are different between the platforms your project supports.</p>
<p>As a result, the file tree may look like this:</p>
<pre><code class="language-bash">├── bin
├── cmd
└── lib
├── process
│ ├── process.go
│ ├── process_darwin.go
│ ├── process_linux.go
│ ├── process_posix_test.go
│ ├── process_windows.go
│ └── process_windows_test.go
├── files
│ ├── files_posix.go
│ ├── files_test.go
│ └── files_windows.go
├── http
│ ├── http.go
│ └── http_test.go
├── os
│ ├── os.go
│ ├── os_posix_test.go
│ └── os_windows_test.go
└── strings
├── strings.go
└── strings_test.go
</code></pre>
<p>When applied and followed carefully, this advice helps to consolidate and maintain low level code, providing one way for accomplishing a particular task, giving more guarantees about the quality and correctness of the implementation, and reducing the maintenance cost.</p>
<h4 id="have-sandbox-for-breaking-changes"><a class="header" href="#have-sandbox-for-breaking-changes">Have Sandbox for Breaking Changes</a></h4>
<p>Another important aspect that is different when working with a monolithic repository, is introduction of breaking changes, or entirely new code that is not proved to be stable. Within a monorepo, a piece of code is relied upon potentially in hundreds of places, so it's better to test a change in isolation, and only then use elsewhere. How do we do about it in a monorepo?</p>
<p>In a monorepo, have a special place for adding potentially unstable code. An <code>experimental</code> or <code>unstable</code> directory at the root level would be a good choice. Then, inside that directory, follow similar structure as if it were the root level.</p>
<h5 id="details-and-discussion-1"><a class="header" href="#details-and-discussion-1">Details and Discussion</a></h5>
<p>In a classic scenario, dependency management tools usually solve this problem. A dependency that is used in many places, is updated, and the change is then gradually introduced. Go modules and/or vendoring are handy tools here, and this is one of the main reasons for their existence.</p>
<p>However, these tools are not available anymore for addressing this problem as all code is in a single repository. At least, not directly. It's impossible to vendor a part of a repository, or use a separate import path for a package that is part of a large module.</p>
<p>A solution to this problem has existed for many years, and is in use by various kinds of software, from private projects to the Linux kernel, and many established Linux distributions and software. A common name for the solution is "experimental".</p>
<p>What do we mean by "experimental"?</p>
<p>Of course, there are different stages in release processes, such as development, alpha and beta versions, release candidates and so on. Somewhere between development and beta there is usually an experimental branch, or a stage. After reaching some level of stability, the project transitions to next stage, usually testing, or beta. This is followed by many projects, yet it's not exactly what current advice is about.</p>
<p>This guideline is about having experimental code included in the stable version, and made available for conscious use. If the reader have ever configured and compiled a Linux kernel, they would recall the <code>EXPERIMENTAL</code> label for drivers that are included in a stable release, but are still in active development, and included for use as is, without any guarantees.</p>
<p>Similarly, even the most stable version of Debian and many other Linux distributions have an <code>experimental</code> section in their repositories, which contain early releases of software. Such sources are turned off by default, but the user is free to use it.</p>
<p>So here's what to do in a monorepo. When introducing a breaking change or something entirely new, and such code is not guaranteed to work or work correctly (hence it's not for general use), consider this:</p>
<ul>
<li>add a package for new code, if it doesn't exist yet, to the <code>experimental</code> or <code>unstable</code> directory</li>
<li>add new code to the package</li>
<li>use it, test it, change it, prove it works</li>
<li>once confirmed working:
<ul>
<li>for new code, move it to the stable tree</li>
<li>for changes to existing code, depending on the situation
<ul>
<li>move changes to the stable tree and make necessary adjustments</li>
<li>alternatively, use type aliases to redirect uses from old code to new
<ul>
<li>test and prove it works</li>
<li>move changes to the stable tree.</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>An important note about the process is that the <code>experimental</code>/<code>unstable</code> tree must be kept in a good order at all times. As with utility code, without discipline, it's too easy to let these places become junk yards. Keep them clear from clutter by making sure everyone on the team is following conventions and the boy scout rule, move working code to the stable tree, eliminate unused and incorrect code.</p>
<h2 id="monorepo-additional-chapters"><a class="header" href="#monorepo-additional-chapters">Monorepo: Additional Chapters</a></h2>
<p>The approach to maintaining a monolithic repository outlined above is a good starting point, and works quite well in most cases. At the same time, larger organisations with a larger number of teams and services, may want or need more flexibility when working with a monorepo. Let's think of potential reasons for that, and how to address the needs.</p>
<h3 id="open-questions"><a class="header" href="#open-questions">Open Questions</a></h3>
<p>What questions are left with no answers by the structured monorepo? Some of them include:</p>
<ul>
<li>What if we need to maintain both, older and newer versions of a package?
For example, this could easily be the case when a large number of services work with the same model, say a core business unit. You have to update the model, you tried to find a way to make a backward and forward-compatible change, but it seems to be either impossible or too resource demanding. Had this model been in a separate module, a new major version with a breaking change could have done the job. But with a monorepo, everything is versioned as a whole, or not versioned at all.</li>
<li>What if some packages are changed too often, such that it's hard for other teams to keep their services up to date with the changes? Or, how to allow quick and frequent changes in code that is relied upon by other code?
While this should not be normally the case in a well-designed system, this happens. There might be places in a codebase which are depended upon by hot execution paths, and for some reason changes to them are made quite often. If several teams are forced to perform routine updates caused by the work of another team, this can quickly become a major bottleneck, a source of anxiety and even lead to a (rather wrong) conclusion that monorepos don't work.</li>
<li>What if infrastructure and utility code changes often enough to bother other teams with the need to keep their services up to date?
Similarly to the previous question, this normally should not happen. Yet, sometimes this happens, especially when the initially written low-level code is not optimal (someone quickly prototyped, and forgot to finalise the work by taking the draft to a solution that is done well), and at some point it needs to be refactored. In such case, there <em>will</em> be breaking changes, and having everything in a single repository dictates the need to adjust all depending services at the same time.</li>
<li>What if all of the above is the case, the number of people working on a project is large, as well as the codebase, which is slightly aged and has got a fair amount of legacy in it?</li>
</ul>
<p>These are just a few questions that you may have, either because it's your case, or you know someone who is in a similar situation, or for any other reason. There are, of course, many other potential questions. Our goal here is not to find a silver bullet that is the ultimate answer to these and all other questions. Instead, we're trying to find out something to start with, where you can begin experimenting and working out your own way.</p>
<p>Further we consider two additional options that you can employ to help in maintaining a healthy monolithic repository as well as to keep your teams happy and efficient, as much as it is possible in this context. It's advised to use them only when the structured approach is not enough, and it has either already limited you in some way, or the analysis shows some evidence that it might happen soon, and other changes have been tried to improve the situation.</p>
<p>As with everything in life, any flexibility comes at a cost of increased complexity and responsibility, and you have to keep the latter two in balance. Abusing the complexity, or neglecting the responsibilities are two of the most common reasons for any failure, especially and very often it is the case in software development.</p>
<p>Having put an extra stress on the importance of being cautious with added complexity, let's look at ways for achieving additional flexibility. The first one offers the use of nested Go modules, and the second is a bit more radical, but still an option for situations when changes happen too often, and are too breaking. And finally, we look at the combination of the two.</p>
<p><em>NOTE: In the context of the next section we use the term "nested modules" for a Go module that is located not at the root of a repository, potentially having siblings in neighbour directories. We deliberately refrain from using the term "submodule" to avoid confusing it with a Git submodule. Git submodules are not considered in and are outside of the scope of the book.</em></p>
<h3 id="more-structure-with-nested-modules"><a class="header" href="#more-structure-with-nested-modules">More Structure with Nested Modules</a></h3>
<p>Consider converting the top-level packages into nested modules when need more fine grained control and flexibility over packages' lifecycle.</p>
<p>This is possible because Go modules support nested modules, which are sometimes referred to as submodules.</p>
<p>As you know, a Go module is identified by the <code>go.mod</code> file. Each module must have this file, and it's location defines the root of the module. While it is the most often use case, and it is strongly advised to have a single module per repository located at the root, it's possible to maintain multiple modules under the same repository. It's also possible to version nested modules independently using Git tags of a slightly different form.</p>
<p>In general, the technique works as follows:</p>
<ul>
<li>some of the top-level packages are separate modules</li>
<li>each such package, i.e. module, is versioned separately</li>
<li>each module can import another module, as if it was just a package</li>
<li>following the conventions set by semantic versioning is mandatory, i.e. each breaking and incompatible change requires a new major version of such module
This allows the use of a separate import path by the code that is interested in the newest version, while the code that is yet to be updated continues using the older version.</li>
</ul>
<p>Now let's look at an example. Here, we have a monolithic repository called <code>monorepo</code> owned by the <code>myorg</code> account. In the repo, we've got two packages - <code>model</code> and <code>controller</code>. Everything up until now has been as a single module, defined at the root of the repository. Our task is to convert the two packages into nested modules, and to version them independently. The execution plan consists of three steps, for each package:</p>
<ul>
<li>initialise a module</li>
<li>commit and push changes to the remote origin</li>
<li>version the new module by creating and pushing a tag, paying special attention to the name of the tag, it's <code>module_name/vX.Y.Z</code>, not just <code>vX.Y.X</code>.</li>
</ul>
<p>The final directory structure looks like this:</p>
<pre><code class="language-bash">.
├── controller
│ ├── controller.go
│ └── go.mod
├── model
│ ├── go.mod
│ └── model.go
└── go.mod
</code></pre>
<p>Let's go look at each step in detail for each of the two packages, starting with <code>controller</code>. For simplicity, we omit creating a separate branch and pull request:</p>
<ul>
<li>first, convert the <code>controller</code> package into module</li>
</ul>
<pre><code class="language-bash">cd controller
go mod init github.com/myorg/monorepo/controller
</code></pre>
<ul>
<li>now commit and push the change</li>
</ul>
<pre><code class="language-bash">git add go.mod
git commit -m "Convert controller package to module"
git push
</code></pre>
<ul>
<li>next, version the module by creating a tag</li>
</ul>
<pre><code class="language-bash">git tag -a controller/v0.1.0 -m "Convert controller to module"
git push --tags
# OR
git push -u origin controller/v0.1.0
cd ..
</code></pre>
<p>Now we handle the <code>model</code> package, assuming it's version is somewhat bigger:</p>
<ul>
<li>convert it into module:</li>
</ul>
<pre><code class="language-bash">cd model
go mod init github.com/myorg/monorepo/model
</code></pre>
<ul>
<li>commit & push</li>
</ul>
<pre><code class="language-bash">git add go.mod
git commit -m "Convert model package to module"
git push
</code></pre>
<ul>
<li>finally, version the module</li>
</ul>
<pre><code class="language-bash">git tag -a model/v0.4.1 -m "Convert model to module"
git push --tags
# OR
git push -u origin model/v0.4.1
cd ..
</code></pre>
<p>And that is it.</p>
<p>This approach allows for versioning packages within the monorepo independently. The biggest advantage is that you can now easily introduce a breaking change under a new major version, starting from <code>v2.0.0</code>. A detailed conversation about versioning Go modules is the subject for next section, right after we finish with monolithic repositories.</p>
<p>All above is, of course, good, but what if you're larger than Google, and one monorepo is too limiting for your needs, or for some other reason the options discussed so far offer not enough flexibility? Well, would two or more monolithic repos help you?</p>
<h3 id="multiple-monorepos"><a class="header" href="#multiple-monorepos">Multiple Monorepos</a></h3>
<p>In a perfect world, and for some teams, a single monorepo works absolutely well. But what if your reality is somewhat less than ideal, and one monolithic repository is not enough? If a single repository is too limiting or restrictive, that is probably not a sign that it does not work. Perhaps you just need opt for two or more monorepos, each carrying unique responsibilities.</p>
<p>We leave aside some obvious and legit cases where a large organisation has several monolithic repositories as a matter of separation of concerns. For example, one company acquires another, and the two companies under the same roof continue development in two monorepos. Legal concerns might be involved too, so this is an absolutely natural reason for multiple monorepos.</p>
<p>Similarly, we leave aside situations where a company starts experimenting in a new field, or a new project. For example, it's natural for a game studio to develop each game in a separate repository.</p>
<p>Nonetheless, there are questions that some may want to ask. What if, for example, the infrastructure-specific code is evolving too quickly? Or what if CI/CD workflows for client and service code are too different? What if services in the monorepo use so different technologies that it somehow resulted to incompatible workflows? Or, what if one team wants to follow GitHub Flow, and the other is willing to follow Git Flow? What if one team is simply not willing to adapt the practice?</p>
<p>The questions above, although sounding real, are a subject to the overall engineering policy and culture at an organisation. Normally, an organisation has some sort of shared vision where everyone is aligned with and agreed on the foundational principles, approaches and tooling. So if those sorts of questions are present within your organisation, and everyone is doing whatever they want, that's probably a topic for a separate conversation. Here we focus on the technical aspects of dealing with a problem where one monorepo is not enough. And while the author has failed to find an example of a real reason for this (besides a few mentioned earlier and the likes), it should not prevent us from exploring what's available.</p>
<p>Below we briefly explore several available combinations for consideration should you need this. It's obvious that all options that are mentioned, as well as anything that you come up with, have pros and cons, and here are a few common. Unless given explicitly, a common advantage is increased flexibility, either technical or organisational/legal, which comes at a cost of the downside in a form of exponentially increased complexity, and increased cost of maintenance. Instead of reducing entropy and chaos, they facilitate it.</p>
<p><em>NOTE: Bear in mind that the options explored further won't work for all teams, and finding a silver bullet is not the goal of the exploration. It should be clear that no one is enforcing anything upon the reader. Some teams consist mostly of full stack developers, while others prefer specialised engineers. Therefore, it's obvious what is good for one case, might not work for another.</em></p>
<h4 id="multiple-single-module-monorepos"><a class="header" href="#multiple-single-module-monorepos">Multiple Single-Module Monorepos</a></h4>
<p>A simplest option, if applying this adjective is even possible once we're here, is to have a few monorepos that follow the structured approach described in detail earlier in the unit.</p>
<p>It's relatively straightforward, and enables for easier separation of concerns. This option might suit for cases when code needs to be split for both, legal and technical reasons, when alternatives are somewhat less optimal.</p>
<p>It might also be helpful when there is a strong desire or need in splitting utility code from business logic code. In such situation, the monorepo with utility code becomes an official in-house extension to the standard library, and the monorepo with business logic and services is focused only on the company's business domain.</p>
<h4 id="architectural-monorepos"><a class="header" href="#architectural-monorepos">Architectural Monorepos</a></h4>
<p>Another option is splitting all code into few layers, and put each layer into a separate monolithic repository.</p>
<p>A large organisation may have a huge project, or many projects, each of them have at the very least three major components: infrastructure, client-side and services. In such case, it might make sense to organise code as follows:</p>
<ul>
<li>all infrastructure and operations code lives in one monorepo</li>
<li>the second monorepo is devoted to client code
<ul>
<li>it's also possible to go further and split web clients from mobile, if needed</li>
</ul>
</li>
<li>finally, the code that is responsible for handling and persisting data, i.e. services, goes into its own repository.</li>
</ul>
<p>With this setup, all major components are split by their nature, and this might be beneficial for the teams, and processes.</p>
<h4 id="technology-specific-monorepos"><a class="header" href="#technology-specific-monorepos">Technology-Specific Monorepos</a></h4>
<p>An organisation may have hundreds of thousands lines of code written using one technology stack, i.e. language, and then switch to another, producing another hundreds of thousands of lines of code, or millions. And while this is not necessarily a goal on its own, it might be beneficial to split code into monolithic repositories focused on a specific technology.</p>
<p>This option may work quite well when a company is transitioning from one stack to another, or works with several in parallel. Go code stands out in that it's much easier to support compared to other languages, besides maybe a case when you have no code at all. So it might be reasonable to put all Go code in a separate repository, and say C++ code in another, and then Javascript in the third.</p>
<h4 id="multiple-monorepos-with-nested-modules"><a class="header" href="#multiple-monorepos-with-nested-modules">Multiple Monorepos with Nested Modules</a></h4>
<p>Potentially the highest degree of flexibility, along with complexity and chaos, can be achieved by employing multiple monolithic repositories with multiple independently versioned modules in them.</p>
<p>It's strongly advised against using this option, but it's still available. If your organisation is ready to pay the price of this flexibility, or if there is a real need in it (which is highly unlikely), then this is technically possible, and, with very high levels of personal responsibility of each individual and collective responsibility of the engineering department as a whole, might work. Depends on you.</p>
<h3 id="closing-notes"><a class="header" href="#closing-notes">Closing Notes</a></h3>
<p>Maintaining a monolithic repository that several teams are working with, is not an easy task. The experience shows that this is not only possible, but is also highly beneficial when approached consciously and organised, when the vision is shared by everyone involved in the process, when the culture is strong, and people understand the importance of keeping things in a good order.</p>
<p>The techniques described in this section are not new, unique, silver bullet, nor a panacea. But they have proved to be helpful for projects of various sizes, in small and larger teams. Give it a go, experiment, adjust to your needs, and find what works and what does not.</p>
<h2 id="versioning-and-go"><a class="header" href="#versioning-and-go">Versioning and Go</a></h2>
<p>The unit won't be complete without considering such an important topic as dependency management, versioning and distribution in the context of the project layout. Thankfully, in Go this is addressed mostly with one of the most significant changes over last several years which is Go Modules.</p>
<p>Although Modules was released more than 2 years ago, many individuals and companies have only started adopting them. Partially this is due to laziness that is often masked under complains about lack of documentation. There is, actually, lots of high quality content on the topic, as Russ Cox published <a href="https://research.swtch.com/vgo">an extensive and exhaustive set of articles on his blog</a> while Modules were going from the proposal to the final release. Partially this is due to a generally slow adoption process when something new is introduced.</p>
<p>Anyway, over the course of last year, the situation with documentation has significantly improved. One of the best resources so far is <a href="https://github.com/golang/go/wiki/Modules">the Go Modules section</a> on the official Go Wiki. Yet, developers are often not confident when it comes to either performing a major dependency update, or releasing a major version. Many projects are still using <code>dep</code>. And while there is nothing wrong with <code>dep</code> or the use of it, the tool is now obsolete, and it will be restricting its user in many ways. So it's better to migrate to Modules, and benefit from what Go has to offer.</p>
<p>This section summarises the learnings and experience gathered since when Modules were first mentioned. If you've already read all of the official materials mentioned above (Russ's posts and all posts in the official Go Wiki and blog), then you probably learn a little from the section. However, there is some practical advice that might be useful. And given the current module is called "The Style Guide", consider everything below as a recommendation.</p>
<p>The outline:</p>
<ul>
<li><a href="u01-08-versioning-and-go.html#notes-on-terminology">Notes on Terminology</a>
<ul>
<li><a href="u01-08-versioning-and-go.html#the-main-branch">The Main Branch</a></li>
<li><a href="u01-08-versioning-and-go.html#the-default-branch">The Default Branch</a></li>
</ul>
</li>
<li><a href="u01-08-versioning-and-go.html#prerequisites-and-assumptions">Prerequisites and Assumptions</a></li>
<li><a href="u01-08-versioning-and-go.html#approaches-to-versioning">Approaches to Versioning</a>
<ul>
<li><a href="u01-08-versioning-and-go.html#review">Review</a></li>
<li><a href="u01-08-versioning-and-go.html#directory-based-versioning">Directory-Based Versioning</a></li>
<li><a href="u01-08-versioning-and-go.html#branch-based-versioning">Branch-Based Versioning</a></li>
<li><a href="u01-08-versioning-and-go.html#branch-based-with-rolling-main-branch">Branch-Based With Rolling Main Branch</a></li>
<li><a href="u01-08-versioning-and-go.html#summary">Summary</a></li>
</ul>
</li>
<li><a href="u01-08-versioning-and-go.html#implementing-versioning">Implementing Versioning</a>
<ul>
<li><a href="u01-08-versioning-and-go.html#overview">Overview</a></li>
<li><a href="u01-08-versioning-and-go.html#initial-migration-to-versioning">Initial Migration to Versioning</a></li>
<li><a href="u01-08-versioning-and-go.html#changes-within-one-major-version">Changes Within One Major Version</a></li>
<li><a href="u01-08-versioning-and-go.html#introducing-a-new-major-version">Introducing a New Major Version</a></li>
<li><a href="u01-08-versioning-and-go.html#notes-on-automation">Notes on Automation</a></li>
</ul>
</li>
<li><a href="u01-08-versioning-and-go.html#conclusion">Conclusion</a></li>
</ul>
<h3 id="notes-on-terminology"><a class="header" href="#notes-on-terminology">Notes on Terminology</a></h3>
<p>This section clarifies the terminology used further around branches in Git and GitHub. This is important because with the introduction of Go Modules, depending on the approach that is taken to versioning, the meaning of branches may change.</p>
<p>When working with Git, and especially with GitHub and the likes, there are two special kinds of branches:</p>
<ul>
<li>the main branch that is considered as the most recent/stable branch</li>
<li>the default branch that is:
<ul>
<li>the default branch in Git</li>
<li>the default base branch for new pull requests</li>
<li>the default branch shown when the repository is visited on the web</li>
<li>in Git, versions prior to <code>v2.28.0</code> defaulted to <code>master</code>. Starting from <code>v2.28.0</code> it allows specifying an alternative name</li>
<li>on GitHub, before October 1, 2020 it was <code>master</code>. Starting from the day it's <code>main</code>.</li>
</ul>
</li>
</ul>
<p>Knowing this two terms is important, as well as understanding the role and meaning for the workflow.</p>
<h4 id="the-main-branch"><a class="header" href="#the-main-branch">The Main Branch</a></h4>
<p>Historically, and until recently, the main branch has been referred to as <code>master</code>. From this point and further in the book, this branch is referred to as "the main branch", or simply <code>main</code>.</p>
<p>The main branch in a project usually contains the most recent version of code. It’s also widely accepted in the open source community that the main branch is the bleeding edge, containing the latest features, thought might not necessarily stable.</p>
<p>The meaning of the main branch in organisations is slightly different. Often, instead of being the cutting edge, the main branch represents the most recent version that is shippable to the customer and/or deployable to production. Of course, if an organisation follows The Git Flow, then the meaning of the main branch is different. We refer to the main branch in a sense that is closest to The GitHub Flow.</p>
<p>It's fair to say that, for the majority of companies, the main branch is the most recent stable branch that is currently in production.</p>
<p>So from now on, when you see "the main branch" or <code>main</code>, and in your case it's still named <code>master</code>, don't be confused.</p>
<h4 id="the-default-branch"><a class="header" href="#the-default-branch">The Default Branch</a></h4>
<p>The default branch in a repository is the one that is:</p>
<ul>
<li>chosen and its contents shown when you visit the project’s homepage</li>
<li>default base branch for a new pull request.</li>
</ul>
<p>There are a few notes about the default branch. The first is that with GitHub, we've become accustomed to seeing the most recent version of a project when we visit the repository. Very often, the default and main branches in a project are the same branch.</p>
<p>On the other hand, sometimes the default branch is not necessarily the same as the main branch. This becomes especially important the introduction of Go Modules.</p>
<p>Your users, including you, probably like the experience that is familiar. Getting immediate access to the most recent version without needing to switch the branch when you visit a project page on GitHub saves some time and energy, and prevents from unnecessary distractions. So it would be great if it's possible to keep things simple.</p>
<p>As we shall see, Go Modules introduces ways that can disrupt the existing workflows under certain conditions. We will also learn how to keep things in a good order, such that versions are properly and conveniently managed, and users are happy with quick and easy access and to the most recent code.</p>
<h3 id="prerequisites-and-assumptions"><a class="header" href="#prerequisites-and-assumptions">Prerequisites and Assumptions</a></h3>
<p>As we're going to look into some practical aspects of versioning after learning it in theory, it's good to introduce agreements about the environment in which all that works.</p>
<p>For the first, the most important thing here is the version of the programming language. It's worth a reminder that in Go only two most recent versions are officially supported. At the time of writing, the latest stable version is <code>v1.15</code>, and the other is <code>v1.14</code>. Anything else is unsupported, and should not be in use. If you do, then it's strongly advised to perform an update to the most recent version, for many reasons.</p>
<p>So the first agreement, and a prerequisite, is the version of the language. In everything that is related to versioning and Modules, <strong>the book assumes that you use at least version <code>v1.14</code>.</strong></p>
<p>The second important thing is what is used for versioning and dependency management. Although there are alternative opinions and tools, Go Modules is the standard, de facto and de jure. Moreover, it has already proved to be a convenient and handy tool for both, managing dependencies and managing the release cycle of a software project.</p>
<p>So the second agreement and a prerequisite is about versioning, dependencies and tooling. From the first letter to the last, <strong>the book assumes that you use Go Modules for dependency management and versioning in Go projects.</strong></p>
<p>The third agreement is implied by the previous, and is about versioning strategy. In modern times, people increment version numbers in any way they like. This kind of freedom is good to some extent. However, in this work, and this is aligned with the official approach Go takes to versioning, <strong>the <a href="https://semver.org/">Semantic Versioning</a> model is used</strong>. In fact, the way Go manages dependencies with the minimal version selection algorithm is built upon conventions established by Semantic Versioning. Similarly, when and what kind of a change triggers increase in what part of the version, is governed by Semver.</p>
<p>As a quick reminder, Semantic Versioning means that for a version number <code>MAJOR.MINOR.PATCH</code> each part is incremented as follows:</p>
<ul>
<li><code>MAJOR</code> when a breaking change is introduced</li>
<li><code>MINOR</code> when a new feature/update is backwards compatible</li>
<li><code>PATCH</code> when backwards compatible bug fixes are made</li>
<li>also, additional labels for pre-release and build metadata can be added and incremented to the <code>MAJOR.MINOR.PATCH</code> format.</li>
</ul>
<p>Semver has proven to be a meaningful and convenient approach to managing software version. It works perfectly well with a standard, slowly increasing versions model. It also works well for one approach that has become quite popular in recent years. You, of course, have seen versions like <code>v2020.08.12</code>, which can be generalised as <code>YEAR.MONTH.DAY</code> or <code>YEAR.RELEASE_IN_THE_YEAR.FIX</code>. It's less flexible as compared to the classic model, but still fits nicely into the semantic model.</p>
<p>Let's quickly recap the prerequisites:</p>
<ul>
<li>Go version is at least <code>v1.14</code></li>
<li>Go Modules for dependency management and versioning</li>
<li>Semantic Versioning is used to decide when and what version is incremented.</li>
</ul>
<h3 id="approaches-to-versioning"><a class="header" href="#approaches-to-versioning">Approaches to Versioning</a></h3>
<p>With modules, there are three main strategies for versioning a project. The official <a href="https://github.com/golang/go/wiki/Modules">Go Wiki provides</a> some guidelines on how and when to use which. In this section we review what the options are, how each works, and when to use it.</p>
<p>Building a firm understanding of how all this works and how it is represented, is required for working with modules efficiently. For this purpose, there is a quick review of important technical aspects of Modules.</p>
<p>After the review, we consider three approaches to versioning a Go project, namely:</p>
<ul>
<li>Directory-Based Versioning</li>
<li>Branch-Based With Changing Default Branch</li>
<li>Branch-Based With Rolling Main Branch.</li>
</ul>
<h4 id="review"><a class="header" href="#review">Review</a></h4>
<p>Getting familiar with the terminology of Modules is helpful in becoming confident when working with it. The bare minimum includes understanding that:</p>
<ul>
<li>anything that has not been tagged yet, is implicitly considered as <code>v0.0.0-YYYYMMDDHHmmSS-first12SymbolsOfCommitHash</code>
<ul>
<li>e.g. <code>v0.0.0-20190827140505-75bee3e2ccb6</code></li>
</ul>
</li>
<li>anything that has been tagged with <code>v2</code>+ but <strong>is not a module</strong> (i.e. just a tagged release), is implicitly considered as <code>vX.Y.Z+incompatible</code>
<ul>
<li>e.g. <code>v2.2.1+incompatible</code></li>
</ul>
</li>
<li>a correct major version must be set in the <code>go.mod</code> file for versioning to work for versions starting from <code>v2</code>
<ul>
<li>e.g. <code>module github.com/awslabs/goformation/v4</code></li>