Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add local module support to the cloudformation package command #9124

Draft
wants to merge 54 commits into
base: develop
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
0438bdc
Initial commit with TODOs for implementing modules
ericzbeard Nov 22, 2024
84cb385
Unit test configured
ericzbeard Dec 4, 2024
9eac23a
Start resolving refs
ericzbeard Dec 4, 2024
0b79526
Resolve Refs
ericzbeard Dec 5, 2024
6f24f03
parse_sub
ericzbeard Dec 5, 2024
86a5d29
Sub strings
ericzbeard Dec 6, 2024
7b72e29
Basic tests pass
ericzbeard Dec 6, 2024
df507cd
Fixed pylint issues
ericzbeard Dec 6, 2024
7808f62
Fix path to unit tests
ericzbeard Dec 6, 2024
e2131d2
Add the ability to specify a module as a Resource
ericzbeard Dec 9, 2024
49fb392
Fix issues with nested Subs and lists
ericzbeard Dec 9, 2024
ec32ecf
Recursive modules
ericzbeard Dec 9, 2024
160be7c
Implement module outputs
ericzbeard Dec 13, 2024
e517720
Handle spaces in parse_sub
ericzbeard Dec 13, 2024
e8e6d96
Enum
ericzbeard Dec 13, 2024
5b0ee49
Code review fixes
ericzbeard Dec 13, 2024
4eef2c4
Code review fixes
ericzbeard Dec 13, 2024
72659f5
Fix merging lists
ericzbeard Dec 16, 2024
5cb126c
Vpc module example with a map
ericzbeard Dec 18, 2024
3ac0577
Use Map in the parent template
ericzbeard Dec 18, 2024
e0cfcfc
Moving docs to a readme
ericzbeard Dec 18, 2024
c6851c9
updating readme
ericzbeard Dec 18, 2024
dd16375
Handle extra dots in getatts
ericzbeard Dec 18, 2024
e119576
Fix manifest error
ericzbeard Dec 18, 2024
39ac9a7
Add basic conditional support
ericzbeard Dec 19, 2024
2f0143b
Added more complete unit test for conditionals
ericzbeard Dec 20, 2024
36305ec
Download modules from a URL
ericzbeard Jan 13, 2025
497fb3d
Removed the requirement for the s3-bucket parameter
ericzbeard Jan 13, 2025
6e36de0
Confirm GetAtt with double quotes works as expected
ericzbeard Jan 14, 2025
6212018
Fix unit test self.uploader
ericzbeard Jan 14, 2025
92ad3bd
Constants section
ericzbeard Jan 15, 2025
fa7c208
Constants example
ericzbeard Jan 16, 2025
30250b3
Minor tweaks to comments
ericzbeard Jan 16, 2025
32fa228
Constants
ericzbeard Jan 27, 2025
49615a8
Use a constant for module source url in test
ericzbeard Jan 30, 2025
ef49cbd
Merge remote-tracking branch 'upstream/develop' into cfn-modules
ericzbeard Feb 3, 2025
23442ce
Test sending arrays to props and fix output bug
ericzbeard Feb 4, 2025
73fd605
Fixing unit tests
ericzbeard Feb 5, 2025
b1c6a75
Fixed conditional test
ericzbeard Feb 5, 2025
6b36ca2
Implement conditionals within resources
ericzbeard Feb 5, 2025
2181e56
Fixed output getatt ref
ericzbeard Feb 5, 2025
db57872
Hide conditional modules
ericzbeard Feb 6, 2025
2a4b64a
Move export
ericzbeard Feb 7, 2025
5792b93
Merge remote-tracking branch 'upstream/develop' into cfn-modules
ericzbeard Feb 7, 2025
20ca09e
Move export metadata back
ericzbeard Feb 7, 2025
16fe030
Added Modules section to output processing
ericzbeard Feb 11, 2025
0713890
Removing Resource LocalModule imports
ericzbeard Feb 12, 2025
064fe42
Replace Parameters with Inputs and Outputs with References
ericzbeard Feb 12, 2025
e43c771
Parent can refer to module properties
ericzbeard Feb 13, 2025
9bf9ec8
Added AwsToolsMetrics
ericzbeard Feb 13, 2025
7795d1d
Add source map
ericzbeard Feb 13, 2025
1bbe6ea
Add line numbers to source map
ericzbeard Feb 13, 2025
f3b7dfe
Constant objects
ericzbeard Feb 14, 2025
85dee15
Replace Constants with Const
ericzbeard Feb 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Use Map in the parent template
  • Loading branch information
ericzbeard committed Dec 18, 2024
commit 3ac0577e8366849cf10b35ab160f2d04725e8e77
62 changes: 45 additions & 17 deletions awscli/customizations/cloudformation/modules.py
Original file line number Diff line number Diff line change
Expand Up @@ -100,9 +100,38 @@ def make_module(template, name, config, base_path, parent_path):
return Module(template, module_config)


def map_placeholders(i, token, val):
"Replace $MapValue and $MapIndex"
if SUB in val:
sub = val[SUB]
r = sub.replace(MAP_PLACEHOLDER, token)
r = r.replace(INDEX_PLACEHOLDER, f"{i}")
words = parse_sub(r)
need_sub = False
for word in words:
if word.t != WordType.STR:
need_sub = True
break
if need_sub:
return {SUB: r}
return r
r = val.replace(MAP_PLACEHOLDER, token)
r = r.replace(INDEX_PLACEHOLDER, f"{i}")
return r


# pylint: disable=too-many-locals,too-many-nested-blocks
def process_resources_section(template, base_path, parent_path, parent_module):
"Recursively process the Resources section of the template"
if parent_module is None:
# Make a fake Module instance to handle find_ref for Maps
# The only valid way to do this at the template level
# is to specify a default for a Parameter, since we need to
# resolve the actual value client-side
parent_module = Module(template, {NAME: "", SOURCE: ""})
if PARAMETERS in template:
parent_module.params = template[PARAMETERS]

for k, v in template[RESOURCES].copy().items():
if TYPE in v and v[TYPE] == LOCAL_MODULE:
# First, pre-process local modules that are looping over a list
Expand All @@ -113,7 +142,6 @@ def process_resources_section(template, base_path, parent_path, parent_module):
if parent_module is None:
msg = "Map is only valid in a module"
raise exceptions.InvalidModuleError(msg=msg)
# TODO: We should be able to fake up a parent module
m = parent_module.find_ref(m[REF])
if m is None:
msg = f"{k} has an invalid Map Ref"
Expand All @@ -126,10 +154,9 @@ def process_resources_section(template, base_path, parent_path, parent_module):
del resource[MAP]
# Replace $Map and $Index placeholders
for prop, val in resource[PROPERTIES].copy().items():
if val == MAP_PLACEHOLDER:
resource[PROPERTIES][prop] = token
if val == INDEX_PLACEHOLDER:
resource[PROPERTIES][prop] = f"{i}"
resource[PROPERTIES][prop] = map_placeholders(
i, token, val
)
template[RESOURCES][logical_id] = resource

del template[RESOURCES][k]
Expand Down Expand Up @@ -682,36 +709,37 @@ def resolve_getatt(self, v, d, n):
logical_id = self.name + v[0]
d[n] = {GETATT: [logical_id, v[1]]}

def find_ref(self, v):
def find_ref(self, name):
"""
Find a Ref.

A Ref might be to a module Parameter with a matching parent
template Property, or a Parameter Default. It could also
be a reference to another resource in this module.

:param name The name to search for
:return The referenced element or None
"""
# print(f"find_ref {v}, props: {self.props}, params: {self.params}")
if v in self.props:
if v not in self.params:
# print(f"find_ref {name}, props: {self.props}, params: {self.params}")
if name in self.props:
if name not in self.params:
# The parent tried to set a property that doesn't exist
# in the Parameters section of this module
msg = f"{v} not found in module Parameters: {self.source}"
msg = f"{name} not found in module Parameters: {self.source}"
raise exceptions.InvalidModuleError(msg=msg)
return self.props[v]
return self.props[name]

if v in self.params:
param = self.params[v]
if name in self.params:
param = self.params[name]
if DEFAULT in param:
# Use the default value of the Parameter
return param[DEFAULT]
msg = f"{v} does not have a Default and is not a Property"
msg = f"{name} does not have a Default and is not a Property"
raise exceptions.InvalidModuleError(msg=msg)

for k in self.resources:
if v == k:
for logical_id in self.resources:
if name == logical_id:
# Simply rename local references to include the module name
return {REF: self.name + v}
return {REF: self.name + logical_id}

return None
17 changes: 17 additions & 0 deletions tests/unit/customizations/cloudformation/modules/map-expect.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
Parameters:
List:
Type: CommaDelimitedList
Default: A,B,C
Resources:
Content0Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: my-bucket-A
Content1Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: my-bucket-B
Content2Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: my-bucket-C
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
Parameters:
Name:
Type: String

Resources:
Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref Name
12 changes: 12 additions & 0 deletions tests/unit/customizations/cloudformation/modules/map-template.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
Parameters:
List:
Type: CommaDelimitedList
Default: A,B,C

Resources:
Content:
Type: LocalModule
Source: ./map-module.yaml
Map: !Ref List
Properties:
Name: !Sub my-bucket-$MapValue
35 changes: 12 additions & 23 deletions tests/unit/customizations/cloudformation/test_modules.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,38 +85,27 @@ def test_main(self):
# The tests are in the modules directory.
# Each test has 3 files:
# test-template.yaml, test-module.yaml, and test-expect.yaml
tests = ["basic", "type", "sub", "modinmod", "output", "policy", "vpc"]
tests = [
"basic",
"type",
"sub",
"modinmod",
"output",
"policy",
"vpc",
"map",
]
for test in tests:
base = "unit/customizations/cloudformation/modules"
t = modules.read_source(f"{base}/{test}-template.yaml")
td = yamlhelper.yaml_parse(t)
e = modules.read_source(f"{base}/{test}-expect.yaml")

# Modules section
if MODULES in td:
for module_name, module_config in td[MODULES].items():
module_config[modules.NAME] = module_name
relative_path = module_config[modules.SOURCE]
module_config[modules.SOURCE] = f"{base}/{relative_path}"
module = modules.Module(td, module_config)
td = module.process()
del td[MODULES]
td = modules.process_module_section(td, base, t)

# Resources with Type LocalModule
for k, v in td[RESOURCES].copy().items():
if TYPE in v and v[TYPE] == LOCAL_MODULE:
module_config = {}
module_config[modules.NAME] = k
relative_path = v[modules.SOURCE]
module_config[modules.SOURCE] = f"{base}/{relative_path}"
props = modules.PROPERTIES
if props in v:
module_config[props] = v[props]
if modules.OVERRIDES in v:
module_config[modules.OVERRIDES] = v[modules.OVERRIDES]
module = modules.Module(td, module_config)
td = module.process()
del td[RESOURCES][k]
td = modules.process_resources_section(td, base, t, None)

processed = yamlhelper.yaml_dump(td)
self.assertEqual(e, processed)
Loading