Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add HTTPS to Sidecar for deployment #8

Merged
merged 52 commits into from
Mar 17, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
51d13c5
add traefik config and update cli
eriktaubeneck Feb 15, 2024
ff09d45
cannot use env variables in traefik.yaml, use static config env varia…
eriktaubeneck Feb 15, 2024
512dd30
must call traefik with sudo to run on these ports
eriktaubeneck Feb 15, 2024
ce75e4c
use traefik cli args instead of env for key/cert
eriktaubeneck Feb 15, 2024
01eab14
move key/cert into dynamic_conf
eriktaubeneck Feb 15, 2024
da0e501
use different env formatting
eriktaubeneck Feb 15, 2024
9ad2140
move tls config into a file created in cli command
eriktaubeneck Feb 16, 2024
c23b5cd
fix bug in dynamic_config
eriktaubeneck Feb 16, 2024
f9fa873
fix bug in dynamic_config
eriktaubeneck Feb 16, 2024
45512b5
use different env format
eriktaubeneck Feb 16, 2024
49b785b
move dynamic config into cli
eriktaubeneck Feb 16, 2024
e087366
make sure to wrap single quotes around double quotes when needed
eriktaubeneck Feb 16, 2024
6459150
remove single/double quotes, add backticks
eriktaubeneck Feb 16, 2024
4cfac07
fix ports
eriktaubeneck Feb 16, 2024
6eb0be1
use adjacent subdomains, not nested
eriktaubeneck Feb 16, 2024
05416d5
add draft-mpc.vercel.app to CORS domains
eriktaubeneck Feb 16, 2024
1ac44f7
make test data directory before generating it
eriktaubeneck Feb 16, 2024
6aed89e
adjust ports, not inferred from network.toml
eriktaubeneck Feb 16, 2024
89458c4
use https not ws for checking status
eriktaubeneck Feb 17, 2024
bffedef
use http not https for checking status
eriktaubeneck Feb 17, 2024
48684ff
turn off verify for status check temporarily
eriktaubeneck Feb 17, 2024
d3e7324
use https for status check
eriktaubeneck Feb 17, 2024
a303c0c
turn off verification for terminate posts
eriktaubeneck Feb 17, 2024
2f7e64a
use https for terminate posts
eriktaubeneck Feb 17, 2024
7e8c5d8
fix traefik bug
eriktaubeneck Feb 17, 2024
ec40bcb
remove tls from helper traefik config
eriktaubeneck Feb 17, 2024
f40f955
readd tls from helper traefik config
eriktaubeneck Feb 17, 2024
6ead567
try a different approach to not using tls for helpers
eriktaubeneck Feb 17, 2024
9273e54
local traefik working. helpers still not working with domains
eriktaubeneck Mar 4, 2024
bc772fc
server updates, use localhost for ipa connections
eriktaubeneck Mar 4, 2024
0a87b6d
remove unneeded helper_domain from cli
eriktaubeneck Mar 5, 2024
5956af0
Update README.md
eriktaubeneck Mar 5, 2024
609a16e
use sidecar0 instead of sidecar-coordinator
eriktaubeneck Mar 5, 2024
656a7e4
removed signed call to /stop. needs to be handled differently
eriktaubeneck Mar 6, 2024
136d5be
add multi-threading to compile features for IPA
eriktaubeneck Mar 7, 2024
9660d9f
add a step to generate the MPC steps file
eriktaubeneck Mar 7, 2024
a093803
typo
eriktaubeneck Mar 8, 2024
cfecf75
fix script path
eriktaubeneck Mar 8, 2024
f727f47
add env option to command
eriktaubeneck Mar 8, 2024
d5873a8
add -m flag to collect_steps
eriktaubeneck Mar 8, 2024
f815f57
add cwd to subclasses of Command
eriktaubeneck Mar 8, 2024
66ccf83
fix pylint errors
eriktaubeneck Mar 12, 2024
0dc32d8
use mkcert CA with httpx
eriktaubeneck Mar 15, 2024
1153d1d
update github.tsx to warn if OCTOKIT_GITHUB_API_KEY isn't present
eriktaubeneck Mar 15, 2024
2dfb717
avoid race condition with getting a query that may be being created
eriktaubeneck Mar 15, 2024
3103593
add -f to git checkout command, as producing steps.txt causes an over…
eriktaubeneck Mar 15, 2024
37984f0
remove verify=False from httpx requests
eriktaubeneck Mar 15, 2024
896820a
update readme for first use of mkcert
eriktaubeneck Mar 15, 2024
6db365a
refresh IPA self signed local_dev keys
eriktaubeneck Mar 16, 2024
180abe2
fix bug with test directory not existing, wrap query run in exception…
eriktaubeneck Mar 16, 2024
158d810
fix pylint and grammer error
eriktaubeneck Mar 16, 2024
591d4b0
update TODO in readme
eriktaubeneck Mar 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
adjust ports, not inferred from network.toml
  • Loading branch information
eriktaubeneck committed Feb 16, 2024
commit 6aed89e9343ae70e29655b36c59367cb982af830
8 changes: 4 additions & 4 deletions local_dev/config/network.toml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ N0Gz2XisE0JNL5f0tEyrJf/PwSlnazeMxw==
-----END CERTIFICATE-----
"""
url = "localhost:7431"
sidecar_port = "17431"
sidecar_url = "localhost:17431"

[peers.hpke]
public_key = "fde0d0c958db9f49d3f1b49cb6830b867cc810bff9e7d0cbf17c777969f3c23e"
Expand All @@ -31,7 +31,7 @@ RwAwRAIgaX95X9bgeZHgbTCl73N2j61AnljyS8DXQ7mWb6fsQXECIFgvumh8TASD
-----END CERTIFICATE-----
"""
url = "localhost:7432"
sidecar_port = "17432"
sidecar_url = "localhost:17432"

[peers.hpke]
public_key = "4e8f1cd4114a8ee8adc58a33050782e2f8ded3336a9c65725f35998e765c4e2d"
Expand All @@ -50,7 +50,7 @@ B6Bgc2gw5JC/G6ahPglwIkjO2ew02/ax6g==
-----END CERTIFICATE-----
"""
url = "localhost:7433"
sidecar_port = "17433"
sidecar_url = "localhost:17433"

[peers.hpke]
public_key = "ebedcfa02354a1d17aed80b0ed55028d0616152d5f8971291e030231dc92063d"
Expand All @@ -61,7 +61,7 @@ version = "http2"

[coordinator]
url = "localhost:7430"
sidecar_port = "17430"
sidecar_url = "localhost:17430"
certificate = """
-----BEGIN CERTIFICATE-----
MIIBHDCBwqADAgECAghMfLQt7MF1IDAKBggqhkjOPQQDAjAUMRIwEAYDVQQDDAls
Expand Down
39 changes: 10 additions & 29 deletions sidecar/app/helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,19 +18,10 @@ class Role(IntEnum):
@dataclass
class Helper:
role: Role
hostname: str
sidecar_port: int
helper_port: int
helper_url: ParseResult
sidecar_url: ParseResult
public_key: EllipticCurvePublicKey

@property
def sidecar_url(self) -> ParseResult:
return urlparse(f"http://{self.hostname}:{self.sidecar_port}")

@property
def helper_url(self) -> ParseResult:
return urlparse(f"http://{self.hostname}:{self.helper_port}")


def load_helpers_from_network_config(network_config_path: Path) -> dict[Role, Helper]:
with network_config_path.open("rb") as f:
Expand All @@ -39,40 +30,30 @@ def load_helpers_from_network_config(network_config_path: Path) -> dict[Role, He
helper_roles = list(r for r in Role if r != Role.COORDINATOR)
helpers = {}
for helper_config, role in zip(helper_configs, helper_roles):
url = urlparse(f"http://{helper_config['url']}")
hostname = str(url.hostname)
helper_port = int(url.port or 0)
sidecar_port = int(helper_config.get("sidecar_port", 0))
if not hostname or not helper_port or not sidecar_port:
raise Exception(f"{network_data=} missing data.")
helper_url = urlparse(f"http://{helper_config['url']}")
sidecar_url = urlparse(f"http://{helper_config['sidecar_url']}")
public_key_pem_data = helper_config.get("certificate")
cert = load_pem_x509_certificate(public_key_pem_data.encode("utf8"))
public_key = cert.public_key()
assert isinstance(public_key, EllipticCurvePublicKey)
helpers[role] = Helper(
role=role,
hostname=hostname,
helper_port=helper_port,
sidecar_port=sidecar_port,
helper_url=helper_url,
sidecar_url=sidecar_url,
public_key=public_key,
)

url = urlparse(f"http://{network_data['coordinator']['url']}")
hostname = str(url.hostname)
helper_port = int(url.port or 0)
sidecar_port = int(network_data["coordinator"].get("sidecar_port", 0))
if not hostname or not helper_port or not sidecar_port:
raise Exception(f"{network_data=} missing data.")
helper_url = urlparse(f"http://{network_data['coordinator']['url']}")
sidecar_url = urlparse(f"http://{network_data['coordinator']['sidecar_url']}")
public_key_pem_data = network_data["coordinator"].get("certificate")
cert = load_pem_x509_certificate(public_key_pem_data.encode("utf8"))
public_key = cert.public_key()
assert isinstance(public_key, EllipticCurvePublicKey)

helpers[Role.COORDINATOR] = Helper(
role=Role.COORDINATOR,
hostname=hostname,
helper_port=helper_port,
sidecar_port=sidecar_port,
helper_url=helper_url,
sidecar_url=sidecar_url,
public_key=public_key,
)
return helpers
7 changes: 4 additions & 3 deletions sidecar/app/query/ipa.py
Original file line number Diff line number Diff line change
Expand Up @@ -202,18 +202,19 @@ def build_from_query(cls, query: IPAQuery):
)

def run(self):
helper_urls = [
sidecar_urls = [
helper.sidecar_url
for helper in settings.helpers.values()
if helper.role != Role.COORDINATOR
]
for helper_url in helper_urls:
for sidecar_url in sidecar_urls:
url = urlunparse(
helper_url._replace(
sidecar_url._replace(
scheme="ws", path=f"/start/ipa-helper/{self.query_id}/status"
),
)
while True:
print(url)
r = httpx.get(url).json()
print(r)
status = r.get("status")
Expand Down
2 changes: 1 addition & 1 deletion sidecar/app/routes/start.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ def start_ipa_helper(
query = IPAHelperQuery(
paths=paths,
query_id=query_id,
port=settings.helper.helper_port,
port=settings.helper_port,
)
background_tasks.add_task(query.start)

Expand Down
1 change: 1 addition & 0 deletions sidecar/app/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ class Settings(BaseSettings):
network_config_path: Annotated[Path, BeforeValidator(gen_path)]
private_key_pem_path: Annotated[Path, BeforeValidator(gen_path)]
role: Role
helper_port: int
_helpers: dict[Role, Helper]
_private_key: EllipticCurvePrivateKey

Expand Down
55 changes: 34 additions & 21 deletions sidecar/cli/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
import yaml

from ..app.command import Command, start_commands_parallel
from ..app.helpers import Role, load_helpers_from_network_config
from ..app.helpers import Role


@click.group()
Expand All @@ -18,18 +18,18 @@ def cli():
def start_helper_sidecar_command(
config_path: Path,
identity: int,
helper_port: int,
sidecar_port: int,
root_path: Optional[Path] = None,
):
role = Role(int(identity))
network_config = config_path / Path("network.toml")
root_path = root_path or Path(f"tmp/sidecar/{role.value}")
root_path.mkdir(parents=True, exist_ok=True)
helpers = load_helpers_from_network_config(network_config)
if role == Role.COORDINATOR:
private_key_pem_path = config_path / Path("coordinator.key")
else:
private_key_pem_path = config_path / Path(f"h{role.value}.key")
helper = helpers[role]
cmd = "uvicorn sidecar.app.main:app"
env = {
**os.environ,
Expand All @@ -38,7 +38,8 @@ def start_helper_sidecar_command(
"CONFIG_PATH": config_path,
"NETWORK_CONFIG_PATH": network_config,
"PRIVATE_KEY_PEM_PATH": private_key_pem_path,
"UVICORN_PORT": str(helper.sidecar_port),
"HELPER_PORT": str(helper_port),
"UVICORN_PORT": str(sidecar_port),
"UVICORN_HOST": "0.0.0.0",
}
return Command(cmd=cmd, env=env)
Expand Down Expand Up @@ -113,6 +114,8 @@ def create_tls_config(cert_path: Path, key_path: Path, config_path: Path):
def start_traefik_command(
config_path: Path,
identity: int,
helper_port: int,
sidecar_port: int,
root_domain: str,
):
role = Role(int(identity))
Expand All @@ -122,9 +125,6 @@ def start_traefik_command(
else:
sidecar_domain = f"sidecar{role.value}.{root_domain}"
helper_domain = f"helper{role.value}.{root_domain}"
network_config = config_path / Path("network.toml")
helpers = load_helpers_from_network_config(network_config)
helper = helpers[role]
cert_path = config_path / Path("cert.pem")
key_path = config_path / Path("key.pem")
tls_config_path = Path("sidecar/traefik/dynamic/tls_conf.yaml")
Expand All @@ -138,8 +138,8 @@ def start_traefik_command(
sidecar_domain=sidecar_domain,
helper_domain=helper_domain,
config_path=dynamic_config_path,
sidecar_port=helper.sidecar_port,
ipa_port=helper.helper_port,
sidecar_port=sidecar_port,
ipa_port=helper_port,
)

env = {
Expand All @@ -149,6 +149,7 @@ def start_traefik_command(
return Command(cmd=cmd, env=env)


# pylint: disable=too-many-arguments
@cli.command
@click.option(
"--config_path",
Expand All @@ -158,22 +159,30 @@ def start_traefik_command(
)
@click.option("--root_path", type=click_pathlib.Path(), default=None)
@click.option("--root_domain", type=str, default="ipa-helper.dev")
@click.option("--helper_port", type=int, default=7430)
@click.option("--sidecar_port", type=int, default=17430)
@click.option("--identity", required=True, type=int)
def start_helper_sidecar(
config_path: Path,
root_path: Optional[Path],
root_domain: str,
helper_port: int,
sidecar_port: int,
identity: int,
):
sidecar_command = start_helper_sidecar_command(
config_path,
identity,
root_path,
config_path=config_path,
identity=identity,
helper_port=helper_port,
sidecar_port=sidecar_port,
root_path=root_path,
)
traefik_command = start_traefik_command(
config_path,
identity,
root_domain,
config_path=config_path,
identity=identity,
helper_port=helper_port,
sidecar_port=sidecar_port,
root_domain=root_domain,
)
start_commands_parallel([sidecar_command, traefik_command])

Expand All @@ -186,9 +195,13 @@ def start_helper_sidecar(
show_default=True,
)
@click.option("--root_path", type=click_pathlib.Path(), default=None)
@click.option("--helper_start_port", type=int, default=7430)
@click.option("--sidecar_start_port", type=int, default=17430)
def start_local_dev(
config_path: Path,
root_path: Optional[Path],
helper_start_port: int,
sidecar_start_port: int,
):
npm_install_command = Command(
cmd="npm --prefix server install",
Expand All @@ -198,15 +211,15 @@ def start_local_dev(
cmd="npm --prefix server run dev",
)

network_config = Path(config_path) / Path("network.toml")
helpers = load_helpers_from_network_config(network_config)
sidecar_commands = [
start_helper_sidecar_command(
config_path,
helper.role,
root_path,
config_path=config_path,
identity=role,
helper_port=helper_start_port + int(role),
sidecar_port=sidecar_start_port + int(role),
root_path=root_path,
)
for helper in helpers.values()
for role in Role
]
commands = [npm_run_dev_command] + sidecar_commands
start_commands_parallel(commands)
Expand Down