Merge branch 'main' into tedi/alllogs

.github/workflows/push-docs.yml

@@ -29,6 +29,12 @@ jobs:
       - name: Append changelog
         run: cat CHANGELOG.md >> docs/docs/changelog.md
         if: github.ref == 'refs/heads/main' || steps.docs-diff.outputs.DOCS_CHANGED != 0
+      - name: Append Go README
+        run: cat api/golang/README.md >> docs/docs/sdk-examples/go-sdk-example.md
+        if: github.ref == 'refs/heads/main' || steps.docs-diff.outputs.DOCS_CHANGED != 0
+      - name: Append TS README
+        run: cat api/typescript/README.md >> docs/docs/sdk-examples/ts-sdk-example.md
+        if: github.ref == 'refs/heads/main' || steps.docs-diff.outputs.DOCS_CHANGED != 0
       - name: Yarn Install
         run: yarn --cwd ./docs --frozen-lockfile install
         if: github.ref == 'refs/heads/main' || steps.docs-diff.outputs.DOCS_CHANGED != 0

api/golang/README.md

@@ -0,0 +1,72 @@
+# Kurtosis Go SDK
+
+This guide provides instructions and code snippets to help you get started with the Kurtosis Go SDK. It enables you to create and manage enclaves programmatically, without having to rely on the Kurtosis Enclave Manager (UI) or the Kurtosis CLI.
+
+The main way to interact with objects from the Kurtosis ecosystem is by getting its *context*. There are three main contexts:
+1. **KurtosisContext**: contains methods for interacting with Kurtosis Engine, allowing manipulation of enclaves.
+2. **EnclaveContext**: contains methods for interacting with an enclave, allowing execution of Starlark scripts.
+3. **ServiceContext**: contains methods for interacting with a service, allowing inspecting its details.
+
+This guide will also help you create and get these contexts.
+
+## Setting Up
+
+To use the Kurtosis Go SDK, you need to add it as a dependency to your Go module. You can do this with the following `go get` command:
+
+```console
+$ go get github.com/kurtosis-tech/kurtosis/api/golang
+```
+
+Please ensure that you have a running Kurtosis Engine instance before executing your code. You can check the status of the Kurtosis Engine using the following command:
+
+```console
+$ kurtosis engine status
+```
+
+Make note of the Engine's version and status information.
+
+## Creating an Enclave
+
+The first step is to obtain a Kurtosis Context, which represents a Kurtosis instance in your Go code:
+
+```go
+kurtosisCtx, err := kurtosis_context.NewKurtosisContextFromLocalEngine()
+```
+
+Next, you can use the Kurtosis Context to create an enclave, which will provide an Enclave Context for managing the enclave:
+
+```go
+enclaveName := "my-enclave"
+enclaveCtx, err := kurtosisCtx.CreateEnclave(ctx, enclaveName)
+```
+
+## Configure for Starlark Runs
+
+Using the Enclave Context, you can perform actions like adding services using Starlark scripts:
+
+```go
+starlarkRunConfig := starlark_run_config.NewRunStarlarkConfig()
+starlarkScript := `
+def run(plan):
+    serviceConfig := ServiceConfig{
+        Image: "httpd",
+    }
+    plan.AddService(name: "my-service", config: serviceConfig)
+`
+starlarkRunResult, err := enclaveCtx.RunStarlarkScriptBlocking(ctx, starlarkScript, starlarkRunConfig)
+```
+## Interacting with services
+After adding a service, you can interact with it by obtaining a service context and running commands:
+
+```go
+serviceCtx, err := enclaveCtx.GetServiceContext("my-service")
+code, output, err := serviceCtx.ExecCommand([]string{"ls"})
+```
+
+For ephemeral enclaves, such as those used in end-to-end testing, you can destroy the created enclave:
+
+```go
+err := kurtosisCtx.DestroyEnclave(ctx, enclaveName)
+```
+
+These instructions should help you get started with using the Kurtosis Go SDK to create and manage enclaves for your projects. If you need further assistance or ahve questions, please open a [Github Discussion](https://github.com/kurtosis-tech/kurtosis/discussions/categories/q-a) or ping us in [Discord](https://discord.com/invite/HUapYX9RvV).

api/typescript/README.md

@@ -0,0 +1,90 @@
+# Kurtosis Typescript SDK
+
+This guide provides instructions and code snippets to help you get started with the Kurtosis Typescript SDK. It enables you to create and manage enclaves programmatically, without having to rely on the Kurtosis Enclave Manager (UI) or the Kurtosis CLI.
+
+The main way to interact with objects from the Kurtosis ecosystem is by getting its *context*. There are three main contexts:
+1. **KurtosisContext**: contains methods for interacting with Kurtosis Engine, allowing manipulation of enclaves.
+2. **EnclaveContext**: contains methods for interacting with an enclave, allowing execution of Starlark scripts.
+3. **ServiceContext**: contains methods for interacting with a service, allowing inspecting its details.
+
+This guide will also help you create and get these contexts.
+
+## Setting Up
+
+To use the Kurtosis Typescript SDK, you need to add it as a dependency to your NPM package. You can do this with the following `npm i` command:
+
+```console
+$ npm i kurtosis-sdk
+```
+
+Please ensure that you have a running Kurtosis Engine instance before executing your code. You can check the status of the Kurtosis Engine using the following command:
+
+```console
+$ kurtosis engine status
+```
+
+Make note of the Engine's version and status information.
+
+## Creating an Enclave
+
+The first step is to obtain a Kurtosis Context, which represents a Kurtosis instance in your Typescript code:
+
+```typescript
+const newKurtosisContextResult = await KurtosisContext.newKurtosisContextFromLocalEngine()
+if(newKurtosisContextResult.isErr()) {
+    // Check for error
+}
+const kurtosisContext = newKurtosisContextResult.value;
+```
+
+Next, you can use the Kurtosis Context to create an enclave, which will provide an Enclave Context for managing the enclave:
+
+```typescript
+const enclaveName = "my-enclave"
+const createEnclaveResult = await kurtosisContext.createEnclave();
+if(createEnclaveResult.isErr()) {
+    // Check for error
+}
+const enclaveContext = createEnclaveResult.value;
+```
+
+## Configure for Starlark Runs
+
+Using the Enclave Context, you can perform actions like adding services using Starlark scripts:
+
+```typescript
+const starlarkRunConfig = new StarlarkRunConfig(
+StarlarkRunConfig.WithSerializedParams(params)
+)
+const starlarkScript = `
+def run(plan):
+    serviceConfig := ServiceConfig{
+        Image: "httpd",
+    }
+    plan.AddService(name: "my-service", config: serviceConfig)
+`
+return enclaveContext.runStarlarkScriptBlocking(starlarkScript, starlarkRunConfig)
+```
+## Interacting with services
+After adding a service, you can interact with it by obtaining a service context and running commands:
+
+```typescript
+const getServiceCtxResult = await enclaveCtx.getServiceContext("my-service")
+if(getServiceCtxResult.isErr()) {
+    // Check for error
+}
+const serviceContext = getServiceCtxResult.value;
+const command = ["ls"]
+serviceContext.execCommand(command)
+```
+
+For ephemeral enclaves, such as those used in end-to-end testing, you can destroy the created enclave:
+
+```typescript
+const destroyEnclaveResponse = await kurtosisContext.destroyEnclave(enclaveName)
+if(destroyEnclaveResponse.isErr()) {
+    // Check for error
+}
+```
+
+These instructions should help you get started with using the Kurtosis Typescript SDK to create and manage enclaves for your projects. If you need further assistance or ahve questions, please open a [Github Discussion](https://github.com/kurtosis-tech/kurtosis/discussions/categories/q-a) or ping us in [Discord](https://discord.com/invite/HUapYX9RvV).

api/typescript/package.json

@@ -25,9 +25,9 @@
     "author": "Kurtosis Technologies Inc <[email protected]> (https://www.kurtosistech.com/)",
     "license": "Apache-2.0",
     "bugs": {
-        "url": "https://github.com/kurtosis-tech/kurtosis-sdk/issues"
+        "url": "https://github.com/kurtosis-tech/kurtosis/issues"
     },
-    "homepage": "https://github.com/kurtosis-tech/kurtosis-sdk#readme",
+    "homepage": "https://github.com/kurtosis-tech/kurtosis",
     "engines": {
         "node": ">=16.13.0"
     },

container-engine-lib/lib/backend_impls/docker/docker_manager/docker_manager.go

@@ -19,6 +19,7 @@ import (
 	"github.com/docker/docker/pkg/jsonmessage"
 	"github.com/docker/docker/pkg/stdcopy"
 	"github.com/docker/go-connections/nat"
+	kurtosis_sdk_version "github.com/kurtosis-tech/kurtosis/api/golang/kurtosis_version"
 	"github.com/kurtosis-tech/kurtosis/container-engine-lib/lib/backend_impls/docker/docker_kurtosis_backend/consts"
 	docker_manager_types "github.com/kurtosis-tech/kurtosis/container-engine-lib/lib/backend_impls/docker/docker_manager/types"
 	"github.com/kurtosis-tech/kurtosis/container-engine-lib/lib/backend_interface/objects/compute_resources"
@@ -304,7 +305,7 @@ func (manager *DockerManager) ListUnusedImages(ctx context.Context) ([]types.Ima
 			continue
 		}
 		for _, tag := range image.RepoTags {
-			if strings.Contains(tag, kurtosisTagPrefix) && containsSemVer(tag) {
+			if strings.Contains(tag, kurtosisTagPrefix) && containsSemVer(tag) && !strings.Contains(tag, kurtosis_sdk_version.KurtosisVersion) {
 				unusedImages = append(unusedImages, image)
 			}
 		}

docs/docs/concepts-reference/kurtosis-yml.md

@@ -14,6 +14,12 @@ The `kurtosis.yml` file is a manifest file necessary to turn a directory into [a
 ```yaml
 # The locator naming this package.
 name: github.com/package-author/package-repo/path/to/directory-with-kurtosis.yml
+# The package's description which will be shown in the Enclave Manager on the UI.
+description: A sentence describing what the package does
+# The package's dependencies replace options
+replace:
+  # Replacing the official Postgres package with my fork
+  github.com/kurtosis-tech/postgres-package: github.com/my-github-user/postgres-package
 ```
 
 Example usage:
@@ -32,6 +38,115 @@ name: github.com/author/package-repo/path/to/directory-with-kurtosis.yml
 The key take away is that `/path/to/directory-with-kurtosis.yml` only needs to be provided if `kurtosis.yml` is not present in the repository's root.
 :::
 
+Replace
+-------
+There are often times when you want to substitute one of your Kurtosis package’s dependencies with another dependency. For example, someone might have forked one of your package's dependencies, and you want to test your package against their fork rather than the normal version. Finding and updating all the dependency-referencing commands (`import_module` , `upload_file`, `read_file`, etc.) in your package is tedious and error-prone, so the `kurtosis.yml` supports a `replace` key to do it for you.
+
+The `replace` key accepts a key-value map where each key is the [locator][locators] of a package to be replaced, and each value is the package locator to replace it with.
+
+For example:
+
+```yaml
+name: github.com/my-github-user/my-package
+replace:
+  # Replacing the official Postgres package with my fork
+  github.com/kurtosis-tech/postgres-package: github.com/my-github-user/postgres-package
+```
+
+This behaves just as if you’d manually updated each Starlark dependency-referencing command that consumes `github.com/kurtosis-tech/postgres-package` and replaced it with `github.com/my-github-user/postgres-package`. This replace includes transitive dependencies: a dependency package that itself uses `github.com/kurtosis-tech/postgres-package` will _also_ instead now use `github.com/my-github-user/postgres-package`!
+
+:::info
+A `replace` entry does nothing if replaced package isn't actually depended upon.
+:::
+
+You may optionally append an `@` and version after the replacement package locator to specify which version of the replacement package ought to be used.
+
+For example:
+
+```yaml
+name: github.com/my-github-user/my-package
+replace:
+  # Replacing the official Postgres package with version 1.2.3
+  github.com/kurtosis-tech/postgres-package: github.com/my-github-user/[email protected]
+```
+
+Like `import_module` and all other dependency-referencing commands, the version can be a tag, branch, or a full commit hash.
+
+:::info
+Go programmers will identify the similarities with the `replace` directive in the `go.mod` file. This is not accidental; the Kurtosis packaging system draws heavy inspiration from the Go module system.
+:::
+
+### Replace In kurtosis.yml Of Dependencies
+`replace` instructions are only evaluated in the `kurtosis.yml` of the root package being called, and are ignored in the `kurtosis.yml`s of package dependencies.
+For example, suppose we had two packages, Dependency and Consumer`, that both use `github.com/kurtosis-tech/postgres-package` in their Starlark.
+Additionally, Consumer depends on Dependency in its Starlark code.
+Their `kurtosis.yml` files look like so:
+
+Dependency:
+
+```yaml
+# NOTE: this package uses github.com/kurtosis-tech/postgres-package in its Starlark
+
+name: github.com/somebody/dependency
+replace:
+  # Replace the official Postgres package with the fork from user 'someboday'
+  github.com/kurtosis-tech/postgres-package: github.com/somebody/postgres-package
+```
+
+Consumer:
+
+```yaml
+# NOTE: this package uses both of these in its Starlark:
+# - github.com/somebody/dependency
+# - github.com/kurtosis-tech/postgres-package
+
+name: github.com/somebody/consumer
+
+# This package does NOT have a 'replace' directive
+```
+
+If the user runs Consumer (`kurtosis run github.com/somebody/consumer`), **Dependency's `replace` will not be evaluated**. This is because `replace` instructions are only executed in the `kurtosis.yml` of the root package.
+
+### Colliding Replace Values
+It is possible to have two Kurtosis packages, one nested within the other.
+For example, suppose we have `github.com/kurtosis-tech/parent` and `github.com/kurtosis-tech/parent/child`.
+To the Kurtosis packaging system, these packages do not have any hierarchical relation and are simply treated as two entirely separate packages with different identifiers for dependency purposes.
+However, this still presents a problem for `replace`: let's suppose you have your own package that depends on both `parent` and `child`, and you want to `replace` both.
+How does this work?
+
+When one package exists in a subpath of another, the more specific (child) package gets replaced first. For example, if your package's `kurtosis.yml` looks like so...
+
+```yaml
+name: github.com/mieubrisse/my-package
+replace:
+  # Replace parent with new-parent
+  github.com/kurtosis-tech/parent: github.com/mieubrisse/new-parent
+
+  # Replace child with new-child
+  github.com/kurtosis-tech/parent/child: github.com/mieubrisse/new-child
+```
+
+...and you have Starlark code inside your package that looks like this...
+
+```python
+child = import_module("github.com/kurtosis-tech/parent/child/main.star")
+
+ 
+def run(plan):
+    child.run(plan)
+```
+
+...then Kurtosis would use the "more specific" rule to replace `github.com/kurtosis-tech/parent/child/main.star` with `github.com/mieubrisse/new-child`.
+The `import_module` call would then be functionally equivalent to the following:
+
+```python
+child = import_module("github.com/mieubrisse/new-child/main.star")
+step_
+```
+
+In other words, replace directives are matched “longest match first”.
+
 <!----------------------- ONLY LINKS BELOW HERE ----------------------------->
 [package]: ./packages.md
-[how-do-kurtosis-imports-work-explanation]: ../explanations/how-do-kurtosis-imports-work.md
+[how-do-kurtosis-imports-work-explanation]: ../explanations/how-do-kurtosis-imports-work.md
+[locators]: ./locators.md

docs/docs/sdk-examples/go-sdk-example.md

@@ -0,0 +1,5 @@
+---
+title: Go SDK Example
+sidebar_label: Go
+slug: /go-sdk-example
+---

docs/docs/sdk-examples/ts-sdk-example.md

@@ -0,0 +1,5 @@
+---
+title: TypeScript SDK Example
+sidebar_label: TypeScript
+slug: /ts-sdk-example
+---

docs/sidebars.js

@@ -35,6 +35,15 @@ const sidebars = {
                 {type: 'autogenerated', dirName: 'guides'}
             ]
         },
+        {
+            type: 'category',
+            label: 'SDK Examples',
+            collapsed: true,
+            items: [
+                'sdk-examples/go-sdk-example',
+                'sdk-examples/ts-sdk-example'
+            ]
+        },
         {
             type: 'category',
             label: 'API Reference',