WIP: Manage Emulators automatically #55

zner0L · 2024-10-28T19:19:21Z

We want to manage emulators automatically, create and delete them as needed and recover failure states (in doubt by creating new emulators on the fly).
This breaks some parts of the current API, but I took care not to change too much. If you think I should get rid of some legacy code, I am open for doing it with this PR, too.

Depends on tweaselORG/appstraction#137 and tweaselORG/appstraction#140 to be released to merge.

baltpeter

I haven't tested this yet since snapshot management isn't fully implemented yet, but I really like the direction this is going!

I could also definitely see the AndroidEmulator class being useful in other contexts, so maybe we should move that to andromatic at some point? But we don't have the time for that now.

README.md

src/version.gen.ts

src/emulator.ts

baltpeter · 2024-11-25T07:46:37Z

src/emulator.ts

+
+export const listSnapshots = async (): Promise<{ [name: string]: string[] }> => {
+    // This returns a list of snapshots of all devices
+    const { stdout } = await runAndroidDevTool('emulator', ['-snapshot-list']);


Does that need a specific emu version?

❯ emulator -snapshot-list INFO | Android emulator version 32.1.12.0 (build_id 9751036) (CL:N/A) ERROR | No AVD specified. Use '@foo' or '-avd foo' to launch a virtual device named 'foo'

Hm, at least for your version, this should be available, according to the documentation close to the release date. I am using 35.2.10.0, though. Have you set the correct ANDROID_HOME?

The snippet above was using my system emu (with correct ANDROID_HOME). Doesn't work with the andromatic one, either:

❯ export ANDROID_HOME=~/.cache/andromatic/ ❯ ~/.cache/andromatic/emulator/emulator -snapshot-list INFO | Android emulator version 33.1.24.0 (build_id 11237101) (CL:N/A) ERROR | No AVD specified. Use '@foo' or '-avd foo' to launch a virtual device named 'foo'

It seems the docs are wrong then. Just tested on my Macbook where I have version 32 as well and got the same error.

Specifying an AVD name as in the docs you linked starts the emu and doesn't list the snapshots:

❯ ~/.cache/andromatic/emulator/emulator @mitm11-honey -snapshot-list INFO | Android emulator version 33.1.24.0 (build_id 11237101) (CL:N/A) INFO | Found systemPath /home/benni/.cache/andromatic//system-images/android-30/google_apis/x86_64/ INFO | Storing crashdata in: , detection is enabled for process: 1494826 INFO | Duplicate loglines will be removed, if you wish to see each individual line launch with the -log-nofilter flag. WARNING | Please update the emulator to one that supports the feature(s): Vulkan INFO | Increasing RAM size to 2048MB WARNING | FeatureControl is requesting a non existing feature. Fontconfig warning: "/usr/share/fontconfig/conf.avail/05-reset-dirs-sample.conf", line 6: unknown element "reset-dirs" library_mode host gpu mode host I1125 13:52:53.193065 1494826 HealthMonitor.cpp:279] HealthMonitor disabled. cannot add library libvulkan.so: failed added library libvulkan.so.1 createGlobalVkEmulation:963 Selecting Vulkan device: NVIDIA GeForce RTX 2060 initialize: Supports id properties, got a vulkan device UUID I1125 13:52:53.423908 1494826 VkCommonOperations.cpp:1225] Initializing VkEmulation features: I1125 13:52:53.423929 1494826 VkCommonOperations.cpp:1226] glInteropSupported: true I1125 13:52:53.423936 1494826 VkCommonOperations.cpp:1227] useDeferredCommands: true I1125 13:52:53.423944 1494826 VkCommonOperations.cpp:1229] createResourceWithRequirements: true I1125 13:52:53.423952 1494826 VkCommonOperations.cpp:1230] useVulkanComposition: false I1125 13:52:53.423958 1494826 VkCommonOperations.cpp:1231] useVulkanNativeSwapchain: false I1125 13:52:53.423964 1494826 VkCommonOperations.cpp:1232] enable guestRenderDoc: false I1125 13:52:53.423970 1494826 VkCommonOperations.cpp:1233] ASTC LDR emulation mode: 2 I1125 13:52:53.423977 1494826 VkCommonOperations.cpp:1234] enable ETC2 emulation: true I1125 13:52:53.423982 1494826 VkCommonOperations.cpp:1235] enable Ycbcr emulation: false I1125 13:52:53.423988 1494826 VkCommonOperations.cpp:1236] guestUsesAngle: false I1125 13:52:53.423995 1494826 VkCommonOperations.cpp:1237] useDedicatedAllocations: false I1125 13:52:53.425509 1494826 FrameBuffer.cpp:486] Graphics Adapter Vendor Google (NVIDIA Corporation) I1125 13:52:53.425517 1494826 FrameBuffer.cpp:487] Graphics Adapter Android Emulator OpenGL ES Translator (NVIDIA GeForce RTX 2060/PCIe/SSE2) I1125 13:52:53.425523 1494826 FrameBuffer.cpp:488] Graphics API Version OpenGL ES 3.0 (4.5.0 NVIDIA 535.183.01) I1125 13:52:53.425541 1494826 FrameBuffer.cpp:489] Graphics API Extensions GL_OES_EGL_sync GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_depth24 GL_OES_depth32 GL_OES_element_index_uint GL_OES_texture_float GL_OES_texture_float_linear GL_OES_compressed_paletted_texture GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth_texture GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_packed_depth_stencil GL_OES_vertex_half_float GL_OES_texture_npot GL_OES_rgb8_rgba8 GL_EXT_color_buffer_float GL_EXT_color_buffer_half_float GL_EXT_texture_format_BGRA8888 GL_APPLE_texture_format_BGRA8888 GL_EXT_texture_buffer I1125 13:52:53.425551 1494826 FrameBuffer.cpp:490] Graphics Device Extensions N/A WARNING | The emulator now requires a signed jwt token for gRPC access! Use the -grpc flag if you really want an open unprotected grpc port INFO | Using security allow list from: /home/benni/.cache/andromatic/emulator/lib/emulator_access.json WARNING | *** Basic token auth should only be used by android-studio *** INFO | The active JSON Web Key Sets can be found here: /run/user/1000/avd/running/1494826/jwks/d41f381f-cacb-409b-9689-b47fcd8dfa97/active.jwk INFO | Scanning /run/user/1000/avd/running/1494826/jwks/d41f381f-cacb-409b-9689-b47fcd8dfa97 for jwk keys. INFO | Started GRPC server at 127.0.0.1:8554, security: Local, auth: +token INFO | Advertising in: /run/user/1000/avd/running/pid_1494826.ini List of snapshots present on all disks: ID TAG VM SIZE DATE VM CLOCK -- default_boot 1.1M 2024-11-25 13:52:40 00:00:04.588 I1125 13:52:53.648558 1494867 FrameBuffer.cpp:3682] setDisplayConfigs w 1080 h 1920 dpiX 420 dpiY 420 I1125 13:52:53.648601 1494867 FrameBuffer.cpp:3695] setDisplayActiveConfig 0 INFO | Loading snapshot 'default_boot'... INFO | Successfully loaded snapshot 'default_boot' INFO | Info: Sandboxing disabled by user. ((null):0, (null)) INFO | Wait for emulator (pid 1494826) 20 seconds to shutdown gracefully before kill;you can set environment variable ANDROID_EMULATOR_WAIT_TIME_BEFORE_KILL(in seconds) to change the default value (20 seconds) [1494826:1495037:1125/135257.138795:ERROR:nss_util.cc(748)] After loading Root Certs, loaded==false: NSS error code: -8018 INFO | Critical: Uncaught SyntaxError: Unexpected token . (https://maps.googleapis.com/maps/api/js?libraries=places&key=AIzaSyABrfflV49aFWF4Kohsf1VSx68td8f-sFY&callback=initMap&v=3.53:30, (null)) INFO | Saving with gfxstream=1 INFO | Critical: Uncaught SyntaxError: Unexpected token . (https://maps.googleapis.com/maps/api/js?libraries=places&key=AIzaSyABrfflV49aFWF4Kohsf1VSx68td8f-sFY&callback=initMap&v=3.53:30, (null)) ERROR | stop: Not implemented

Unless I am doing something really, really wrong, -snapstorage doesn't seem to work at all anyway.

I tried:

emulator -avd testdeleteme -snapstorage /tmp/snapstorage emulator -avd testdeleteme -snapstorage /tmp/snapstorage.img export ANDROID_HOME=~/.cache/andromatic/ ~/.cache/andromatic/emulator/emulator -avd testdeleteme -snapstorage /tmp/snapstorage.img

And created snapshots using adb emu avd snapshot save tmp123 as well as the GUI. In both cases, they were stored in ~/.android/avd/testdeleteme.avd/snapshots. /tmp/snapstorage is empty and /tmp/snapstorage.img was never created.

It’s hard to tell, there is a -no-snapstorage options, but I don’t care to test this. I guess I’ll just use the folders as a fallback?

But again, there is no way to pass to pass any of these options to CA anyway, is there?

Not really, no. We also don’t use this function right now, so maybe I shouldn’t care really? As I said in the comment, I don’t know if I really want to introduce the added complexity of snapshot management anyway.

As I said in the comment, I don’t know if I really want to introduce the added complexity of snapshot management anyway.

But you included most of that already, how much more is there to do? Also, we decided to move the honey data automation from cli to CA.

src/emulator.ts

src/util.ts

src/index.ts

examples/multiple-apps.ts

baltpeter

This is great, but as I've said before, I don't think it can be merged without snapshot management.

src/index.ts

baltpeter · 2024-11-28T14:35:21Z

README.md

@@ -47,7 +47,9 @@ The following example collects the traffic for an app in the Android emulator. I



I feel like a major feature like managed emulators deserves an actual mention in the README, doesn't it? :D

README.md

src/emulator.ts

src/index.ts

src/emulator.ts

baltpeter · 2024-12-02T07:44:37Z

src/index.ts

@@ -337,39 +329,38 @@ export type AndroidEmulatorRunTargetOptionsManaged = {
    managedEmulatorOptions: {
        /**
         * A key to distinguish the emulator from other ones created by cyanoacrylate. All analyses using the same key
-         * will share an emulator. The created emulator will be named `cyanoacrylate-{key}-{MD5 hash of the options}`.
+         * will share an emulator. The created emulator will be named `cyanoacrylate-{key}-{SHA1 hash of the


Suggested change

* will share an emulator. The created emulator will be named `cyanoacrylate-{key}-{SHA1 hash of the

* will share an emulator. The created emulator will be named `cyanoacrylate-{key}-{MD5 hash of the

baltpeter · 2024-12-02T07:52:59Z

src/index.ts

+                    !emulator?.resetSnapshotName &&
+                    targetOptions?.managed &&
+                    (analysisOptions?.capabilities as unknown as SupportedCapability<'android'> | undefined)?.includes(
+                        'frida'


We should error if honeyData is specified but frida isn't in the capabilities.

baltpeter · 2024-12-02T07:56:19Z

src/index.ts

-                            emulator.resetSnapshotName = snapshotName;
-                        }
-                    })(),
-                    abortSignal && rejectOnAbort(abortSignal),


Can you explain why we don't need the rejectOnAbort anymore? Is this handled by the abortSignal?.throwIfAborted(); and it was never needed?

I realized that rejectOnAbort() contributed to the memory leakage, as we never de-registered the listener if there was no abort (and the promise never resolved). I mainly put it there, because we didn’t have so many methods accepting abortSignals, yet. But now that we abort down the line, that is better anyway (since rejecting here doesn’t stop any of the running processes).

zner0L added 7 commits October 28, 2024 20:15

WIP: Create emulators automatically and manage their failure states

20fa014

Keep created emulators, manage snapshots as well

483fe35

Move emulator state into a separate class

9d9c325

Asynchronously handle errors via AbortSignal

4690b62

Send AbortSignals to new appstraction endpoints

84808d8

adapt examples

cb8abd2

Think about snapshot management

fe2d0ed

zner0L requested a review from baltpeter November 22, 2024 23:44

zner0L marked this pull request as ready for review November 22, 2024 23:44

zner0L mentioned this pull request Nov 22, 2024

Sprinkle AbortSignals over the code to make it cancellable tweaselORG/appstraction#140

Open

squash me: Document the new AndroidEmulator class

2119d02

baltpeter reviewed Nov 25, 2024

View reviewed changes

zner0L added 4 commits November 25, 2024 15:32

Address issues from code review

af01e7c

Update README

fbd378e

Address issues from code review

df9f489

Change API slightly and move restart logic

9250b58

zner0L requested a review from baltpeter November 27, 2024 14:56

baltpeter reviewed Nov 28, 2024

View reviewed changes

Snapshot management

6e2b52f

zner0L requested a review from baltpeter November 30, 2024 00:12

baltpeter reviewed Dec 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Manage Emulators automatically #55

WIP: Manage Emulators automatically #55

zner0L commented Oct 28, 2024 •

edited

Loading

baltpeter left a comment

baltpeter Nov 25, 2024

zner0L Nov 25, 2024

baltpeter Nov 25, 2024

zner0L Nov 25, 2024

baltpeter Nov 25, 2024

baltpeter Nov 25, 2024

zner0L Nov 25, 2024

baltpeter Nov 25, 2024

zner0L Nov 25, 2024 •

edited

Loading

baltpeter Nov 26, 2024

baltpeter left a comment

baltpeter Nov 28, 2024

baltpeter Dec 2, 2024

baltpeter Dec 2, 2024

baltpeter Dec 2, 2024

zner0L Dec 4, 2024

		@@ -47,7 +47,9 @@ The following example collects the traffic for an app in the Android emulator. I

	* will share an emulator. The created emulator will be named `cyanoacrylate-{key}-{SHA1 hash of the
	* will share an emulator. The created emulator will be named `cyanoacrylate-{key}-{MD5 hash of the

WIP: Manage Emulators automatically #55

Are you sure you want to change the base?

WIP: Manage Emulators automatically #55

Conversation

zner0L commented Oct 28, 2024 • edited Loading

baltpeter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zner0L Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

baltpeter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zner0L commented Oct 28, 2024 •

edited

Loading

zner0L Nov 25, 2024 •

edited

Loading