Use Better Screenshot Means

[JakeWharton]

...to get awesome full-screen screenshots.

Socket to the host which captures using normal means through ddmlib. Could be a simple REST API or just dumb text communication. Client would block until a response was received. Fall back to decor view.

[JakeWharton]

Root detection added in jw/root branch.

[edenman]

Alternatively, try to use this: http://code.google.com/p/android-screenshot-library/ then we wouldn't need root!

[brnhffmnn]

Just thinking out loud: Would it be possible to take the screenshots via ddms? Somehow the instrumenation app would need to send a command up to the runner, so it could invoke IDevice.getScreenshot.

[JakeWharton]

The test would also have to block until the screenshot was taken which is the harder problem.

[brnhffmnn]

I managed to get DDMS take screenshots. Tests are also blocking until the screenshot is taken.

I hooked up android-screenshot-lib (not the same as the one on google code). Basically it listens for logs with a special tag. To get the test to block, a file will be created on the device which gets deleted via adb. As soon as the file is gone the test continues. Root is obviously not needed on the device.

I didn't request a pull yet, because it's not fully tested yet (+I didn't write tests) and there is still one issue: screenshots appear to be duplicated in the resulting report. But I didn't find the cause yet. Also the screenshots are not rotated with respect to the current device orientation, but I actually don't mind that.

May I ask for a review? See my screenshot-ddms branch

[JakeWharton]

@panzerfahrer We explicitly avoided going that route since it's very fragile and the blocking is awkward. You may have noticed that the library is where we stole the animated GIF creation from, however.

One way we were going to approach it was to start a web server on the desktop and use adb to port forward it to all of the devices. This way the Spoon.screenshot method could make a synchronous HTTP connection with the tag and test info and wait for the response.

The other would be to use a native service deployed via adb on test run which allows taking full screen captures on the device and then pulling at the end in the normal fashion.

For something like what you describe to be added we definitely would need things like proper orientation support. I'll try to take a look at what you have tonight.

[RichardGuion]

I am following this issue as I would like to see dialogs in the screenshots. I am using Robotium in addition to Spoon, when I used their solo.takeScreenshot method it captured the dialog fine. However, it just captured the view for the dialog and not the entire screen. You probably don't like that approach as it would mess up the animated gif.

[JakeWharton]

Correct. We also don't want to depend on Robotium!

We'll likely go the route of starting a native service over ADB and use it to take screenshots of the entire screen (including the status bar and soft buttons, woot).

[JakeWharton]

Started work on the jw/full-screen-capture branch. The idea is that the runner deploys a binary that listens on a port for connections and a full path to the target image. The server captures the framebuffer and encodes it to a PNG at the destination file. If any exceptions are encountered they are sent back along the socket connection, otherwise it is simply closed.

AOSP has two on-device binaries for capturing the framebuffer screencap and screenshot. Both have dependencies on internal AOSP things which we probably want to tear out into a standalone tool. We also would need to get compilation working for all architectures and have detection (using #76) to deploy the correct binary. I'm fairly certain the maven-android-plugin supports this but we'd have to figure out how to bundle the results as part of the final runner jar.

We also need an effective way of not starting duplicate servers and killing the server once execution has completed. A simple PID file should suffice.

I don't have the time to work on this now. Maybe in 4-6 weeks.

[brnhffmnn]

After my first attempt with the quirky "lock file" approach, I tried it with running a server on the device.

I'm starting a server within the InstrumentationTestRunner. spoon-runner houses a client which connects to the server via an ADB port forwarding. I'm not using a native service, but plain Java socket channels.

Unfortunately, it's not really working and I couldn't yet figure out what I'm missing. Anyhow, maybe it can be of help. I had to postpone my work on this, so I'm not sure when can get a look into it again.

panzerfahrer/screenshot-ddms-socket

[JakeWharton]

@panzerfahrer That's the route we initially wanted to go as well. Haven't really had time to explore it. If I get back around to work on it before you do I will definitely take a look. Thanks!

[JakeWharton]

BOOM https://developer.android.com/reference/android/app/UiAutomation.html#takeScreenshot()

[RichardGuion]

New in 4.3 eh?

[matt-oakes]

Has there been any progress on the issue of not being able to see dialogs in screenshots?

[JakeWharton]

No. Free time is fleeting nowadays.

[holmes]

But we do accept pull requests.

[matt-oakes]

Thanks for the reply! No problem I understand. We're looking to get it working for a project at work so I'll see if I can get the time to fix this.

Looking at the code inside the takeScreenshot method of Robotium it seems at first glance like a more sophisticated version of what you use in Spoon. Would it be ok if we tried to port their stuff over into Spoon? There wouldn't be any dependency, I'd just move their code over to spoon.

[JakeWharton]

You mean the gross, grab-all-the-windows-with-reflection-and-render-them-in-reverse method? I suppose there's worse things... go for it.

[matt-oakes]

Pretty much. The goal for us to to have screenshots which show things like dialogs as well as the foreground activity. Ideally we'd just use the UIAutomation class but we'd like it to work on older devices too.

I'll take a look and see what I can do with it.

[RichardGuion]

Here is some code that I wrote. I have a higher level automation api which encapsulates both Spoon and Robotium. I should probably do a pull request.

Two methods for taking screenshots:

void takeScreenshot(String tag);
void takeScreenshot(String tag, boolean isDialog);

First one will call Spoon's method. Second flavor calls the ScreenshotTaker class below. You can see that I put the screenshots in the same folder as Spoon, and I use the same convention for naming the files. This way when the test is run, Spoon pulls the screenshots taken by both methods.

        final String imageName = System.currentTimeMillis() + "_" + tag;
        final String imageFolder = String.format("%s%s/%s/",
                ScreenshotTaker.SPOON_SCREEN_SHOTS_LOCATION,
                mClassName,
                mMethodName);

        runOnMainSync(new Runnable() {
            @Override
            public void run() {
                try {
                    ScreenshotTaker screenshotTaker = new ScreenshotTaker(imageFolder);
                    screenshotTaker.takeScreenShot(imageName);
                } catch (Exception e) {
                    logMsg("Error taking screenshot: ", e.getMessage());
                }
            }});

ScreenshotTaker class which has pretty much the same code as what Robotium uses...

import android.graphics.Bitmap;
import android.view.View;

import java.io.File;
import java.io.FileOutputStream;
import java.lang.reflect.Field;

import static com.evernote.test.framework.TestLogger.logTestError;

public class ScreenshotTaker {

private String screenShotFolderLocation;
private String windowManagerString;

public static final String SPOON_SCREEN_SHOTS_LOCATION = "/data/data/com.mycompany/app_spoon-screenshots/";


public ScreenshotTaker(String folderLocation){
    screenShotFolderLocation = folderLocation;
    setWindowManagerString();
}

public void takeScreenShot(String tag) throws Exception {
    takeScreenShot(getRecentDecorView(getWindowDecorViews()), tag);
}

public void takeScreenShot(View view, String name) throws Exception {

    if (view == null) {
        logTestError("takeScreenShot: view from getWindowDecorViews() is null.");
        return;
    }

    File sddir = new File(screenShotFolderLocation);
    if (!sddir.exists()) {
        logTestError("Spoon screenshot folder does not exist: %s", screenShotFolderLocation);
        return;
    }

    view.setDrawingCacheEnabled(true);
    view.buildDrawingCache();
    Bitmap b = view.getDrawingCache();
    File screenShotFile = new File(screenShotFolderLocation + name + ".png");

    FileOutputStream fos = new FileOutputStream(screenShotFile);
    if (fos != null) {
        b.compress(Bitmap.CompressFormat.PNG, 100, fos);
        screenShotFile.setReadable(true, false);

        fos.flush();
        fos.close();
    }
}


/**
 * Returns the WindorDecorViews shown on the screen.
 *
 * @return the WindorDecorViews shown on the screen
 */

public View[] getWindowDecorViews()
{

    Field viewsField;
    Field instanceField;
    try {
        viewsField = windowManager.getDeclaredField("mViews");
        instanceField = windowManager.getDeclaredField(windowManagerString);
        viewsField.setAccessible(true);
        instanceField.setAccessible(true);
        Object instance = instanceField.get(null);
        return (View[]) viewsField.get(instance);
    } catch (SecurityException e) {
        e.printStackTrace();
    } catch (NoSuchFieldException e) {
        e.printStackTrace();
    } catch (IllegalArgumentException e) {
        e.printStackTrace();
    } catch (IllegalAccessException e) {
        e.printStackTrace();
    }
    return null;
}

/**
 * Returns the most recent DecorView
 *
 * @param views the views to check
 * @return the most recent DecorView
 */

public final View getRecentDecorView(View[] views) {
    if (views == null) {
        logTestError("Error in getRecentDecorView: 0 views passed in.");
        return null;
    }

    final View[] decorViews = new View[views.length];
    int i = 0;
    View view;

    for (int j = 0; j < views.length; j++) {
        view = views[j];
        if (view != null && view.getClass().getName()
                .equals("com.android.internal.policy.impl.PhoneWindow$DecorView")) {
            decorViews[i] = view;
            i++;
        }
    }
    return getRecentContainer(decorViews);
}

/**
 * Returns the most recent view container
 *
 * @param views the views to check
 * @return the most recent view container
 */

private final View getRecentContainer(View[] views) {
    View container = null;
    long drawingTime = 0;
    View view;

    for(int i = 0; i < views.length; i++){
        view = views[i];
        if (view != null && view.isShown() && view.hasWindowFocus() && view.getDrawingTime() > drawingTime) {
            container = view;
            drawingTime = view.getDrawingTime();
        }
    }
    return container;
}


private static Class<?> windowManager;
static{
    try {
        String windowManagerClassName;
        if (android.os.Build.VERSION.SDK_INT >= 17) {
            windowManagerClassName = "android.view.WindowManagerGlobal";
        } else {
            windowManagerClassName = "android.view.WindowManagerImpl";
        }
        windowManager = Class.forName(windowManagerClassName);

    } catch (ClassNotFoundException e) {
        throw new RuntimeException(e);
    } catch (SecurityException e) {
        e.printStackTrace();
    }
}


/**
 * Sets the window manager string.
 */
private void setWindowManagerString(){

    if (android.os.Build.VERSION.SDK_INT >= 17) {
        windowManagerString = "sDefaultWindowManager";

    } else if(android.os.Build.VERSION.SDK_INT >= 13) {
        windowManagerString = "sWindowManager";

    } else {
        windowManagerString = "mWindowManager";
    }
}

}

[yogurtearl]

Any plans for "UiAutomation.html#takeScreenshot()" support ( with fallback for older API versions )?

Also, it would be great if there was a Spoon.logImage(String tag, Bitmap bitmap) which puts any bitmap in the Spoon report. This could be used for testing a standalone custom view and for snapshot Animations in progress.

[andaag]

Just tried quickly hacking in takeScreenshot support. Took about 5 minutes, but it turns out takeScreenshot occasionally (and for no reason I can predict..) returns null.

(We had a fun issue where our tests would fail due to a "please update your google play services version" dialog. All our screenshots looked perfect, but when we ran without headless mode we could spot the huge dialog over the entire screen :P )

[andaag]

#175

Better than the decorview on devices that supports it atleast.

[yogurtearl]

Caution, UiAutomation.html#takeScreenshot() is flaky and can sometimes cause issues with OpenGL memory allocations... We have had cases where it causes other apps to crash ( like the launcher ), presumably because it is triggering system wide issues with OpenGL memory allocations.

[Takhion]

UiAutomation.takeScreenshot() is so broken, for me every time it either returns null on the emulator or crashes the process on a real device...
I guess the only viable option is a screenshot service (like this one)!

[JakeWharton]

There's other means of taking a screenshot like MediaProjection. We tried those services with Spoon a year or two ago and they were pretty awful.

[LucioC]

We use the UiAutomation.takeScreenshot() method for our tests without problems. Not sure what you guys have different, but it never returned null or crashed the app for us. We use it in different real devices and we have something like 50+ test cases.

[rafaelaazevedo]

@RichardGuion Thank you for your help ! It's working for me 👍

[richardradics]

I just tried out this library, and it works like a charm with dialogs.

https://github.com/jraska/Falcon

[JakeWharton]

It just loops over all the windows like Robotium and smashes them together.
We can copy their implementation in or just link against the library at
worst.

On Sat, Mar 19, 2016, 8:56 AM Richard Radics [email protected]
wrote:

I just tried out this library, and it works like a charm with dialogs.

https://github.com/jraska/Falcon

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#4 (comment)

[sarmadali20]

I am using UiAutomation.takeScreenshot() without any issues.. with spoon 1.7.1 don't like the fact that I have to click on a link in the report....
#501 allows spoon to show any user defined png in the report... I think it allows for any implementation you like!

[anhduy5689h]

I use both Spoon library and Falcon library and it works well.
In Gradle:
androidTestImplementation 'com.squareup.spoon:spoon-client:1.7.1'
//Spoon is really great library, but it also does not include dialogs and other extra windows in its screenshots
//Falcon support capturing dialog: https://github.com/jraska/Falcon
androidTestImplementation 'com.jraska:falcon:2.1.1'
androidTestImplementation 'com.jraska:falcon-spoon-compat:2.1.1'

And use this code to capture dialog:
Falcon.takeScreenshot(activity, file)