Speeding up gcov collection from test device

[joefradley]

Some background context first. I’ve been looking into ways of speeding up the gather_on_test.sh script. Depending on the device I’m testing, the best case is 1.5 minutes, the worst is 11 minutes. This ultimately affects the granularity of collection. I’m generally forced to collect coverage from a group of tests instead of per test.

The primary slow point in gather_on_test.sh is the find + cat of each *.gcda into temp files:

find $GCDA -name '*.gcda' -exec sh -c 'cat < $0 > '$TEMPDIR'/$0' {} \; 

The FAQ states this is required because of issues of using the seq_file interface. And sure enough I’m able to reproduce the empty files when I attempt to place them straight into a tar file with a command such as:

find $GCDA -name '*.gcno' -o -name '*.gcda' | tar cfz $DEST -T -

I familiarized myself a bit with the seq_file interface and looked over the code in gcov/fs.c. I was hoping to see why tar doesn’t play well with seq_file. I thought perhaps if tar ended up using one of the defined file_operations (such as read or llseek) maybe we could use gcov specialized versions instead. But tar didn’t seem to be using either of those. And nothing else glaring jumped out at me.

I then moved to testing with a cp command instead of the find/cat approach, such as:

cp -rf $GCDA $TEMPDIR

This turned out to be WAY faster and no sign of empty files. 9 seconds versus 1.5 minutes. I did not find any issues with the collected files when compare to those collected the previous way.

Questions regarding going the cp approach:

• Even though I haven’t reproduced it yet, is the usage of cp from debugfs susceptible to creating empty files?
• The FAQ mentions both empty and incomplete files. Empty files are easy to detect, incomplete I’m not sure. Is there an easy way to detect incomplete files?

Questions regarding updating gcov to work with tar:

• Are the issues prohibiting the use of tar directly with seq_file documented somewhere? seq_file documentation is from 2003, the gcov FAQ on it is from 2009, maybe something new has been introduced to address it.

If there’s a known solution for it, I’d be happy to work on a patch for it.

[xaizek]

I think you've missed that pseudo-files backed by seq_file have zero size. tar simply believes result of stat.st_size and probably doesn't even open those files. It also does use read() through safe_read() wrapper from gnulib (an alternative is to use mmap(), but I think it won't work for seq_file).

cat works because it's dump. It can just call read() until EOF. Anything that does the same should do. You might have to check coreutils/busybox/etc. for anything more than that which could lead to incomplete files (re-reading parts of a file? not sure what can be the cause here, but maybe look at handling of sparse files).

[joefradley]

I think you've missed that pseudo-files backed by seq_file have zero size. tar simply believes result of stat.st_size and probably doesn't even open those files.

Thanks, I did not know that and that would explain it. I looked if by chance tar had an option to ignore that check and go straight to read() (or safe_read()) but it doesn't. The closest is the --sparse option which will look for empty spots in a file.

However, maybe a patch could be created for tar itself, something along the lines of "ignore the reported size, just read it". I just looked at the tar source for the first time, it doesn't seem out of the realm of possibilities. I have no idea how the contribution process works in comparison to Linux. Also, I don't have any idea how long it would take an updated tar to make it into Linux.

cat works because it's dump. It can just call read() until EOF.

Do you happen to know how cp works in this regard? Because from my anecdotal testing it appears to work the same as cat.

[xaizek]

Do you happen to know how cp works in this regard? Because from my anecdotal testing it appears to work the same as cat.

At least in the case of coreutils it's complecated, because of various optimizations. The implementation is in src/copy.c, look for lseek_copy() and sparse_copy().

• It first attempts to do a BTRFS clone
• Uses copy_file_range() instead of read() when that works
• I don't like lseek(, SEEK_HOLES), but maybe it has no effect on seq_file

On the positive side it doesn't seem to trust file's size and there is this comment:

            /* copy_file_range incorrectly returns 0 when reading from
               the proc file system on the Linux kernel through at
               least 5.6.19 (2020), so fall back on 'read' if the
               input file seems empty.  */

So developers might be aiming to support copying from pseudo file-systems.

[joefradley]

I'm selecting this as the answer, boiled down from the proceeding comments:
I feel better running with the cp command. I'll test further internally for a bit to confirm all still works. After which, I can send an upstream patch with the proposed gather_on_device.sh tweak.

Thank you for looking into copy and pointing me at the code. That time frame, 5.6.19 (2020), would explain how at the time of that gather_on_device.sh change stating to use find + cat, cp would not have worked if it was early'ing out on empty files (what tar still does).

(There's still room for speed improvement I'm sure, but this 90 second to 9 second, is a good increase for now.)