Clutch fills up disk space

[stas-smedia]

i have an automated process of using Clutch
i upload the files to the device and install with ipaintaller
decrypt it with clutch
the remove the app using ideviceinstaller
(and delete all the IPA files in the process, So /var/mobile/Documents/Dumped is empty)
the thing is after about 100 apps i ran out of disk space.
running Clutch --clean doesn't help.
after i reboot the device all the disk space is "returned" so i'm not sure what's causing this.
one idea i have is that some the apps fail the decrypt process so maybe there is an issue of cleaning up after failure.

[NinjaLikesCheez]

Clutch may be failing to clean /var/tmp/clutch correctly - having checked the source quickly it doesn't notify you if it fails to clean that directory correctly which it should. Try removing /var/tmp/clutch manually (you can do this whenever, /var/tmp/ is removed automatically upon reboot which is why this disk space is being returned to you)

[stas-smedia]

Hi, thanks for the answer,
but that doesn't seem to be the issue (the /var/tmp/clutch size is only about 100M) the "lost" space is more than 10GB.
what other directories are emptied on reboot?
maybe some system cache directories Clutch is filling up?

[stas-smedia]

here is some output to show the "weirdness" of the "lost" space
#~ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/disk0s1s1 2.9G 2.7G 80M 98% /
devfs 53K 53K 0 100% /dev
/dev/disk0s1s2 12G 4.9G 7.2G 41% /private/var

#~ du -sh /private/var
920M /private/var

[ttwj]

Clutch might accidentally leave modified binaries behind in the app folder, are you sure the app folders are being properly emptied? 

[Tatsh]

@stas-smedia Please do this as root and paste the output.

du -hsx /var/containers/* | sort -rh
du -hsx /var/mobile/Dumped/* | sort -rh
du -hsx /var/tmp/* | sort -rh

@ttwj Might I suggest keeping a list of files created and written to by Clutch and catching all signals to clean them up? (note you cannot listen on KILL/9 or STOP). Basically assume any signal other than normal exit (TERM) means error and therefore, clean up everything.

[stas-smedia]

~# du -hsx /var/containers | sort -rh
184M /var/containers

~# du -hsx /var/mobile/Documents/Dumped | sort -rh
0 /var/mobile/Documents/Dumped

i also looked at at apps directory like ttwj suggested , this is the output:
~# du -sh /var/containers/Bundle/Application
167M /var/containers/Bundle/Application

so it appeats to be ok...

while my disk space keeps decreasing... :/

~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/disk0s1s1 2.9G 2.7G 80M 98% /
devfs 53K 53K 0 100% /dev
/dev/disk0s1s2 12G 7.3G 4.7G 61% /private/var
~# du -sh /private/var
922M /private/var

[Tatsh]

What about (note the *):

du -hsx /var/containers/* | sort -rh
du -hsx /var/mobile/Containers/* | sort -rh

[stas-smedia]

~ root# du -hsx /var/containers/* | sort -rh
167M /var/containers/Bundle
17M /var/containers/Shared
1.2M /var/containers/Data

~ root# du -hsx /var/mobile/Containers/* | sort -rh
13M /var/mobile/Containers/Data
480K /var/mobile/Containers/Shared

[Tatsh]

Well then start narrowing it down. That command is usually what I use to figure out where space is going.

du -hsx /* | sort -rh

Once you see what is taking up the most space, which will be the first item, cd into it and run:

du -hsx * | sort -rh

Keep going until you find the worst offenders. The standard directories where your data gets stored (by apps and their data, and iOS) does not seem to be it. This may not be a Clutch issue.

[stas-smedia]

@Tatsh
that's what i've tried to do but it doesn't even make sense the output of df
since
~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/disk0s1s1 2.9G 2.7G 80M 98% /
devfs 53K 53K 0 100% /dev
/dev/disk0s1s2 12G 7.3G 4.7G 61% /private/var

so /private/var should use atleast 7.3GB but when i run du on it
~# du -sh /private/var
932M /private/var

so i thinking it has to be some system file that du ignore or something.

when i search manually this is what i've found:
/System/Library/Caches/com.apple.dyld is about 1.1 GB
(but this seems to normal)
/private/var/MobileAsset/Assets/com_apple_MobileAsset_VoiceServicesVocalizerVoice is about 170MB
(which is also ok)

so i have no idea what could be causing this

[Tatsh]

You have to look at what is taking up space on /. Everything else looks fine. This is a 16 GB iOS device I am presuming and it is almost inevitable you are going to run out of space sooner than if you had a larger size device.

You do have stashing for tweaks set up?

For me on a 128 GB iPhone:

# df -B 1 -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/disk0s1s1  4.4G  3.4G  922M  79% /
devfs            50K   50K     0 100% /dev
/dev/disk0s1s2  114G   41G   74G  36% /private/var
/dev/disk0s1s3   10M  2.3M  7.8M  23% /private/var/wireless/baseband_data
/dev/disk3      224M   64M  161M  29% /Developer

# cd /
# du -bhsx * | sort -rh
3.0G    System
478M    usr
358M    Applications
341M    Library
48M     Developer
5.7M    bin
1.9M    sbin
999K    private
1.3K    dev
68      mnt
68      lib
68      cores
68      boot
15      tmp
11      var
11      etc
11      User

# cd /private/var
# du -bhsx * | sort -rh
20G     containers
18G     mobile
1022M   MobileAsset
306M    root
220M    db
129M    vm
104M    cache
83M     lib
45M     tmp
8.2M    Keychains
2.5M    networkd
1.7M    wireless
1.6M    installd
787K    logs
...

Unless it turns out /var/tmp/clutch is taking up a lot of space (which you can just delete it, or run clutch --clean), this is not a Clutch issue.

[ttwj]

Did you switch between using iCloud or iTunes to sync your device? That affected the available space for me a while ago

[stas-smedia]

@ttwj
No, the device is pretty empty and i haven't setup iCloud or sync with iTunes.

right now i thinking that maybe ideviceinstaller is doing some "funny" stuff when uninstalling the apps, but i'm not sure.

[stas-smedia]

@ttwj @Tatsh
Thanks, i think you are right and it's not Clutch's fault.
i switched to mobiledevice instead of ideviceinstaller and it seems to fill up slower.
so the problem is in the uninstall process, guess i'll have to write some code to delete the apps on the device itself.

[ttwj]

Phew. I can get Clutch to output the location of the app you're cracking. We used to have that feature in Clutch 1 to facilitate scripting. 

[stas-smedia]

That would be great :)

[ttwj]

Ok I'll push an update soon :) Any other variables you'd like to grab?

[stas-smedia]

Thanks! nothing else i can think of right now

[stas-smedia]

@ttwj
I've rechecked the issue multiple times (switched to using installipa to both install and uninstall)
and there is some "wasted" space (around 200M) because of ipainstaller but this happens also with iTunes and that space is not increasing over time, i've tested the process with and without Clutch multiple and has to be Cluch that is "wasting" the disk space.

[ofirkris]

Same issue here, devices get chocked with virtual Data, when I use Clutch on many apps.

[stas-smedia]

one idea i have on what could be causing this:
if i run
ps aux
i see the app running after already uninstalled it

[ttwj]

Hmm looks like a zombie process, they can only be removed by forcibly rebooting the device. 

[stas-smedia]

@ttwj
i've tried to do
killall -u mobile
to respring before uninstalling the app but it doesn't help the disk space is still "wasted"

[mikeburragejr]

I see the same on IOS 9.3.3 (but not 9.0 or 9.1) with clutch 2.0.2 or 2.0.4, with "(appName)" records with process state 'U' sticking around in 'ps aux' output, eventually making it impossible to start any processes and requiring phone hard reset (assuming from 'top' output it's because of lack of available memory). Respring, kill/killall does nothing.

Site note for others (in the mean time) that dumpdecrypted using steps in stefanesser/dumpdecrypted#19 doesn't have the same problem.

[ttwj]

Looks like the new Pangu jailbreak messed up with the zombie process killing mechanism. You can remove them by performing a reboot 

[ofirkris]

Any update on this issue? is this fixable at all?

[NinjaLikesCheez]

I'm looking into it, no update other than what mikeburragejr has said really. This might have something to do with how we open the binaries to read and write from the disk that's causing the process to lock up into the uninterruptible state - but that's just a theory. I'll update the ticket with more once I've had time to look into it further.

[ofirkris]

@NinjaLikesCheez Thx!

[Songchunbo]

Same issue. Any update on this? After decryption, two zombie processes produced. They hold the handle of decrypted ipa files even if the files are deleted, and the space won't be released.

de-iPad:/private/var/mobile/Documents root# ps -ef|grep 741 |grep -v grep
0 741 1 0 4:26PM ttys000 0:00.00 (Chaabi2016)
de-iPad:/private/var/mobile/Documents root# ps -ef|grep 740 |grep -v grep
0 740 1 0 4:26PM ttys000 0:00.00 (Chaabi2016_armv7)

[Tatsh]

Clutch might stay running in 2.0.4 because it was using CFRunLoopRun() which means infinite loop in some odd cases as the run loop queue fails to empty (probably due to lack of space? or lack of memory?). You need to try the latest version (compile it yourself for now).

[Tatsh]

Need updates from people: @ofirkris @Songchunbo @mikeburragejr . Please report if this is still happening with the latest version on master (build it please) and report what device, jailbreak and iOS version you have.

[mikeburragejr]

Using the latest clutch code - the issue still exists (no change) on pangu 9.3.3 jailbroken devices (iphone6/6s). Sorry for the very delayed response here.

[Tatsh]

@mikeburragejr Is this about zombie processes or disk space? The original ticket was about disk space, then seemed to get to become about zombie processes. If it is about zombie processes, please make a new ticket and I will close this one.

[Tatsh]

Closing. 2+ months since last activity.