Support local Anisette generation

[malmeloo]

Support local anisette header generation using pyprovision.

[biemster]

I doubt this can be included in the pypi package, since building pyprovision requires a D compiler which is not really a default install usually.
This might be possible however (on x86-64) if this could run with cosmopolitan: Smoothstep/apple-gen#1 (and if they finally add anisette support)

[malmeloo]

I have don't have much experience with lower-lever languages, but I was actually messing with this yesterday and it's pretty simple to build a python wheel that includes libprovision.so. The downside is that it still relies on externally installed libraries; I did briefly try to "repair" the wheel using https://github.com/pypa/auditwheel, but it wasn't able to find all external dependencies. That would still require a separate wheel for each architecture / OS / python version though, and would probably be tricky in terms of licensing, to say the least.

That project looks quite interesting though, worth keeping an eye at!

[JayFoxRox]

I've just made https://github.com/JayFoxRox/pyprovision-uc public.
I'm not sure when I'll get around to finalizing and packaging it up. For now, you can probably install it via pip git install:

pip3 install --user -U git+https://github.com/JayFoxRox/pyprovision-uc.git#egg=pyprovision-uc

I made this when I wasn't able to make pyprovision / D compiler work on my setup within a couple of hours.

It's still unfinished and still needs to be cleaned up.
But I've been using a version of this (similar to the one I made public) for a couple of weeks by now.

I had a lot of problems with the stat syscall emulation, so the code is particularly unclean there.

There's also good chances that there's issues with endianess or 32-bit (because I'm using host ctypes for ARM guest code).
I'm running this on macOS on M1 (so far) and most things appear to be right.
I'll move my local installation to a raspberry pi 4 soon (and plan to test armv7 soon and aarch64 in a couple of weeks) - I'll fix up any issues I'll find.

There's also a chance that the VM will become unstable over time.
The emulated memory layout shows that I had to workaround some of the worst issues.
So what I have working now is the bare minimum.
I usually spawn a new Python process for every request, so improving it much further is not of interest to me.

I also plan to modify the API a bit in the future - I don't like how it operates on a real filesystem.

Also CC @Dadoum and @biemster

[malmeloo]

Oh that's really cool! I just tried it out myself and while I did have to fix some things (create the directories, some debugPrint issues), it runs perfectly otherwise. Most of the code looks like black magic to me, but if you need help with anything else (refactoring?) let me know.

[hkfuertes]

I was able to make dadoum (D written) work in my dockerfile:

FROM python:3.11-slim
RUN pip install --upgrade pip

ENV DEBIAN_FRONTEND=noninteractive
RUN apt update && apt install dub git gcc -y

RUN git clone https://github.com/Dadoum/pyprovision /tmp/pyprovision
RUN cd /tmp/pyprovision && pip install .
...

And I managed (I believe) to write the LocalAnisetteProvider:

class LocalAnisetteProvider(BaseAnisetteProvider):
    """Anisette provider. Generates headers without a remote server using pyprovision."""
    
    def __init__(self, libary_path, provisioning_path, device_json_path):
        self.adi = ADI(libary_path)
        self.adi.provisioning_path = provisioning_path
        self.device = Device(device_json_path)
        if not self.device.initialized:
            # Pretend to be a MacBook Pro
            self.device.server_friendly_description = "<MacBookPro13,2> <macOS;13.1;22C65> <com.apple.AuthKit/1 (com.apple.dt.Xcode/3594.4.19)>"
            self.device.unique_device_identifier = str(uuid.uuid4()).upper()
            self.device.adi_identifier = secrets.token_hex(8).lower()
            self.device.local_user_uuid = secrets.token_hex(32).upper()
        self.adi.identifier = self.device.adi_identifier
        self.dsid = c_ulonglong(-2).value
        self.provisioning_session = ProvisioningSession(self.adi, self.device)
        self.provisioning_session.provision(self.dsid)
        self.otpObj = self.adi.request_otp(self.dsid)
        

    @property
    @override
    def otp(self) -> str:
        return self.otpObj.one_time_password

    @property
    @override
    def machine(self) -> str:
        return self.otpObj.machine_identifier

    @override
    async def close(self) -> None:
        """See `BaseAnisetteProvider.close`_."""

Based of @Dadoum's example... but I get this error:

email?  >****
passwd? > ****
Traceback (most recent call last):
  File "/app/_login.py", line 77, in get_account_sync
    with acc_store.open() as f:
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/pathlib.py", line 1044, in open
    return io.open(self, mode, buffering, encoding, errors, newline)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'account.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/do_login.py", line 13, in <module>
    acc = get_account_sync(anisette)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/_login.py", line 80, in get_account_sync
    _login_sync(acc)
  File "/app/_login.py", line 20, in _login_sync
    state = account.login(email, password)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/findmy/reports/account.py", line 960, in login
    return self._evt_loop.run_until_complete(coro)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/findmy/reports/account.py", line 468, in login
    new_state = await self._gsa_authenticate(username, password)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/findmy/reports/account.py", line 731, in _gsa_authenticate
    raise InvalidCredentialsError(msg)
findmy.errors.InvalidCredentialsError: Password authentication failed: This action could not be completed due to possible environment mismatch.
Exception ignored in: <function Closable.__del__ at 0x7aeb37d64400>
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/findmy/util/closable.py", line 31, in __del__
AttributeError: 'LocalAnisetteProvider' object has no attribute '_loop'
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7aeb3794fc50>
ERROR:asyncio:Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7aeb37bb73f0>, 1307.841477031)]']
connector: <aiohttp.connector.TCPConnector object at 0x7aeb3794ff50>

findmy.errors.InvalidCredentialsError: Password authentication failed: This action could not be completed due to possible environment mismatch.

Any clue? I know that this is not even implemented, but maybe knows what this error indicates. Also, If what I did can help you, I can try and open a PR...

[biemster]

@hkfuertes you might want to change your password, people subscribed to this got it in their email.

[hkfuertes]

Thank you! It was a mistake... I didn't thought of email subscription...

Edit: Changed!, @biemster thank you for the heads up!

[hkfuertes]

I think I'm able to answer myself:

FindMy.py/findmy/reports/anisette.py

Line 71 in d82b1fb

def client(self) -> str:

😆
I'll keep working... I'm trying to make a flask API compatible with OpenHaystack Mobile app, to be able to track my airtags from android...

[malmeloo]

Oh yes indeed, sorry for taking so long to respond. But glad you figured it out yourself!

The issue here is not necessarily getting local anisette to work, but rather packaging it so that people don't need a local D compiler. Unless your solution works outside of Docker as well?

Edit: you could also use anisette-v3-server in a separate docker container if you're deploying FindMy.py in docker anyway; that way you can simply use RemoteAnisetteProvider with the container's ip address.

Also also, if you have a functional API implementation, let me know because I'm also interested ;-)

[hkfuertes]

Ohhh I'm way behind you guys... hahahaha I just now how to follow steps and connect dots, I dont really understand Anisette or Findmy... hahahahah.

The only reason I tried the local anisette is just to try... you see, I tried remote anisete with the anisette server in the same docker-compose, and it worked fine... but only after first login. Once I restart the server, for some reason, I needed to re-login. I was hoping that local anisette would solve this issue...

[malmeloo]

Hey @JayFoxRox, I'd finally like to tackle this issue and I think your implementation is currently looking the most promising in terms of flexibility and ease of integration. My plan is to release a separate python package with a nice API to act as an anisette provider. Just checking in to see if that'd be OK with you as 80-90% of it will probably be consisting of your code (credit will be given, of course!)

[JayFoxRox]

Sure, do whatever you want with it 👍

Consider it MIT licensed for now, but I'll happily add another license to suit your needs.
If you need any assistance or have questions about the code you can also contact me (http://www.jannikvogel.de/).
I'm a bit busy with life right now, so I might take a while to respond - also it's been a while so I'd have to read up on my own code again.

[malmeloo]

Perfect, thanks! MIT is fine, I don't really care too much about the exact license specifics. Just want the package to be out there so it can be used :-)

[douniwan5788]

Hi @JayFoxRox , I noticed that hook_free = hook_emptyStub. Won't this cause a memory leak if it runs for a long time?

[JayFoxRox]

It uses an "Allocator"-class which doesn't even have the option to free pages: https://github.com/JayFoxRox/pyprovision-uc/blob/70254670674c7e54d6e4a07483248ed124aa016f/src/pyprovision/__init__.py#L168

Note that this is a VM which runs Apples Android code (specifically a very small portion of it).
In general the allocator does not matter because we can always kill the VM and start a new one, so I just went the lazy route.

However, I don't think restarting the VM would work yet, because https://github.com/JayFoxRox/pyprovision-uc/blob/70254670674c7e54d6e4a07483248ed124aa016f/src/pyprovision/__init__.py#L737 would only instantiate it once (so the malloc area would be shared by all VMs and it would be impossible to reset it).
This allocator probably has to be moved into the VM, too.

But yes, if you keep calling functions in the VM there'd be problems - it hasn't been designed to run more than a handful of calls. The allocator actually uses a fixed amount of space: https://github.com/JayFoxRox/pyprovision-uc/blob/70254670674c7e54d6e4a07483248ed124aa016f/src/pyprovision/__init__.py#L144 . So instead of leaking memory, you'd just run out of memory (if the functions you call in the VM actually do malloc on each call).

I'm also not sure if it's enough to lose the reference to the ADI to actually free the unicorn-engine machine, so there might be an actual memory leak there (leaking the the unicorn-engine memory).

[malmeloo]

Ta-daaa: https://github.com/malmeloo/Anisette.py

It works, using basically the same code as @JayFoxRox wrote, but in a nice API. There are still a couple of things to fix and finish, but it appears to work perfectly so far.

My intention is to also add a small command-line interface to this tool to quickly create and destroy provisioning sessions and save them to disk. Like anisette-v3-server, but without the server, and mostly suited for developers. And I think it would be nice to be able to quickly spin up a server as well. Also like anisette-v3-server, but without Docker :-). It could even work with pipx!

None of the stability issues mentioned above have been fixed yet, except for the dangling malloc allocator. I'm curious to see how long a single instance of the VM will survive until it comes crashing down.

[malmeloo]

A quick progress update: I managed to implement free() operations in the allocator, and after manually freeing some supposed-to-be-temporary allocations here and there, most of the memory leaks have been fixed. The only source that remains appears to be caused by Apple's library refusing to free some of the memory it malloc'd. This results in roughly 4-5 kilobytes of leaked memory per call to get Anisette headers. With the current size of the memory area dedicated to malloc, that means that the allocator will be "full" after roughly 3200 requests for headers. I'm not sure to what extent that is really fixable without applying some sort of heuristics to clean up after the library.

However, I can confirm that unicorn appears to release all system memory when the VM is garbage collected by Python. So I've configured my wrapper to simply destroy and recreate the VM once any memory region has been > 50% allocated, which might be a band-aid fix but is totally fine IMO considering how rarely it occurs. And we can always adjust the size of the memory region to tweak that frequency.

I've been benchmarking it and am able to hit ~40 header requests per second on my laptop, with the entire process consuming roughly 75 MB of system memory. I think that's very respectable, so my next focus will probably be on actually publishing the library so it can be integrated into FindMy.py.

I've also set up a Cloudflare worker to expose a URL that creates a bundle consisting of only the necessary libraries, which should make it very easy to get started. That bundle is only a few megabytes, which is tiny compared to the ~130 MB of the full Apple Music APK.