Android 86.1.1 client will not connect to custom sync server over insecure connection (http://)

[majurgens]

I have a custom sync server running and working just fine with Firefox v86 Windows 10 desktop clients and Firefox v68 Android clients.
I do the following on Android:

• Install Firefox
• Enable debugging mode
• Specify a custom sync server
• Sign in via Firefox account
• Press Sync Now

I have a tcpdump running on the sync server machine and I can see all the traffic from the desktop clients. Yet, when I press "sync Now" on Android I do not even get a single packet hitting the sync server. Its like the Android client does not even try to initiate a connection at all, and hence I cannot sync from Android.

It does not matter if the server even exists, is running etc. The client will not even send one network packet to the sync server hosts. Seems like a 100% sync client issue.

I have tried all the following URLs, all with the same result ie zero network packets

• http://server/token
• http://server/token/1.0/sysnc/1.5
• http://server/token/1.0

[rfk]

If possible, please try using https:// URL, I believe there are some checks in the client code that refuse by default to connect to insecure servers.

[majurgens]

Sorry, it did also try that. I have tried it again also - but the same result - zero network packets towards the syncserver
So adding the following attempt:

• https://server/token/1.0/sync/1.5

So just on this, if there are such checks (allow https only) then that should be able to be worked around to support development environments etc. It is sometimes a valid use case to not use https and the risk can be acceptable.

[Atera]

Same here. I am using HTTPS with a self‐signed certificate (installed on phone). Latest version makes no effort to connect, and instantly stops when trying to initiate manually.

[zilexa]

Please try:
https://server/token/1.0/sync/1.5/
That slash at the end is how it worked for me. It was driving me crazy too. I believe I posted this earlier in an issue.
Like this, with proper https (I use caddy-docker-proxy + a cheap own domain) this works on Android 11, Pixel 4a and a Samsung S20.

[majurgens]

I added / to the end of the URL but still zero network packets seen from a tcpdump running on the sync server. I have now tried the following URLs with the same result of zero network packets:

• http://server/token
• http://server/token/1.0
• http://server/token/1.0/sysnc/1.5
• https://server/token/1.0/sync/1.5
• http://server/token/1.0/sysnc/1.5/
• https://server/token/1.0/sync/1.5/

[zilexa]

Are you sure you added the URL to the sync area, not the Account part? One of the other mistakes I made and didn't notice for days..
This is how it works for me:

[majurgens]

Thanks for the screenshot. Yes, this matches what I have tried as per my list of URLs

[rfk]

(Sorry lack of replies here, I don't have a lot of time to look into self-hosting stuff these days).

I think we'll probably need debug output from the Android device to figure out what's going on here. Any chance you can try connecting adb to the phone and obtaining client-side logs? There are some instructions for how to do this on Fenix here:

• https://github.com/mozilla-mobile/fenix/wiki/Logging-Crash-Information#using-adb-logcat-to-get-crash-information

(They say they're for getting debug info about crashes, but if you mentally replace "Reproduce the crash" with "Sync now" it should help produce debug logs for the sync process).

[majurgens]

I assume that "submit the crash report" is translated to collect the stdout from the adb output.

That output is available here:
https://www.edcint.co.nz/tmp/expire_14/info.txt (please let me know when downloaded as this will expire in 14 days)

What I did:
Start adb command line at Tue 16 Mar 2021 14:39:46 AEDT
Started Firefox
Went to Settings->Account
Pressed "Sync now" several times
End adb command line at Tue 16 Mar 2021 14:40:07 AEDT

[rfk]

(please let me know when downloaded as this will expire in 14 days)

Thanks, I've grabbed a copy to investigate further 👍

[rfk]

Unfortunately the logs aren't giving me any additional clues, but some notes from my first look at them:

We see several instances of Successfully executed: getTokenServerEndpointURL, which correspond to FxA setting up the sync auth info here. Notably, we do not see any of the warnings that we'd expect to see if this step failed, so I infer that it's working correctly.

But then...nothing. No logs from the sync system at all, except for a single line from syncmanager telling us it's starting up:

I/BgSyncManager(16650): Periodic syncing enabled: PeriodicSyncConfig(periodMinutes=240, initialDelayMinutes=5)

The next log line I would expect to see is here when the syncmanager kicks off a sync. I don't see any of those, and I don't know why - it's logged at debug level but we see plenty of other debug-level log entries in the file, so that doesn't seem like it should be a problem.

So...I can't quite explain what's happening here :-(

I'll see if I can get @grigoryk to take a look at the logs for additional insight.

[majurgens]

Successfully executed: getTokenServerEndpointURL

I think this might be telling lies or it is telling only a partial truth. Whilst the timing of those log entries looks about right, a packet capture running on the sync server sees zero packets on port 5000 from the test phone. The same packet capture sees network packets on port 5000 from other firefox clients (Windows, Android Firefox v68). I can't see any reason why I wouldn't see the packets on my network

[grigoryk]

At a first glance, logs are consistent with us not entering this block: https://github.com/mozilla-mobile/android-components/blob/4982e9ba1b1a72a149f4137ebaa828ba19ebf214/components/service/firefox-accounts/src/main/java/mozilla/components/service/fxa/manager/FxaAccountManager.kt#L302-L305

... which maybe could happen if the lock we're using for synchronized is never released? this in that code is actually pointing to an instance of CoroutineScope, not the account manager... I'm not sure what that'll cause (who else can hold a lock on that scope object..? for how long?).

Furthermore, it seems like we're never actually hitting syncManager.start call to initialize the dispatcher. See https://github.com/mozilla-mobile/android-components/blob/4982e9ba1b1a72a149f4137ebaa828ba19ebf214/components/service/firefox-accounts/src/main/java/mozilla/components/service/fxa/manager/FxaAccountManager.kt#L904-L910

A null dispatcher could also be consistent with these logs. (it's initialized as part of that start method - https://github.com/mozilla-mobile/android-components/blob/4982e9ba1b1a72a149f4137ebaa828ba19ebf214/components/service/firefox-accounts/src/main/java/mozilla/components/service/fxa/sync/SyncManager.kt#L152-L154)

For this to not be hit, either we're not notifying account state observers when we should as we're going through the state machine, or that AccountsToSyncIntegration instance isn't registered as an observer.

In the logs, it does seems like we're taking the correct path through the state machine, and should be calling the observers at
https://github.com/mozilla-mobile/android-components/blob/4982e9ba1b1a72a149f4137ebaa828ba19ebf214/components/service/firefox-accounts/src/main/java/mozilla/components/service/fxa/manager/FxaAccountManager.kt#L535 - but that doesn't seem to have an effect. getProfile is mentioned in the logs at just the right moment, as expected.

I'll keep looking tomorrow.

[grigoryk]

@majurgens I've tried out the custom sync server setup locally, using our production instance as the custom server (it worked fine!). Would it be possible for you to try that, and see if that works? This is the url: https://token.services.mozilla.com (as discoverable via https://accounts.firefox.com/.well-known/fxa-client-configuration)

If that works, it's likely that the problem is connected to your local setup. Otherwise there's something off about our custom server handling logic (I just don't see what, yet).

Also, the logcat you've supplied before lacked most of the Debug level logging. I'm assuming you didn't filter out these log statements yourself, right?

[majurgens]

Ok, so now it gets interesting.
I set the URL as you suggested (good idea) and can confirm that Firefox client syncs (well, it says it does and I can see network packets). Certainly different to my specified problem.

So now I am looking at what is different between all the URLs I have tried vs that one.

I checked local name resolution and network access by pasting http://server/token/1.0/sysnc/1.5 into the firefox search/URL bar as though this was a website I wanted to visit. I got a JSON response effectively telling me that the request was unauthorized.

So if Firefox can resolve the local name and access it then the other differences I see between the URLs are:

• specification of a port number
• use of a URI rather than just a hostname

I tried a few more things:

1. used http://server as the custom sync server URL
   Result: zero packets
2. used https://server as the custom sync server URL
   Result: some packets hit the sync server when sync is first configured. Will not send any more packets when trying to manually sync, even after reboot.

So it does appear that it fails under the following conditions:

• the use of a URI
• the use of HTTP (instead of HTTPS)

At this point I will just restate that, if there are check that allow https only then that should be able to be worked around to support development environments etc. It is sometimes a valid use case to not use https and the risk can be acceptable. I'd suggest that this limitation should be removed.

[rfk]

pasting http://server/token/1.0/sysnc/1.5 into the firefox search/URL bar as though this was
a website I wanted to visit. I got a JSON response effectively telling me that the request was unauthorized.

I think you correctly understood this, but commenting explicitly for completeness: getting an "unauthorized" response here is good and expected. You should be able to visit http://server/ or http://server/token/ to see a human-readable success message rather than an authorization error.

used https://server as the custom sync server URL
Result: some packets hit the sync server when sync is first configured. Will not send any more packets
when trying to manually sync, even after reboot.

You mentioned using a self-signed certificate, so I wonder if the initial packets are from trying to connect at the TLS level, but then we don't see any more packets because the certificate is rejected. If you browse to the https:// version of your server (e.g. https://server/token) in the urlbar, does it succeed?

[rfk]

At this point I will just restate that, if there are check that allow https only then that should be able to be worked around to
support development environments etc. It is sometimes a valid use case to not use https and the risk can be acceptable.

We do indeed have such a check, and I don't see any way currently that it could be disabled:

• https://github.com/mozilla/application-services/blob/b0170c6f23099f1fcb067b7985c11e93cbe5094e/components/viaduct/src/backend.rs#L46-L54

We should at the very least add some better log output for when this happens. I'm not opposed to letting the user override this, but we'd have to figure out a lot of plumbing to expose that all the way through to Fenix UI.

[majurgens]

Also, the logcat you've supplied before lacked most of the Debug level logging. I'm assuming you didn't filter out these log statements yourself, right?

I confirm that I did not filter anything out

[majurgens]

You mentioned using a self-signed certificate, so I wonder if the initial packets are from trying to connect at the TLS level, but then we don't see any more packets because the certificate is rejected. If you browse to the https:// version of your server (e.g. https://server/token) in the urlbar, does it succeed?

It was not me that mentioned using a self-signed certificate. However , I tried going to https://server/token/1.0/sync/1.5 and as expected, network packets hit the server and it could not establish a connection

[rfk]

Thanks for checking. So IIUC, there are two issues at play here:

• We don't currently support accessing the sync server over http://; fixing this would require some non-trivial work to plumb that through as an option all the way into the Fenix UI.
• There is some connectivity issue when accessing your server over https://, which happens even when browsing directly to the site; we're probably not the right folks to help debug that further unfortunately.

Does that sound right? (I don't want to try to propose any next steps if I've misunderstood where things are currently at).

[majurgens]

* There is some connectivity issue when accessing your server over `https://`, which happens even when browsing directly to the site; we're probably not the right folks to help debug that further unfortunately.

This is not the correct understanding. That test showed that when Firefox was asked to access https://server/token/1.0/sync/1.5 that it can (I see network packets reach the target server) EXCEPT if Firefox is asked to reach this URL as a custom sync server URL in which case Firefox does not actually even attempt to send any network packets to the target.

Let me restate:
Case 1) Use https://server/token/1.0/sync/1.5 in Firefox search/URL bar

• Result: Firefox sends network packets to the target server
• All this really proves is that network resolution and transport works ok.
• We all know that Firefox works with HTTPS URLs - that is not in question

Case 2) Use https://server/token/1.0/sync/1.5 as a custom sync server URL

• Result: Firefox does not even send network packets to the target server
• We know that the network is ok and can support the packets so the problem seems to lie with Firefox not even trying

[rfk]

Case 1) Use https://server/token/1.0/sync/1.5 in Firefox search/URL bar

But does it correctly load the resulting webpage? (Which as you noted earlier, should be a numeric JSON response with a 401 status code). I took your earlier message "network packets hit the server and it could not establish a connection" to mean that the page failed to load properly, but perhaps I misunderstood.

Case 2) Use https://server/token/1.0/sync/1.5 as a custom sync server URL
Result: Firefox does not even send network packets to the target server

Earlier you said "some packets hit the sync server when sync is first configured. Will not send any more packets when trying to manually sync, even after reboot" for this scenario, which might indicate that it's somehow caching a lower-level connection failure rather than trying afresh on each sync.

I'm not trying to nitpick, I just want to be sure that I fully understand the described symptoms.

[majurgens]

But does it correctly load the resulting webpage?

There is nothing to load, there is nothing listening on the server. That is why I have stopped at saying "network packets hit the server"

Earlier you said "some packets hit the sync server when sync is first configured. Will not send any more packets when trying to manually sync, even after reboot"

That was referring to using https://server as the custom sync server URL

Let me restate, adding Case 3:
Case 1) Use https://server/token/1.0/sync/1.5 in Firefox search/URL bar - Firefox sends packets. Assumed to be ok as this is fundamental Firefox functionality
Case 2) Use https://server/token/1.0/sync/1.5 as a custom sync server URL - no packets received at server
Case 3) Use https://server as a custom sync server URL - some packets received at server when sync first setup. Manual syncs seem to send no packets.

[rfk]

Thanks, that helps clarify.

Case 2) Use https://server/token/1.0/sync/1.5 as a custom sync server URL - no packets received at server

It's possible that this case was hitting the issue mentioned in earlier comments in the thread, where URLs without a trailing / prevent things from working correctly. That bug should be fixed in the latest versions of Firefox for Android but perhaps not yet in the release version. Adding a trailing slash here might make it behave like Case (3).

But of course Case (3) won't help you if the server doesn't actually serve over https://.

So it sounds like the remaining issue here is the lack of support for http:// URLs. I'm going to edit the title of the bug to reflect this but please correct me if I've misunderstood anything else here.

Unfortunately I don't think we're likely to prioritize the work of supporting http:// URLs anytime soon. If anyone felt motivated to try to add this as a contributor though, I would be happy to help offer some pointers on the moving parts involved.

[majurgens]

That bug should be fixed in the latest versions of Firefox for Android but perhaps not yet in the release version

Do you know what version that is and where I can get it so I can try it?

[rfk]

Either of "Firefox Nightly" or "Firefox Beta" in the play store should have the fix, I'm not sure exactly what version number those correspond to off the top of my head.

[majurgens]

I've just install Nightly and
Case 2) Use https://server/token/1.0/sync/1.5 - now sends packets to the server (no trailing slash).

Let me configure a TLS server to frontend the sync service and confirm the end-to-end . Is there any good docs for that?
https://mozilla-services.readthedocs.io/en/latest/howtos/run-sync-1.5.html is not that great

[majurgens]

Note that the documentation at https://github.com/mozilla-services/syncserver talks about using HTTP://

[Atera]

You mentioned using a self-signed certificate, so I wonder if the initial packets are from trying to connect at the TLS level, but then we don't see any more packets because the certificate is rejected. If you browse to the https:// version of your server (e.g. https://server/token) in the urlbar, does it succeed?

The person trying to use https with custom certificates with me. I actually checked and it seems my previous attempt to access the address only worked because of an exception, not because it was properly detected in Firefox, so this could very well be a relevant problem. I know this is a bit of a tangent, but what is the best way to ensure Firefox on Android correctly reads custom certificates (ergo, what is the procedure to add them to its store)? I've added the certificates in question to the Android store, and have followed instructions for older versions of Firefox for Android that involve putting the .crt files on a website and serving them with MIME application/x-x509-ca-cert, but that no longer seems to work (this just downloads the crt files). Could you point me in the right direction? If so, that may solve my syncing issue, though I'd have to test again to be sure.

[zilexa]

Why don't you simply use Sync as it is attended and obtain proper https certificate from an authority like letsencrypt for your domain, your reverse proxy can take care of that easily. If you use Docker, the setup can be extremely simple with just 2 containers, one for Caddy and one for Firefox-Sync. Nothing else needed except for your own properly redirected domain:
https://github.com/zilexa/Homeserver/blob/5bbb45d282aa2ffd8f97023d999dcaa9224bc2b6/docker/docker-compose.yml#L88

Notice how the Firefox container only needs 3 labels and that's really it.. zero other config is needed, that includes caddy, what you see there is everything.

The time you spend on documenting this issue seems like a waste when a properly configured server works just fine with Firefox Sync..

[virgoparna]

Letsencrypt is no use, if sync server is internal network only accessible from Internet via VPN.

[virgoparna]

And after configuring server to ssl (own ca, ca certificate installed on android) neither old nor new versions of Firefox sync. Old version shows no connection in server log, new Firefox reports 401 error (is at least able to connect server).

[virgoparna]

Firefox 68.11.0 - worked with http, certificate seems to work in browser (I do not remember, if I needed to accept it in Firefox), sync shows not connections in apache log (security.enterprise.roots.enabled = true).
Firefox 87.0.0-beta.6 - can open web page, attempt to sync results in 401 error in apache log on GET https://SERVER/PATH/sync/token/1.0/sync/1.5 (security.enterprise.roots.enabled = true).

Firefox 86.1.1 - ignores android installed ca certificate, even after manually opening page on server and allowing certificate refuses to sync. After every restart needs to accept certificate opening page again. No about:config, cannot check security.enterprise.roots.enabled value.

Desktop Firefox works with https, ca certificate was installed to firefox.

[rfk]

what is the best way to ensure Firefox on Android correctly reads custom certificates

I'll ask around and see what I can find out.

[rfk]

No about:config, cannot check security.enterprise.roots.enabled value.

From folks on the Fenix team: you do still need to check security.enterprise.roots.enabled in order to use the Android cert store, but in order to do that you need to be running Firefox Beta or Firefox Nightly, because about:config is disabled in Firefox Release.

[rfk]

There is also a tracking bug for letting you enable this on release here: https://bugzilla.mozilla.org/show_bug.cgi?id=1678191

[virgoparna]

I suspect, that current issue with newest beta versions is #263. Symptoms are the same. Only thing remaining is: why Firefox 68.11.0 is unable to connect to https sync server...

[virgoparna]

After creating new install of sync server Firefox 87.0.0-beta.6 works (with enterprise roots enabled) with ssl. issue was probably old syncserver version (last time I tried update, it was complete failure and sync server became unworkable (I needed to restore from backup to working state).
Latest stable needs security.enterprise.roots.enabled support.
And Firefox 68.11.0 still does not show up in apache logs, when attempting to sync. I'll try to upgrade to latest stable and hope, that security.enterprise.roots.enabled survives the transefer.

[virgoparna]

And this setting survived the upgrade. Following that bugzilla bug about that setting.

[majurgens]

Just a note to tell you what I have ended up doing.

I installed a Letsencrypt SSL certificate and ran the sync server behind an apache httpd frontend. I am now using the following format URL successfully:
https://server/token/1.0/sync/1.5

I did also try https://server but that did not work.

Note that https://server/token/1.0/sync/1.5 did not work at first as my first attempt (can't remember the exact details). I had the client setup with https://server/token/1.0/sync/1.5 but the server was not working. It would not work again until I signed out, changed the URL to the same thing and signed in again. I am guessing but maybe the client registered the server as dead and some part of the signout/URL change/signin made it reattempt.

A little more detail on what I did that might help others with internal servers.

• I am on Fedora 30 (old version so that the python 2 sync server works better)
• I decided to use Apache HTTPD frontend
• I have a public domain mydomain.com hosted on a Cpanel service
• I made an internal DNS entry server.mydomain.com
• dnf install certbot python-requests certbot-apache
• cd /opt
• git clone https://github.com/letsdebug/certbot-cpanel-dns-auth-hook
  -- this is a Cpanel client that can dynamically make DNS entries on the Cpanel hosted server. You need to be able to do this for internal servers since the only way you can register a Letsencrypt certificate is by using a DNS check. The HTTP check will not work because Letsencrypt serves cannot see the internal server
  -- if you are using some other hosting you will need to find a way to make dynamic DNS entries at your DNS hoster
• Now get the certificate
• certbot certonly --manual --manual-auth-hook "/opt/certbot-cpanel-dns-auth-hook/cpanel-dns.py create" --manual-cleanup-hook "/opt/certbot-cpanel-dns-auth-hook/cpanel-dns.py delete" -d "server.mydomain.com" --preferred-challenges dns-01
• during this command the script /opt/certbot-cpanel-dns-auth-hook/cpanel-dns.py adds a DNS TXT entry on my public DNS for _acme-challenge.server.mydomain.com, loads it with a key, Letsencrypt servers check the key (to verify you own the domain), then the scripts deletes the DNS entry
• this places the certificate into my file system
• now all I have to do is make sure my Apache config points to that certificate
• I used the follow Apache config (added this to ssl.conf)

<Directory /opt/mozilla-sync/syncserver>
  Require all granted
</Directory>

<VirtualHost *:443>
  ServerName server.mydomain.com
  DocumentRoot /opt/mozilla-sync/syncserver
  SSLCertificateFile /etc/letsencrypt/live/server.mydomain.com/fullchain.pem
  SSLCertificateKeyFile /etc/letsencrypt/live/server.mydomain.com/privkey.pem
  WSGIProcessGroup sync
  WSGIDaemonProcess sync user=apache group=apache processes=2 threads=5 python-path=/opt/mozilla-sync/syncserver/local/lib/python2.7/site-packages/
  WSGIPassAuthorization On
  WSGIScriptAlias / /opt/mozilla-sync/syncserver/syncserver.wsgi
</VirtualHost>

• Configure sync clients to use https://server.mydomain.com/token/1.0/sync/1.5 (which works internally only because if your internal DNS entry)

[majurgens]

Letsencrypt is no use, if sync server is internal network only accessible from Internet via VPN.

This is not true. As long as the internal server can get out to the internet, it does not matter if it cannot be seen from the Internet. The Letsencrypt HTTP will not work (requires server that is visible from the Internet) but the DNS will work if the server can get to the Internet. My previous comment in this thread goes into some detail on how I made this work

[virgoparna]

Letsencrypt is no use, if sync server is internal network only accessible from Internet via VPN.

This is not true. As long as the internal server can get out to the internet, it does not matter if it cannot be seen from the Internet. The Letsencrypt HTTP will not work (requires server that is visible from the Internet) but the DNS will work if the server can get to the Internet.

Only, if service provider has DNS update API and that API is supported. At least that is my understanding.

[virgoparna]

It is my understanding, that Letsencrypt DNS requires, that DNS provider has API to update DNS data. Anyway, browser should support custom CA-s anyway (Desktop Firefox supports it by installing them directly in Firefox). Anyway, that is Firefox issue and not syncserver one.