Drop in Snowflake users from Russia and evidence of blocking starting 2024-11

[cohosh]

We've noticed a significant drop in Snowflake users from Russia, starting earlier this month, along with some vantage point testing that suggests a different kind of block than what we've seen before.

At first we thought this was due to several recent certificate renewals of Fastly front domains that were used for Snowflake, but we've since ruled out blocking of rendezvous channel. The timing of the drop in users also doesn't quite match up with the domain renewals.

From our vantage point tests, it does not look like the DTLS fingerprinting of Snowflake that happened in Russia in 2021. The DTLS handshake completes and several bytes of application data are received from the proxy before the client suddenly stops receiving data. This doesn't rule out fingerprinting. There are a few proxies that do not appear to be blocked and Tor can be fully bootstrapped through these proxies, but we haven't noticed any difference in the DTLS fingerprint between working proxies and proxies that go stale. It's also possible that our vantage point tests aren't offering an accurate view of why there is a drop in users.

Related Links:

• Snowflake GitLab issue: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40407
• Full analysis of previous DTLS fingerprinting: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40014#note_2765061
• ntc.party topic: https://ntc.party/t/drop-in-snowflake-users-from-russia-and-evidence-of-blocking-starting-2024-11/13083

[PricacyPro0]

on my Android Orbot Snowflake Relay I saw a recent sudden increase in twice as much russian connections as usual per day. I already thought, that something must be going on.
Just spread that more people start installing the snowflake browser extension.

[IrradiatedKiwi]

I find the following result interesting. Because earlier this year in January I experience the similiar situation in China.

What's weird is that data channel connections to proxies appear to be opening successfully, and some data is being sent bidirectionally before the connections become stale. From one of the snowflake probe logs (edited to remove extraneous messages)

Here is the post I made at that time:

#325

and @wkrp wrote this in that post:

The anti-censorship team looked into this report a little. The cause is uncertain. The logs show STUN, rendezvous, and DTLS connection establishment working correctly, but apparently not much data is transferred over the DTLS connection before it is closed.

My experience at that time was that the snowflake bootstrap could not even finish most of time.
But in rare case they did finish and wireshark would start to show some DTLS Application Data transfer. But no websites could be opened.
And in even rarer case website could work, It would be very slow and the connection would only last for a really short time (no more than 1 to 3 minutes). Then the connection would be blocked and wireshark would show Encrypted Alert messages.

The problem lasted for more than one month then gone, We couldn't figure out the exact reason back then. At that time we even couldn't figure out the width of the blocking.
I am not sure whether that case is related with this one. But still maybe the experience is kinda similar?

[cohosh]

I've written a patch for snowflake to more easily test Snowflake connections from vantage points. Manual WebRTC signalling can be used to test client connections to a specific Snowflake proxy under your control that also has manual signalling enabled. Hopefully this is useful to anyone who wants to try out theories on how it is being blocked or test fixes.

You will need the following patch for both the client and the proxy: https://gitlab.torproject.org/cohosh/snowflake/-/commit/9523d9bbed0baf44e8779c1c78bb7ebd4e501a8a

Proxy setup

To run the proxy, you will need 2 terminals:

• Terminal A: Run the proxy with ./proxy --copy-paste --verbose --unsafe-logging
• Terminal B: Run cat > signal

Client setup

To run the client, you will need 3 terminals, and the following torrc file:

UseBridges 1
DataDirectory datadir

ClientTransportPlugin snowflake exec ./client -copy-paste -log snowflake.log --unsafe-logging

Bridge snowflake 192.0.2.3:80 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 ice=stun:stun.l.google.com:19302,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478 utls-imitate=hellorandomizedalpn

SocksPort auto

• Terminal A: Run the client with tor -f torrc
• Terminal B: Run tail -f snowflake.log
• Terminal C: Run cat > signal

Manual signalling

Once the client has started, you will see something like the following output in the client's Terminal B.

2024/11/20 00:45:58 Please Copy & Paste the following to the peer:
2024/11/20 00:45:58 ----------------
2024/11/20 00:45:58 

{"type":"offer","sdp":"v=0\r\no=- 7792309864421436583 1732063557 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=msid-semantic:WMS*\r\na=fingerprint:sha-256 82:33:BE:0E:A0:DC:FC:24:85:B5:7B:D5:9D:64:32:C9:A4:2F:56:B0:A6:47:35:B0:A4:4F:5D:96:E9:A6:7F:DA\r\na=extmap-allow-mixed\r\na=group:BUNDLE 0\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=setup:actpass\r\na=mid:0\r\na=sendrecv\r\na=sctp-port:5000\r\na=ice-ufrag:vggBpjGDVijaxuQY\r\na=ice-pwd:skiZVUcgjPZpSyENHMHFBtRGUszeInMU\r\na=candidate:1408731006 1 udp 1694498815 XX.XX.XX.XX 40705 typ srflx raddr 0.0.0.0 rport 40705\r\na=end-of-candidates\r\n"}

2024/11/20 00:45:58 ----------------

Copy that, and paste it into the proxy's Terminal B. Soon after, you should see the following output in the proxy's Terminal A:

2024/11/20 00:46:05 Please Copy & Paste the following to the peer:
2024/11/20 00:46:05 ----------------
2024/11/20 00:46:05 

{"type":"answer","sdp":"v=0\r\no=- 415248651781399192 1732218246 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=msid-semantic:WMS*\r\na=fingerprint:sha-256 25:C6:51:5D:35:61:F6:21:40:39:60:DB:47:85:9F:78:EB:A6:8E:B5:27:91:8C:AA:34:4E:73:DD:A4:1B:65:85\r\na=extmap-allow-mixed\r\na=group:BUNDLE 0\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=setup:active\r\na=mid:0\r\na=sendrecv\r\na=sctp-port:5000\r\na=ice-ufrag:zyyLdoWNpFiIRUXn\r\na=ice-pwd:AzLMxJJrliwDFEHzpSwODjFrqlCtYsbv\r\na=candidate:1408731006 1 udp 1694498815 XX.XX.XX.XX 52584 typ srflx raddr 0.0.0.0 rport 52584\r\na=candidate:1408731006 2 udp 1694498815 XX.XX.XX.XX 52584 typ srflx raddr 0.0.0.0 rport 52584\r\na=end-of-candidates\r\n"}

2024/11/20 00:46:05 ----------------

Copy that and paste it into the client's Terminal C. If everything goes right, the WebRTC datachannel should open at both the client and the proxy.

[cohosh]

It looks like the blocking has at least temporarily stopped since Thursday. Most snowflake connections from our vantage point have been succeeding for the last few days. The number of snowflake users from Russia seems to be climbing again.