How can I remember these two data storing modes? It is quite simple. First, remember that the addresses of the memory units we are talking about are always arranged from low to high. For a multi-byte number, if the first byte in the low address you see is the least-significant byte, the system is Little Endian, where Little matches low. On the contrary is Big Endian, where Big corresponds to "high".
Program Example
To deepen our understanding of Endianness, let's look at the following example of a C program:
-
1 2 3 4
char a = 1; char b = 2; short c = 255; /* 0x00ff */ long d = 0x44332211;
+
1 2 3 4
char a = 1; char b = 2; short c = 255; /* 0x00ff */ long d = 0x44332211;
On Intel 80x86 based systems, the memory content corresponding to variables a, b, c, and d are shown in the following table：
@@ -356,11 +356,11 @@
Program Example
Little Endian: Intel x86, AMD64, DEC VAX
How to detect the Endianess of local system in the program? The following function can be called for a quick check. If the return value is 1, it is Little Endian, else Big Endian：
-
1 2 3 4
inttest_endian(){ int x = 1; return *((char *)&x); }
+
1 2 3 4
inttest_endian() { int x = 1; return *((char *)&x); }
Network Order
Endianness is also important for computer communications. Imagine that when a Little Endian system communicates with a Big Endian system, the receiver and sender will interpret the data completely differently if not handled properly. For example, for the variable d in the C program segment above, the Little Endian sender sends 11 22 33 44 four bytes, which the Big Endian receiver converts to the value 0x11223344. This is very different from the original value. To solve this problem, the TCP/IP protocol specifies a special "network byte order" (referred to as "network order"), which means that regardless of the Endian supported by the computer system, the most-significant byte is always sent first while transmitting data. From the definition, we can see that the network order corresponds to the Big Endian.
To avoid communication problems caused by Endianness and to facilitate software developers to write portable programs, some C preprocessing macros are defined for conversion between network bytes and local byte order. htons() and htonl() are used to convert local byte order to network byte order, the former works with 16-bit unsigned numbers and the latter for 32-bit unsigned numbers. ntohs() and ntohl() implement the conversion in the opposite direction. The prototype definitions of these four macros can be found as follows (available in the netinet/in.h file on Linux systems).
Notice at step (1) we replace the checksum field with 0x0000. As can be seen, the calculated header checksum 0x598f is the same as the value in the captured packet. This calculating process is only used for the sender to generate the initial checksum. In practice, for the intermediate forwarding router and the final receiver, they can just sum up all header fields of the received IP packet by the same algorithm. If the result is 0xffff, the checksum verification passes.
C Program Implementation
How to program IPv4 header checksum computing? RFC 1071 (Computing the Internet Checksum) shows a reference "C" language implementation:
{ /* Compute Internet Checksum for "count" bytes * beginning at location "addr". */ registerlong sum = 0;
while( count > 1 ) { /* This is the inner loop */ sum += * (unsignedshort *) addr++; count -= 2; }
/* Add left-over byte, if any */ if ( count > 0 ) sum += * (unsignedchar *) addr;
/* Fold 32-bit sum to 16 bits */ while (sum>>16) sum = (sum & 0xffff) + (sum >> 16);
checksum = ~sum; }
In a real network connection, the source device can call the above code to generate the initial IPv4 header checksum. This checksum is then updated at each step of the routing hop because the router must decrement the Time To Live (TTL) field. RFC 1141 (Incremental Updating of the Internet Checksum) gives a reference implementation of fast checksum update：
For TCP segment and UDP datagram, both have 16-bit header checksum fields used for error-checking by the destination host. The checksum computing algorithm is the same as the IP header, except for the difference of covered data. Here the checksum is calculated over the whole TCP/UDP header and the payload, plus a pseudo-header that mimics the IPv4 header as shown below:
For multimedia applications running VoIP or streaming video data transmission protocols, it'd better receive data with some degree of corruption than not receiving any data at all. Another example is the CAPWAP protocol used to connect Cisco wireless controller and access points. It specifies UDP-Lite as the default transport protocol for the CAPWAP Data channel, while the connection is established over the IPv6 network.
At last, share a C program snippet to present how to initialize a Berkeley socket to establish an IPv6 UDP-Lite connection:
int udplite_conn = socket(AF_INET6, SOCK_DGRAM, IPPROTO_UDPLITE); int val = 8; /* checksum only covers 8-byte UDP-Lite header */ (void)setsockopt(udplite_conn, IPPROTO_UDPLITE, UDPLITE_SEND_CSCOV, &val, sizeof val); (void)setsockopt(udplite_conn, IPPROTO_UDPLITE, UDPLITE_RECV_CSCOV, &val, sizeof val);
Here IPPROTO_UDPLITE is protocol number 136, which is used together with AF_INET6 address family parameter in socket() function call for IPv6 socket creation. The UDPLITE_SEND_CSCOV(10) and UDPLITE_RECV_CSCOV(11) are the control parameters of socket options configuration function setsockopt(), used for setting the Checksum Coverage value in the sender and the receiver respectively. Remember that both the sender and the receiver must set the same value, otherwise, the receiver will not be able to verify the checksum properly.
Network Attached Storage (NAS) provides data access to a heterogeneous group of clients over computer networks. As hard drive prices continue to drop, NAS devices have made their way into the homes of the masses. Leading brands in the SMB and home NAS market, such as Synology, have their products range in price from as low as $300 to $ 700 for the high models. But if you are a Raspberry Pi player, you can build a very nice home NAS and streaming service for only about half the cost of the lowest price.
+
Network Attached Storage (NAS) provides data access to a heterogeneous group of clients over computer networks. As hard drive prices continue to drop, NAS devices have made their way into the homes of the masses. Leading brands in the SMB and home NAS market, such as Synology, have their products range in price from as low as $300 to $700 for the high models. But if you are a Raspberry Pi player, you can build a very nice home NAS and streaming service for only about half the cost of the lowest price.
Knowledge obtained on the papers always feels shallow, must know this thing to practice. — LU You (Chinese historian and poet of the Southern Song Dynasty)
This blog records the whole process of building a Raspberry Pi NAS and home media server, including project planning, system implementation, and performance review. It also covers some important experiences and lessons that could hopefully benefit anyone interested in this DIY project.
@@ -440,7 +440,7 @@
Prepare Internal SSD
PWM Fan Control
Before the actual hardware assembly, a special software provided by Geekworm - PWM fan control script - must be installed. PWM fan speed adjustment to temperature change is a major feature that lets NASPi stand out from other hardware solutions. So this step is critical.
Referring to Geekworm's X-C1 software wiki page, the installation command sequence on the SSH session connected to the Raspberry Pi 4B system is as follows
If you can't do git clone directly on Raspberry Pi 4B, you can first download the X-C1 software on the SSH client, then transfer it to Raspberry Pi 4B using scp. After that, continue to execute the subsequent commands
1
❯ scp -r x-c1 pi@192.168.2.4:/home/pi/
How does X-C1 software control PWM fan?
@@ -451,7 +451,7 @@
PWM Fan Control
In addition, the following pi-temp.sh script, which reads out the GPU and CPU temperatures, is also useful
-
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
pi@raspberrypi:~ $ cat ./pi-temp.sh #!/bin/bash # Script: pi-temp.sh # Purpose: Display the ARM CPU and GPU temperature of Raspberry Pi # ------------------------------------------------------- cpu=$(</sys/class/thermal/thermal_zone0/temp) echo"$(date) @ $(hostname)" echo"-------------------------------------------" echo"GPU => $(vcgencmd measure_temp)" echo"CPU => temp=$((cpu/1000))’C"
pi@raspberrypi:~ $ ./pi-temp.sh Mon 29 Nov 06:59:17 GMT 2021 @ raspberrypi ------------------------------------------- GPU => temp=33.1'C CPU => temp=32’C
+
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
pi@raspberrypi:~ $ cat ./pi-temp.sh #!/bin/bash # Script: pi-temp.sh # Purpose: Display the ARM CPU and GPU temperature of Raspberry Pi # ------------------------------------------------------- cpu=$(</sys/class/thermal/thermal_zone0/temp) echo"$(date) @ $(hostname)" echo"-------------------------------------------" echo"GPU => $(vcgencmd measure_temp)" echo"CPU => temp=$((cpu/1000))’C"
pi@raspberrypi:~ $ ./pi-temp.sh Mon 29 Nov 06:59:17 GMT 2021 @ raspberrypi ------------------------------------------- GPU => temp=33.1'C CPU => temp=32’C
Hardware Assembly Process
Below is a snapshot of the Geekworm NASPi parts out of the box (except for the Raspberry Pi 4B on the far right of the second row and the screwdriver in the lower right corner)
The three key components in the second row, from left to right, are
@@ -499,7 +499,7 @@
Configure File Services
Scan for mounted disk drives
Click Storage > Disks from the sidebar menu to enter the hard drive management page. If there is an external USB storage device just plugged in, you can click 🔍 here to scan it out. The scan results for this system are as follows. The internal Samsung 500GB SSD and external Seagate 2TB HDD are detected, and the 32GB microSD that contains the entire software system is listed at the top:
On the SSH terminal, we could see the information for the same set of mounted drivers
Click Storage > File Systems from the sidebar menu to enter the file system management page. If the storage device does not have a file system yet, click ⨁ to Create or Mount the file system. OMV can create/mount ext4, ext3, JFS, and xfs file systems, but only mounts are supported for the NTFS file system. The following figure shows that OMV correctly mounts NTFS file systems for SSDs and HDDs:
Set Shared Folders
@@ -551,13 +551,13 @@
Performance Review
By connecting a macOS laptop to one of the remaining ports of the TL-SG105, we could perform some simple same-subnet tests to evaluate the performance of this NAS system fully.
System Stress Test
Referring to Geekworm NASPi Stress Test Wiki page, we executed the following command over SSH connection, which cloned the test script from GitHub and ran the stress test:
-
1 2 3 4
git clone https://github.com/geekworm-com/rpi-cpu-stress cd rpi-cpu-stress chmod +x stress.sh sudo ./stress.sh
+
1 2 3 4
git clone https://github.com/geekworm-com/rpi-cpu-stress cd rpi-cpu-stress chmod +x stress.sh sudo ./stress.sh
Simultaneously we established a second SSH session and ran htop to monitor system status. The screenshot below was taken while close to the 5-minute mark (left is the htop real-time display, and right is the stress test output) Dividing the temp value on the right side by 1000 gave the CPU temperature. All 4 CPU cores reached 100% full load during the test, while the maximum temperature did not exceed 70°C. At this moment, there was no obvious heat sensation when touching the case. Typing ctrl-c to stop the stress test, and then executing the temperature measurement script again
1 2 3 4 5
pi@raspberrypi:~ $ ./pi-temp.sh Fri Dec 24 15:59:21 PST 2021 @ raspberrypi ------------------------------------------- GPU => temp=39.9'C CPU => temp=40'C
The system temperature returned to a low range value. This test result assures the system meets the design goal.
File Transfer Speed Test
The file transfer speed can be roughly measured with the secure remote copy tool SCP. First, create a 1GB size file by running the mkfile command on the macOS client, then copy it to the user directory of the remote NAS system with the scp command
After the copy was done, it would print the time spent and the deduced speed. Running the command with the source and the destination reversed would give us the speed of receiving a file from the NAS system.
In this software implementation, the first step can generate odd numbers directly. Also for demonstration purposes, the second step uses the first 50 prime numbers greater than 2 for the basic primality test. The whole process is shown in the following flowchart.
For the first step, Python function programming requires importing the library function randrange() from the random library. The function uses the input number of bits n in the exponents of 2, which specify the start and end values of randrange(). It also sets the step size to 2 to ensure that only n-bit random odd values are returned.
-
1 2 3 4 5 6
from random import randrange
defgenerate_n_bit_odd(n: int): '''Generate a random odd number in the range [2**(n-1)+1, 2**n-1]''' assert n > 1 return randrange(2 ** (n - 1) + 1, 2 ** n, 2)
+
1 2 3 4 5 6
from random import randrange
defgenerate_n_bit_odd(n: int): '''Generate a random odd number in the range [2**(n-1)+1, 2**n-1]''' assert n > 1 return randrange(2 ** (n - 1) + 1, 2 ** n, 2)
The code for the second step is simple. It defines an array with elements of 50 prime numbers after 2, then uses a double loop in the function to implement the basic primality test. The inner for loop runs the test with the elements of the prime array one by one. It aborts back to the outer loop immediately upon failure, from there it calls the function in the first step to generate the next candidate odd number and test again.
-
1 2 3 4 5 6 7 8 9 10 11 12 13 14
defget_lowlevel_prime(n): """Generate a prime candidate not divisible by first primes""" whileTrue: # Obtain a random odd number c = generate_n_bit_odd(n)
# Test divisibility by pre-generated primes for divisor in first_50_primes: if c % divisor == 0and divisor ** 2 <= c: break else: # The for loop did not encounter a break statement, # so it passes the low-level primality test. return c
+
1 2 3 4 5 6 7 8 9 10 11 12 13 14
defget_lowlevel_prime(n): """Generate a prime candidate not divisible by first primes""" whileTrue: # Obtain a random odd number c = generate_n_bit_odd(n)
# Test divisibility by pre-generated primes for divisor in first_50_primes: if c % divisor == 0and divisor ** 2 <= c: break else: # The for loop did not encounter a break statement, # so it passes the low-level primality test. return c
The Miller-Rabin primality test1 in the third step is a widely used method for testing prime numbers. It uses a probabilistic algorithm to determine whether a given number is a composite or possibly a prime number. Although also based on Fermat's little theorem, the Miller-Rabin primality test is much more efficient than the Fermat primality test. Before showing the Python implementation of the Miller-Rabin prime test, a brief description of how it works is given here.
By Fermat's little theorem, for a prime \(n\), if the integer \(a\) is not a multiple of \(n\), then we have \(a^{n-1}\equiv 1\pmod n\). Therefore if \(n>2\), \(n-1\) is an even number and must be expressed in the form \(2^{s}*d\), where both \(s\) and \(d\) are positive integers and \(d\) is odd. This yields \[a^{2^{s}*d}\equiv 1\pmod n\] If we keep taking the square root of the left side of the above equation and then modulo it, we will always get \(1\) or \(-1\)2. If we get \(1\), it means that the following equation ② holds; if we never get \(1\), then equation ① holds: \[a^{d}\equiv 1{\pmod {n}}{\text{ ①}}\]\[a^{2^{r}d}\equiv -1{\pmod {n}}{\text{ ②}}\] where \(r\) is some integer that lies in the interval \([0, s-1]\). So, if \(n\) is a prime number greater than \(2\), there must be either ① or ② that holds. The conditional statement of this law is also true, i.e.** if we can find a \(\pmb{a}\) such that for any \(\pmb{0\leq r\leq s-1}\) the following two equations are satisfied: \[\pmb{a^{d}\not \equiv 1\pmod n}\]\[\pmb{a^{2^{r}d}\not \equiv -1\pmod n}\] Then \(\pmb{n}\) must not be a prime number**. This is the mathematical concept of the Miller-Rabin primality test. For the number \(n\) to be tested, after calculating the values of \(s\) and \(d\), the base \(a\) is chosen randomly and the above two equations are tested iteratively. If neither holds, \(n\) is a composite number, otherwise, \(n\) may be a prime number. Repeating this process, the probability of \(n\) being a true prime gets larger and larger. Calculations show that after \(k\) rounds of testing, the maximum error rate of the Miller-Rabin primality test does not exceed \(4^{-k}\).
The Miller-Rabin primality test function implemented in Python is as follows, with the variables n,s,d,k in the code corresponding to the above description.
defmiller_rabin_primality_check(n, k=20): '''Miller-Rabin Primality Test with a specified round of test Input: n - n > 3, an odd integer to be tested for primality k - the number of rounds of testing to perform Output: True - passed (n is a strong probable prime) False - failed (n is a composite)''' # For a given odd integer n > 3, write n as (2^s)*d+1, # where s and d are positive integers and d is odd. assert n > 3 if n % 2 == 0: returnFalse s, d = 0, n - 1 while d % 2 == 0: d >>= 1 s += 1
for _ inrange(k): a = randrange(2, n - 1) x = pow(a, d, n) if x == 1or x == n - 1: continue for _ inrange(s): x = pow(x, 2, n) if x == n - 1: break else: # The for loop did not encounter a break statement, # so it fails the test, it must be a composite returnFalse
# Passed the test, it is a strong probable prime returnTrue
defmiller_rabin_primality_check(n, k=20): '''Miller-Rabin Primality Test with a specified round of test Input: n - n > 3, an odd integer to be tested for primality k - the number of rounds of testing to perform Output: True - passed (n is a strong probable prime) False - failed (n is a composite)''' # For a given odd integer n > 3, write n as (2^s)*d+1, # where s and d are positive integers and d is odd. assert n > 3 if n % 2 == 0: returnFalse s, d = 0, n - 1 while d % 2 == 0: d >>= 1 s += 1
for _ inrange(k): a = randrange(2, n - 1) x = pow(a, d, n) if x == 1or x == n - 1: continue for _ inrange(s): x = pow(x, 2, n) if x == n - 1: break else: # The for loop did not encounter a break statement, # so it fails the test, it must be a composite returnFalse
# Passed the test, it is a strong probable prime returnTrue
Putting all of the above together, the whole process can be wrapped into the following function, where the input of the function is the number of bits and the output is a presumed random large prime number.
-
1 2 3 4 5
defget_random_prime(num_bits): whileTrue: pp = get_lowlevel_prime(num_bits) if miller_rabin_primality_check(pp): return pp
+
1 2 3 4 5
defget_random_prime(num_bits): whileTrue: pp = get_lowlevel_prime(num_bits) if miller_rabin_primality_check(pp): return pp
Utility Functions
Greatest Common Divisor (GCD) gcd(a,b) and Least Common Multiple lcm(a,b):
The RSA encryption algorithm needs to calculate the Carmichael function \(\lambda(N)\) of modulus \(N\), with the formula \(\lambda(pq)= \operatorname{lcm}(p - 1, q - 1)\), where the least common multiple function is used. The relationship between the least common multiple and the greatest common divisor is: \[\operatorname{lcm}(a,b)={\frac{(a\cdot b)}{\gcd(a,b)}}\] There is an efficient Euclidean algorithm for finding the greatest common divisor, which is based on the principle that the greatest common divisor of two integers is equal to the greatest common divisor of the smaller number and the remainder of the division of the two numbers. The specific implementation of Euclid's algorithm can be done iteratively or recursively. The iterative implementation of the maximum convention function is applied here, and the Python code for the two functions is as follows:
-
1 2 3 4 5 6 7 8 9
defgcd(a, b): '''Computes the Great Common Divisor using the Euclid's algorithm''' while b: a, b = b, a % b return a
deflcm(a, b): """Computes the Lowest Common Multiple using the GCD method.""" return a // gcd(a, b) * b
+
1 2 3 4 5 6 7 8 9
defgcd(a, b): '''Computes the Great Common Divisor using the Euclid's algorithm''' while b: a, b = b, a % b return a
deflcm(a, b): """Computes the Lowest Common Multiple using the GCD method.""" return a // gcd(a, b) * b
Extended Euclidean Algorithm exgcd(a,b) and Modular Multiplicative Inverse invmod(e,m):
The RSA key pair satisfies the equation \((d⋅e)\bmod \lambda(N)=1\), i.e., the two are mutually modular multiplicative inverses with respect to the modulus \(\lambda(N)\). The extended Euclidean algorithm can be applied to solve the modular multiplicative inverse \(d\) of the public key exponent \(e\) quickly. The principle of the algorithm is that given integers \(a,b\), it is possible to find integers \(x,y\) (one of which is likely to be negative) while finding the greatest common divisor of \(a,b\) such that they satisfy Bézout's identity: \[a⋅x+b⋅y=\gcd(a, b)\] substituted into the parameters \(a=e\) and \(b=m=\lambda(N)\) of the RSA encryption algorithm, and since \(e\) and \(\lambda(N)\) are coprime, we can get: \[e⋅x+m⋅y=1\] the solved \(x\) is the modulo multiplicative inverse \(d\) of \(e\). The Python implementations of these two functions are given below:
defexgcd(a, b): """Extended Euclidean Algorithm that can give back all gcd, s, t such that they can make Bézout's identity: gcd(a,b) = a*s + b*t Return: (gcd, s, t) as tuple""" old_s, s = 1, 0 old_t, t = 0, 1 while b: q = a // b s, old_s = old_s - q * s, s t, old_t = old_t - q * t, t a, b = b, a % b return a, old_s, old_t
definvmod(e, m): """Find out the modular multiplicative inverse x of the input integer e with respect to the modulus m. Return the minimum positive x""" g, x, y = exgcd(e, m) assert g == 1
# Now we have e*x + m*y = g = 1, so e*x ≡ 1 (mod m). # The modular multiplicative inverse of e is x. if x < 0: x += m return x
Similarly, an iterative approach is applied here to implement the extended Euclidean algorithm, with the modular inverse multiplicative function calling the former.
defexgcd(a, b): """Extended Euclidean Algorithm that can give back all gcd, s, t such that they can make Bézout's identity: gcd(a,b) = a*s + b*t Return: (gcd, s, t) as tuple""" old_s, s = 1, 0 old_t, t = 0, 1 while b: q = a // b s, old_s = old_s - q * s, s t, old_t = old_t - q * t, t a, b = b, a % b return a, old_s, old_t
definvmod(e, m): """Find out the modular multiplicative inverse x of the input integer e with respect to the modulus m. Return the minimum positive x""" g, x, y = exgcd(e, m) assert g == 1
# Now we have e*x + m*y = g = 1, so e*x ≡ 1 (mod m). # The modular multiplicative inverse of e is x. if x < 0: x += m return x
Similarly, an iterative approach is applied here to implement the extended Euclidean algorithm, with the modular inverse multiplicative function calling the former.
Implementing RSA Class
Note: Textbook RSA has inherent security vulnerabilities. The reference implementation in the Python language given here is for learning and demonstration purposes only, by no means to be used in actual applications. Otherwise, it may cause serious information security incidents. Keep this in mind!
Encryption and Decryption Methods
RSA encryption and regular decryption formulas are \[\begin{align}
c\equiv m^e\pmod N\\
m\equiv c^d\pmod N
\end{align}\] Python built-in pow() function supports modular exponentiation. The above two can be achieved by simply doing the corresponding integer to byte sequence conversions and then calling pow() with the public or private key exponent:
For fast descryption, a few extra steps are needed: \[\begin{align}
m_1&=c^{d_P}\pmod {p}\tag{1}\label{eq1}\\
m_2&=c^{d_Q}\pmod {q}\tag{2}\label{eq2}\\
h&=q_{\text{inv}}(m_1-m_2)\pmod {p}\tag{3}\label{eq3}\\
m&=m_{2}+hq\pmod {pq}\tag{4}\label{eq4}
\end{align}\] In practice, if \(m_1-m_2<0\) in the step \((3)\), \(p\) needs to be added to adjust to a positive number. It can also be seen that the acceleration ratio would theoretically be close to \(4\) because the fast decryption method decreases the modulus and exponent by roughly half the order. Considering the additional computational steps, the actual speedup ratio estimate is subtracted by a correction \(\varepsilon\), noted as \(4-\varepsilon\). The code of the fast decryption function is as follows:
-
1 2 3 4 5 6 7 8 9 10 11 12
defdecrypt_fast(self, encrypted_int_data: int): # Use Chinese Remaider Theorem + Fermat's Little Theorem to # do fast RSA description assert self.fast == True m1 = pow(encrypted_int_data, self.d_P, self.p) m2 = pow(encrypted_int_data, self.d_Q, self.q) t = m1 - m2 if t < 0: t += self.p h = (self.q_Inv * t) % self.p m = (m2 + h * self.q) % self.n return uint_to_bytes(m)
+
1 2 3 4 5 6 7 8 9 10 11 12
defdecrypt_fast(self, encrypted_int_data: int): # Use Chinese Remaider Theorem + Fermat's Little Theorem to # do fast RSA description assert self.fast == True m1 = pow(encrypted_int_data, self.d_P, self.p) m2 = pow(encrypted_int_data, self.d_Q, self.q) t = m1 - m2 if t < 0: t += self.p h = (self.q_Inv * t) % self.p m = (m2 + h * self.q) % self.n return uint_to_bytes(m)
Signature Generation and Verification Methods
The RSA digital signature generation and verification methods are very similar to encryption and regular decryption functions, except that the public and private exponents are used in reverse. The signature generation uses the private exponent, while the verification method uses the public key exponent. The implementation of fast signature generation is the same as the fast decryption steps, but the input and output data are converted and adjusted accordingly. The specific implementations are presented below：
defgenerate_signature(self, encoded_msg_digest: bytes): """Use RSA private key to generate Digital Signature for given encoded message digest""" int_data = uint_from_bytes(encoded_msg_digest) returnpow(int_data, self.d, self.n) defgenerate_signature_fast(self, encoded_msg_digest: bytes): # Use Chinese Remaider Theorem + Fermat's Little Theorem to # do fast RSA signature generation assert self.fast == True int_data = uint_from_bytes(encoded_msg_digest) s1 = pow(int_data, self.d_P, self.p) s2 = pow(int_data, self.d_Q, self.q) t = s1 - s2 if t < 0: t += self.p h = (self.q_Inv * t) % self.p s = (s2 + h * self.q) % self.n return s
defverify_signature(self, digital_signature: int): """Use RSA public key to decrypt given Digital Signature""" int_data = pow(digital_signature, self.e, self.n) return uint_to_bytes(int_data)
defgenerate_signature(self, encoded_msg_digest: bytes): """Use RSA private key to generate Digital Signature for given encoded message digest""" int_data = uint_from_bytes(encoded_msg_digest) returnpow(int_data, self.d, self.n) defgenerate_signature_fast(self, encoded_msg_digest: bytes): # Use Chinese Remaider Theorem + Fermat's Little Theorem to # do fast RSA signature generation assert self.fast == True int_data = uint_from_bytes(encoded_msg_digest) s1 = pow(int_data, self.d_P, self.p) s2 = pow(int_data, self.d_Q, self.q) t = s1 - s2 if t < 0: t += self.p h = (self.q_Inv * t) % self.p s = (s2 + h * self.q) % self.n return s
defverify_signature(self, digital_signature: int): """Use RSA public key to decrypt given Digital Signature""" int_data = pow(digital_signature, self.e, self.n) return uint_to_bytes(int_data)
Functional Tests
Once the RSA class is completed, it is ready for testing. To test the basic encryption and decryption functions, first initialize an RSA object with the following parameters
@@ -418,7 +418,7 @@
Performance Tests
decrypt_fast() - Fast descryption method
Both use the assert statement to check the result, as shown in the code below:
The outer loop iterates over different key lengths klen, from 512 bits to 4096 bits in 5 levels, and the corresponding RSA object obj is initialized with:
Once the shadow memory design is determined, the implementation of compiler instrumentation to detect dynamic memory access errors is easy. For memory accesses of 8 bytes, the shadow memory bytes are checked by inserting instructions before the original read/write code, and an error is reported if they are not zero. For memory accesses of less than 8 bytes, the instrumentation is a bit more complicated, where the shadow memory byte values are compared with the last three bits of the read/write address. This situation is also known as the "slow path" and the sample code is as follows.
-
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
// Check the cases where we access first k bytes of the qword // and these k bytes are unpoisoned. boolSlowPathCheck(shadow_value, address, kAccessSize){ last_accessed_byte = (address & 7) + kAccessSize - 1; return (last_accessed_byte >= shadow_value); } ...
// Check the cases where we access first k bytes of the qword // and these k bytes are unpoisoned. boolSlowPathCheck(shadow_value, address, kAccessSize) { last_accessed_byte = (address & 7) + kAccessSize - 1; return (last_accessed_byte >= shadow_value); } ...
For global and stack (local) objects, ASan has designed different instrumentation to detect their out-of-bounds access errors. The red zone around a global object is added by the compiler at compile time and its address is passed to the runtime library at application startup, where the runtime library function then poisons the red zone and writes down the address needed in error reporting. The stack object is created at function call time, and accordingly, its red zone is created and poisoned at runtime. In addition, because the stack object is deleted when the function returns, the instrumentation code must also zero out the shadow memory it is mapped to.
In practice, the ASan compiler instrumentation process is placed at the end of the compiler optimization pipeline so that instrumentation only applies to the remaining memory access instructions after variable and loop optimization. In the latest GCC distribution, the ASan compiler stubbing code is located in two files in the gcc subdirectory gcc/asan.[ch].
Runtime Library Replacement
@@ -467,7 +467,7 @@
Application Examples
ASan is very easy to use. The following is an example of an Ubuntu Linux 20.4 + GCC 9.3.0 system running on an x86_64 virtual machine to demonstrate the ability to detect various memory access errors.
Test Cases
As shown below, the test program writes seven functions, each introducing a different error type. The function names are cross-referenced with the error types one by one:
intstack_buffer_overflow() { int s[10]; s[0] = 10; return s[10]; }
int *gp;
voidstack_use_after_return() { int r[10]; r[0] = 10; gp = &r[0]; return; }
voidstack_use_after_scope() { { int c = 0; gp = &c; } *gp = 10; return; }
The test program calls the getopt library function to support a single-letter command line option that allows the user to select the type of error to be tested. The command line option usage information is as follows.
1 2 3 4 5 6 7 8 9 10 11 12
$ ./asan-test
Test AddressSanitizer usage: asan-test [ -bfloprs ]
-b heap buffer overflow -f heap use after free -l heap memory leak -o global buffer overflow -p stack use after scope -r stack use after return -s stack buffer overflow
The GCC compile command for the test program is simple, just add two compile options
@@ -484,7 +484,7 @@
OOB Test
The following Global OOB test result also clearly shows the error line asan-test.c:16, the global variable name ga and its definition code location asan-test.c:13:5, and you can also see that the global object has a red zone poisoning value of 0xf9.
$ ./asan-test -o ================================================================= ==57367==ERROR: AddressSanitizer: global-buffer-overflow on address 0x564363ea4048 at pc 0x564363ea1383 bp 0x7ffc0d6085d0 sp 0x7ffc0d6085c0 READ of size 4 at 0x564363ea4048 thread T0 #0 0x564363ea1382 in global_buffer_overflow /home/zixi/coding/asan-test.c:16 #1 0x564363ea1a6c in main /home/zixi/coding/asan-test.c:98 #2 0x7f8cb43890b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) #3 0x564363ea128d in _start (/home/zixi/coding/asan-test+0x128d)
Surprisingly, ASan does not report the obvious Global OOB error here. Why?
The reason has to do with the way GCC treats global variables. The compiler treats functions and initialized variables as Strong symbols, while uninitialized variables are Weak symbols by default. Since the definition of weak symbols may vary from source file to source file, the size of the space required is unknown. The compiler cannot allocate space for weak symbols in the BSS segment, so it uses the COMMON block mechanism so that all weak symbols share a COMMON memory region, thus ASan cannot insert the red zone. During the linking process, after the linker reads all the input target files, it can determine the size of the weak symbols and allocate space for them in the BSS segment of the final output file.
Fortunately, GCC's -fno-common option turns off the COMMON block mechanism, allowing the compiler to add all uninitialized global variables directly to the BSS segment of the target file, also allowing ASan to work properly. This option also disables the linker from merging weak symbols, so the linker reports an error directly when it finds a compiled unit with duplicate global variables defined in the target file.
That is, we not only give the product of the two prime numbers used by RSA but also tell you their sum. How are these given? You need to discover by yourself from the rest of the information. After clicking the two links and downloading the file, open the first Python file:
FLAG = open('flag.txt').read().strip() FLAG = int(hexlify(FLAG.encode()), 16) SEED = int(hexlify(os.urandom(32)).decode(), 16) STATE = random_state(SEED)
FLAG = open('flag.txt').read().strip() FLAG = int(hexlify(FLAG.encode()), 16) SEED = int(hexlify(os.urandom(32)).decode(), 16) STATE = random_state(SEED)
If you have basic Python programming skills and understand the principles of the RSA algorithm, you should be able to read the above program quickly. What it does is:
Open the file flag.txt to read the content. Then use the hexlify and int functions to convert it to an integer and store the result in a variable FLAG.
@@ -370,7 +370,7 @@
Conventional Solution
Obviously, \(p\) and \(q\) are its two roots. According to the quadratic formula
We can get \(p\) and \(q\). The rest of the work is easy. The code to compute \(d\) from \(p\) and \(q\) can be copied directly from lines 28, 30, and 31 in gen.py. The final complete Python problem-solving code is as follows:
x = int((Lines[0].split())[2], 16) # x = p + q n = int((Lines[1].split())[2], 16) # n = p * q c = int((Lines[2].split())[2], 16) # Ciphertext
defsolve_rsa_primes(s: int, m: int) -> tuple: ''' Solve RSA prime numbers (p, q) from the quadratic equation p^2 - s * p + m = 0 with the formula p = s/2 +/- sqrt((s/2)^2 - m) Input: s - sum of primes, m - product of primes Output: (p, q) ''' half_s = s >> 1 tmp = math.isqrt(half_s ** 2 - m) returnint(half_s + tmp), int(half_s - tmp);
# Now run with the real input p, q = solve_rsa_primes(x, n) m = math.lcm(p - 1, q - 1) e = 65537 d = pow(e, -1, m) FLAG = pow(c, d, n) print(FLAG.to_bytes((FLAG.bit_length() + 7) // 8, 'big'))
x = int((Lines[0].split())[2], 16) # x = p + q n = int((Lines[1].split())[2], 16) # n = p * q c = int((Lines[2].split())[2], 16) # Ciphertext
defsolve_rsa_primes(s: int, m: int) -> tuple: ''' Solve RSA prime numbers (p, q) from the quadratic equation p^2 - s * p + m = 0 with the formula p = s/2 +/- sqrt((s/2)^2 - m) Input: s - sum of primes, m - product of primes Output: (p, q) ''' half_s = s >> 1 tmp = math.isqrt(half_s ** 2 - m) returnint(half_s + tmp), int(half_s - tmp);
# Now run with the real input p, q = solve_rsa_primes(x, n) m = math.lcm(p - 1, q - 1) e = 65537 d = pow(e, -1, m) FLAG = pow(c, d, n) print(FLAG.to_bytes((FLAG.bit_length() + 7) // 8, 'big'))
The above program defines a general function solve_rsa_primes to solve two large prime numbers. After it gets d, the same pow function is called to decrypt, and finally the plaintext is converted from a large integer to a byte sequence and printed out. The result of running this program is
STM1:/# find . -name "*lib*2.23*" | xargs ls -alh -rwxr-xr-x 1 root root 9.6K Jan 1 1970 ./lib/libanl-2.23.so -rwxr-xr-x 1 root root 1.1M Jan 1 1970 ./lib/libc-2.23.so -rwxr-xr-x 1 root root 177.5K Jan 1 1970 ./lib/libcidn-2.23.so -rwxr-xr-x 1 root root 29.5K Jan 1 1970 ./lib/libcrypt-2.23.so -rwxr-xr-x 1 root root 9.5K Jan 1 1970 ./lib/libdl-2.23.so -rwxr-xr-x 1 root root 429.4K Jan 1 1970 ./lib/libm-2.23.so -rwxr-xr-x 1 root root 65.8K Jan 1 1970 ./lib/libnsl-2.23.so -rwxr-xr-x 1 root root 17.5K Jan 1 1970 ./lib/libnss_dns-2.23.so -rwxr-xr-x 1 root root 33.6K Jan 1 1970 ./lib/libnss_files-2.23.so -rwxr-xr-x 1 root root 90.5K Jan 1 1970 ./lib/libpthread-2.23.so -rwxr-xr-x 1 root root 65.7K Jan 1 1970 ./lib/libresolv-2.23.so -rwxr-xr-x 1 root root 25.9K Jan 1 1970 ./lib/librt-2.23.so -rwxr-xr-x 1 root root 9.5K Jan 1 1970 ./lib/libutil-2.23.so
STM2:/# find . -name "*lib*0.9.33*" | xargs ls -alh -rwxr-xr-x 1 root root 28.0K Jan 1 1970 ./lib/ld-uClibc-0.9.33.2.so -rwxr-xr-x 1 root root 36.1K Jan 1 1970 ./lib/libcrypt-0.9.33.2.so -rwxr-xr-x 1 root root 16.2K Jan 1 1970 ./lib/libdl-0.9.33.2.so -rwxr-xr-x 1 root root 72.1K Jan 1 1970 ./lib/libm-0.9.33.2.so -rwxr-xr-x 1 root root 116.4K Jan 1 1970 ./lib/libpthread-0.9.33.2.so -rwxr-xr-x 1 root root 16.2K Jan 1 1970 ./lib/librt-0.9.33.2.so -rwxr-xr-x 1 root root 28.3K Jan 1 1970 ./lib/libthread_db-0.9.33.2.so -rwxr-xr-x 1 root root 621.4K Jan 1 1970 ./lib/libuClibc-0.9.33.2.so -rwxr-xr-x 1 root root 8.1K Jan 1 1970 ./lib/libubacktrace-0.9.33.2.so -rwxr-xr-x 1 root root 4.1K Jan 1 1970 ./lib/libutil-0.9.33.2.so
IPv6 and Interface API
With the steady growth of IPv6 deployment, adding IPv6 protocol stack support for embedded systems has become necessary. In a software project that adds IPv4/IPv6 dual-stack function to devices using uClibc, it is found that there is an application link error - undefined reference to getifaddrs. getifaddrs() is a very useful function, we can call it to get the address information of all the network interfaces of the system. Query the Linux programming manual:
intgetifaddrs(struct ifaddrs **ifap); ... DESCRIPTION The getifaddrs() function creates a linked list of structures describing the network interfaces of the local system, and stores the address of the first item of the list in *ifap. ... VERSIONS The getifaddrs() function first appeared in glibc 2.3, but before glibc 2.3.3, the implementation supported only IPv4 addresses; IPv6 support was added in glibc 2.3.3. Support of address families other than IPv4 is available only on kernels that support netlink. ...
intgetifaddrs(struct ifaddrs **ifap); ... DESCRIPTION The getifaddrs() function creates a linked list of structures describing the network interfaces of the local system, and stores the address of the first item of the list in *ifap. ... VERSIONS The getifaddrs() function first appeared in glibc 2.3, but before glibc 2.3.3, the implementation supported only IPv4 addresses; IPv6 support was added in glibc 2.3.3. Support of address families other than IPv4 is available only on kernels that support netlink. ...
The last sentence above is key: only kernels supporting netlink can support address families other than IPv4. The Linux kernel version running on this system is 3.x, which supports netlink. So, could there be a problem with uClibc's support for netlink that causes getifaddrs() not to get compiled?
With this question in mind, search the source code directory of uClibc and find the C file that implements the function getifaddrs():
... #if __ASSUME_NETLINK_SUPPORT #ifdef __UCLIBC_SUPPORT_AI_ADDRCONFIG__ /* struct to hold the data for one ifaddrs entry, so we can allocate everything at once. */ structifaddrs_storage { structifaddrsifa; union { /* Save space for the biggest of the four used sockaddr types and avoid a lot of casts. */ structsockaddrsa; structsockaddr_llsl; structsockaddr_ins4; #ifdef __UCLIBC_HAS_IPV6__ structsockaddr_in6s6; #endif } addr, netmask, broadaddr; char name[IF_NAMESIZE + 1]; }; #endif/* __UCLIBC_SUPPORT_AI_ADDRCONFIG__ */ ... #ifdef __UCLIBC_SUPPORT_AI_ADDRCONFIG__ ... int getifaddrs(struct ifaddrs **ifap) ... #endif/* __UCLIBC_SUPPORT_AI_ADDRCONFIG__ */ ... #endif/* __ASSUME_NETLINK_SUPPORT */
... #if __ASSUME_NETLINK_SUPPORT #ifdef __UCLIBC_SUPPORT_AI_ADDRCONFIG__ /* struct to hold the data for one ifaddrs entry, so we can allocate everything at once. */ structifaddrs_storage { structifaddrsifa; union { /* Save space for the biggest of the four used sockaddr types and avoid a lot of casts. */ structsockaddrsa; structsockaddr_llsl; structsockaddr_ins4; #ifdef __UCLIBC_HAS_IPV6__ structsockaddr_in6s6; #endif } addr, netmask, broadaddr; char name[IF_NAMESIZE + 1]; }; #endif/* __UCLIBC_SUPPORT_AI_ADDRCONFIG__ */ ... #ifdef __UCLIBC_SUPPORT_AI_ADDRCONFIG__ ... int getifaddrs(struct ifaddrs **ifap) ... #endif/* __UCLIBC_SUPPORT_AI_ADDRCONFIG__ */ ... #endif/* __ASSUME_NETLINK_SUPPORT */
Just as expected! The implementation of the entire function and the definition of the associated data structure ifaddrs_storageare are placed under three nested conditional compilation directives with macros defined as
__ASSUME_NETLINK_SUPPORT
@@ -350,7 +350,7 @@
IPv6 and Interface API
1 2 3 4 5 6 7 8 9 10
--- a/toolchain/uClibc/config-0.9.33.2/common +++ b/toolchain/uClibc/config-0.9.33.2/common @@ -147,7 +147,8 @@ UCLIBC_HAS_RPC=y UCLIBC_HAS_FULL_RPC=y -# UCLIBC_HAS_IPV6 is not set +UCLIBC_HAS_IPV6=y -# UCLIBC_USE_NETLINK is not set +UCLIBC_USE_NETLINK=y +UCLIBC_SUPPORT_AI_ADDRCONFIG=y UCLIBC_HAS_BSD_RES_CLOSE=y
SHA-2 Hash Function
Embedded systems often need to provide remote SSH login services for system administrators, which requires the creation of system users and their passwords. Linux saves the user name and the hashed password in the /etc/shadow file. The storage format of the hash value follows a de facto standard called the Modular Crypt Format (MCF for short), and its format is as follows:
id: indicates the identifier of the hash algorithm (eg 1 for MD5, 5 for SHA-256, 6 for SHA-512)
@@ -360,16 +360,16 @@
SHA-2 Hash Function
With the rapid increase of computing power following Moore's Law, the previously commonly used MD5-based hashing scheme has become obsolete because it is too vulnerable to attack. Newly designed systems are now switched to the SHA-512 hashing scheme, corresponding to $6$ seen in the /etc/shadow file.
Both generation and verification of user password hash values can be implemented with the POSIX C library function named crypt. This function is defined as follows:
-
1
char *crypt(constchar *key, constchar *salt)
+
1
char *crypt(constchar *key, constchar *salt)
The input parameter key points to the string containing the user's password, and salt points to a string in the format $<id>$<salt> indicating the hash algorithm and salt to be used. Most Linux distributions use the crypt function provided by the glibc library. The following figure summarizes the augmented crypt function in Glibc:
In an embedded Linux system integrating uClibc, uClibc provides support for the crypt function. But the test found that it returned a null pointer for the correct \(6\) input! What's going on here?
The answer lies in the uClibc's implementation of the crypt function. Find the corresponding C source code:
if (salt[0] == '$') { if (salt[1] && salt[2] == '$') { /* no blowfish '2X' here ATM */ if (*++salt == '1') return __md5_crypt(ukey, usalt); #ifdef __UCLIBC_HAS_SHA256_CRYPT_IMPL__ elseif (*salt == '5') return __sha256_crypt(ukey, usalt); #endif #ifdef __UCLIBC_HAS_SHA512_CRYPT_IMPL__ elseif (*salt == '6') return __sha512_crypt(ukey, usalt); #endif } /* __set_errno(EINVAL);*//* ENOSYS might be misleading */ returnNULL; } return __des_crypt(ukey, usalt); }
Aha! It turns out that it only does MD5 hashing by default, and the codes of SHA-256 and SHA-512 need their own conditional compilation macro definitions. This is easy to handle, just edit the configuration file of uClibc and open the latter two.
1 2 3 4 5 6 7 8 9
--- a/toolchain/uClibc/config-0.9.33.2/common +++ b/toolchain/uClibc/config-0.9.33.2/common @@ -151,8 +151,8 @@ UCLIBC_HAS_REGEX_OLD=y UCLIBC_HAS_RESOLVER_SUPPORT=y -# UCLIBC_HAS_SHA256_CRYPT_IMPL is not set -# UCLIBC_HAS_SHA512_CRYPT_IMPL is not set +UCLIBC_HAS_SHA256_CRYPT_IMPL=y +UCLIBC_HAS_SHA512_CRYPT_IMPL=y UCLIBC_HAS_SHADOW=y
Finally, take a look at the program that comes with uClibc to test the SHA-512 hash algorithm. It clearly lists the data structures defined by the test code, including the salt, the input password, and the expected output, as well as several test vectors:
-
test/crypt/sha512c-test.c
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
staticconststruct { constchar *salt; constchar *input; constchar *expected; } tests[] = { { "$6$saltstring", "Hello world!", "$6$saltstring$svn8UoSVapNtMuq1ukKS4tPQd8iKwSMHWjl/O817G3uBnIFNjnQJu" "esI68u4OTLiBFdcbYEdFCoEOfaS35inz1" }, { "$6$rounds=10000$saltstringsaltstring", "Hello world!", "$6$rounds=10000$saltstringsaltst$OW1/O6BYHV6BcXZu8QVeXbDWra3Oeqh0sb" "HbbMCVNSnCM/UrjmM0Dp8vOuZeHBy/YTBmSK6H9qs/y3RnOaw5v." }, ... { "$6$rounds=10$roundstoolow", "the minimum number is still observed", "$6$rounds=1000$roundstoolow$kUMsbe306n21p9R.FRkW3IGn.S9NPN0x50YhH1x" "hLsPuWGsUSklZt58jaTfF4ZEQpyUNGc0dqbpBYYBaHHrsX." }, };
+
test/crypt/sha512c-test.c
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
staticconststruct { constchar *salt; constchar *input; constchar *expected; } tests[] = { { "$6$saltstring", "Hello world!", "$6$saltstring$svn8UoSVapNtMuq1ukKS4tPQd8iKwSMHWjl/O817G3uBnIFNjnQJu" "esI68u4OTLiBFdcbYEdFCoEOfaS35inz1" }, { "$6$rounds=10000$saltstringsaltstring", "Hello world!", "$6$rounds=10000$saltstringsaltst$OW1/O6BYHV6BcXZu8QVeXbDWra3Oeqh0sb" "HbbMCVNSnCM/UrjmM0Dp8vOuZeHBy/YTBmSK6H9qs/y3RnOaw5v." }, ... { "$6$rounds=10$roundstoolow", "the minimum number is still observed", "$6$rounds=1000$roundstoolow$kUMsbe306n21p9R.FRkW3IGn.S9NPN0x50YhH1x" "hLsPuWGsUSklZt58jaTfF4ZEQpyUNGc0dqbpBYYBaHHrsX." }, };
It can be seen that the last test case defines the round value 10 ($6$rounds=10$roundstoolow), while the output shows that the round is 1000 (rounds=1000). This confirms that the crypt function implementation of uClibc matches the augmented function of Glibc - in order to ensure security, if the input specified round is too small, crypt will automatically set to the minimum round of 1000.
DNS Security Patch
In early May 2022, Nozomi Networks, a company focused on providing security solutions for industrial and critical infrastructure environments, released a newly discovered uClibc security vulnerability CVE-2022-30295. This vulnerability exists in the Domain Name System (DNS) implementation of all versions of uClibc and its fork uClibc-ng (prior to version 1.0.41). Since the implementation uses predictable transaction IDs when making DNS requests, there is a risk of DNS cache poisoning attacks.
The user \(i\) can decompose \(N\) using his own key pair \((e_i,d_i)\). Whether \(d\) is generated using the Euler function \(\varphi(N)\) or the Carmichael function \(\lambda(N)\), there are algorithms that quickly derive the prime factors \(p\) and \(q\) from a given \(d\)2. And once \(p\) and \(q\) are known, user \(i\) can compute any other user's private key \(d_j\) with one's public key \((N,e_j)\). At this point, the other users have no secrets from user \(i\).
Even if all users do not have the knowledge and skill to decompose \(N\), or are "nice" enough not to know the other users' private keys, a hacker can still perform a common modulus attack to break the users' messages. If the public keys of two users, Alice and Bob, are \(e_1\) and \(e_2\), and \(e_1\) and \(e_2\) happen to be mutually prime (which is very likely), then by Bézout's identity, the eavesdropper Eve can find that \(s\) and \(t\) satisfy: \[e_{1}s+e_{2}t=gcd(e_1,e_2)=1\] At this point, if someone sends the same message \(m\) to Alice and Bob, Eve can decrypt \(m\) after recording the two ciphertexts \(c_1\) and \(c_2\) and performing the following operation: \[c_1^s⋅c_2^t\equiv(m^{e _1})^s⋅(m^{e_2})^t\equiv m^{e_{1}s+e_{2}t}\equiv m\pmod N\] The corresponding Python function code is shown below.
-
1 2 3 4 5 6 7 8 9 10 11 12 13 14
defcommon_modulus(e1, e2, N, c1, c2): # Call the extended Euclidean algorithm function g, s, t = gymp2.gcdext(e1, e2) assert g == 1 if s < 0: # Find c1's modular multiplicative inverse re = int(gmpy2.invert(c1, N)) c1 = pow(re, s*(-1), N) c2 = pow(c2, t, N) else: # t is negative, find c2's modular multiplicative inverse re = int(gmpy2.invert(c2, N)) c2 = pow(re, t*(-1), N) c1 = pow(c1, a, N) return (c1*c2) % N
Two library functions of gmpy23 are called here: gcdext() to implement the extended Euclidean algorithm, and invert() to find the modular multiplicative inverse element. Note that Python's exponential function pow() supports modular exponentiation, but the exponent must not be negative. Since one of \(s\) or \(t\) must be negative, you have to first call invert() to convert \(c_1\) or \(c_2\) to the corresponding modular multiplicative inverse, then invert the negative number to calculate the modular exponent. For example, lines 7 and 8 above implement \(c_1^s=(c_1^{-1})^{-s}\bmod N\).
+
1 2 3 4 5 6 7 8 9 10 11 12 13 14
defcommon_modulus(e1, e2, N, c1, c2): # Call the extended Euclidean algorithm function g, s, t = gymp2.gcdext(e1, e2) assert g == 1 if s < 0: # Find c1's modular multiplicative inverse re = int(gmpy2.invert(c1, N)) c1 = pow(re, s*(-1), N) c2 = pow(c2, t, N) else: # t is negative, find c2's modular multiplicative inverse re = int(gmpy2.invert(c2, N)) c2 = pow(re, t*(-1), N) c1 = pow(c1, a, N) return (c1*c2) % N
Two library functions of gmpy23 are called here: gcdext() to implement the extended Euclidean algorithm, and invert() to find the modular multiplicative inverse element. Note that Python's exponential function pow() supports modular exponentiation, but the exponent must not be negative. Since one of \(s\) or \(t\) must be negative, you have to first call invert() to convert \(c_1\) or \(c_2\) to the corresponding modular multiplicative inverse, then invert the negative number to calculate the modular exponent. For example, lines 7 and 8 above implement \(c_1^s=(c_1^{-1})^{-s}\bmod N\).
Is it possible to reuse only \(p\) or \(q\) since the shared modulus \(N\) is proven to be insecure? This seems to avoid the common-modulus attack and ensure that each user's public key \(N\) value is unique. Big mistake! This is an even worse idea! The attacker gets the public \(N\) values of all users and simply combines \((N_1,N_2)\) pairwise to solve Euclid's algorithm for the great common divisor, and a successful solution gives a prime factor \(p\), and a simple division gives the other prime factor \(q\). With \(p\) and \(q\), the attacker can immediately compute the user's private key \(d\). This is the non-coprime modulus attack.
When applying textbook RSA, if both the public exponent \(e\) and the plaintext \(m\) are small, such that \(c=m^e<N\), the plaintext \(m\) can be obtained by directly calculating the \(e\)th root of the ciphertext \(c\). Even if \(m^e>N\) but not large enough, then since \(m^e=c+k⋅N\), you can loop through the small \(k\) values to perform brute-force root extraction cracking. Here is the Python routine:
-
1 2 3 4 5 6 7 8 9 10
defcrack_small(c, e, N, repeat) times = 0 msg = 0 forkinrange(repeat): m, is_exact = gmpy2.iroot(c + times, e) if is_exact andpow(m, e, N) == c: msg = int(m) break times += N return msg
Here the gmpy2 library function iroot() is called to find the \(e\)th root.
+
1 2 3 4 5 6 7 8 9 10
defcrack_small(c, e, N, repeat) times = 0 msg = 0 for k inrange(repeat): m, is_exact = gmpy2.iroot(c + times, e) if is_exact andpow(m, e, N) == c: msg = int(m) break times += N return msg
Here the gmpy2 library function iroot() is called to find the \(e\)th root.
Textbook RSA is deterministic, meaning that the same plaintext \(m\) always generates the same ciphertext \(c\). This makes codebook attack possible: the attacker precomputes all or part of the \(m\to c\) mapping table and saves, then simply searches the intercepted ciphertext for a match. Determinism also means that textbook RSA is not semantically secure and that the ciphertext can reveal some information about the plaintext. Repeated occurrences of the ciphertext indicate that the sender is sending the same message over and over again.
Textbook RSA is malleable, where a particular form of algebraic operation is performed on the ciphertext and the result is reflected in the decrypted plaintext. For example, if there are two plaintexts \(m_1\) and \(m_2\), and encryption yields \(c_1=m_1^e\bmod N\) and \(c_2=m_2^e\bmod N\), what does \((c_1⋅c_2)\) decryption yield? Look at the following equation: \[(c_1⋅c_2)^d\equiv m_1^{ed}⋅m_2^{ed}\equiv m_1⋅m_2\pmod N\] So the plaintext obtained after decrypting the product of the two ciphertexts is equal to the product of the two plaintexts. This feature is detrimental to RSA encryption systems in general and provides an opportunity for chosen-ciphertext attack. The following are two examples of attack scenarios:
@@ -438,7 +438,7 @@
Broadcast Attack
Håstad further proves that even if padding is used to prevent broadcast attacks, if the messages generated by the padding scheme are linearly related to each other, such as using the formula \(m_i=i2^b+m\) (\(b\) is the number of bits of \(m\)) to generate the message sent to the receiver \(i\), then the broadcast attack can still recover the plaintext \(m\) as long as \(k>e\). The broadcast attack in this case is still based on the Chinese remainder theorem, but the specific cracking method depends on the information of the linear relationship.
To summarize the above analysis, to prevent the broadcast attack, we must use a higher public exponent \(e\) and apply random padding at the same time. Nowadays, the common public key exponent \(e\) is $65537 (\(2^{16}+1\)), which can balance the efficiency and security of message encryption or signature verification operations.
Last, Python routines for simulating broadcast attacks are given as follows:
defsolve_crt(ai: list, mi: list): '''mi and ai are the list of modulus and remainders. The precondition of the function is that the modulus in the mi list are pairwise coprime.''' M = reduce(lambda x, y: x * y, mi) ti = [a * (M//m) * int(gmpy2.invert(M//m, m)) for (m, a) inzip(mi, ai)] return reduce(lambda x, y: x + y, ti) % M
defuint_to_bytes(x: int) -> bytes: '''convert unsigned integer to byte array''' if x == 0: returnbytes(1) return x.to_bytes((x.bit_length() + 7) // 8, 'big')
quote = b'The cosmos is within us. We are made of star stuff. - Carl Sagan' bob = RSA(1024, 3) carol = RSA(1024, 3) dave = RSA(1024, 3) cipher_list = [bob.encrypt(quote), carol.encrypt(quote), dave.encrypt(quote)] modulus_list = [bob.n, carol.n, dave.n]
defsolve_crt(ai: list, mi: list): '''mi and ai are the list of modulus and remainders. The precondition of the function is that the modulus in the mi list are pairwise coprime.''' M = reduce(lambda x, y: x * y, mi) ti = [a * (M//m) * int(gmpy2.invert(M//m, m)) for (m, a) inzip(mi, ai)] return reduce(lambda x, y: x + y, ti) % M
defuint_to_bytes(x: int) -> bytes: '''convert unsigned integer to byte array''' if x == 0: returnbytes(1) return x.to_bytes((x.bit_length() + 7) // 8, 'big')
quote = b'The cosmos is within us. We are made of star stuff. - Carl Sagan' bob = RSA(1024, 3) carol = RSA(1024, 3) dave = RSA(1024, 3) cipher_list = [bob.encrypt(quote), carol.encrypt(quote), dave.encrypt(quote)] modulus_list = [bob.n, carol.n, dave.n]
This code uses two methods to simulate the broadcast attack. One calls the generic Chinese remainder theorem solver function solve_crt() and then gets the cube root of the result; the other calls the special broadcast attack function rsa_broadcast_attack() for the public key index \(e=3\), which directly outputs the cracked plaintext value. The internal implementation of these two functions is based on the generalized formula of the Chinese remainder theorem, and the output results should be identical. The cracked plaintext value is then input to the uint_to_bytes() function, which is converted into a byte array to compare with the original quote. Note that the program uses objects generated by the RSA class to simulate the receivers Bob, Carroll, and Dave, and the implementation of the RSA class is omitted here given the limitation of space.
To be continued, please look forward to the next part: RSA: Attack and Defense (2)
Lighttpd is a lightweight open-source web server software. It focuses on high performance, low memory footprint, and fast responsiveness. Lighttpd is suitable for serving web applications and static content of all sizes. Its design goal is to provide an efficient, flexible, and scalable web server, especially suitable for high-load and resource-constrained (such as embedded systems) environments.
The first Lighttpd release to support TLS 1.3 is version 1.4.56. Starting with this version, the minimum version of TLS that Lighttpd supports by default is TLS 1.2. That is to say, Lighttpd supports TLS 1.2 and TLS 1.3 if no corresponding configuration file modification is made.
To limit the use of Lighttpd to only the TLS 1.3 feature, first make sure the mod_openssl module is loaded. Then in the configuration file lighttpd.conf, find the server.modules section, and add the following ssl.openssl.ssl-conf-cmd line:
This will set the minimum version supported by Lighttpd to be TLS 1.3. Finally, save and reload the Lighttpd configuration for the changes to take effect:
diff --git a/archives/2021/12/index.html b/archives/2021/12/index.html
index 99741cef..5a151a24 100644
--- a/archives/2021/12/index.html
+++ b/archives/2021/12/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/archives/2021/index.html b/archives/2021/index.html
index ba196499..169dd65a 100644
--- a/archives/2021/index.html
+++ b/archives/2021/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/archives/2022/01/index.html b/archives/2022/01/index.html
index 887e109b..65b67acb 100644
--- a/archives/2022/01/index.html
+++ b/archives/2022/01/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/archives/2022/03/index.html b/archives/2022/03/index.html
index a17ca96a..94cce1b1 100644
--- a/archives/2022/03/index.html
+++ b/archives/2022/03/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/archives/2022/04/index.html b/archives/2022/04/index.html
index 4d81ef33..be35cb95 100644
--- a/archives/2022/04/index.html
+++ b/archives/2022/04/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/archives/2022/08/index.html b/archives/2022/08/index.html
index 89b92aec..584273f1 100644
--- a/archives/2022/08/index.html
+++ b/archives/2022/08/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/archives/2022/11/index.html b/archives/2022/11/index.html
index e9ce2075..db193493 100644
--- a/archives/2022/11/index.html
+++ b/archives/2022/11/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/archives/2022/index.html b/archives/2022/index.html
index 81e4c90f..867f6799 100644
--- a/archives/2022/index.html
+++ b/archives/2022/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/archives/2023/03/index.html b/archives/2023/03/index.html
index 3dfa4767..209f9ab3 100644
--- a/archives/2023/03/index.html
+++ b/archives/2023/03/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/archives/2023/08/index.html b/archives/2023/08/index.html
index c4c5d0ea..b96a9767 100644
--- a/archives/2023/08/index.html
+++ b/archives/2023/08/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/archives/2023/index.html b/archives/2023/index.html
index 6add53ea..a016f3da 100644
--- a/archives/2023/index.html
+++ b/archives/2023/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/archives/index.html b/archives/index.html
index 25a32f24..7e4ba8cc 100644
--- a/archives/index.html
+++ b/archives/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/archives/page/2/index.html b/archives/page/2/index.html
index d3fd30ee..d2d7d3b7 100644
--- a/archives/page/2/index.html
+++ b/archives/page/2/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/atom.xml b/atom.xml
index a20a109a..59ad72fd 100644
--- a/atom.xml
+++ b/atom.xml
@@ -81,10 +81,10 @@
-
-
+
+
@@ -222,10 +222,10 @@
https://www.packetmania.net/en/2021/12/29/RPi-NAS-Plex/2021-12-29T18:53:08.000Z
- 2023-11-11T02:10:14.447Z
+ 2023-11-11T02:20:03.171Z
- <p>Network Attached Storage (NAS) provides data access to a heterogeneous group of clients over computer networks. As hard drive prices continue to drop, NAS devices have made their way into the homes of the masses. Leading brands in the SMB and home NAS market, such as Synology, have their products range in price from as low as $300 to $ 700 for the high models. But if you are a Raspberry Pi player, you can build a very nice home NAS and streaming service for only about half the cost of the lowest price.
+ <p>Network Attached Storage (NAS) provides data access to a heterogeneous group of clients over computer networks. As hard drive prices continue to drop, NAS devices have made their way into the homes of the masses. Leading brands in the SMB and home NAS market, such as Synology, have their products range in price from as low as $300 to $700 for the high models. But if you are a Raspberry Pi player, you can build a very nice home NAS and streaming service for only about half the cost of the lowest price.
diff --git a/categories/DIY-Projects/index.html b/categories/DIY-Projects/index.html
index 2cfb1034..ba308f29 100644
--- a/categories/DIY-Projects/index.html
+++ b/categories/DIY-Projects/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/categories/Study-Notes/index.html b/categories/Study-Notes/index.html
index 095bdeb6..dff60c4b 100644
--- a/categories/Study-Notes/index.html
+++ b/categories/Study-Notes/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/categories/Technical-Know-how/index.html b/categories/Technical-Know-how/index.html
index 0b376c21..e9282979 100644
--- a/categories/Technical-Know-how/index.html
+++ b/categories/Technical-Know-how/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/categories/Technology-Review/index.html b/categories/Technology-Review/index.html
index 08008b3b..d1e0ef57 100644
--- a/categories/Technology-Review/index.html
+++ b/categories/Technology-Review/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/categories/Tool-Guide/index.html b/categories/Tool-Guide/index.html
index 6b57133e..f63c5229 100644
--- a/categories/Tool-Guide/index.html
+++ b/categories/Tool-Guide/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/categories/index.html b/categories/index.html
index 27597e5e..5afb50e9 100644
--- a/categories/index.html
+++ b/categories/index.html
@@ -5,7 +5,7 @@
-
+
diff --git a/index.html b/index.html
index c945abab..700118d3 100644
--- a/index.html
+++ b/index.html
@@ -5,7 +5,7 @@
-
+
@@ -272,7 +272,7 @@
Edited on
-
+
@@ -309,7 +309,7 @@
-
Network Attached Storage (NAS) provides data access to a heterogeneous group of clients over computer networks. As hard drive prices continue to drop, NAS devices have made their way into the homes of the masses. Leading brands in the SMB and home NAS market, such as Synology, have their products range in price from as low as $300 to $ 700 for the high models. But if you are a Raspberry Pi player, you can build a very nice home NAS and streaming service for only about half the cost of the lowest price.
+
Network Attached Storage (NAS) provides data access to a heterogeneous group of clients over computer networks. As hard drive prices continue to drop, NAS devices have made their way into the homes of the masses. Leading brands in the SMB and home NAS market, such as Synology, have their products range in price from as low as $300 to $700 for the high models. But if you are a Raspberry Pi player, you can build a very nice home NAS and streaming service for only about half the cost of the lowest price.
ASan itself cannot detect heap memory leaks. But when ASan is integrated into the compiler, as it replaces the memory allocation/free functions, the original leak detection feature of the compiler tool is consolidated with ASan. So, adding the ASan option to the compilation command line also turns on the leak detection feature by default.
\n
\n
This covers all common memory access errors except for \"uninitialized memory reads\" (UMR). ASan detects them with a false positive rate of 0, which is quite impressive. In addition, ASan detects several C++-specific memory access errors such as
\n
\n
Initialization Order Fiasco: When two static objects are defined in different source files and the constructor of one object calls the method of the other object, a program crash will occur if the former compilation unit is initialized first.
\n
Container Overflow: Given libc++/libstdc++ container, access [container.end(), container.begin() + container.capacity())], which crosses the [container.begin(), container.end()] range but still within the dynamically allocated memory area.
\n
Delete Mismatch: For the array object created by new foo[n], should not call delete foo for deletion, use delete [] foo instead.
\n
\n
ASan's high reliability and performance have made it the preferred choice of compiler and IDE developers since its introduction. Today ASan is integrated into all four major compilation toolsets:
ASan's developers first used the Chromium open-source browser for routine testing and found more than 300 memory access errors over 10 months. After integration into mainstream compilation tools, it reported long-hidden bugs in numerous popular open-source software, such as Mozilla Firefox, Perl, Vim, PHP, and MySQL. Interestingly, ASan also identified some memory access errors in the LLVM and GCC compilers' code. Now, many software companies have added ASan run to their mandatory quality control processes.
\n
Working Principle
\n
The USENIX conference paper 3, published by Serebryany in 2012, comprehensively describes the design principles, algorithmic ideas, and programming implementation of ASan. In terms of the overall structure, ASan consists of two parts.
\n\n
Compiler instrumentation - modifies the code to verify the shadow memory state at each memory access and creates poisoned red zones at the edges of global and stack objects to detect overflows or underflows.
\n
Runtime library replacement - replaces malloc/free and its related functions to create poisoned red zones at the edge of dynamically allocated heap memory regions, delay the reuse of memory regions after release, and generate error reports.
\n\n
Here shadow memory, compiler instrumentation, and memory allocation function replacement are all previously available techniques, so how has ASan innovatively applied them for efficient error detection? Let's take a look at the details.
\n
Shadow Memory
\n
Many inspection tools use separated shadow memory to record metadata about program memory, and then apply instrumentation to check the shadow memory during memory accesses to confirm that reads and writes are safe. The difference is that ASan uses a more efficient direct mapping shadow memory.
\n
The designers of ASan noted that typically the malloc function returns a memory address that is at least 8-byte aligned. For example, a request for 20 bytes of memory would divide 24 bytes of memory, with the last 3 bits of the actual return pointer being all zeros. in addition, any aligned 8-byte sequence would only have 9 different states: the first \\(k\\,(0\\leq k \\leq 8)\\) bytes are accessible, and the last \\(8-k\\) are not. From this, they came up with a more compact shadow memory mapping and usage scheme:
\n
\n
Reserve one-eighth of the virtual address space for shadow memory
\n
Directly map application memory to shadow memory using a formula that divides by 8 plus an offset\n
Each byte of shadow memory records one of the 9 states of the corresponding 8-byte memory block\n
\n
0 means all 8 bytes are addressable
\n
Any negative value indicates that the entire 8-byte word is unaddressable (poisoned )
\n
k (1 ≤ k ≤ 7) means that the first k bytes are addressable
\n
\n
\n
The following figure shows the address space layout and mapping relationship of ASan. Pay attention to the Bad area in the middle, which is the address segment after the shadow memory itself is mapped. Because shadow memory is not visible to the application, ASan uses a page protection mechanism to make it inaccessible.
\n
\n
Compiler Instrumentation
\n
Once the shadow memory design is determined, the implementation of compiler instrumentation to detect dynamic memory access errors is easy. For memory accesses of 8 bytes, the shadow memory bytes are checked by inserting instructions before the original read/write code, and an error is reported if they are not zero. For memory accesses of less than 8 bytes, the instrumentation is a bit more complicated, where the shadow memory byte values are compared with the last three bits of the read/write address. This situation is also known as the \"slow path\" and the sample code is as follows.
\n
// Check the cases where we access first k bytes of the qword // and these k bytes are unpoisoned. boolSlowPathCheck(shadow_value, address, kAccessSize){ last_accessed_byte = (address & 7) + kAccessSize - 1; return (last_accessed_byte >= shadow_value); } ...
For global and stack (local) objects, ASan has designed different instrumentation to detect their out-of-bounds access errors. The red zone around a global object is added by the compiler at compile time and its address is passed to the runtime library at application startup, where the runtime library function then poisons the red zone and writes down the address needed in error reporting. The stack object is created at function call time, and accordingly, its red zone is created and poisoned at runtime. In addition, because the stack object is deleted when the function returns, the instrumentation code must also zero out the shadow memory it is mapped to.
\n
In practice, the ASan compiler instrumentation process is placed at the end of the compiler optimization pipeline so that instrumentation only applies to the remaining memory access instructions after variable and loop optimization. In the latest GCC distribution, the ASan compiler stubbing code is located in two files in the gcc subdirectory gcc/asan.[ch].
\n
Runtime Library Replacement
\n
The runtime library needs to include code to manage shadow memory. The address segment to which shadow memory itself is mapped is to be initialized at application startup to disable access to shadow memory by other parts of the program. The runtime library replaces the old memory allocation and free functions and also adds some error reporting functions such as __asan_report_load8.
\n
The newly replaced memory allocation function malloc will allocate additional storage as a red zone before and after the requested memory block and set the red zone to be non-addressable. This is called the poisoning process. In practice, because the memory allocator maintains a list of available memory corresponding to different object sizes, if the list of a certain object is empty, the OS will allocate a large set of memory blocks and their red zones at once. As a result, the red zones of the preceding and following memory blocks will be connected, as shown in the following figure, where \\(n\\) memory blocks require only \\(n+1\\) red zones to be allocated.
\n
\n
The new free function needs to poison the entire storage area and place it in a quarantine queue after the memory is freed. This prevents the memory region from being allocated any time soon. Otherwise, if the memory region is reused immediately, there is no way to detect incorrect accesses to the recently freed memory. The size of the quarantine queue determines how long the memory region is in quarantine, and the larger it is the better its capability of detecting UAF errors!
\n
By default, both the malloc and free functions log their call stacks to provide more detailed information in the error reports. The call stack for malloc is kept in the red zone to the left of the allocated memory, so a large red zone can retain more call stack frames. The call stack for free is stored at the beginning of the allocated memory region itself.
\n
Integrated into the GCC compiler, the source code for the ASan runtime library replacement is located in the libsanitizer subdirectory libsanitizer/asan/*, and the resulting runtime library is compiled as libasan.so.
\n
Application Examples
\n
ASan is very easy to use. The following is an example of an Ubuntu Linux 20.4 + GCC 9.3.0 system running on an x86_64 virtual machine to demonstrate the ability to detect various memory access errors.
\n
Test Cases
\n
As shown below, the test program writes seven functions, each introducing a different error type. The function names are cross-referenced with the error types one by one:
intstack_buffer_overflow(){ int s[10]; s[0] = 10; return s[10]; }
int *gp;
voidstack_use_after_return(){ int r[10]; r[0] = 10; gp = &r[0]; return; }
voidstack_use_after_scope(){ { int c = 0; gp = &c; } *gp = 10; return; }
\n
The test program calls the getopt library function to support a single-letter command line option that allows the user to select the type of error to be tested. The command line option usage information is as follows.
\n
\b$ ./asan-test
Test AddressSanitizer usage: asan-test [ -bfloprs ]
-b\theap buffer overflow -f\theap use after free -l\theap memory leak -o\tglobal buffer overflow -p\tstack use after scope -r\tstack use after return -s\tstack buffer overflow
\n
The GCC compile command for the test program is simple, just add two compile options
\n
\n
-fsanitize=address: activates the ASan tool
\n
-g: enable debugging and keep debugging information
\n
\n
OOB Test
\n
For Heap OOB error, the run result is
\n
$ ./asan-test -b ================================================================= ==57360==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x604000000038 at pc 0x55bf46fd64ed bp 0x7ffced908dc0 sp 0x7ffced908db0 READ of size 4 at 0x604000000038 thread T0 #0 0x55bf46fd64ec in heap_buffer_overflow /home/zixi/coding/asan-test.c:34 #1 0x55bf46fd6a3f in main /home/zixi/coding/asan-test.c:88 #2 0x7fd16f6560b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) #3 0x55bf46fd628d in _start (/home/zixi/coding/asan-test+0x128d)
0x604000000038 is located 0 bytes to the right of 40-byte region [0x604000000010,0x604000000038) allocated by thread T0 here: #0 0x7fd16f92ebc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8) #1 0x55bf46fd646c in heap_buffer_overflow /home/zixi/coding/asan-test.c:32 #2 0x55bf46fd6a3f in main /home/zixi/coding/asan-test.c:88 #3 0x7fd16f6560b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
SUMMARY: AddressSanitizer: heap-buffer-overflow /home/zixi/coding/asan-test.c:34 in heap_buffer_overflow Shadow bytes around the buggy address: 0x0c087fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c087fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c087fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c087fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c087fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0c087fff8000: fa fa 00 00 00 00 00[fa]fa fa fa fa fa fa fa fa 0x0c087fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c087fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c087fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c087fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c087fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd ... ==57360==ABORTING
\n
Referring to the heap-buffer-overflow function implementation, you can see that it requests 40 bytes of memory to hold 10 32-bit integers. However, on the return of the function, the code overruns to read the data after the allocated memory. As the above run log shows, the program detects a Heap OOB error and aborts immediately. ASan reports the name of the source file and line number asan-test.c:34 where the error occurred, and also accurately lists the original allocation function call stack for dynamically allocated memory. The \"SUMMARY\" section of the report also prints the shadow memory data corresponding to the address in question (observe the lines marked by =>). The address to be read is 0x604000000038, whose mapped shadow memory address 0x0c087fff8007 holds the negative value 0xfa (poisoned and not addressable). Because of this, ASan reports an error and aborts the program.
\n
The Stack OOB test case is shown below. ASan reports an out-of-bounds read error for a local object. Since the local variables are located in the stack space, the starting line number asan-test.c:37 of the function stack_buffr_overflow is listed. Unlike the Heap OOB report, the shadow memory poisoning values for the front and back redzone of the local variable are different, with the previous Stack left redzone being 0xf1 and the later Stack right redzone being 0xf3. Using different poisoning values (both negative after 0x80) helps to quickly distinguish between the different error types.
\n
$ ./asan-test -s ================================================================= ==57370==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f1cf5044058 at pc 0x55d8b7e9d601 bp 0x7ffc830c29e0 sp 0x7ffc830c29d0 READ of size 4 at 0x7f1cf5044058 thread T0 #0 0x55d8b7e9d600 in stack_buffer_overflow /home/zixi/coding/asan-test.c:40 #1 0x55d8b7e9daec in main /home/zixi/coding/asan-test.c:108 #2 0x7f1cf87760b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) #3 0x55d8b7e9d28d in _start (/home/zixi/coding/asan-test+0x128d)
Address 0x7f1cf5044058 is located in stack of thread T0 at offset 88 in frame #0 0x55d8b7e9d505 in stack_buffer_overflow /home/zixi/coding/asan-test.c:37
This frame has 1 object(s): [48, 88) 's' (line 38) <== Memory access at offset 88 overflows this variable HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork (longjmp and C++ exceptions *are* supported) SUMMARY: AddressSanitizer: stack-buffer-overflow /home/zixi/coding/asan-test.c:40 in stack_buffer_overflow Shadow bytes around the buggy address: 0x0fe41ea007b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe41ea007c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe41ea007d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe41ea007e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe41ea007f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0fe41ea00800: f1 f1 f1 f1 f1 f1 00 00 00 00 00[f3]f3 f3 f3 f3 0x0fe41ea00810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe41ea00820: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe41ea00830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe41ea00840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe41ea00850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 ... ==57370==ABORTING
\n
The following Global OOB test result also clearly shows the error line asan-test.c:16, the global variable name ga and its definition code location asan-test.c:13:5, and you can also see that the global object has a red zone poisoning value of 0xf9.
\n
$ ./asan-test -o ================================================================= ==57367==ERROR: AddressSanitizer: global-buffer-overflow on address 0x564363ea4048 at pc 0x564363ea1383 bp 0x7ffc0d6085d0 sp 0x7ffc0d6085c0 READ of size 4 at 0x564363ea4048 thread T0 #0 0x564363ea1382 in global_buffer_overflow /home/zixi/coding/asan-test.c:16 #1 0x564363ea1a6c in main /home/zixi/coding/asan-test.c:98 #2 0x7f8cb43890b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) #3 0x564363ea128d in _start (/home/zixi/coding/asan-test+0x128d)
Surprisingly, ASan does not report the obvious Global OOB error here. Why?
\n
The reason has to do with the way GCC treats global variables. The compiler treats functions and initialized variables as Strong symbols, while uninitialized variables are Weak symbols by default. Since the definition of weak symbols may vary from source file to source file, the size of the space required is unknown. The compiler cannot allocate space for weak symbols in the BSS segment, so it uses the COMMON block mechanism so that all weak symbols share a COMMON memory region, thus ASan cannot insert the red zone. During the linking process, after the linker reads all the input target files, it can determine the size of the weak symbols and allocate space for them in the BSS segment of the final output file.
\n
Fortunately, GCC's -fno-common option turns off the COMMON block mechanism, allowing the compiler to add all uninitialized global variables directly to the BSS segment of the target file, also allowing ASan to work properly. This option also disables the linker from merging weak symbols, so the linker reports an error directly when it finds a compiled unit with duplicate global variables defined in the target file.
\n
This is confirmed by a real test. Modify the GCC command line for the previous code segment
then compile, link, and run. ASan successfully reported the Global OOB error.
\n
UAF Test
\n
The following is a running record of UAF error detection. Not only is the information about the code that went wrong reported here, but also the call stack of the original allocation and free functions of the dynamic memory is given. The log shows that the memory was allocated by asan-test.c:25, freed at asan-test.c:27, and yet read at asan-test.c:28. The shadow memory data printed later indicates that the data filled is negative 0xfd, which is also the result of the poisoning of the memory after it is freed.
\n
$ \u0007./asan-test -\bf ================================================================= ==57363==ERROR: AddressSanitizer: heap-use-after-free on address 0x604000000034 at pc 0x558b4a45444e bp 0x7ffccf4ca790 sp 0x7ffccf4ca780 READ of size 4 at 0x604000000034 thread T0 #0 0x558b4a45444d in heap_use_after_free /home/zixi/coding/asan-test.c:28 #1 0x558b4a454a4e in main /home/zixi/coding/asan-test.c:91 #2 0x7fc7cc98b0b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) #3 0x558b4a45428d in _start (/home/zixi/coding/asan-test+0x128d)
0x604000000034 is located 36 bytes inside of 40-byte region [0x604000000010,0x604000000038) freed by thread T0 here: #0 0x7fc7ccc637cf in __interceptor_free (/lib/x86_64-linux-gnu/libasan.so.5+0x10d7cf) #1 0x558b4a454412 in heap_use_after_free /home/zixi/coding/asan-test.c:27 #2 0x558b4a454a4e in main /home/zixi/coding/asan-test.c:91 #3 0x7fc7cc98b0b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
previously allocated by thread T0 here: #0 0x7fc7ccc63bc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8) #1 0x558b4a4543bd in heap_use_after_free /home/zixi/coding/asan-test.c:25 #2 0x558b4a454a4e in main /home/zixi/coding/asan-test.c:91 #3 0x7fc7cc98b0b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
SUMMARY: AddressSanitizer: heap-use-after-free /home/zixi/coding/asan-test.c:28 in heap_use_after_free Shadow bytes around the buggy address: 0x0c087fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c087fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c087fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c087fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c087fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0c087fff8000: fa fa fd fd fd fd[fd]fa fa fa fa fa fa fa fa fa 0x0c087fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c087fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c087fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c087fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c087fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd ... ==57363==ABORTING
\n
HML Test
\n
The results of the memory leak test are as follows. Unlike the other test cases, ABORTING is not printed at the end of the output record. This is because, by default, ASan only generates a memory leak report when the program terminates (process ends). If you want to check for leaks on the fly, you can call ASan's library function __lsan_do_recoverable_leak_check, whose definition is located in the header file sanitizer/lsan_interface.h.
Direct leak of 40 byte(s) in 1 object(s) allocated from: #0 0x7f06b85b1bc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8) #1 0x5574a8bcd3a0 in heap_leak /home/zixi/coding/asan-test.c:20 #2 0x5574a8bcda5d in main /home/zixi/coding/asan-test.c:94 #3 0x7f06b82d90b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
SUMMARY: AddressSanitizer: 40 byte(s) leaked in 1 allocation(s).
\n
UAS Test
\n
See the stack_use_after_scope function code, where the memory unit holding the local variable c is written outside of its scope. The test log accurately reports the line number line 54 where the variable is defined and the location of the incorrect writing code asan-test.c:57:
\n
./asan-test -\bp ================================================================= ==57368==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7f06f0a9b020 at pc 0x56121a7548d9 bp 0x7ffd1de0d050 sp 0x7ffd1de0d040 WRITE of size 4 at 0x7f06f0a9b020 thread T0 #0 0x56121a7548d8 in stack_use_after_scope /home/zixi/coding/asan-test.c:57 #1 0x56121a754a7b in main /home/zixi/coding/asan-test.c:101 #2 0x7f06f42cd0b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) #3 0x56121a75428d in _start (/home/zixi/coding/asan-test+0x128d)
Address 0x7f06f0a9b020 is located in stack of thread T0 at offset 32 in frame #0 0x56121a7547d0 in stack_use_after_scope /home/zixi/coding/asan-test.c:52
This frame has 1 object(s): [32, 36) 'c' (line 54) <== Memory access at offset 32 is inside this variable HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork (longjmp and C++ exceptions *are* supported) SUMMARY: AddressSanitizer: stack-use-after-scope /home/zixi/coding/asan-test.c:57 in stack_use_after_scope Shadow bytes around the buggy address: 0x0fe15e14b5b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe15e14b5c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe15e14b5d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe15e14b5e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe15e14b5f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0fe15e14b600: f1 f1 f1 f1[f8]f3 f3 f3 00 00 00 00 00 00 00 00 0x0fe15e14b610: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe15e14b620: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe15e14b630: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe15e14b640: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe15e14b650: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 ... ==57368==ABORTING
\n
UAR Test
\n
The UAR test has its peculiarities. Because the stack memory of a function is reused immediately after it returns, to detect local object access errors after return, a \"pseudo-stack\" of dynamic memory allocation must be set up, for details check the relevant Wiki page of ASan4. Since this algorithm change has some performance impact, ASan does not detect UAR errors by default. If you really need to, you can set the environment variable ASAN_OPTIONS to detect_stack_use_after_return=1 before running. The corresponding test logs are as follows.
\n
$ export ASAN_OPTIONS=detect_stack_use_after_return=1 $ env | grep ASAN ASAN_OPTIONS=detect_stack_use_after_return=1 $ ./asan-test -\br ================================================================= ==57369==ERROR: AddressSanitizer: stack-use-after-return on address 0x7f5493e93030 at pc 0x55a356890ac9 bp 0x7ffd22c5cf30 sp 0x7ffd22c5cf20 READ of size 4 at 0x7f5493e93030 thread T0 #0 0x55a356890ac8 in main /home/zixi/coding/asan-test.c:105 #1 0x7f54975c50b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) #2 0x55a35689028d in _start (/home/zixi/coding/asan-test+0x128d)
Address 0x7f5493e93030 is located in stack of thread T0 at offset 48 in frame #0 0x55a356890682 in stack_use_after_return /home/zixi/coding/asan-test.c:45
ASan supports many other compiler flags and runtime environment variable options to control and tune the functionality and scope of the tests. For those interested please refer to the ASan flags Wiki page5.
\n
A zip archive of the complete test program is available for download here: asan-test.c.gz
The problem of Endianness is essentially a question about how computers store large numbers.
\n
I do not fear computers. I fear lack of them. — Isaac Asimov (American writer and professor of biochemistry, best known for his hard science fiction)
\n
\n
We know that one basic memory unit can hold one byte, and each memory unit has its address. For an integer larger than decimal 255 (0xff in hexadecimal), more than one memory unit is required. For example, 4660 is 0x1234 in hexadecimal and requires two bytes. Different computer systems use different methods to store these two bytes. In our common PC, the least-significant byte 0x34 is stored in the low address memory unit and the most-significant byte 0x12 is stored in the high address memory unit. While in Sun workstations, the opposite is true, with 0x34 in the high address memory unit and 0x12 in the low address memory unit. The former is called Little Endian and the latter is Big Endian.
\n
How can I remember these two data storing modes? It is quite simple. First, remember that the addresses of the memory units we are talking about are always arranged from low to high. For a multi-byte number, if the first byte in the low address you see is the least-significant byte, the system is Little Endian, where Little matches low. On the contrary is Big Endian, where Big corresponds to \"high\".
\n
Program Example
\n
To deepen our understanding of Endianness, let's look at the following example of a C program:
\n
char a = 1; \t \t \t char b = 2; short c = 255;\t/* 0x00ff */ long d = 0x44332211;
\n
On Intel 80x86 based systems, the memory content corresponding to variables a, b, c, and d are shown in the following table：
\n
\n\n
\n
Address Offset
\n
Memory Content
\n
\n\n\n
\n
0x0000
\n
01 02 FF 00
\n
\n
\n
0x0004
\n
11 22 33 44
\n
\n\n
\n
We can immediately tell that this system is Little Endian. For a 16-bit integer short c, we see the least-significant byte 0xff first, and the next one is 0x00. Similarly for a 32-bit integer long d, the least-significant byte 0x11 is stored at the lowest address 0x0004. If this is in a Big Endian computer, memory content would be 01 02 00 FF 44 33 22 11.
\n
At the run time all computer processors must choose between these two Endians. The following is a shortlist of processor types with supported Endian modes:
\n
\n
Pure Big Endian: Sun SPARC, Motorola 68000, Java Virtual Machine
\n
Bi-Endian running Big Endian mode: MIPS with IRIX, PA-RISC, most Power and PowerPC systems
\n
Bi-Endian running Little Endian mode: ARM, MIPS with Ultrix, most DEC Alpha, IA-64 with Linux
\n
Little Endian: Intel x86, AMD64, DEC VAX
\n
\n
How to detect the Endianess of local system in the program? The following function can be called for a quick check. If the return value is 1, it is Little Endian, else Big Endian：
\n
inttest_endian(){ int x = 1; return *((char *)&x); }
\n
Network Order
\n
Endianness is also important for computer communications. Imagine that when a Little Endian system communicates with a Big Endian system, the receiver and sender will interpret the data completely differently if not handled properly. For example, for the variable d in the C program segment above, the Little Endian sender sends 11 22 33 44 four bytes, which the Big Endian receiver converts to the value 0x11223344. This is very different from the original value. To solve this problem, the TCP/IP protocol specifies a special \"network byte order\" (referred to as \"network order\"), which means that regardless of the Endian supported by the computer system, the most-significant byte is always sent first while transmitting data. From the definition, we can see that the network order corresponds to the Big Endian.
\n
To avoid communication problems caused by Endianness and to facilitate software developers to write portable programs, some C preprocessing macros are defined for conversion between network bytes and local byte order. htons() and htonl() are used to convert local byte order to network byte order, the former works with 16-bit unsigned numbers and the latter for 32-bit unsigned numbers. ntohs() and ntohl() implement the conversion in the opposite direction. The prototype definitions of these four macros can be found as follows (available in the netinet/in.h file on Linux systems).
#error"Either BIG_ENDIAN or LITTLE_ENDIAN must be #defined, but not both."
#endif
\n","categories":["Study Notes"],"tags":["C/C++ Programming","System Programming","Computer Architecture","Computer Communications"]},{"title":"Does Diffie-Hellman Key Exchange Use a Technology Similar to RSA?","url":"/en/2022/11/21/DH-and-RSA/","content":"
Recently, at a WPA3 technology introduction meeting within the R&D team, the speaker mentioned that the OWE technology for encrypted wireless open networks is based on Diffie-Hellman key exchange, and casually said that Diffie-Hellman key exchange is using technology similar to RSA. This statement is wrong! Although Diffie-Hellman key exchange and RSA encryption algorithms belong to public key cryptography, their working mechanisms and application scenarios are different. As a research and development engineer and technician supporting network security, it is necessary to clearly understand the working mechanism and mathematical principles of the two, as well as the differences and connections between them.
\n
A cryptographic system should be secure even if everything about the system, except the key, is public knowledge. — Auguste Kerckhoffs (Dutch linguist and cryptographer, best known for his “Kerckhoffs's principle” of cryptography)
\n
\n
Diffie-Hellman Key Exchange
\n
Diffie-Hellman key exchange (DH for short) is a secure communication protocol that allows two communicating parties to exchange messages over an insecure public channel to create a shared secret without any foreknowledge. This secret can be used to generate keys for subsequent communications between the two parties using symmetric encryption techniques (e.g. AES).
\n
The idea of this kind of public key distribution to achieve shared secrets was first proposed by Ralph Merkle, a doctoral student of Stanford University professor Martin Hellman, and then Professor Hellman's research assistant Whitfield Diffie and Professor Herman jointly invented a practical key exchange protocol. In 1976, Diffie and Hellman were invited to publish their paper \"New Directions in Cryptography\" in IEEE Transactions on Information Theory, which laid the foundation for the public key cryptography system and officially announced the birth of the new Diffie-Herman key exchange technology.
Alice chooses a prime number \\(p=71\\), and then a primitive root \\(g=7\\) of the multiplicative group of integers modulo \\(p\\)
\n
Alice chooses a random number \\(a=17\\) that is less than \\(p\\), calculate \\(A=g^a\\bmod\\;p=7^{17}\\bmod\\;71 = 62\\)
\n
Alice sends all \\((p,g,A)\\) to Bob
\n
Bob also chooses a random number \\(b=39\\) that is less than \\(p\\), calculate \\(B=g^b\\bmod\\;p=7^{39}\\bmod\\;71 = 13\\)
\n
Bob sends \\(B\\) back to Alice
\n
Alice calculates \\(s=B^a\\bmod\\;p=13^{17}\\bmod\\;71 = 42\\)
\n
Bob calculate \\(s=A^b\\bmod\\;p=62^{39}\\bmod\\;71 = 42\\)
\n\n
Is it troublesome calculating \\(\\color{#93F}{\\bf62^{39}\\bmod\\;71}\\)? It is actually very easy……
\n\n
Remember that modular arithmetic has the property of preserving primitive operations: \\[(a⋅b)\\bmod\\;m = [(a\\bmod\\;m)⋅(b\\bmod\\;m)]\\bmod\\;m\\] Combining with the principle of Exponentiation by Squaring, and applying the right-to-left binary method to do fast calculation: \\[\\begin{align}\n62^{39}\\bmod\\;71 & = (62^{2^0}⋅62^{2^1}⋅62^{2^2}⋅62^{2^5})\\bmod\\;71\\\\\n& = (62⋅10⋅(62^{2^1}⋅62^{2^1})⋅(62^{2^4}⋅62^{2^4}))\\bmod\\;71\\\\\n& = (62⋅10⋅(10⋅10)⋅(62^{2^3}⋅62^{2^3}⋅62^{2^4}))\\bmod\\;71\\\\\n& = (62⋅10⋅29⋅(29⋅29⋅62^{2^3}⋅62^{2^4}))\\bmod\\;71\\\\\n& = (62⋅10⋅29⋅(60⋅60⋅62^{2^4}))\\bmod\\;71\\\\\n& = (62⋅10⋅29⋅(50⋅50))\\bmod\\;71\\\\\n& = (62⋅10⋅29⋅15)\\bmod\\;71\\\\\n& = 42\n\\end{align}\\]
\n\n\n
As if by magic, both Alice and Bob get the same \\(s\\) value of \\(42\\). This is the shared secret of two people! After this, Alice and Bob can use the hash value of \\(s\\) as a symmetric key for encrypted communication, which is unknown to any third party.
\n
Why? Because of the nature of the modular exponentiation of the multiplicative group, \\(g^{ab}\\) and \\(g^{ba}\\) are equal with the modulo \\(p\\):
So calculated \\(s\\) values must be the same. Of course, real applications would use much larger \\(p\\), otherwise the attacker can exhaust all the remainder to try to crack the ciphertext encrypted by the symmetric key.
\n
Notice \\((p,g,A,B)\\) is public and \\((a,b,s)\\) is secret. Now suppose an eavesdropper Eve can see all the messages between Alice and Bob, can she deduce \\(s\\)? The answer is that this is only practically possible if the values of \\((p,a,b)\\) are very small. Eve must first invert \\((a,b)\\) from what she knows about \\((p,g,A,B)\\):
This is the famous discrete logarithm problem. It is a recognized computational challenge and no polynomial-time efficient algorithm is currently found to compute the discrete logarithm. So this protocol is considered eavesdropping-safe as long as the appropriate \\((p,a,b)\\) is chosen. RFC 3526 recommends 6 Modular Exponential (MODP) DH groups of large prime numbers for practical applications, the smallest of which has 1536 bits!
\n
It should also be emphasized that Diffie-Hellman key exchange itself does not require authentication of both communicating parties, so it is vulnerable to man-in-the-middle attacks. If an attacker can tamper with the messages sent and received by both sides in the middle of the channel, he can complete Diffie-Hellman key exchange twice by pretending to be an identity. The attacker can then decrypt the entire message. Therefore, usually practical applications need to incorporate authentication mechanisms to prevent such attacks.
\n
Diffie-Hellman key exchange technique is a crucial contribution to modern cryptography. In 2015, 39 years after the announcement of this invention, Diffie and Hellman jointly won the ACM Turing Award, known as the \"Nobel Prize of Computing\". The ACM award poster directly stated that they \"invented public key cryptography\".
\n
\n
RSA Encryption Algorithm
\n
RSA is a public key encryption algorithm. The public key encryption system with the same name as the core technology is widely used in secure data transmission. Today, the comprehensive development of the Internet has provided great convenience to the public in all aspects of society. Whether you are surfing, gaming, entertaining, shopping, instant messaging with friends and family, managing a bank account, investing in financial securities, or simply sending and receiving email, RSA is working behind the scenes to protect your privacy and data security.
\n
RSA is actually an acronym for the last names of three people: American cryptographer Ronald Rivest, Israeli cryptographer Adi Shamir, and American computer scientist Leonard Max Adleman. In 1977, Levister, Shamir, and Adleman collaborated at the Massachusetts Institute of Technology (MIT) to invent the RSA encryption algorithm. The algorithm was first published in a public technical report at MIT, and later compiled and published in the February 1978 issue of ACM Communications under the title \"A Method for Obtaining Digital Signatures and Public Key Cryptosystems\".
\n
The basic idea of RSA is that the user creates a key pair consisting of a public key and a private key. The public key is freely distributed and the private key must be kept secret. Anyone can encrypt a message with the public key, and the resulting ciphertext can only be deciphered by the private key holder. On the other hand, any message encrypted with the private key can be decrypted by the public key. Since we assume that the private key can only be held by a specific object, encrypting with the private key is equivalent to generating a digital signature, and decrypting with the public key is equivalent to verifying the signature.
\n
The RSA encryption algorithm consists of a four-step operational process: key generation, key distribution, encryption, and decryption. A simple and concrete example is also given below to illustrate.
\n\n
Alice randomly chooses two prime numbers \\(p=127\\) and \\(q=5867\\), computes \\(N=pq=745109\\)
When \\(p\\) and \\(q\\) are both primes, \\(\\lambda(pq)=\\mathrm{lcm}(p − 1, q − 1)\\)
\n
\\(\\mathrm{lcm}\\) represents the function for the least common multiple, which may be calculated through the Euclidean algorithm
\n
\\(\\mathrm{lcm}(126,5866)=52794\\)
\n
\n
Alice chooses an integer \\(e=5\\) less than \\(\\lambda(N)\\) but also coprime with \\(\\lambda(N)\\), and calculates the modular multiplicative inverse of \\(e\\) modulo \\(\\lambda(N)\\). That is \\(d\\equiv e^{-1}\\pmod {\\lambda(N)}\\), \\(d=10559\\)\n
\n
The definition of modular multiplicative inverse is, determine \\(d\\) such that \\((d⋅e)\\;\\bmod\\;\\lambda(N)=1\\)
\n
\\(d=10559\\equiv 5^{-1}\\pmod {52794}\\)
\n
\n
\\(\\pmb{(N,e)}\\)is Alice's public key，\\(\\pmb{(N,d)}\\)is her private key\n
\n
Alice sends her public key \\((745109,5)\\) to Bob
\n
Alice saves her private key \\((745109,10559)\\) in a secret place
\n
Alice distroies all records of \\(p,q,\\lambda(N)\\)
\n
\n
When Bob wants to send Alice a message \\(M\\), according to the encoding format agreed upon by both parties, he first translates \\(M\\) to one or more positive integers \\(m\\) that are all less than \\(N\\), and then uses Alice's public key to compute the ciphertext \\(c\\) one by one. The calculation formula is \\(\\pmb{c\\equiv m^e\\pmod N}\\)\n
\n
Assume \\(M\\) is \"CACC 9678\", and the encoding scheme is 0 for spaces, 1-26 for a-z/A-Z (ignoring case), and 27-36 for 0-9
\n
Encoding yields the positive integer string \"030103 030036 333435\". Note that each integer is less than 745109
\n
After encryption, it becomes ciphertext integer string \"184539 741303 358095\"\n
\n
\\(184539 \\equiv 30103^5\\pmod {745109}\\)
\n
\\(741303 \\equiv 30036^5\\pmod {745109}\\)
\n
\\(358095 \\equiv 333435^5\\pmod {745109}\\)
\n
\n
\n
After Alice receives the ciphertext integer string, she uses her private key to compute the plaintext one by one, the calculation formula is \\(\\pmb{m\\equiv c^d\\pmod N}\\)\n
The third step above works out \\(d\\) from \\(\\color{#93F}{\\bf(d\\cdot 5)\\;mod\\;52794=1}\\), here's how
\n\n
The modular multiplicative invers can be solved quickly by applying the Extended Euclidean algorithm. Referring to this Wiki page, with the precondition of coprime, the following equation can be written (\\(gcd\\) is the function for the greatest common divisor function):
\n
\\[52794s+5t=\\mathrm{gcd}(5, 52794)=1\\]
\n
The goal is to find the smallest positive integer \\(t\\) that satisfies the above equation. The following table shows the iterative process of the algorithm:
\n
\n\n
\n
Index \\(i\\)
\n
Quotient \\(q_{i-1}\\)
\n
Remainder \\(r_i\\)
\n
\\(s_i\\)
\n
\\(t_i\\)
\n
\n\n\n
\n
0
\n
\n
\\(52794\\)
\n
\\(1\\)
\n
\\(0\\)
\n
\n
\n
1
\n
\n
\\(5\\)
\n
\\(0\\)
\n
\\(1\\)
\n
\n
\n
2
\n
\\(52794 \\div5 = 10558\\)
\n
\\(4\\)
\n
\\(1 - 10558\\times 0 = 1\\)
\n
\\(0 - 10558\\times 1 = -10558\\)
\n
\n
\n
3
\n
\\(5 \\div4 = 1\\)
\n
\\(1\\)
\n
\\(0-1\\times1 = -1\\)
\n
\\(1 - 1\\times (-10558) = \\bf10559\\)
\n
\n\n
\n
It only takes two iterations to get the remainder \\(1\\) and the algorithm ends. The final \\(t\\) is the \\(5^{-1}\\pmod {52794}\\) we want.
\n\n\n
String together after decoding to get the same information \"CACC 9678\". Why does Alice's decrypted message match exactly the one sent by Bob? The reason lies in the modular exponentiation operation. First of all, because \\(c\\equiv m^e\\pmod N\\), we can get \\(c^d\\equiv (m^e)^d \\equiv m^{ed} \\pmod N\\). Since \\((d⋅e)\\;mod\\;\\lambda(N)=1\\), it is deduced that \\(ed = 1 + h\\lambda(N)\\) (\\(h\\) is a non-negative integer为非负整数). Combine these two
The penultimate congruence above (symbol \\(\\equiv\\)) is based on Euler's theorem). This proves the correctness of the decryption formula \\({m\\equiv c^d\\pmod N}\\)! You can also see that the order of \\(e\\) and \\(d\\) is irrelevant for the result of \\(m^{ed}\\pmod N\\), so the message that Alice encrypted with her private key can be decrypted by Bob with Alice's public key. This also proves the feasibility of digital signatures.
\n
In terms of security, if a third party can derive \\(d\\) from Alice's public key \\((N,e)\\), then the algorithm is broken. But the prerequisite for cracking is to first identify \\(p\\) and \\(q\\) from \\(N\\), which is very difficult when \\(N\\) is big. In fact, this is the famous problem of factoring large numbers, another recognized computational challenge. So far, \"the best-known algorithms are faster than exponential order of magnitude times and slower than polynomial order of magnitude times.\" The latest record, published on the RSA Factoring Challenge website, is the February 2020 crack of RSA-250, a large number of 829 bits. This development indicates that the security of 1024-bit \\(N\\)-valued public keys is already in jeopardy. In view of this, National Institute of Standards and Technology (NIST) recommends that RSA keys be at least 2048 bits in length for real-world applications.
\n
On the other hand, although the public key does not need to be transmitted confidentially, it is required to be reliably distributed. Otherwise, Eve could pretend to be Alice and send her own public key to Bob. If Bob believes it, Eve can intercept all messages passed from Bob to Alice and decrypt them with her own private key. Eve will then encrypt this message with Alice's public key and pass it to her. Alice and Bob cannot detect such a man-in-the-middle attack. The solution to this problem is to establish a trusted third-party authority to issue certificates to ensure the reliability of public keys. This is the origin of the Public Key Infrastructure (PKI).
\n
The RSA public key encryption algorithm is the genius creation of three cryptographers and computer scientists. Its invention is a new milestone in public key cryptography and has become the cornerstone of modern secure Internet communication. The outstanding contribution of Levister, Shamir, and Adelman earned them the ACM Turing Award in 2002, a full 13 years before Diffie and Herman!
\n
\n
Difference and Connection
\n
The following table summarizes the comparison of Diffie-Hellman key exchange and RSA public key encryption algorithm:
Modular exponentiation, exponentiation by squaring
\n
Modular exponentiation, exponentiation by squaring, extended Euclidean algorithms
\n
\n
\n
Public Key
\n
\\((p,g,A,B)\\)
\n
\\((N,e)\\)
\n
\n
\n
Private Key
\n
\\((a,b,s)\\)
\n
\\((N,d)\\)
\n
\n
\n
Security
\n
Discrete logarithm problem
\n
Large number prime factorization problem
\n
\n
\n
Typical Applications
\n
Key Exchange
\n
Encryption/Decryption, Digital Signature
\n
\n
\n
Key Kength
\n
\\(\\ge2048\\) bits
\n
\\(\\ge2048\\) bits
\n
\n
\n
Authentication
\n
Requires external support
\n
Requires PKI support for public key distribution
\n
\n
\n
Forward Secrecy
\n
Support
\n
Not support
\n
\n\n
\n
As can be seen, both are asymmetric public key techniques, and both have a public and private key pair. They both use Modular exponentiation and exponentiation by squaring mathematical operations, and the RSA public-key encryption algorithm also requires the application of the extended Euclidean algorithm to solve the modular multiplicative inverse. Despite these similarities, the mathematical principles underlying them are different, and the computational challenges corresponding to their security are different in nature. These characteristics determine that the Diffie-Hellman key exchange can be used for key exchange, but not for encryption/decryption, while the RSA public key encryption algorithm can not only encrypt/decrypt but also support digital signatures. Therefore, the argument that the two use similar technologies cannot be established in general.
\n
ElGamal encryption based on the evolution of the Diffie-Hellman key exchange can be used to encrypt/decrypt messages, but due to some historical reasons and the great commercial success of the RSA public key encryption algorithm, ElGamal encryption is not popular.
\n
\n
In modern cryptography, key length is defined as the number of bits of a key used by an encryption algorithm. Theoretically, since all algorithms may be cracked by brute force, the key length determines an upper limit on the security of an encryption algorithm. Cryptanalytic study shows that the key strengths of Diffie-Hellman key exchange and RSA public key encryption algorithm are about the same. The computational intensities for breaking discrete logarithms and factoring large numbers are comparable. Therefore, the recommended key length for both cryptographic technologies in practical applications is at least 2048 bits.
\n
For authentication, Diffie-Hellman key exchange requires external support, otherwise it is not resistant to man-in-the-middle attacks. RSA public key encryption algorithm can be used to verify digital signatures, but only if there is a PKI supporting reliable public key distribution. The current system of PKI is quite mature, and there is a special Certificate Authority (CA) that undertakes the responsibility of public key legitimacy checking in the public key system, as well as issues and manages public key digital certificates in X.509 format.
\n
One problem with the RSA public key encryption algorithm in practice is that it does not have Forward Secrecy. Forward Secrecy, sometimes referred to as Perfect Forward Secrecy, is a security property of confidential communication protocols, meaning that the leakage of the long-term used master key does not result in the leakage of past session information. If the system has forward secrecy, it can protect the historical communication records in case of private key leakage. Imagine a situation where, although Eve cannot decrypt the RSA-encrypted messages between Alice and Bob, Eve can archive the entire past message ciphertext. One day in the future, Alice's private key for some reason was leaked, then Eve can decrypt all the message records.
\n
The solution to this problem is Diffie-Hellman key exchange! Remember that the \\((A,B)\\) in the public key of the Diffie-Hellman key exchange is generated by both parties from their respective private keys \\((a,b)\\), so if a random \\((a,b)\\) value is generated at each session, future key leaks will not crack the previous session key. This shows that Diffie-Hellman key exchange supports forward secrecy! If we combine the forward secrecy of Diffie-Hellman key exchange with the digital signature feature of the RSA public key encryption algorithm, we can implement a key exchange with authentication protection. This process can be simplified by the following example.
\n\n
Alice and Bob exchange authenticated RSA public key certificates
\n
Alice and Bob each generate a random \\((a,b)\\) value and compute \\((A,B)\\) using the shared Diffie-Hellman \\((p,g)\\).
\n
Alice encrypts \\(A\\) with her RSA private key to generate a digital signature, which she sends to Bob along with \\(A\\)
\n
Bob encrypts \\(B\\) with his own RSA private key to generate a digital signature and sends it to Alice along with \\(B\\).
\n
Alice verifies the signature with Bob's RSA public key, confirms that \\(B\\) came from Bob, and computes \\(s\\) using \\((p,a,B)\\). 6.
\n
Bob verifies the signature with Alice's RSA public key, confirms that \\(A\\) came from Alice, and computes \\(s\\) using \\((p,b,A)\\)
\n
Alice and Bob agree to share a secret and generate a subsequent symmetric encryption (AES) session key for confidential communication
\n\n
Here the RSA digital signature safeguards the key exchange from man-in-the-middle attacks. Also in the second step above, if a new random number is generated for each session, then even if Alice's or Bob's RSA private keys are leaked one day, it does not threaten the security of previous sessions because the eavesdropper still has to solve the discrete logarithm puzzle. We have also achieved forward secrecy. In fact, this is the working mechanism of the DHE-RSA cipher suite as defined by the ubiquitous Transport Layer Security (TLS) protocol.
\n
DHE-RSA Cipher Suite
\n
Transport Layer Security (TLS) and its predecessor Secure Sockets Layer (SSL) is a security protocol that provides security and data integrity for Internet communications. TLS is widely used in applications such as browsers, email, instant messaging, VoIP, and virtual private networks (VPNs), and has become the de facto industry standard for secure Internet communications. Currently, TLS 1.2 is the commonly supported version of the protocol, supporting secure connections over TCP. Datagram Transport Layer Security (DTLS) protocol is also defined for UDP applications. DTLS is much the same as TLS, with some extensions for connectionless UDP transport in terms of reliability and security. DTLS 1.2 matches the functionality of TLS 1.2.
\n
The TLS protocol uses a client-server architectural model. It works by using X.509 authentication and asymmetric encryption algorithms to authenticate the communicating parties, after which keys are exchanged to generate a symmetric encryption session key. This session key is then used to encrypt the data exchanged between the two communicating parties, ensuring the confidentiality and reliability of the information without fear of attack or eavesdropping by third parties. For identification purposes, the TLS 1.2 protocol combines the authentication, key exchange, bulk encryption, and message authentication code algorithms used into the Cipher Suite name. Each Cipher Suite is given a double-byte encoding. The TLS Cipher Suite Registry provides a reference table of all registered Cipher Suite names, sorted by encoding value from small to large.
\n
Since the computation intensity of asymmetric encryption algorithms (RSA, etc.) is much higher than that of symmetric encryption algorithms (AES, etc.), practical applications almost always use symmetric encryption algorithms to encrypt messages in batches in terms of performance.
\n
\n
TLS 1.2 protocol supports a series of cipher suites that combine the Diffie-Hellman key exchange with the RSA public key encryption algorithm. They all start with TLS_DH_RSA or TLS_DHE_RSA`. The \"E\" in DHE stands for \"Ephemeral\", which means that a random \\((a,b)\\) value is required to be generated for each session. So TLS_DHE_RSA cipher suite can provide forward secrecy, while TLS_DH_RSA cannot, and the former should be preferred in practical applications.
\n
Here we take a typical TLS_DHE_RSA_WITH_AES_128_CBC_SHA (encoding 0x00,0x33) cipher suite as an example to explain the process of Diffie-Hellman working with RSA to establish a DTLS session. First, explain the composition of the cipher suite.
\n
\n
DHE: ephemeral DH to implement key exchange
\n
RSA: public key for signing and certifying the DHE
Referring to the packet file dtls-dhe-rsa.pcap captured from the network port, the following handshake protocol message sequence chart can be obtained
\n
\nsequenceDiagram\n\nautonumber\nparticipant C as Client\nparticipant S as Server\nNote over C,S: Handshake Protocol\nrect rgb(230, 250, 255)\nC->>S: Client Hello (Cr, Cipher Suites))\nS-->>C: Hello Verify Request (Cookie)\nC->>S: Client Hello (Cr, Cookie, Cipher Suites)\nS-->>C: Server Hello (Sr, Cipher Suite), Certificate (Sn, Se)\nS-->>C: Server Key Exchange (p,g,A,Ss)\nS-->>C: Certificate Request, Server Hello Done\nC->>S: Certificate (Cn, Ce)\nC->>S: Client Key Exchange (B)\nC->>S: Certificate Verify (Cs)\nend\nNote over C,S: Establish Secure Channel\nrect rgb(239, 252, 202)\nC->>S: Change Cipher Spec, Encrypted Handshake Message\nS-->>C: Change Cipher Spec, Encrypted Handshake Message\nC->>S: Application Data\nS-->>C: Application Data\nend\n \n
\n
Below is the analysis with regard to the data package numbers in the message sequence chart:
\n
\n
Packets \\(\\require{enclose}\\enclose{circle}{1}-\\enclose{circle}{3}\\) present the initial handshake message exchange.\n
\n
The client first sends a Hello message containing a random number \\(C_r\\) and a list of supported cipher suites
\n
The server responds with a Hello Verify Request message containing a block of information (cookie)
\n
The client receives the Hello Verify Request and resends the Hello message with the entire contents of the previous message plus a copy of the cookie
\n
\n
\n
Hello verification is specific to DTLS to prevent denial of service attacks. The protocol stipulates that the server will not continue to serve the client until it receives a hello message containing the copied cookie.
\n
\n
\n
Packets \\(\\require{enclose}\\enclose{circle}{4}-\\enclose{circle}{6}\\) shows the server enters verification and key exchange stage:\n
\n
The server responds with a Hello message first, which contains the random number \\(S_r\\) and the selected cipher suite\n
\n
As shown below, the server selects TLS_DHE_RSA_WITH_AES_128_CBC_SHA!
\n
\n
The same packet also contains the Server Certificate message, which is typically large and divided into multiple fragments
\n
The server certificate provides the RSA public key \\((S_N,\\;S_e)\\) that verifies its signature
\n
Next, the server sends a Key Exchange message containing its DH public key \\((p,g,A)\\) and signature \\(Ss\\)\n
\n
The length of \\(p\\) in the figure below is 256 bytes, which means that the key length is 2048 bits and \\(Pubkey\\) is \\(A\\).
\n
You can also see in the figure that the algorithms chosen for the signature are SHA512 and RSA.
\n
The operation is to first compute \\(\\operatorname{SHA512}(Cr,Sr,p,g,A)\\) and then encrypt it with the server RSA private key
\n
\n
After that, the server sends a Certificate Request message and a Hello Done message\n
\n
The server requests the client to send an RSA public key certificate that verifies its signature
\n
\n
\n
\n
Note: If DH-RSA cipher suite is used, the server-side DH public key parameters \\((p,g,A)\\) are unchanged and will be included directly in its certificate message. At this time, the server will not issue a Key Exchange message \\(\\require{enclose}\\enclose{circle}{5}\\). For DHE-RSA, the \\(A\\) value is different for each session.
\n
\n
\n
Packets \\(\\require{enclose}\\enclose{circle}{7}-\\enclose{circle}{9}\\) shows the client enters verification and key echange stage:\n
\n
The client first sends a Certificate message, which contains the RSA public key \\((C_N,\\;C_e)\\) and also splits into multiple fragments
\n
The client then sends a Key Exchange message, which contains its DH public key \\(B\\)\n
\n
The \\(Pubkey\\) in the following figure is \\(B\\)
\n
\n
The client finally sends a Certificate Verify message, which contains the signature \\(Cs\\)\n
\n
The signature covers all previous messages except for the initial Client Hello \\(\\require{enclose}\\enclose{circle}{1}\\) and the Hello Verify Request \\(\\require{enclose}\\enclose{circle}{2}\\)
\n
The signature operation also computes SHA512 and encrypts it with the client's RSA private key
\n
\n
\n
Packets \\(\\require{enclose}\\enclose{circle}{10}-\\enclose{circle}{11}\\) completes handshake and establishs the secure channel:\n
\n
Each side first verifies the signature sent by the other side
\n
After successful verification, DH algorithm is run to generate the same premaster key
\n
Both parties call pseudo-random function (PRF) to generate a 48-byte master key from the premaster key \\[master\\_secret = \\operatorname{PRF}(pre\\_master\\_secret,\\unicode{x201C}master\\;secret\\unicode{x201D},Cr+Sr)[0..47]\\]
\n
Both parties call PRF again to generate a 72-byte key block from the master key \\[key\\_block = \\operatorname{PRF}(master\\_secret,\\unicode{x201C}key\\;expansion\\unicode{x201D},Sr+Cr)[0..71]\\]
\n
Key blocks are assigned to HMAC-SHA1 and AES_128_CBC function blocks.\n
Server Write Message Authentication Code (MAC) key: 20 bytes
\n
Client Write Encryption Key: 16 bytes
\n
Server write encryption key: 16 bytes
\n
\nNote that TLS/DTLS 1.2 specifies that this cipher suite uses an explicit initial vector (IV) and does not require the allocation of a key block
\n
The client generates a Change Cipher Spec message indicating the start of the encryption and MAC modules
\n
The client invokes PRF a third time to generate the 12-byte end-of-handshake authentication code used for master key and handshake message authentication, which is packaged into an end-of-handshake message and entered into the encryption and MAC modules \\[\\operatorname{PRF}(master\\_secret,finished\\_label,\\operatorname{SHA256}(handshake\\_messages))[0..11]\\]
\n
The client sends the Change Cipher Spec message and the encrypted end-of-handshake message to the server
\n
The server verifies the received client end-of-handshake message and repeats the above three steps to generate its own Change Cipher Spec message and encrypted an end-of-handshake message, then send them to the client
\n
The client completes the handshake by verifying the received server end-of-handshake message. Now the encrypted secure channel is established
\n
\n
Packets \\(\\require{enclose}\\enclose{circle}{12}-\\enclose{circle}{13}\\) shows that the encrypted application data exchange has officially started
\n
\n
This is the complete process of establishing a secure message channel using the TLS_DHE_RSA_WITH_AES_128_CBC_SHA (encoding 0x00,0x33) cipher suite, where DHE implements a key exchange with forward secrecy protection and RSA digital signature provides authentication for DHE, creating a solution for secure communication. With a clear understanding of this, we will better grasp the working mechanism of Diffie-Hellman and RSA, effectively apply them in practice and avoid unnecessary mistakes.
\n","categories":["Study Notes"],"tags":["Cryptography","Network Security"]},{"title":"IPv4 and IPv6 Header Checksum Algorithm Explained","url":"/en/2021/12/26/IPv4-IPv6-checksum/","content":"
About the IP packet header checksum algorithm, simply put, it is 16-bit ones' complement of the ones' complement sum of all 16-bit words in the header. However, not many sources show exactly how this is done. The same checksum algorithm is used by TCP segment and UDP datagram, but the data involved in the checksum computing is different from that in the IP header. In addition, the checksum operation of the IPv6 packet is different from that of IPv4. Therefore, it is necessary to make a comprehensive analysis of the checksum algorithm of IP packets.
\n
Nothing in life is to be feared, it is only to be understood. — Marie Curie (Polish and naturalized-French physicist and chemist, twice Nobel Prize winner)
Here the 16-bit Header Checksum field is used for error-checking of the IPv4 header. While computing the IPv4 header checksum, the sender first clears the checksum field to zero, then calculates the sum of each 16-bit value within the header. The sum is saved in a 32-bit value. If the total number of bytes is odd, the last byte is added separately.
\n
After all additions, the higher 16 bits saving the carry is added to the lower 16 bits. Repeat this till all higher 16 bits are zeros. Finally, the sender takes the ones' complement of the lower 16 bits of the result and writes it to the IP header checksum field.
\n
The following demonstrates the entire calculation process using actual captured IPv4 packets.
At the beginning of the above 16-bit hex dump is the Ethernet frame header. The IP packet header starts from offset 0x000e, with the first byte 0x45 and the last byte 0xe9. Based on the previous description of the algorithm, we can make the following calculations:
Notice at step (1) we replace the checksum field with 0x0000. As can be seen, the calculated header checksum 0x598f is the same as the value in the captured packet. This calculating process is only used for the sender to generate the initial checksum. In practice, for the intermediate forwarding router and the final receiver, they can just sum up all header fields of the received IP packet by the same algorithm. If the result is 0xffff, the checksum verification passes.
\n
C Program Implementation
\n
How to program IPv4 header checksum computing? RFC 1071 (Computing the Internet Checksum) shows a reference \"C\" language implementation:
\n
{ /* Compute Internet Checksum for "count" bytes * beginning at location "addr". */ registerlong sum = 0;
while( count > 1 ) { /* This is the inner loop */ sum += * (unsignedshort *) addr++; count -= 2; }
/* Add left-over byte, if any */ if ( count > 0 ) sum += * (unsignedchar *) addr;
/* Fold 32-bit sum to 16 bits */ while (sum>>16) sum = (sum & 0xffff) + (sum >> 16);
checksum = ~sum; }
\n
In a real network connection, the source device can call the above code to generate the initial IPv4 header checksum. This checksum is then updated at each step of the routing hop because the router must decrement the Time To Live (TTL) field. RFC 1141 (Incremental Updating of the Internet Checksum) gives a reference implementation of fast checksum update：
For TCP segment and UDP datagram, both have 16-bit header checksum fields used for error-checking by the destination host. The checksum computing algorithm is the same as the IP header, except for the difference of covered data. Here the checksum is calculated over the whole TCP/UDP header and the payload, plus a pseudo-header that mimics the IPv4 header as shown below:
It consists of the source and destination IP addresses, the protocol number (TCP:6/UDP:17), and the total length of the TCP/UDP header and payload (in bytes). The purpose of including the pseudo-header in the checksum computing is to confirm the packet reaches the expected destination and avoid IP spoofing attacks. Besides, for IPv4 UDP header checksum is optional, it carries all-zeros if unused.
\n
IPv6 Difference
\n
IPv6 is IP protocol version 6, and its main design goal was to resolve the problem of IPv4 address exhaustion. Of course, it provides many benefits in other aspects. Although IPv6 usage is growing slowly, the trend is unstoppable. The latest IPv6 standard is published in RFC 8200(Internet Protocol, Version 6 (IPv6) Specification).
Notice that the IPv6 header does not include a checksum field, a significant difference from IPv4. The absence of a checksum in the IPv6 header furthers the end-to-end principle of Internet design, to simplify router processing and speed up the packet transmission. Protection for data integrity can be accomplished by error detection at the link layer or the higher-layer protocols between endpoints (such as TCP/UDP on the transport layer). This is why IPv6 forces the UDP layer to set the header checksum.
\n
For IPv6 TCP segment and UDP datagram header checksum computing, the pseudo-header that mimics the IPv6 header is shown below
In actual IPv6 network applications, UDP-Lite (Lightweight UDP) can be used to balance error detection and transmission efficiency. UDP-Lite has its own protocol number 136, and its standard is described in RFC 3828 (The Lightweight User Datagram Protocol (UDP-Lite)).
\n
Referring to the following header format, UDP-Lite uses the same set of port number values assigned by the IANA for use by UDP. But it redefines the Length field in the UDP header to a Checksum Coverage, which allows the application layer to control the length of checksummed data. This is useful for the application that can be tolerant of the potentially lossy transmission of the uncovered portion of the data.
UDP-Lite protocol defines the values of \"Checksum Coverage\" (in bytes) as shown in the following table:
\n
\n
\n
\n
\n
\n
\n\n
\n
Checksum Coverage
\n
Coverage Area
\n
Description
\n
\n\n\n
\n
0
\n
entire UDP-Lites datagram
\n
Calculation covers IP pseudo-header
\n
\n
\n
1-7
\n
(invalid)
\n
The receiver has to drop the datagram
\n
\n
\n
8
\n
UDP-Lites header
\n
Calculation covers IP pseudo-header
\n
\n
\n
> 8
\n
UDP-Lites header + portion of payload data
\n
Calculation covers IP pseudo-header
\n
\n
\n
> IP datagram length
\n
(invalid)
\n
The receiver has to drop the datagram
\n
\n\n
\n
For multimedia applications running VoIP or streaming video data transmission protocols, it'd better receive data with some degree of corruption than not receiving any data at all. Another example is the CAPWAP protocol used to connect Cisco wireless controller and access points. It specifies UDP-Lite as the default transport protocol for the CAPWAP Data channel, while the connection is established over the IPv6 network.
\n
At last, share a C program snippet to present how to initialize a Berkeley socket to establish an IPv6 UDP-Lite connection:
int udplite_conn = socket(AF_INET6, SOCK_DGRAM, IPPROTO_UDPLITE); int val = 8; /* checksum only covers 8-byte UDP-Lite header */ (void)setsockopt(udplite_conn, IPPROTO_UDPLITE, UDPLITE_SEND_CSCOV, &val, sizeof val); (void)setsockopt(udplite_conn, IPPROTO_UDPLITE, UDPLITE_RECV_CSCOV, &val, sizeof val);
\n
Here IPPROTO_UDPLITE is protocol number 136, which is used together with AF_INET6 address family parameter in socket() function call for IPv6 socket creation. The UDPLITE_SEND_CSCOV(10) and UDPLITE_RECV_CSCOV(11) are the control parameters of socket options configuration function setsockopt(), used for setting the Checksum Coverage value in the sender and the receiver respectively. Remember that both the sender and the receiver must set the same value, otherwise, the receiver will not be able to verify the checksum properly.
IPv6 supports multiple addresses, making address assignments more flexible and convenient. Unlike IPv4, which relied solely on the DHCP protocol for address assignment, IPv6 incorporates a native Stateless Address AutoConfiguration SLAAC) protocol. SLAAC can either work alone to provide IPv6 addresses to hosts, or it can work with DHCPv6 to generate new assignment schemes. Here is a comprehensive analysis of the dynamic address allocation mechanism for IPv6.
\n
Who the hell knew how much address space we needed? — Vint Cerf (American Internet pioneer and one of \"the fathers of the Internet\")
\n
\n
IPv6 Address Overview
\n
Address Formats
\n
The most significant difference between IPv6 and IPv4 is its large address space. IPv4 has 32 bits (4 bytes) and allows for approximately 4.29 (232) billion addresses. IPv6, on the other hand, defines 128 bits (16 bytes) and supports approximately 340 x 1036 addresses. This is a pretty impressive number, and there will be no address depletion for the foreseeable future. A typical IPv6 address can be divided into two parts. As shown in the figure below, the first 64 bits are used to represent the network, and the next 64 bits are used as the interface identifier.
\n
The interface identifier can be generated in several ways:
Automatically established randomly or cryptographically
\n
\n
IETF recommends a canonical textual representation format for ease of writing. It includes leading zeros suppression and compression of consecutive all-zero fields. With the network prefix length at the end, the above address can be shortened to 2001:db8:130f::7000:0:140b/64.
Unicast: A network address corresponds to a single network node, point-to-point connection.
\n
Anycast: The target address corresponds to a group of receiving nodes, but only the \"nearest\" one receives.
\n
Multicast: The target address corresponds to a group of nodes that can receive replicated messages.
\n\n
Note that there are no broadcast addresses in IPv6, their function being superseded by multicast addresses. Anycast addresses are syntactically indistinguishable from unicast addresses and have very limited applications. A typical application for anycast is to set up a DNS root server to allow hosts to look up domain names in close proximity. For unicast and multicast addresses, they can be identified by different network prefixes:
\n
\n
\n
\n
\n
\n
\n
\n\n
\n
Address Type
\n
Binary Form
\n
Hexadecimal Form
\n
Application
\n
\n\n\n
\n
Link-local address (unicast)
\n
1111 1110 10
\n
fe80::/10
\n
Use on a single link, non-routable
\n
\n
\n
Unique local address (unicast)
\n
1111 1101
\n
fd00::/8
\n
Analogous to IPv4 private network addressing
\n
\n
\n
Global unicast address
\n
001
\n
2000::/3
\n
Internet communications
\n
\n
\n
Multicast address
\n
1111 1111
\n
ff00::/8
\n
Group communications, video streaming
\n
\n\n
\n
Each interface of a host must have a link-local address. Additionally, it can be manually or dynamically autoconfigured to obtain a unique local address and a global unicast address. Thus, IPv6 interfaces naturally have multiple unicast addresses. Unique local addresses are managed by the local network administrator, while the global unicast addresses are allocated by the IANA-designated regional registry. Referring to the following diagram, all current global unicast addresses are assigned from the 2000::/3 address block, with the first 48 bits of the address identifying the service provider's global routing network and the next 16 bits identifying the enterprise or campus internal subnet: Because an IPv6 multicast address can only be used as a destination address, its bit definition is different from that of unicast. Referring to RFC 4291, a multicast address containing 4 bits of the feature flags, 4 bits of the group scope, and the last 112 bits of the group identifier: Furthermore the same protocol specifies a few pre-defined IPv6 multicast addresses, the most important of which are
\n
\n
All Nodes Addresses on the local link — ff02::1
\n
All Routers Addresses on the local link — ff02::2
\n
Solicited-Node Address on local link — ff02::1:ffxx:xxxx
\n
\n
Dynamic Allocation Schemes
\n
NDP Protocol
\n
IPv6 dynamic address assignment depends on Neighbor Discovery Protocol (NDP). NDP acts at the data link layer and is responsible for discovering other nodes and corresponding IPv6 addresses on the link and determining available routes and maintaining information reachability to other active nodes. It provides the IPv6 network with the equivalent of the Address Resolution Protocol (ARP) and ICMP router discovery and redirection protocols in IPv4 networks. However, NDP adds many improvements and new features. NDP defines five ICMPv6 message types:
\n\n
Router Solicitation (RS)
\n
Router Advertisement (RA)
\n
Neighbor Solicitation (NS)
\n
Neighbor Advertisement (NA)
\n
Redirect
\n\n
The first two message types here, RS and RA, are the keys to implementing dynamic IPv6 address assignment. The host sends an RS message to the multicast address ff02::2 of all routers in the local network segment to request routing information. When the router receives the RS from the network node, it sends an immediate RA in response. The message format of the RA is as follows
It defines two special bits, M and O, with the following meaning:
\n
\n
M — \"Managed address configuration\" flag, set to 1 when the address is obtained from DHCPv6.
\n
O — \"Other configuration\" flag, set to 1 to indicate that other configuration information is available via DHCPv6
\n
\n
The RA message ends with the Options section, which originally had three possible options: Source Link-Layer Address, MTU, and Prefix Information. Later, RFC 8106 (which replaced RFC 6106) added the Recursive DNS Server (RDNSS) and DNS Search List (DNSSL) options. The Prefix Information option directly provide hosts with on-link prefixes and prefixes for Address Autoconfiguration, and it has the following format
Here the Prefix Length and the Prefix jointly determine the network prefix of the IPv6 address. In addition, the Prefix Information option also defines two special bits, L and A:
\n
\n
L — on-link flag. When set, indicates that this prefix can be used for on-link determination.
\n
A — autonomous address-configuration flag. When set, indicates that this prefix can be used for SLAAC.
\n
\n
Similar to the IPv4 subnet mask feature, the purpose of the \"on-link\" determination is to allow the host to determine which networks an interface can access. By default, the host only considers the network where the link-local address is located as \"on-link\". If the \"on-link\" status of a destination address cannot be determined, the host forwards the IPv6 datagram to the default gateway (or default router) by default. When the host receives an RA message, if the \"on-link\" flag for a prefix information option is set to 1 and the Valid Lifetime is also a non-zero value, the host creates a new prefix network entry for it in the prefix list. All unexpired prefix network entries are \"on-link\".
\n
Message Sequence
\n
After understanding the NDP protocol and the information conveyed by the RA messages, let's see how they guide the network nodes to achieve dynamic address assignment.
\n
Routers in the network periodically send RA messages to the multicast addresses (ff02::1) of all nodes in the local subnet. However, to avoid latency, the host sends one or more RS messages to all routers in the local subnet as soon as it has finished booting. The protocol requires the routers to respond to the RA messages within 0.5 seconds. Then, based on the values of the M/O/A bits in the received RA messages, the host decides how to dynamically configure the unique local and global unicast addresses of the interface and how to obtain other configuration information. With certain combinations of bit fetch values, the host needs to run DHCPv6 client software to connect to the server to obtain address assignment and/or other configuration information. The entire process is shown in the following message sequence diagram.
\n
\nsequenceDiagram\n\nparticipant R as Router\nparticipant H as Host\nparticipant S as DHCPv6 Server\nNote over R,H: Router Request\nrect rgb(239, 252, 202)\nH->>R: Router Solicitation\nR-->>H: Router Advertisement\nend\nNote over H,S: Address Request\nrect rgb(230, 250, 255)\nH->>S: DHCPv6 Solicit\nS-->>H: DHCPv6 Advertise\nH->>S: DHCPv6 Request\nS-->>H: DHCPv6 Reply\nend\nNote over H,S: Other Information Request\nrect rgb(230, 250, 255)\nH->>S: DHCPv6 Information-request\nS-->>H: DHCPv6 Reply\nend\n\n
\n
Note: Unlike the IPv4 DHCP protocol, DHCPv6 clients use UDP port 546 and servers use UDP port 547.
\n
\n
Next explain in detail three dynamic allocation schemes determined by the combination of the M/O/A-bit values:
\n
\n
SLAAC
\n
SLAAC + Stateless DHCPv6
\n
Stateful DHCPv6
\n
\n
SLAAC
\n
SLAAC is the simplest automatic IPv6 address assignment scheme and does not require any server. It works by sending an RS message request after the host starts up and the router sends back RA messages to all nodes in the local network segment. If the RA message contains the following configuration
\n
\n
M-bit and O-bit all clear in the message header
\n
L-bit and A-bit all set in Prefix Information option
\n
\n
Then the host receives this RA message and performs the following operations to implement SLAAC:
\n\n
Combine the network prefix with the local interface identifier to generate a unique local address or global unicast address.
\n
Install the default gateway (or default route) to point to the router address (source address of the RA message).
\n
Set this interface as the \"on-link\" corresponding to the network prefix, which is also the next-hop interface of the default gateway above.
\n
If the RDNSS and/or DNSSL options are included, install the name servers and domain name suffixes.
\n\n
This way, the host gets one or more IPv6 unique local addresses or global unicast addresses, plus the default gateway and domain name service information to complete various Internet connections.
\n
The following is an example of the SLAAC configuration on a Cisco Catalyst 9300 Multilayer Access Switch:
\n
ipv6 unicast-routing interface Vlan10 ipv6 enable ipv6 address 2001:ABCD:1000::1/64 ipv6 nd ra dns server 2001:4860:4860::8888 infinite ipv6 nd ra dns search-list example.com
\n
The Layer 3 interface of the Cisco Multilayer Switch provides routing functionality. As you can see, when IPv6 is activated on the Layer 3 interface in VLAN 10, its default address auto-assignment scheme is SLAAC. the control bits of RA messages from this interface are all set according to the SLAAC scheme, and the network prefixes for each IPv6 address it configures are automatically added to the RA prefix information options list. Of course, the network administrator can also exclude certain network prefixes with a separate interface configuration command. The last two lines of the example configuration command specify RDNSS and DNSSL, which are also added to the RA message options.
\n
If a host connects to a port in VLAN 10, it immediately gets a global unicast address with the network prefix of 2001:ABCD:1000::/64, and its default gateway address is set to 2001:ABCD:1000::1. Open a browser and enter a URL, and it will send a message to the specified domain name server 2001:4860:4860::8888 (Google's public name server address) to obtain the IPv6 address of the destination URL to establish a connection.
\n
SLAAC + Stateless DHCPv6
\n
SLAAC automatic address assignment is fast and easy, providing a plug-and-play IPv6 deployment solution for small and medium-sized network deployments. However, if a network node needs access to additional configuration information, such as NTP/SNTP server, TFTP server, and SIP server addresses, or if its functionality relies on certain Vendor-specific Information Options, it must choose SLAAC + stateless DHCPv6 scheme.
\n
This scenario still uses SLAAC automatic address assignment, but the router instructs the host to connect to a DHCPv6 server for additional configuration information. At this point, the RA message sent back by the router has
\n
\n
M-bit clear and O-bit set in the message header
\n
L-bit and A-bit all set in Prefix Information option
\n
\n
After receiving this RA message, the host performs the following actions:
\n\n
Combine the network prefix with the local interface identifier to generate a unique local address or global unicast address.
\n
Install a default gateway (or default route) pointing to the router address (source address of the RA message).
\n
Set this interface as the \"on-link\" corresponding to the network prefix, which is also the next-hop interface of the default gateway above.
\n
If the RDNSS and/or DNSSL options are included, install the name servers and domain name suffixes.
\n
Start the DHCPv6 client and connect to the DHCPv6 server to request additional configuration information.
\n
Save the additional configuration information replied by the DHCPv6 server.
\n\n
As you can see, SLAAC + stateless DHCPv6 is not different from SLAAC in terms of address assignment. DHCPv6 only provides additional configuration information and does not assign IPv6 addresses. So the DHCPv6 server does not track the address assignment status of network nodes, which is what \"stateless\" means.
\n
The corresponding configuration commands on the Catalyst 9300 switch are as follows.
The difference with the SLAAC example is that the VLAN 10 interface configuration command ipv6 nd other-config-flag explicitly specifies to set the O-bit of the RA message. Its next command, ipv6 dhcp server vlan-10-clients, activates the DHCPv6 server response feature of the interface, corresponding to the server's pool name of vlan-10-clients. The DHCPv6 server is configured above the interface configuration, starting at ipv6 dhcp pool vlan-10-clients, and contains the DNS server address, DNS domain name, and SNTP server address.
\n
If you are using a separate DHCPv6 server located on a network segment, you can remove the ipv6 dhcp server command and enable the ipv6 dhcp relay destination command on the next line of the example to specify the address to forward DHCPv6 requests to the external server.
\n
Stateful DHCPv6
\n
Many large enterprises use DHCP to manage the IPv4 addresses of their devices, so deploying DHCPv6 to centrally assign and manage IPv6 addresses is a natural preference. This is where Stateful DHCPv6 comes into play. This scenario also requires RA messages sent by the router but does not rely solely on network prefixes for automatic address assignment. The control bits of the RA messages are configured to
\n
\n
M-bit set in the message header, O-bit does not matter
\n
L-bit and A-bit can be set or clear as desired in Prefix Information option
\n
\n
Upon receiving this RA message, the host performs the following actions:
\n\n
Generate a unique local address or a global unicast address if there is a Prefix Information option with the A-bit set.
\n
Install a default gateway (or default route) pointing to the router address (source address of the RA message).
\n
If there is a Prefix Information option with the L-bit set, set this interface to \"on-link\" with the corresponding network prefix.
\n
If the RDNSS and/or DNSSL options are included, install the name servers and domain suffixes.
\n
Start the DHCPv6 client and connect to the server to request addresses and other configuration information.
\n
Set the address assigned by the DHCPv6 server to this interface.
\n
Save additional configuration information from the DHCPv6 server response.
\n\n
An example of the Stateful DHCPv6 configuration command on a Catalyst 9300 switch is as follows.
Compared to SLAAC + Stateless DHCPv6, the interface configuration here removes the ipv6 nd other-config-flag and replaces it with the ipv6 nd managed-config-flag command. This corresponds to setting the M-bit of the RA message header. The DHCPv6 server configuration adds two address prefix commands to set the network prefix. Also, the ipv6 nd prefix 2001:ABCD:1:1::/64 no-advertise configured for the interface specifies that the router does not include the 2001:ABCD:1:1::/64 prefix information option into the RA. So, this example host interface will not generate SLAAC addresses, but only two addresses from DHPCv6: a unique local address with the network prefix FD09:9:5:90::/64, and a global unicast address with the network prefix 2001:9:5:90::/64. The interface identifier for each of these two addresses is also specified by DHPCv6.
\n
How to distinguish the source of dynamically assigned addresses for host interfaces? The method is simple. One thing to remember is that DHPCv6 does not send the network prefix length to the requestor, so the network prefix length of the addresses received from DHPCv6 is 128, while the network prefix length of the addresses generated by SLAAC will not be 128. See the following example of the wired0 interface on a Linux host:
We can immediately determine that the interface is using Stateful DHCPv6 address assignment, but also generates the SLAAC address with the same network prefix 2001:20::/64 received.
\n
\n
2001:20::53c7:1364:a4d8:fd91/128 — DHCPv6 address, random interface identifer
\n
2001:20::a2ec:f9ff:fe6c:d930/64 — SLAAC addeess, interface identifer is MAC in EUI-64 format
\n
fe80::a2ec:f9ff:fe6c:d930/64 — Link-local address, interface identifer is MAC in EUI-64 format
\n
\n
Note: DHPCv6 server also does not provide any IPv6 default gateway information. The host needs to be informed of the dynamic default gateway from the RA message.
\n
\n
Summary and Comparison
\n
The following table shows the control bit combinations of RA messages concerning different address allocation and other configuration acquisition methods.
\n
\n\n
\n
M-bit
\n
O-bit
\n
A-bit
\n
Host Address
\n
Other Configuration
\n
\n\n\n
\n
0
\n
0
\n
0
\n
Static Settings
\n
Manual Configuration
\n
\n
\n
0
\n
0
\n
1
\n
Prefix specified by RA, automatically generated
\n
manually configured
\n
\n
\n
0
\n
1
\n
0
\n
Static Settings
\n
DHCPv6
\n
\n
\n
0
\n
1
\n
1
\n
Prefix specified by RA, automatically generated
\n
DHCPv6
\n
\n
\n
1
\n
0
\n
0
\n
Stateful DHCPv6
\n
DHCPv6
\n
\n
\n
1
\n
0
\n
1
\n
Stateful DHCPv6 and/or automatically generated
\n
DHCPv6
\n
\n
\n
1
\n
1
\n
0
\n
Stateful DHCPv6
\n
DHCPv6
\n
\n
\n
1
\n
1
\n
1
\n
Stateful DHCPv6 and/or automatically generated
\n
DHCPv6
\n
\n\n
\n
Summarize three dynamic allocation schemes:
\n
\n
\n
\n
\n
\n
\n\n
\n
Allocation Scheme
\n
Features
\n
Appiccation Scenarios
\n
\n\n\n
\n
SLAAC
\n
Simple and practical, fast deployment
\n
SMB, Consumer Product Networking, Internet of Things (IoT)
\n
\n
\n
SLAAC + Stateless DHCPv6
\n
Auto Configuration, Extended Services
\n
SMBs need additional network services
\n
\n
\n
Stateful DHCPv6
\n
Centralized management and control
\n
Large enterprises, institutions, and campus networks
\n
\n\n
\n
Note: Since IPv6 network interfaces can have multiple addresses (a link-local address, plus one or more unique local addresses and/or global unicast addresses), it becomes important how the source address is selected when establishing an external connection. RFC 6724 gives detailed IPv6 source address selection rules. In the development of embedded systems, the control plane and the data plane connected to the same remote device are often implemented by different functional components. For example, the control plane directly calls a Linux userspace socket to establish the connection, and the IPv6 source address used for the connection is selected by the TCP/IP stack, while the data plane directly implements data encapsulation processing and transmission in kernel space. In this case, the IPv6 source address selected by the control plane has to be synchronized to the data plane in time, otherwise, the user data might not be delivered to the same destination.
\n
\n
Troubleshooting Guide
\n
The common IPv6 dynamic address assignment debugging and troubleshooting commands on Cisco routers and switches are listed in the following table.
\n
\n
\n
\n
\n
\n\n
\n
Command
\n
Description
\n
\n\n\n
\n
show ipv6 interface brief
\n
Displays a short summary of IPv6 status and configuration for each interface
\n
\n
\n
show ipv6 interface [type] [num]
\n
Displays IPv6 and NDP usability status information for single interface
\n
\n
\n
show ipv6 interface [type] [num] prefix
\n
Displays IPv6 network prefix information for single interface
\n
\n
\n
show ipv6 dhcp pool
\n
Display DHCPv6 configuration pool information
\n
\n
\n
show ipv6 dhcp binding
\n
Displays all automatic client bindings from the DHCPv6 server binding table
\n
\n
\n
show ipv6 dhcp interface [type] [num]
\n
Display DHCPv6 interface information
\n
\n
\n
debug ipv6 nd
\n
Debug IPv6 NDP protocol
\n
\n
\n
debug ipv6 dhcp
\n
Debug DHCPv6 server
\n
\n\n
\n
The following console NDP protocol debug log shows that the router received an RS message from host FE80::5850:6D61:1FB:EF3A and responded with an RA message to the multicast address FF02::1 of all nodes in this network:
\n
Router# debug ipv6 nd ICMP Neighbor Discovery events debugging is on Router# show logging | include RS ICMPv6-ND: Received RS on GigabitEthernet0/0/0 from FE80::5850:6D61:1FB:EF3A Router# show logging | include RA ICMPv6-ND: Sending solicited RA on GigabitEthernet0/0/0 ICMPv6-ND: Request to send RA for FE80::C801:EFFF:FE5A:8 ICMPv6-ND: Setup RA from FE80::C801:EFFF:FE5A:8 to FF02::1 on GigabitEthernet0/0/0
\n
And the next log shows an example of Stateless DHCPv6 observed after entering the debug ipv6 dhcp debug command. Host FE80::5850:6D61:1FB:EF3A sends an INFORMATION-REQUEST message to the DHCPv6 server, which selects the source address FE80::C801:B9FF:FEF0:8 and sends a response message.
\n
Router#debug ipv6 dhcp IPv6 DHCP debugging is on
IPv6 DHCP: Received INFORMATION-REQUEST from FE80::5850:6D61:1FB:EF3A on FastEthernet0/0 IPv6 DHCP: Option VENDOR-CLASS(16) is not processed IPv6 DHCP: Using interface pool LAN_POOL IPv6 DHCP: Source Address from SAS FE80::C801:B9FF:FEF0:8 IPv6 DHCP: Sending REPLY to FE80::5850:6D61:1FB:EF3A on FastEthernet0/0
\n
The following debug log of Stateful DHCPv6 shows the complete process of two message exchanges (SOLICIT/ADVERTISE, REQUEST/REPLY) on lines 1, 15, 16, and 26.
\n
IPv6 DHCP: Received SOLICIT from FE80::5850:6D61:1FB:EF3A on FastEthernet0/0 IPv6 DHCP: Option UNKNOWN(39) is not processed IPv6 DHCP: Option VENDOR-CLASS(16) is not processed IPv6 DHCP: Using interface pool LAN_POOL IPv6 DHCP: Creating binding for FE80::5850:6D61:1FB:EF3A in pool LAN_POOL IPv6 DHCP: Binding for IA_NA 0E000C29 not found IPv6 DHCP: Allocating IA_NA 0E000C29 in binding for FE80::5850:6D61:1FB:EF3A IPv6 DHCP: Looking up pool 2001:ABCD::/64 entry with username '000100011F3E8772000C29806CCC0E000C29' IPv6 DHCP: Poolentry for the user not found IPv6 DHCP: Allocated new address 2001:ABCD::D9F7:61C:D803:DCF1 IPv6 DHCP: Allocating address 2001:ABCD::D9F7:61C:D803:DCF1 in binding for FE80::5850:6D61:1FB:EF3A, IAID 0E000C29 IPv6 DHCP: Updating binding address entry for address 2001:ABCD::D9F7:61C:D803:DCF1 IPv6 DHCP: Setting timer on 2001:ABCD::D9F7:61C:D803:DCF1 for 60 seconds IPv6 DHCP: Source Address from SAS FE80::C801:B9FF:FEF0:8 IPv6 DHCP: Sending ADVERTISE to FE80::5850:6D61:1FB:EF3A on FastEthernet0/0 IPv6 DHCP: Received REQUEST from FE80::5850:6D61:1FB:EF3A on FastEthernet0/0 IPv6 DHCP: Option UNKNOWN(39) is not processed IPv6 DHCP: Option VENDOR-CLASS(16) is not processed IPv6 DHCP: Using interface pool LAN_POOL IPv6 DHCP: Looking up pool 2001:ABCD::/64 entry with username '000100011F3E8772000C29806CCC0E000C29' IPv6 DHCP: Poolentry for user found IPv6 DHCP: Found address 2001:ABCD::D9F7:61C:D803:DCF1 in binding for FE80::5850:6D61:1FB:EF3A, IAID 0E000C29 IPv6 DHCP: Updating binding address entry for address 2001:ABCD::D9F7:61C:D803:DCF1 IPv6 DHCP: Setting timer on 2001:ABCD::D9F7:61C:D803:DCF1 for 172800 seconds IPv6 DHCP: Source Address from SAS FE80::C801:B9FF:FEF0:8 IPv6 DHCP: Sending REPLY to FE80::5850:6D61:1FB:EF3A on FastEthernet0/0
\n
For complex cases where it is difficult to identify whether the problem is with the host, router, or DHCPv6 server, we recommend using the free open-source network packet analysis software Wireshark to capture packets of the entire process for analysis. While analyzing packets with Wireshark, you can apply the keyword filtering function.
\n
\n\n
\n
Filter String
\n
Only Show
\n
\n\n\n
\n
icmpv6.type=133
\n
ICMPv6 RS
\n
\n
\n
icmpv6.nd.ra.flag
\n
ICMPv6 RA
\n
\n
\n
dhcpv6
\n
DHCPv6 packets
\n
\n\n
\n
We can either run Wireshark directly on the host side, or we can use the Switched Port Analyzer (SPAN) provided with the switch. Running on the network side, SPAN can collectively redirect packets from a given port to the monitor port running Wireshark for capturing. Cisco Catalyst 9300 Series switches also directly integrate with Wireshark software to intercept and analyze filtered packets online, making it very easy to use.
Accurate and effective testing of IPv6 products is key to ensuring high interoperability, security, and reliability of IPv6 infrastructure deployments. The IPv6 Ready logo is an IPv6 testing and certification program created by the IPv6 Forum. Its goals are to define IPv6 conformance and interoperability test specifications, provide a self-testing toolset, establish Global IPv6 Test Centers and provide product validation services, and finally, issue IPv6 Ready logo.
IPv6 Core Protocols Test Specification (Conformance)
\n
IPv6 Core Protocols Interoperability Test Specification (Interoperability)
\n
\n
Along with these two new test specifications, the project team also affirmed two permanent changes:
\n\n
Testing must be done in an IPv6-only environment, without any IPv4 being used for the device to function.
\n
The device under test must have IPv6 on and enabled on all IP interfaces by default.
\n\n
Not surprisingly, the new version 5.0 core protocols test specification has a section dedicated to defining SLAAC test cases to validate this core IPv6 protocol.
\n
IPv6 Core Protocol RFC List
\n
In the list below, the RFCs shown in bold are directly covered by the IPv6 Ready Version 5.0 Core Protocol Test Specification:
\n
\n
RFC 4191 Default Router Preferences and More-Specific Routes
\n
RFC 4193 Unique Local IPv6 Unicast Addresses
\n
RFC 4291 IP Version 6 Addressing Architecture
\n
RFC 4443 Internet Control Message Protocol (ICMPv6) for the Internet Protocol Version 6 (IPv6) Specification
\n
RFC 4861 Neighbor Discovery for IP version 6 (IPv6)
\n
RFC 4862 IPv6 Stateless Address Autoconfiguration
\n
RFC 4941 Privacy Extensions for Stateless Address Autoconfiguration in IPv6
\n
RFC 5095 Deprecation of Type 0 Routing Headers in IPv6
\n
RFC 6724 Default Address Selection for Internet Protocol Version 6 (IPv6)
\n
RFC 6980 Security Implications of IPv6 Fragmentation with IPv6 Neighbor Discovery
\n
RFC 7217 A Method for Generating Semantically Opaque Interface Identifiers with IPv6 Stateless Address Autoconfiguration (SLAAC)
\n
RFC 8064 Recommendation on Stable IPv6 Interface Identifiers
\n
RFC 8106 IPv6 Router Advertisement Options for DNS Configuration
\n
RFC 8200 Internet Protocol, Version 6 (IPv6) Specification
\n
RFC 8201 Path MTU Discovery for IP version 6
\n
RFC 8415 Dynamic Host Configuration Protocol for IPv6 (DHCPv6)
\n
\n","categories":["Study Notes"],"tags":["TCP/IP","Cisco Technology"]},{"title":"Implement Textbook RSA in Python","url":"/en/2022/01/22/Python-Textbook-RSA/","content":"
RSA encryption algorithm is one of the core technologies of modern public-key cryptography and is widely used on the Internet. As a classical algorithm of public-key cryptography, the programming implementation of textbook RSA can help us quickly grasp its mathematical mechanism and design ideas, and accumulate important experience in the software implementation of cryptography. Here is a detailed example of textbook RSA implementation in Python 3.8 programming environment.
\n
Random numbers should not be generated with a method chosen at random. — Donald Knuth（American computer scientist, mathematician, and professor emeritus at Stanford University, the 1974 recipient of the ACM Turing Award, often called the \"father of the analysis of algorithms\"）
\n
\n
Generating Large Primes
\n
The security of the RSA encryption algorithm is built on the mathematical challenge of factoring the product of two large prime numbers. The first step in constructing the RSA encryption system is to generate two large prime numbers \\(p\\) and \\(q\\), and calculate the modulus \\(N=pq\\). \\(N\\) is the length of the RSA key, the larger the more secure. Nowadays, practical systems require the key length to be no less than 2048 bits, with corresponding \\(p\\) and \\(q\\) about 1024 bits each.
\n
A general effectiveness method for generating such large random prime numbers is a probability-based randomization algorithm, which proceeds as follows:
In this software implementation, the first step can generate odd numbers directly. Also for demonstration purposes, the second step uses the first 50 prime numbers greater than 2 for the basic primality test. The whole process is shown in the following flowchart.
\n
\n
For the first step, Python function programming requires importing the library function randrange() from the random library. The function uses the input number of bits n in the exponents of 2, which specify the start and end values of randrange(). It also sets the step size to 2 to ensure that only n-bit random odd values are returned.
\n
from random import randrange
defgenerate_n_bit_odd(n: int): '''Generate a random odd number in the range [2**(n-1)+1, 2**n-1]''' assert n > 1 return randrange(2 ** (n - 1) + 1, 2 ** n, 2)
\n
The code for the second step is simple. It defines an array with elements of 50 prime numbers after 2, then uses a double loop in the function to implement the basic primality test. The inner for loop runs the test with the elements of the prime array one by one. It aborts back to the outer loop immediately upon failure, from there it calls the function in the first step to generate the next candidate odd number and test again.
\n
defget_lowlevel_prime(n): """Generate a prime candidate not divisible by first primes""" whileTrue: # Obtain a random odd number c = generate_n_bit_odd(n)
# Test divisibility by pre-generated primes for divisor in first_50_primes: if c % divisor == 0and divisor ** 2 <= c: break else: # The for loop did not encounter a break statement, # so it passes the low-level primality test. return c
\n
The Miller-Rabin primality test1 in the third step is a widely used method for testing prime numbers. It uses a probabilistic algorithm to determine whether a given number is a composite or possibly a prime number. Although also based on Fermat's little theorem, the Miller-Rabin primality test is much more efficient than the Fermat primality test. Before showing the Python implementation of the Miller-Rabin prime test, a brief description of how it works is given here.
\n
By Fermat's little theorem, for a prime \\(n\\), if the integer \\(a\\) is not a multiple of \\(n\\), then we have \\(a^{n-1}\\equiv 1\\pmod n\\). Therefore if \\(n>2\\), \\(n-1\\) is an even number and must be expressed in the form \\(2^{s}*d\\), where both \\(s\\) and \\(d\\) are positive integers and \\(d\\) is odd. This yields \\[a^{2^{s}*d}\\equiv 1\\pmod n\\] If we keep taking the square root of the left side of the above equation and then modulo it, we will always get \\(1\\) or \\(-1\\)2. If we get \\(1\\), it means that the following equation ② holds; if we never get \\(1\\), then equation ① holds: \\[a^{d}\\equiv 1{\\pmod {n}}{\\text{ ①}}\\]\\[a^{2^{r}d}\\equiv -1{\\pmod {n}}{\\text{ ②}}\\] where \\(r\\) is some integer that lies in the interval \\([0, s-1]\\). So, if \\(n\\) is a prime number greater than \\(2\\), there must be either ① or ② that holds. The conditional statement of this law is also true, i.e.** if we can find a \\(\\pmb{a}\\) such that for any \\(\\pmb{0\\leq r\\leq s-1}\\) the following two equations are satisfied: \\[\\pmb{a^{d}\\not \\equiv 1\\pmod n}\\]\\[\\pmb{a^{2^{r}d}\\not \\equiv -1\\pmod n}\\] Then \\(\\pmb{n}\\) must not be a prime number**. This is the mathematical concept of the Miller-Rabin primality test. For the number \\(n\\) to be tested, after calculating the values of \\(s\\) and \\(d\\), the base \\(a\\) is chosen randomly and the above two equations are tested iteratively. If neither holds, \\(n\\) is a composite number, otherwise, \\(n\\) may be a prime number. Repeating this process, the probability of \\(n\\) being a true prime gets larger and larger. Calculations show that after \\(k\\) rounds of testing, the maximum error rate of the Miller-Rabin primality test does not exceed \\(4^{-k}\\).
\n
The Miller-Rabin primality test function implemented in Python is as follows, with the variables n,s,d,k in the code corresponding to the above description.
\n
defmiller_rabin_primality_check(n, k=20): '''Miller-Rabin Primality Test with a specified round of test Input: n - n > 3, an odd integer to be tested for primality k - the number of rounds of testing to perform Output: True - passed (n is a strong probable prime) False - failed (n is a composite)''' # For a given odd integer n > 3, write n as (2^s)*d+1, # where s and d are positive integers and d is odd. assert n > 3 if n % 2 == 0: returnFalse s, d = 0, n - 1 while d % 2 == 0: d >>= 1 s += 1
for _ inrange(k): a = randrange(2, n - 1) x = pow(a, d, n) if x == 1or x == n - 1: continue for _ inrange(s): x = pow(x, 2, n) if x == n - 1: break else: # The for loop did not encounter a break statement, # so it fails the test, it must be a composite returnFalse
# Passed the test, it is a strong probable prime returnTrue
\n
Putting all of the above together, the whole process can be wrapped into the following function, where the input of the function is the number of bits and the output is a presumed random large prime number.
\n
defget_random_prime(num_bits): whileTrue: pp = get_lowlevel_prime(num_bits) if miller_rabin_primality_check(pp): return pp
\n
Utility Functions
\n\n
Greatest Common Divisor (GCD) gcd(a,b) and Least Common Multiple lcm(a,b): \nThe RSA encryption algorithm needs to calculate the Carmichael function \\(\\lambda(N)\\) of modulus \\(N\\), with the formula \\(\\lambda(pq)= \\operatorname{lcm}(p - 1, q - 1)\\), where the least common multiple function is used. The relationship between the least common multiple and the greatest common divisor is: \\[\\operatorname{lcm}(a,b)={\\frac{(a\\cdot b)}{\\gcd(a,b)}}\\] There is an efficient Euclidean algorithm for finding the greatest common divisor, which is based on the principle that the greatest common divisor of two integers is equal to the greatest common divisor of the smaller number and the remainder of the division of the two numbers. The specific implementation of Euclid's algorithm can be done iteratively or recursively. The iterative implementation of the maximum convention function is applied here, and the Python code for the two functions is as follows:
\n
defgcd(a, b): '''Computes the Great Common Divisor using the Euclid's algorithm''' while b: a, b = b, a % b return a
deflcm(a, b): """Computes the Lowest Common Multiple using the GCD method.""" return a // gcd(a, b) * b
\n
Extended Euclidean Algorithm exgcd(a,b) and Modular Multiplicative Inverse invmod(e,m): \nThe RSA key pair satisfies the equation \\((d⋅e)\\bmod \\lambda(N)=1\\), i.e., the two are mutually modular multiplicative inverses with respect to the modulus \\(\\lambda(N)\\). The extended Euclidean algorithm can be applied to solve the modular multiplicative inverse \\(d\\) of the public key exponent \\(e\\) quickly. The principle of the algorithm is that given integers \\(a,b\\), it is possible to find integers \\(x,y\\) (one of which is likely to be negative) while finding the greatest common divisor of \\(a,b\\) such that they satisfy Bézout's identity: \\[a⋅x+b⋅y=\\gcd(a, b)\\] substituted into the parameters \\(a=e\\) and \\(b=m=\\lambda(N)\\) of the RSA encryption algorithm, and since \\(e\\) and \\(\\lambda(N)\\) are coprime, we can get: \\[e⋅x+m⋅y=1\\] the solved \\(x\\) is the modulo multiplicative inverse \\(d\\) of \\(e\\). The Python implementations of these two functions are given below:
\n
defexgcd(a, b): """Extended Euclidean Algorithm that can give back all gcd, s, t such that they can make Bézout's identity: gcd(a,b) = a*s + b*t Return: (gcd, s, t) as tuple""" old_s, s = 1, 0 old_t, t = 0, 1 while b: q = a // b s, old_s = old_s - q * s, s t, old_t = old_t - q * t, t a, b = b, a % b return a, old_s, old_t
definvmod(e, m): """Find out the modular multiplicative inverse x of the input integer e with respect to the modulus m. Return the minimum positive x""" g, x, y = exgcd(e, m) assert g == 1
# Now we have e*x + m*y = g = 1, so e*x ≡ 1 (mod m). # The modular multiplicative inverse of e is x. if x < 0: x += m return x
Similarly, an iterative approach is applied here to implement the extended Euclidean algorithm, with the modular inverse multiplicative function calling the former.
\n\n
Implementing RSA Class
\n
Note: Textbook RSA has inherent security vulnerabilities. The reference implementation in the Python language given here is for learning and demonstration purposes only, by no means to be used in actual applications. Otherwise, it may cause serious information security incidents. Keep this in mind!
\n
\n
Based on the object-oriented programming idea, it can be designed to encapsulate the RSA keys and all corresponding operations into a Python class. The decryption and signature generation of the RSA class are each implemented in two ways, regular and fast. The fast method is based on the Chinese Remainder Theorem and Fermat's Little Theorem. The following describes the implementation details of the RSA class.
\n\n
Object Initialization Method \nInitialization method __init__() has the user-defined paramaters with default values shown as below：
\n
\n
Key bit-length (\\(N\\))：2048
\n
Public exponent (\\(e\\))：65537
\n
Fast decryption or signature generation：False
\n
\n
This method internally calls the get_random_prime() function to generate two large random prime numbers \\(p\\) and \\(q\\) that are about half the bit-length of the key. It then calculates their Carmichael function and verifies that the result and \\(e\\) are coprime. If not, it repeats the process till found. Thereafter it computes the modulus \\(N\\) and uses the modular multiplicative inverse function invmod() to determine the private exponent \\(d\\). If a fast decryption or signature generation function is required, three additional values are computed as follows: \\[\\begin{align}\nd_P&=d\\bmod (p-1)\\\\\nd_Q&=d\\bmod (q-1)\\\\\nq_{\\text{inv}}&=q^{-1}\\pmod {p}\n\\end{align}\\]
Encryption and Decryption Methods \nRSA encryption and regular decryption formulas are \\[\\begin{align}\nc\\equiv m^e\\pmod N\\\\\nm\\equiv c^d\\pmod N\n\\end{align}\\] Python built-in pow() function supports modular exponentiation. The above two can be achieved by simply doing the corresponding integer to byte sequence conversions and then calling pow() with the public or private key exponent:
For fast descryption, a few extra steps are needed: \\[\\begin{align}\nm_1&=c^{d_P}\\pmod {p}\\tag{1}\\label{eq1}\\\\\nm_2&=c^{d_Q}\\pmod {q}\\tag{2}\\label{eq2}\\\\\nh&=q_{\\text{inv}}(m_1-m_2)\\pmod {p}\\tag{3}\\label{eq3}\\\\\nm&=m_{2}+hq\\pmod {pq}\\tag{4}\\label{eq4}\n\\end{align}\\] In practice, if \\(m_1-m_2<0\\) in the step \\((3)\\), \\(p\\) needs to be added to adjust to a positive number. It can also be seen that the acceleration ratio would theoretically be close to \\(4\\) because the fast decryption method decreases the modulus and exponent by roughly half the order. Considering the additional computational steps, the actual speedup ratio estimate is subtracted by a correction \\(\\varepsilon\\), noted as \\(4-\\varepsilon\\). The code of the fast decryption function is as follows:
\n
defdecrypt_fast(self, encrypted_int_data: int): # Use Chinese Remaider Theorem + Fermat's Little Theorem to # do fast RSA description assert self.fast == True m1 = pow(encrypted_int_data, self.d_P, self.p) m2 = pow(encrypted_int_data, self.d_Q, self.q) t = m1 - m2 if t < 0: t += self.p h = (self.q_Inv * t) % self.p m = (m2 + h * self.q) % self.n return uint_to_bytes(m)
\n
Signature Generation and Verification Methods \nThe RSA digital signature generation and verification methods are very similar to encryption and regular decryption functions, except that the public and private exponents are used in reverse. The signature generation uses the private exponent, while the verification method uses the public key exponent. The implementation of fast signature generation is the same as the fast decryption steps, but the input and output data are converted and adjusted accordingly. The specific implementations are presented below：
\n
defgenerate_signature(self, encoded_msg_digest: bytes): """Use RSA private key to generate Digital Signature for given encoded message digest""" int_data = uint_from_bytes(encoded_msg_digest) returnpow(int_data, self.d, self.n) \t defgenerate_signature_fast(self, encoded_msg_digest: bytes): # Use Chinese Remaider Theorem + Fermat's Little Theorem to # do fast RSA signature generation assert self.fast == True int_data = uint_from_bytes(encoded_msg_digest) s1 = pow(int_data, self.d_P, self.p) s2 = pow(int_data, self.d_Q, self.q) t = s1 - s2 if t < 0: t += self.p h = (self.q_Inv * t) % self.p s = (s2 + h * self.q) % self.n return s
defverify_signature(self, digital_signature: int): """Use RSA public key to decrypt given Digital Signature""" int_data = pow(digital_signature, self.e, self.n) return uint_to_bytes(int_data)
\n\n
Functional Tests
\n
Once the RSA class is completed, it is ready for testing. To test the basic encryption and decryption functions, first initialize an RSA object with the following parameters
\n
\n
Key length (modulo \\(N\\)): 512 bits
\n
Public exponent (\\(e\\)): 3
\n
Fast decryption or signature generation: True
\n
\n
Next, we can call the encryption method encrypt() of the RSA object instance to encrypt the input message, and then feed the ciphertext to the decryption method decrypt() and the fast decryption method decrypt_fast() respectively. We use the assert statement to compare the result with the original message. The code snippet is as follows.
\n
# ---- Test RSA class ---- alice = RSA(512, 3, True) msg = b'Textbook RSA in Python' ctxt = alice.encrypt(msg) assert alice.decrypt(ctxt) == msg assert alice.decrypt_fast(ctxt) == msg print("RSA message encryption/decryption test passes!")
\n
Likewise, we can also test the signature methods. In this case, we need to add the following import statement to the beginning of the file
\n
from hashlib import sha1
\n
This allows us to generate the message digest with the library function sha1() and then call the generate_signature() and generate_signature_fast() methods of the RSA object instance to generate the signature, respectively. Both signatures are fed to the verify_signature()` function and the result should be consistent with the original message digest. This test code is shown below.
If no AssertionError is seen, we would get the following output, indicating that both the encryption and signature tests passed.
\n
RSA message encryption/decryption test passes! RSA signature generation/verification test passes!
\n
Performance Tests
\n
Once the functional tests are passed, it is time to see how the performance of fast decryption is. We are interested in what speedup ratio we can achieve, which requires timing the execution of the code. For time measurements in Python programming, we have to import the functions urandom() and timeit() from the Python built-in libraries os and timeit, respectively:
\n
from os import urandom from timeit import timeit
\n
urandom() is for generaring random bype sequence, while timeit() can time the execution of a given code segment. For the sake of convenience, the RSA decryption methods to be timed are first packed into two functions:
\n
\n
decrypt_norm() - Regular decryption method
\n
decrypt_fast() - Fast descryption method
\n
\n
Both use the assert statement to check the result, as shown in the code below:
The outer loop iterates over different key lengths klen, from 512 bits to 4096 bits in 5 levels, and the corresponding RSA object obj is initialized with:
\n
\n
Key length (modular \\(N\\)): klen
\n
Public exponent (\\(e\\)): 65537
\n
Fast decryption or signature generation: True
\n
\n
The variable rpt is also set in the outer loop to be the square root of the key length, and the timing variables t_n and t_f are cleared to zeros.
\n
The inner layer also loops 5 times, each time executing the following operations:
\n
\n
Call urandom() to generate a random sequence of bytes mg with bits half the length of the key
\n
Call obj.encrypt() to generate the ciphertext ct
\n
call timeit() and enter the packing functions decrypt_norm() and decrypt_fast() with the decryption-related parameters obj, ct and mg, respectively, and set the number of executions to rpt
\n
The return values of the timeit() function are stored cumulatively in t_n and t_f
\n
\n
\n
At the end of each inner loop, the current key length, the mean value of the timing statistics, and the calculated speedup ratio t_n/t_f are printed. The actual program segment is printed below:
Start RSA fast decryption profiling... Key size 512 => norm 0.0008s, fast 0.0003s Speedup: 2.43 Key size 1024 => norm 0.0043s, fast 0.0015s Speedup: 2.88 Key size 2048 => norm 0.0273s, fast 0.0085s Speedup: 3.19 Key size 3072 => norm 0.0835s, fast 0.0240s Speedup: 3.48 Key size 4096 => norm 0.1919s, fast 0.0543s Speedup: 3.53
\n
The test results confirm the effectiveness of the fast decryption method. As the key length increases, the computational intensity gradually increases and the running timeshare of the core decryption operation becomes more prominent, so the speedup ratio grows correspondingly. However, the final speedup ratio tends to a stable value of about 3.5, which is consistent with the upper bound of the theoretical estimate (\\(4-\\varepsilon\\)).
\n
The Python code implementation of the textbook RSA helps reinforce the basic number theory knowledge we have learned and also benefits us with an in-depth understanding of the RSA encryption algorithm. On this basis, we can also extend to experiment some RSA elementary attack and defense techniques to further master this key technology of public-key cryptography. For the complete program click here to download: textbook-rsa.py.gz
\n\n\n\n
Gary Lee Miller, a professor of computer science at Carnegie Mellon University, first proposed a deterministic algorithm based on the unproven generalized Riemann hypothesis. Later Professor Michael O. Rabin of the Hebrew University of Jerusalem, Israel, modified it to obtain an unconditional probabilistic algorithm.↩︎
\n
This is because it follows from \\(x^2\\equiv 1\\pmod n\\) that \\((x-1)(x+1)=x^{2}-1\\equiv 0\\pmod n\\). Since \\(n\\) is a prime number, by Euclid's Lemma, it must divide either \\(x- 1\\) or \\(x+1\\), so \\(x\\bmod n\\) must be \\(1\\) or \\(-1\\).↩︎
\n\n\n","categories":["Technical Know-how"],"tags":["Cryptography","Python Programming"]},{"title":"Build an Awesome Raspberry Pi NAS for Home Media Streaming","url":"/en/2021/12/29/RPi-NAS-Plex/","content":"
Network Attached Storage (NAS) provides data access to a heterogeneous group of clients over computer networks. As hard drive prices continue to drop, NAS devices have made their way into the homes of the masses. Leading brands in the SMB and home NAS market, such as Synology, have their products range in price from as low as $300 to $ 700 for the high models. But if you are a Raspberry Pi player, you can build a very nice home NAS and streaming service for only about half the cost of the lowest price.
\n
Knowledge obtained on the papers always feels shallow, must know this thing to practice. — LU You (Chinese historian and poet of the Southern Song Dynasty)
\n
\n
This blog records the whole process of building a Raspberry Pi NAS and home media server, including project planning, system implementation, and performance review. It also covers some important experiences and lessons that could hopefully benefit anyone interested in this DIY project.
\n
Project Planning
\n
Raspberry Pi 4B features an upgraded 1.8GHz Broadcom BCM2711（quad-core Cortex-A72）processor and onboard RAM up to 8GB. It includes two new USB 3.0 ports and a full-speed Gigabit Ethernet interface. The power supply is also updated to a USB-C connector. All these greatly improve system throughput and overall comprehensive performance, and we can use them to create a full-featured home NAS.
\n
For NAS system software, OpenMediaVault (OMV) is a complete NAS solution based on Debian Linux. It is a Linux rewrite of the well-known free and open-source NAS server system FreeNAS (based on FreeBSD). The salient features of OMV are
\n
\n
Simple and easy-to-use out-of-the-box solution, no need for expert-level knowledge of computer networking and storage systems
\n
Available for x86-64 and ARM platforms with a full Web Administration interface
\n
Supports a variety of different protocols (SFTP、SMB/CIFS, NFS, etc.) for file storage access
\n
Can be controlled via SSH (if enabled), and provides Access Right Management for users and groups
\n
\n
While primarily designed for home environments or small home offices, OMV's use is not limited to those scenarios. The system is built on a modular design. It can be easily extended with available plugins right after the installation of the base system. OMV is the NAS server system software we are looking for.
\n
The NAS system with media playback services provides an excellent audio/video-on-demand experience in a home network environment. Plex Media Server software integrates Internet media services (YouTube, Vimeo, TED, etc.) and local multimedia libraries to provide streaming media playback on users' various devices. The features of Plex for managing local libraries are
\n
\n
Centralized management and easy sharing of a single library
\n
Web interface with media resource navigation, streaming playback
\n
Real-time saving and resuming of playback progress
\n
Multi-user support and hierarchical playback rights settings
\n
\n
The Plex Media Server software itself is free and supports a wide range of operating systems, making it ideal for integration with home NAS.
\n
These cover all the software needed for our NAS project, but they are not enough for a complete NAS system. We also need a preferred case, otherwise, the Raspberry Pi NAS will only run bare metal. Although there are many cases available in the market for Raspberry Pi 4B, as a NAS system we need a case kit that can accommodate at least 1-2 internal SSD/HDD and must also have a good heat dissipation design.
\n
After some review and comparison, we chose Geekworm's NASPi Raspberry Pi 4B NAS storage kit. NASPi is a NUC (Next Unit of Computing) style NAS storage kit designed for the latest Raspberry Pi 4B. It consists of three components:
\n\n
X823 shield board, which provides storage function for 2.5-inch SDD/HDD
\n
X-C1 adapter board, which adjusts all Raspberry Pi 4B interfaces to the back of the case and provides power management and safe shutdown function
\n
Temperature-controlled PWM (Pulse-Width Modulation) fan as the cooling system
\n\n
All these components are packed into a case made of aluminum alloy with an anodized surface.
\n
Thereon our NAS project can be planned with the following subsystems:
\n
\n
Hardware System:\n
\n
Raspberry Pi 4B 8GB RAM
\n
32GB microSD for OS storage
\n
NASPi NAS storage kit
\n
15-20W USB-C power adaptor
\n
500GB internal SSD（USB 3.0）
\n
2TB external HDD（USB 3.0）
\n
\n
Software System:\n
\n
Raspberry Pi OS Lite（with no desktop environment）
\n
OMV for NAS file server
\n
Plex media server providing streaming service
\n
\n
\n
It is important to note that NAS servers are generally headless systems without a keyboard, mouse, or monitor. This poses some challenges for the installation, configuration, and tuning of hardware and software systems. In practice, as described in the next section, we run an SSH terminal connection to complete the basic project implementation process.
\n
System Implementation
\n
The execution of this project was divided into four stages, which are described in detail as follows.
\n
Prepare Raspberry Pi 4B
\n
In the first stage, we need to prepare the Raspberry Pi OS and do some basic unit tests. This is important, if we delay the OS test until the entire NSAPi kit is assembled, it will be troublesome to find problems with the Raspberry Pi then.
\n
Bake Raspberry Pi OS
\n
First, insert the microSD card into the USB adapter and connect it to the macOS computer, then go to the Raspberry Pi website and download the Raspberry Pi Imager software to run. From the application screen, click CHOOSE OS > Raspberry Pi OS (other) > Raspberry Pi OS Lite (32-bit) step by step. This selects the lightweight Raspberry Pi OS that does not require a desktop environment, and then click CHOOSE STORAGE to pick the microSD card.
\n
Next is a trick - hit the ctrl-shift-x key combination and the following advanced options dialog box will pop up Here is exactly the option we need to enable SSH on boot up - Enable SSH. It also allows the user to pre-set a password for the default username pi (default is raspberry). Once set up, click SAVE to return to the main page and then click WRITE to start formatting the microSD card and writing OS to it. When finished, remove the microSD card and insert it into the Raspberry Pi, connect the Ethernet cable then power it up.
\n
Probe IP Address
\n
At this point we encountered a problem: since the installed system does not have a desktop environment, it cannot connect to the keyboard, mouse, and monitor, so how do we find its IP address? There are two ways:
\n\n
connect to the home router's management WebUI and find the address for the hostname 'raspberry'.
\n
run the Nmap tool to scan the target subnet and check the changes before and after the Raspberry Pi boots up
\n\n
The log of the Nmap tool run can be seen below. Notice that a new IP address 192.168.2.4 is showing up in the scan report. Rerunning Nmap against this address alone, we saw that TCP port 22 was open. We could roughly determine that this might be our newly online Raspberry Pi:
\n
❯ nmap -sn 192.168.2.0/24 Starting Nmap 7.92 ( https://nmap.org ) at 2021-11-28 21:07 PST Nmap scan report for router.sx.com (192.168.2.1) Host is up (0.0050s latency). Nmap scan report for 192.168.2.3 Host is up (0.0048s latency). Nmap scan report for 192.168.2.4 ## New IP after Raspberry Pi boots up Host is up (0.0057s latency). Nmap done: 256 IP addresses (3 hosts up) scanned in 15.31 seconds
❯ nmap 192.168.2.4 Nmap scan report for 192.168.2.4 Host is up (0.0066s latency). Not shown: 999 closed tcp ports (conn-refused) PORT STATE SERVICE 22/tcp open ssh
\n
System Update and Upgrade
\n
Next, try SSH connection
\n
❯ ssh pi@192.168.2.4 pi@192.168.2.4's password: Linux raspberrypi 5.10.63-v7l+ #1488 SMP Thu Nov 18 16:15:28 GMT 2021 armv7l The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Fri Dec 24 19:46:15 2021 from 192.168.2.3 pi@raspberrypi:~ $
\n
Once confirmed, we executed the following commands in the Raspberry Pi to update and upgrade the system:
This stage concluded with the stability test of the Raspberry Pi 4B system Ethernet connection. The test was executed on a macOS computer using the simple ping command, setting the -i 0.05 option to specify 20 packets per second and the -t 3600 option for one hour run
\n
❯ sudo ping -i 0.05 192.168.2.4 -t 3600
\n
There should be no more than 1% packet loss or timeout on a subnet with no wireless connectivity, otherwise, it should be checked for troubleshooting. As a matter of fact, in our test, it was happening that nearly 10% of ping packets got lost and the SSH connection dropped intermittently. Searching the Internet, we found that there have been quite a few reports of similar issues with the Raspberry Pi 4B Ethernet connection. The analysis and suggestions given by people on the relevant forums focus on the following
\n\n
Unstable power supply accounts for packet loss, and needs to be replaced with a reliable USB-C power adapter of 15W or more.
\n
Energy-efficient Ethernet (Energy-Efficient Ethernet) malfunction, can be fixed by disabling it.
\n
The full-speed Gigabit Ethernet connection function is faulty and has to be downgraded to 100Mbit/s for stable use.
\n\n
Practically, we tried all of the above with little success. Later, we found that the home router connected to the Raspberry Pi 4B was a Belkin N750 DB made in 2011. Although it provides Wi-Fi dual-band 802.11n and 4 Gigabit Ethernet ports, the manufacturing date is too long ago, which makes people doubt its interoperability. Also points 2 and 3 of the above report are essentially interoperability issues. Thinking of these, we immediately ordered the TP-Link TL-SG105 5-port Gigabit Ethernet switch. After receiving it, we extended the Gigabit Ethernet port of N750 with TL-SG105, connected Raspberry Pi 4B to TL-SG105, and retested it. Sure enough, this time the ping packet loss rate was less than 0.1% and the SSH connection became solid.
\n
The conclusion is that the Raspberry Pi 4B Gigabit Ethernet interface may have compatibility issues with some older devices, which can be solved by inserting a desktop switch with good interoperability between the two.
\n
NSAPi Kit Assembly
\n
In the second stage, we assembled the NSAPi storage kit, intending to finish all hardware installation and complete the standalone NAS body.
\n
Prepare Internal SSD
\n
The NSAPi supports either an internal SSD or HDD. The project picked a Samsung 870 EVO 500GB internal SSD, here we ought to first make sure the SSD works properly on its own, otherwise, we would have to disassemble the NASPi to replace it. The SSD can be hooked up to Windows for file systems and basic read/write operation checks. In the case of a newly purchased SSD, the following steps can be done on Windows to quickly format it:
\n\n
Click on Start or the Windows button, select Control Panel > System and Security
\n
Select Administrative Tools > Computer Management > Disk management
\n
Choose the disk to be formatted, right-click then select Format
\n
Check the following in the Dialog box pop-up\n
\n
File System → NTFS
\n
Allocation Unit Size → Default
\n
Volume Label → (enter volume name)
\n
Perform a quick format
\n
\n
Click the OK button to start a fast format for the SSD
\n\n
⚠️Note: Here the chosen file system is NTFS. OMV supports NTFS mounting and reads/writes.
\n
PWM Fan Control
\n
Before the actual hardware assembly, a special software provided by Geekworm - PWM fan control script - must be installed. PWM fan speed adjustment to temperature change is a major feature that lets NASPi stand out from other hardware solutions. So this step is critical.
\n
Referring to Geekworm's X-C1 software wiki page, the installation command sequence on the SSH session connected to the Raspberry Pi 4B system is as follows
If you can't do git clone directly on Raspberry Pi 4B, you can first download the X-C1 software on the SSH client, then transfer it to Raspberry Pi 4B using scp. After that, continue to execute the subsequent commands
\n
❯ scp -r x-c1 pi@192.168.2.4:/home/pi/
\n
How does X-C1 software control PWM fan?
\n\n
The core of X-C1 software is a Python script named fan.py, which is presented below
Its logic is quite simple. With the pigpio module imported, it first initializes a PWM control object and then starts a while loop with a 1-second sleep cycle inside. The CPU temperature is read at each cycle, and the duty cycle of PWM is set according to the temperature level to control the fan speed. The duty cycle is 0 when it is lower than 30℃, and the fan stops; when it is higher than 75℃, the duty cycle is 100, and the fan spins at full speed. Users can modify the temperature threshold and duty cycle parameters in the program to customize the PWM fan control.
\n\n\n
In addition, the following pi-temp.sh script, which reads out the GPU and CPU temperatures, is also useful
\n
pi@raspberrypi:~ $ cat ./pi-temp.sh #!/bin/bash # Script: pi-temp.sh # Purpose: Display the ARM CPU and GPU temperature of Raspberry Pi # ------------------------------------------------------- cpu=$(</sys/class/thermal/thermal_zone0/temp) echo"$(date) @ $(hostname)" echo"-------------------------------------------" echo"GPU => $(vcgencmd measure_temp)" echo"CPU => temp=$((cpu/1000))’C"
pi@raspberrypi:~ $ ./pi-temp.sh Mon 29 Nov 06:59:17 GMT 2021 @ raspberrypi ------------------------------------------- GPU => temp=33.1'C CPU => temp=32’C
\n
Hardware Assembly Process
\n
Below is a snapshot of the Geekworm NASPi parts out of the box (except for the Raspberry Pi 4B on the far right of the second row and the screwdriver in the lower right corner)
\n
The three key components in the second row, from left to right, are
\n
\n
X-C1 V1.3 adapter board provides power management, interface adaptation, and security shutdown functions
\n
X823 V1.5 shield board provides a 2.5-inch SSD/HDD storage function (UASP supported)
\n
4010 PWM fan and metal fan bracket
\n
\n
The assembly process was done step-by-step mainly by referring to NASPi installation video on Youtube, and the steps are generalized as follows.
\n\n
Insert the SSD into the SATA III connector of X823, flip it to the other side, and fix it with screws.
\n
Install the Raspberry Pi 4B after fixing the spacers on this side, and place the 7-pin cable between the two
\n
Install the PWM fan on top of the Raspberry Pi 4B with the additional spacers
\n
Connect X-C1 and Raspberry Pi 4B, insert a 7-pin connector right to the X-C1 GPIO port and a 3-pin connector to the X-C1 FAN port
\n
Align and insert the 2x7-pin daughterboard to the GPIO port of the Raspberry Pi 4B and fix it with screws
\n
Plug in the USB 3.0 connector to connect the X823 USB 3.0 port to the corresponding Raspberry Pi 4B USB 3.0
\n\n
Now the installation of the internal accessories has been completed, we have a view of this
\n
\n
At this point, we added the USB-C power and pressed the front button to start the system, we could see the PWM fan started to spin. It was also observed that the fan spin rate was not constant, which demonstrated that the temperature controller PWM fan was working properly.
\n
The front button switch with embedded blue LED decides the whole system's on/off state and can be tested below
\n
\n
Press the switch after power-on, and the system starts
\n
Press and hold the switch for 1-2 seconds while running, then the system restarts
\n
Press and hold the switch for 3 seconds during operation to shut down the system safely.
\n
Press and hold the switch for 7-8 seconds during operation to force shutdown
\n
\n
Running the off command on the SSH connection can also trigger a safe shutdown. Be cautious that we should not use the Linux shutdown command, as that would not power down the X-C1 board.
\n
After the button switch test, we now unplugged the USB 3.0 connector and inserted the entire module into the case. Next was to add the back panel and tighten the screws, then re-insert the USB 3.0 connector. This completed the whole NASPi storage kit assembly process. Below are the front and rear views of the final system provided by Geekworm (all interfaces and vents are marked).
\n
\n
OMV Installation and Configuration
\n
The third stage is for installing and configuring the key software package of the NAS system - PMV. The goal is to bring up the basic network file access service. Before restarting the NAS, we plugged a Seagate 2TB external HDD into the remaining USB 3.0 port. After booting, connected SSH to NASPi from macOS and performed the following process.
\n
Install OMV Package
\n
Installing OMV is as simple as running the following command line directly from a terminal with an SSH connection.
Due to the large size of the entire OMV package, this installation process can take a long time. After the installation, the IP address of the system may change and you will need to reconnect to SSH at this time.
\n
(Reading database ... 51781 files and directories currently installed.) Purging configuration files for dhcpcd5 (1:8.1.2-1+rpt3) ... Purging configuration files for raspberrypi-net-mods (1.3.2) ... Enable and start systemd-resolved ... Unblocking wifi with rfkill ... Adding eth0 to openmedivault database ... IP address may change and you could lose connection if running this script via ssh. client_loop: send disconnect: Broken pipe\t
\n
After reconnecting, you can use dpkg to view the OMV packages. As you can see, the latest version of OMV installed here is 6.0.5.
\n
pi@raspberrypi:~ $ dpkg -l | grep openme ii openmediavault 6.0.5-1 all openmediavault - The open network attached storage solution ii openmediavault-flashmemory 6.0.2 all folder2ram plugin for openmediavault ii openmediavault-keyring 1.0 all GnuPG archive keys of the OpenMediaVault archive ii openmediavault-omvextrasorg 6.0.4 all OMV-Extras.org Package Repositories for OpenMediaVault
\n
OMV Management UI
\n
At this point OMV's workbench is live. Launching a browser on a macOS computer and typing in the IP address will open the beautiful login screen (click on the 🌍 icon in the upper right corner to select the user interface language): After logging in with the default username and password shown above, you will see the Workbench screen. The first thing you should do at this point is to click the ⚙️ icon in the top right corner to bring up the settings menu and click \"Change Password\". You can also change the language here Clicking on \"Dashboard\" in the settings menu allows you to select the relevant components to be enabled. The menu on the left side provides task navigation for administrators and can be hidden when not needed. The complete OMV administration manual can be found in the online documentation
\n
Configure File Services
\n
Next is the key process for configuring the NAS, which consists of the following 5 steps.
\n\n
Scan for mounted disk drives
\n
Click Storage > Disks from the sidebar menu to enter the hard drive management page. If there is an external USB storage device just plugged in, you can click 🔍 here to scan it out. The scan results for this system are as follows. The internal Samsung 500GB SSD and external Seagate 2TB HDD are detected, and the 32GB microSD that contains the entire software system is listed at the top:
\n
On the SSH terminal, we could see the information for the same set of mounted drivers
Click Storage > File Systems from the sidebar menu to enter the file system management page. If the storage device does not have a file system yet, click ⨁ to Create or Mount the file system. OMV can create/mount ext4, ext3, JFS, and xfs file systems, but only mounts are supported for the NTFS file system. The following figure shows that OMV correctly mounts NTFS file systems for SSDs and HDDs:
\n
Set Shared Folders
\n
From the sidebar menu, click Storage > File Systems to access the shared folder management page. Here, click ⨁ to create a shared folder. When creating it, specify the name, corresponding file system, and relative path, and you can also add comments. Select the created folder and click the pencil icon again to edit the related information. This system sets the relative paths of shared folders Zixi-Primary and Zixi-Secondary for SSD and HDD respectively Notice the orange alert at the top of the figure above, which alerts the administrator that the configurations have changed and must click on the ✔️ icon to take effect.
\n
Add shared folder access users
\n
Click User Management > Users from the sidebar menu to enter the user management page. The system's default user pi has root privileges and cannot be used for file-sharing access due to security concerns. So you need to add a new user separately. On this page, click ⨁ to Create or Import user, only user name and password are required when creating a new user, others are optional. Once created, select this user and click the third folder+key icon (prompting \"Shared folder privileges\") to enter the following privileges settings page As shown in the figure, for this new user zixi, the administrator can set the read and write access permissions for each shared folder.
\n
Start file share services
\n
If you expand the \"Services\" item in the navigation menu, you can see that OMV manages five services: FTP, NFS, Rsync, SMB/CIFS, and SSH. SSH is enabled at the beginning of the system OS image preparation. NFS and SMB/CIFS are the most common network file-sharing protocols, and both are supported by macOS. Take SMB/CIFS as an example here. Click Services > SMB/CIFS from the sidebar menu to enter the management page. The page contains two buttons: Settings and Shares. Click \"Settings\" first to activate the SMB/CIFS service and configure the workgroup name on the new page, other options can be left as default. After saving, it returns to the SMB/CIFS administration page. Then enter \"Shares\", click ⨁ to Create shared folders Zixi-Primary and Zixi-Secondary on the new page then save. After that, click the ✔️ icon in the orange warning bar to make all configuration updates take effect, and you will end up with the following result
\n\n
Now our Raspberry Pi NAS system is ready for file sharing and the SMB/CIFS service is started. After checking the relevant components to turn on, our dashboard live monitoring looks like this
\n
Set Up Client Device
\n
Once the server side is ready, we need to add the network share folder on the client side as follows.
\n
\n
Windows PC client\n
\n
Open File Explore, click “This PC”
\n
Right-click on the blank area at the right pane, select \"Add a network location” on the popup menu
\n
Enter “\\\\<IP-address>\\” in the “Internet or network address\" input box
\n
Enter username and password when prompted
\n
\n
MacBook client (screenshot below)\n
\n
Open Finder, click the menu item Go
\n
Click “Connect to Server...”
\n
Enter URL “smb://<IP-address>/”, then click Connect
\n
Enter username and password when prompted \n
\n
\n
\n
Once the client side is set up, users can perform various operations on the network share folder as if it were a local directory, such as previewing, creating new, opening or copying files, creating new subdirectories, or deleting existing subdirectories.
\n
Plex Installation and Configuration
\n
The last stage is to install and configure the Plex Media Server, and then start a network streaming service.
\n
Install Media Server
\n
The process of installing Plex Media Server requires HTTPS transport support, so we must first install the https-transport package. SSH to our Raspberry Pi NAS and execute the install command
\n
sudo apt-get install apt-transport-https
\n
Next add the Plex repository to the system, which requires downloading the Plex sign key first. Here are the related commands and run logs
\n
pi@raspberrypi:~ $ curl https://downloads.plex.tv/plex-keys/PlexSign.key | sudo apt-key add - % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 100 3072 100 3072 0 0 10039 0 --:--:-- --:--:-- --:--:-- 10039 OK
\n
Use the same apt-key command to check the newly added Plex sign key
\n
pi@raspberrypi:~ $ apt-key list Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). /etc/apt/trusted.gpg ... pub rsa4096 2015-03-22 [SC] CD66 5CBA 0E2F 88B7 373F 7CB9 9720 3C7B 3ADC A79D uid [ unknown] Plex Inc. sub rsa4096 2015-03-22 [E] ...
\n
You can see that Plex uses 4096-bit RSA keys. For the warning message \"apt-key is deprecated...\" in the above log, you can ignore it for now. Go to read some discussion on the askubuntu forum if you are interested.
\n
The next step is to add the Plex repository to the system repository list, and then update the packages
echo deb https://downloads.plex.tv/repo/deb public main | sudo tee /etc/apt/sources.list.d/plexmediaserver.list sudo apt-get update
Now we can start the actual Plex Media Server installation with the following installation commands
\n
pi@raspberrypi:~ $ sudo apt install plexmediaserver Reading package lists... Done Building dependency tree... Done Reading state information... Done The following NEW packages will be installed: plexmediaserver 0 upgraded, 1 newly installed, 0 to remove and 20 not upgraded. Need to get 66.1 MB of archives. After this operation, 146 MB of additional disk space will be used. Get:1 https://downloads.plex.tv/repo/deb public/main armhf plexmediaserver armhf 1.25.0.5282-2edd3c44d [66.1 MB] Fetched 66.1 MB in 28s (2392 kB/s) Selecting previously unselected package plexmediaserver. (Reading database ... 51783 files and directories currently installed.) Preparing to unpack .../plexmediaserver_1.25.0.5282-2edd3c44d_armhf.deb ... PlexMediaServer install: Pre-installation Validation. PlexMediaServer install: Pre-installation Validation complete. Unpacking plexmediaserver (1.25.0.5282-2edd3c44d) ... Setting up plexmediaserver (1.25.0.5282-2edd3c44d) ...
Configuration file '/etc/apt/sources.list.d/plexmediaserver.list' ==> File on system created by you or by a script. ==> File also in package provided by package maintainer. What would you like to do about it ? Your options are: Y or I : install the package maintainer's version N or O : keep your currently-installed version D : show the differences between the versions Z : start a shell to examine the situation The default action is to keep your current version. *** plexmediaserver.list (Y/I/N/O/D/Z) [default=N] ? PlexMediaServer install: PlexMediaServer-1.25.0.5282-2edd3c44d - Installation starting. PlexMediaServer install: PlexMediaServer install: Now installing based on: PlexMediaServer install: Installation Type: New PlexMediaServer install: Process Control: systemd PlexMediaServer install: Plex User: plex PlexMediaServer install: Plex Group: plex PlexMediaServer install: Video Group: video PlexMediaServer install: Metadata Dir: /var/lib/plexmediaserver/Library/Application Support PlexMediaServer install: Temp Directory: /tmp PlexMediaServer install: Lang Encoding: en_US.UTF-8 PlexMediaServer install: Nvidia GPU card: Not Found PlexMediaServer install: PlexMediaServer install: Completing final configuration. Created symlink /etc/systemd/system/multi-user.target.wants/plexmediaserver.service → /lib/systemd/system/plexmediaserver.service. PlexMediaServer install: PlexMediaServer-1.25.0.5282-2edd3c44d - Installation successful. Errors: 0, Warnings: 0
\n
The log shows a question is asked about the Plex media server list (plexmediaserver.list), just choose the default N. When we see \"Installation successful\", we know that the installation was successful. At this point, the Plex streaming service is up and running. Invoking the Nmap scan again from the macOS side, we find that TCP port 32400 for Plex service is open.
\n
❯ nmap -p1-65535 192.168.2.4 | grep open 22/tcp open ssh 80/tcp open http 111/tcp open rpcbind 139/tcp open netbios-ssn 445/tcp open microsoft-ds 2049/tcp open nfs 5357/tcp open wsdapi 32400/tcp open plex
\n
Configure Media Server
\n
The configuration of the Plex Media Server has been done on the web GUI. Launch a browser on the macOS computer and type in the URL http://<IP-address>:32400/web, now we can see the following page if no surprise We can sign in with a Google, Facebook, or Apple account, or we can enter an email to create a new account. Follow the instructions on the page step by step, no need for any payment, soon we reach the Server Setup page. Here we can configure the server name and add libraries. Normally we don't need to access our home media server from outside, so remember to uncheck the \"Allow me to access my media outside my home\" box in this step. To add a library, first select the type of library (movies, TV episodes, music, photos, etc.), then click the \"BROWSE FOR MEDIA FOLDER\" button to browse and select the corresponding folder. Once the library is added, the included media files will immediately appear in the local service directory, as shown in the screenshot below Here we have a local server named ZIXI-RPI-NAS for our Raspberry Pi NAS, the movie directory in the library shows The Matrix trilogy and is playing the first one The Matrix. Move your mouse over the server name and ➕ icon will appear to the right, click on it to continue adding new media libraries.
\n
Once the Plex Media Server is configured, we can open a browser from any device on our home network to do streaming on-demand, without the need to download additional applications. The whole experience is just like our own proprietary home Netflix service. This is awesome!
\n
Performance Review
\n
By connecting a macOS laptop to one of the remaining ports of the TL-SG105, we could perform some simple same-subnet tests to evaluate the performance of this NAS system fully.
\n
System Stress Test
\n
Referring to Geekworm NASPi Stress Test Wiki page, we executed the following command over SSH connection, which cloned the test script from GitHub and ran the stress test:
\n
git clone https://github.com/geekworm-com/rpi-cpu-stress cd rpi-cpu-stress chmod +x stress.sh sudo ./stress.sh
\n
Simultaneously we established a second SSH session and ran htop to monitor system status. The screenshot below was taken while close to the 5-minute mark (left is the htop real-time display, and right is the stress test output) Dividing the temp value on the right side by 1000 gave the CPU temperature. All 4 CPU cores reached 100% full load during the test, while the maximum temperature did not exceed 70°C. At this moment, there was no obvious heat sensation when touching the case. Typing ctrl-c to stop the stress test, and then executing the temperature measurement script again
\n
pi@raspberrypi:~ $ ./pi-temp.sh Fri Dec 24 15:59:21 PST 2021 @ raspberrypi ------------------------------------------- GPU => temp=39.9'C CPU => temp=40'C
\n
The system temperature returned to a low range value. This test result assures the system meets the design goal.
\n
File Transfer Speed Test
\n
The file transfer speed can be roughly measured with the secure remote copy tool SCP. First, create a 1GB size file by running the mkfile command on the macOS client, then copy it to the user directory of the remote NAS system with the scp command
After the copy was done, it would print the time spent and the deduced speed. Running the command with the source and the destination reversed would give us the speed of receiving a file from the NAS system.
As can be seen, the speed of remote write is around 20MB/s, while the speed of remote file read can reach over 60MB/s. Considering that scp-related encryption and decryption are implemented in software on general-purpose Raspberry Pi systems, this result should be considered passable.
\n
Disk Access Speed Test
\n
The real test of the NAS's performance is the network drive read/write speed test. For this, we downloaded the AmorphousDiskMark app from Apple's App Store. This is an easy and efficient drive speed test that measures the read/write performance of a storage device in terms of MB/s and IOPS (input/output operations per second). It has four types of tests:
\n\n
sequential read/write, 1MB block, queue depth 8
\n
sequential read/write, 1MB block, queue depth 1
\n
random read/write, 4KB block, queue depth 64
\n
random read/write, 4KB block, queue depth 1
\n\n
The above queue depths are the default values, but other values are also available. In addition, users can also modify the test file size and duration.
\n
Run the application on the macOS client and select the remote SMB folders Zixi-Primary (Samsung SSD) and Zixi-Secondary (Seagate HDD) respectively at the top, then click the All button in the upper left corner to start the NAS drive speed test process. A side-by-side comparison of the two test results is shown below
\n
\n
This gives a few observations:
\n
\n
Reads are faster than writes for NAS drives, and the difference under random access is huge.
\n
SSD outperforms HDD for both sequential and random accesses.
\n
Large queue depth speeds up reads, especially for random accesses, but there is little impact on writes.
\n
For both SSDs and HDDs, sequential reads/writes are significantly more efficient than random reads/writes.
\n
For both SSDs and HDDs, sequential reads/writes reach their highest speeds at large queue depths.
\n
\n
These are not surprising and are consistent with the test results on macOS laptops with direct external SSDs and HDDs, only with the lower numbers. With this NAS system, both the SSD and HDD are connected via the USB 3.0 interface. USB 3.0 supports transfer speeds of up to 5Gbit/s, so the performance bottleneck of the system is the network interface bandwidth and processor power.
\n
That being said, for both SSDs and HDDs, the transfer speeds have been more than 900Mbit/s at 1MB sequential read and queue depth 8, close to the upper bandwidth limit of the Gigabit Ethernet interface. This read speed can support a single 1080p60 video stream at a frame rate of 60fps or 2 parallel 1080i50 video streams at a frame rate of 25fps, which is sufficient for home streaming services. In another media service test, the NAS system performs satisfactorily with three computers playing HD video on demand and one phone playing MP3 music without any lag.
\n
Project Summary
\n
This completes our Raspberry Pi home NAS project. Now we can move our NAS to a more permanent location to provide network file and streaming services for the whole family.
\n
\n
Economically, our home NAS has the cost summarized in the table below (excluding SSD/HDD)
\n
\n
\n
\n
\n
\n
\n\n
\n
Devices
\n
Functions
\n
Cost（$）
\n
\n\n\n
\n
Raspberry Pi 4B 2/4/8GB RAM
\n
Primary hardware system
\n
45/55/75
\n
\n
\n
Samsung 32GB EVO+ Class-10 Micro SDHC
\n
OS storage
\n
10
\n
\n
\n
Geekworm NASPi Raspberry Pi 4B NAS Storage Kit
\n
Case, extending board and PWM fan
\n
60
\n
\n
\n
Geekworm 20W 5V 4A USB-C Power Adaptor
\n
Power supply
\n
15
\n
\n
\n
TP-Link TL-SG105 5-Port Gigabit Ethernet Switch
\n
Desktop switch
\n
15
\n
\n\n
\n
Even with the choice of 8GB RAM Raspberry Pi 4B, the whole cost is only $175, a little more than half of the price of the low-end brand NAS sold in the market. Unless there are a lot of client devices that need streaming services, the memory consumption is usually under 2GB, so the 2GB Raspberry Pi 4B should be able to work in most home scenarios. That cuts the cost down to $145, less than half the MSRP.
\n
On the other hand, this DIY project was a very good exercise of hands-on practice, helping us gain valuable intuitive experience in building network connections, configuring system hardware and software, and tuning and testing application layer services. To sum up, the home NAS system built with Raspberry Pi 4B and OMV, combined with a Plex media server, provides a cost-effective solution for file backup and streaming media services in the home network.
\n
Appendix: List of related devices and Amazon links
\n
\n
Disclosure: This blog site is reader-supported. When you buy through the affiliate links below, as an Amazon Associate, I earn a tiny commission from qualifying purchases. Thank you.
\n","categories":["DIY Projects"],"tags":["Raspberry Pi","NAS"]},{"title":"RSA: Attack and Defense (1)","url":"/en/2023/03/16/RSA-attack-defense/","content":"
RSA is a public-key cryptosystem built on top of an asymmetric encryption algorithm, which was jointly invented by three cryptographers and computer scientists at the Massachusetts Institute of Technology in 1977. The RSA public-key encryption algorithm and cryptosystem provide data confidentiality and signature verification functions widely used on the Internet. Since its birth, RSA has become a major research object of modern cryptography. Many cryptanalysts and information security experts have been studying its possible theoretical flaws and technical loopholes to ensure security and reliability in practical applications.
\n
There are certain things whose number is unknown. If we count them by threes, we have two left over; by fives, we have three left over; and by sevens, two are left over. How many things are there? — Sunzi Suanjing, Volume 2.26
\n
\n
Fortunately, after more than 40 years of extensive research and practical application tests, although many sophisticated attack methods have been discovered, RSA is generally safe. These attack methods all take advantage of the improper use of RSA or the vulnerability of software and hardware implementations, and cannot shake the security foundation of its encryption algorithm. On the other hand, the research on these attack methods shows that implementing a safe and robust RSA application is not a simple task. A consensus in cryptography and network security hardware and software engineering practice is: never roll your own cryptography!1 The appropriate solution is to use an existing, well-tested, and reliably maintained library or API to implement the RSA algorithm and protocol application.
\n
Here is a brief survey of the common means of attacking RSA, the mathematical mechanism on which the attack is based, and the corresponding protective measures. Referring to the previous article, let’s start by reviewing the working mechanism and process of RSA:
\n\n
Choose two large prime numbers \\(p\\) and \\(q\\), compute \\(N=pq\\)
When both \\(p\\) and \\(q\\) are prime, \\(\\lambda(pq)=\\operatorname {lcm}(p − 1, q − 1)\\)
\n
\\(\\operatorname{lcm}\\) is a function to find the least common multiple, which can be calculated by the Euclidean algorithm
\n
\n
Choose a number \\(e\\) that is less than \\(\\lambda(N)\\) and also coprime with it, then calculate the modular multiplicative inverse of \\(e\\) modulo \\(\\lambda(N)\\). That is \\(d\\equiv e^{-1}\\pmod {\\lambda(N)}\\)\n
\n
Per the definition of modular multiplicative inverse, find \\(d\\) such that \\((d⋅e)\\bmod\\lambda(N)=1\\)
\\(\\pmb{(N,e)}\\) is the public key，\\(\\pmb{(N,d)}\\) is the private key\n
\n
The public key can be known by everyone, but the private key must be kept secret
\n
The records of \\(p,q,\\lambda(N)\\) can all be discarded
\n
\n
The sender first converts the message into a positive integer less than \\(N\\) according to the agreed encoding format, then uses the receiver's public key to compute the ciphertext with the formula \\(\\pmb{c\\equiv m^e\\pmod N}\\)
\n
After receiving the ciphertext, the receiver uses its private key to compute the plaintext \\(m\\) with the formula \\(\\pmb{m\\equiv c^d\\pmod N}\\), then decodes it into the original message
\n
A message encrypted with the private key can also be decrypted by the public key, i.e. if \\(\\pmb{s\\equiv m^d\\pmod N}\\), \\(\\pmb{m\\equiv s^e\\pmod N}\\). This is the supported digital signature feature
\n\n
Note that the second and third steps in the original RSA paper use Euler's totient function\\(\\varphi(N)\\) instead of \\(\\lambda(N)\\). The relationship between these two functions is: \\[\\varphi(N)=\\lambda(N)⋅\\operatorname{gcd}(p-1,q-1)\\] Here \\(\\operatorname{gcd}\\) is the greatest common divisor function. Using \\(\\lambda(N)\\) can yield the minimum workable private exponent \\(d\\), which is conducive to efficient decryption and signature operations. Implementations that follow the above procedure, whether using Euler's or Carmichael's functions, are often referred to as \"textbook RSA \".
\n
Textbook RSA is insecure, and there are many simple and effective means of attack. Before discussing the security holes of the textbook RSA in detail, it is necessary to review the first known attack method - integer factorization!
\n
Integer Factorization
\n
The theoretical cornerstone of the security of the RSA encryption algorithm is the problem of factoring large numbers. If we can separate \\(p\\) and \\(q\\) from the known \\(N\\), we can immediately derive the private exponent \\(d\\) and thus completely crack RSA. Factoring large numbers is a presumed difficult computational problem. The best-known asymptotic running time algorithm is General Number Field Sieve, and its time complexity is \\({\\displaystyle \\exp \\left(\\left(c+o(1)\\right)(\\ln N)^{\\frac {1}{3}}(\\ln \\ln N)^{\\frac {2}{3}}\\right)}\\), where the constant \\(c = 4/\\sqrt[3]{9}\\)，\\(\\displaystyle \\exp\\) and \\(\\displaystyle \\exp\\) is the exponential function of Euler's number (2.718).
\n
For a given large number, it is difficult to accurately estimate the actual complexity of applying the GNFS algorithm. However, based on the heuristic complexity empirical estimation, we can roughly see the increasing trend of computational time complexity:
\n
\n
For a large number of 1024 bits, there are two prime factors of about 500 bits each, and the decomposition requires basic arithmetic operations of order \\(2^{70}\\)
\n
For a large number of 2048 bits, there are two prime factors of about 1000 bits each, and the decomposition requires basic arithmetic operations of order \\(2^{90}\\), a million times slower than the 1024-bit number
\n
\n
The rapid development of computer software and hardware technology has made many tasks that seemed impossible in the past become a reality. Check the latest record released by the RSA Factoring Challenge website. In February 2020, a team led by French computational mathematician Paul Zimmermann successfully decomposed the large number RSA-250 with 250 decimal digits (829 bits):
According to the announcement of the factorization released by Zimmerman, using a 2.1GHz Intel Xeon Gold 6130 processor, the total computing time to complete this task is about 2700 CPU core-years. This number may seem large, but in today's era of cluster computing, grid computing, and cloud computing for the masses, it's not a stretch to think that organizations with strong financial backing can reduce computing time to hours or even minutes. As an example, go to the online tool website of the free open-source mathematical software system SageMath and enter the following first 5 lines of Sage Python code:
The result was obtained within minutes, and a large number of 72 decimal digits (240 bits) was decomposed. You know, in the 1977 RSA paper, it is mentioned that it takes about 104 days to decompose a 75-digit decimal number. The technological progress of mankind is so amazing!
\n
As the attacker's spear becomes sharper and sharper, the defender's shield must become thicker and thicker. Therefore, 1024-bit RSA is no longer secure, and applications should not use public key \\(N\\) values that are less than 2048 bits. And when high security is required, choose 4096-bit RSA.
\n
Elementary Attacks
\n
Although the decomposition of large numbers is an attack method known to everyone, the security vulnerabilities caused by some low-level errors commonly found in RSA applications make it possible to use simple attacks to succeed, and some typical ones are explained below.
\n
\n
In the early development of RSA, finding large prime numbers took quite a bit of time based on the backward computing power of the time. Therefore, some system implementations tried to share the modulus \\(N\\). The idea was to generate only one set \\((p,q)\\), and then all users would use the same \\(N=pq\\) values, with a central authority that everyone trusted assigning key pairs \\((e_i,d_i)\\) to each user \\(i\\), and nothing would go wrong as long as the respective private keys \\(d_i\\) were kept. Unfortunately, this is a catastrophic mistake! This implementation has two huge security holes：
\n\n
The user \\(i\\) can decompose \\(N\\) using his own key pair \\((e_i,d_i)\\). Whether \\(d\\) is generated using the Euler function \\(\\varphi(N)\\) or the Carmichael function \\(\\lambda(N)\\), there are algorithms that quickly derive the prime factors \\(p\\) and \\(q\\) from a given \\(d\\)2. And once \\(p\\) and \\(q\\) are known, user \\(i\\) can compute any other user's private key \\(d_j\\) with one's public key \\((N,e_j)\\). At this point, the other users have no secrets from user \\(i\\).
\n
Even if all users do not have the knowledge and skill to decompose \\(N\\), or are \"nice\" enough not to know the other users' private keys, a hacker can still perform a common modulus attack to break the users' messages. If the public keys of two users, Alice and Bob, are \\(e_1\\) and \\(e_2\\), and \\(e_1\\) and \\(e_2\\) happen to be mutually prime (which is very likely), then by Bézout's identity, the eavesdropper Eve can find that \\(s\\) and \\(t\\) satisfy: \\[e_{1}s+e_{2}t=gcd(e_1,e_2)=1\\] At this point, if someone sends the same message \\(m\\) to Alice and Bob, Eve can decrypt \\(m\\) after recording the two ciphertexts \\(c_1\\) and \\(c_2\\) and performing the following operation: \\[c_1^s⋅c_2^t\\equiv(m^{e _1})^s⋅(m^{e_2})^t\\equiv m^{e_{1}s+e_{2}t}\\equiv m\\pmod N\\] The corresponding Python function code is shown below.
\n
defcommon_modulus(e1, e2, N, c1, c2): # Call the extended Euclidean algorithm function g, s, t = gymp2.gcdext(e1, e2) assert g == 1 if s < 0: # Find c1's modular multiplicative inverse\t\t re = int(gmpy2.invert(c1, N)) c1 = pow(re, s*(-1), N) c2 = pow(c2, t, N) else: # t is negative, find c2's modular multiplicative inverse re = int(gmpy2.invert(c2, N)) c2 = pow(re, t*(-1), N) c1 = pow(c1, a, N) return (c1*c2) % N
Two library functions of gmpy23 are called here: gcdext() to implement the extended Euclidean algorithm, and invert() to find the modular multiplicative inverse element. Note that Python's exponential function pow() supports modular exponentiation, but the exponent must not be negative. Since one of \\(s\\) or \\(t\\) must be negative, you have to first call invert() to convert \\(c_1\\) or \\(c_2\\) to the corresponding modular multiplicative inverse, then invert the negative number to calculate the modular exponent. For example, lines 7 and 8 above implement \\(c_1^s=(c_1^{-1})^{-s}\\bmod N\\).
\n
\n
Is it possible to reuse only \\(p\\) or \\(q\\) since the shared modulus \\(N\\) is proven to be insecure? This seems to avoid the common-modulus attack and ensure that each user's public key \\(N\\) value is unique. Big mistake! This is an even worse idea! The attacker gets the public \\(N\\) values of all users and simply combines \\((N_1,N_2)\\) pairwise to solve Euclid's algorithm for the great common divisor, and a successful solution gives a prime factor \\(p\\), and a simple division gives the other prime factor \\(q\\). With \\(p\\) and \\(q\\), the attacker can immediately compute the user's private key \\(d\\). This is the non-coprime modulus attack.
\n
When applying textbook RSA, if both the public exponent \\(e\\) and the plaintext \\(m\\) are small, such that \\(c=m^e<N\\), the plaintext \\(m\\) can be obtained by directly calculating the \\(e\\)th root of the ciphertext \\(c\\). Even if \\(m^e>N\\) but not large enough, then since \\(m^e=c+k⋅N\\), you can loop through the small \\(k\\) values to perform brute-force root extraction cracking. Here is the Python routine:
\n
defcrack_small(c, e, N, repeat) times = 0 msg = 0 forkinrange(repeat): m, is_exact = gmpy2.iroot(c + times, e) if is_exact andpow(m, e, N) == c: msg = int(m) break times += N return msg
Here the gmpy2 library function iroot() is called to find the \\(e\\)th root.
\n
Textbook RSA is deterministic, meaning that the same plaintext \\(m\\) always generates the same ciphertext \\(c\\). This makes codebook attack possible: the attacker precomputes all or part of the \\(m\\to c\\) mapping table and saves, then simply searches the intercepted ciphertext for a match. Determinism also means that textbook RSA is not semantically secure and that the ciphertext can reveal some information about the plaintext. Repeated occurrences of the ciphertext indicate that the sender is sending the same message over and over again.
\n
Textbook RSA is malleable, where a particular form of algebraic operation is performed on the ciphertext and the result is reflected in the decrypted plaintext. For example, if there are two plaintexts \\(m_1\\) and \\(m_2\\), and encryption yields \\(c_1=m_1^e\\bmod N\\) and \\(c_2=m_2^e\\bmod N\\), what does \\((c_1⋅c_2)\\) decryption yield? Look at the following equation: \\[(c_1⋅c_2)^d\\equiv m_1^{ed}⋅m_2^{ed}\\equiv m_1⋅m_2\\pmod N\\] So the plaintext obtained after decrypting the product of the two ciphertexts is equal to the product of the two plaintexts. This feature is detrimental to RSA encryption systems in general and provides an opportunity for chosen-ciphertext attack. The following are two examples of attack scenarios:
\n\n
Imagine that there is an RSA decryption machine that can decrypt messages with an internally saved private key \\((N,d)\\). For security reasons, the decryptor will reject repeated input of the same ciphertext. An attacker, Marvin, finds a piece of ciphertext \\(c\\) that is rejected by the decryptor when he enters it directly because the ciphertext \\(c\\) has been decrypted before. Marvin finds a way to crack it. He prepares a plaintext \\(r\\) himself, encrypts it with the public key \\((N,e)\\) to generate a new ciphertext \\(c'={r^e}c\\bmod N\\), and then feeds the ciphertext \\(c'\\) to the decryptor. The decryption machine has not decrypted this new ciphertext, so it will not reject it. The result of the decryption is \\[m'\\equiv (c')^d\\equiv r^{ed}c^d\\equiv rm\\pmod N\\] Now that Marvin has \\(m'\\), he can calculate \\(m\\) using the formula \\(m\\equiv m'r^{-1}\\pmod N\\).
\n
Suppose Marvin wants Bob to sign a message \\(m\\), but Bob refuses to do so after reading the message content. Marvin can achieve his goal by using an attack called blinding4. He picks a random message \\(r\\), generates \\(m'={r^e}m\\bmod N\\), and then takes \\(m'\\) to Bob to sign. Bob probably thinks \\(m'\\) is irrelevant and signs it. The result of Bob's signature is \\(s'=(m')^d\\bmod N\\). Now Marvin has Bob's signature on the original message \\(m\\) using the formula \\(s=s'r^{-1}\\bmod N\\). Why? The reason is that \\[s^e\\equiv (s')^er^{-e}\\equiv (m')^{ed}r^{-e}\\equiv m'r^{-e}\\equiv m\\pmod N\\]
\n
\n
\n
The above is by no means a complete list of elementary attack methods, but they are illustrative. In practical RSA applications, we must be very careful and should do the following:
\n
\n
generate a unique public key modulus \\(N\\) for each user individually to prevent common-mode attacks
\n
not reuse the prime factor to generate the public key modulus \\(N\\), to eliminate the non-coprime modulus attack
\n
\n
For the textbook RSA deterministic and malleable flaws, and possible brute-force root extraction cracking vulnerabilities, the padding with random elements method can be used to protect against them, and the protection is valid due to the following:
\n
\n
Padding ensures that the number of bits in the encrypted message is close to \\(N\\), while not using small \\(e\\) values, making possible brute-force root extraction cracking ineffective
\n
Random padding makes the same plaintext produce different ciphertexts, guaranteeing semantic security and making ciphertext attacks impossible
\n
Strictly format-defined padding destroys malleability and reduces the possibility of ciphertext selection attacks. For example, if the first few bytes after padding must be a given value, the decrypted data will most likely not conform to the predefined format after the algebraic operation on the corresponding ciphertext, which disrupts the ciphertext selection attack.
\n
\n
Low Public Exponent Attacks
\n
Using low public exponent is dangerous, and there are advanced attacks in the case of non-padding or improper padding, even if brute-force root extraction cracking does not succeed.
\n
Broadcast Attack
\n
Discovered by Swedish theoretical computer scientist Johan Håstad 5, hence the name Håstad's Broadcast Attack. Consider this simplified scenario, assuming that Alice needs to send the same message \\(m\\) to Bob, Carol, and Dave. The public keys of the three recipients are \\((N_1,3)\\), \\((N_2,3)\\), and \\((N_3,3)\\), i.e., the public exponent is all 3 and the public key modulus is different for each. The messages are not padded and Alice directly encrypts and sends three ciphertexts \\(c_1,c_2,c_3\\) using the public keys of the other three:
At this point Eve secretly writes down the three ciphertexts, marking \\(M=m^3\\), and if she can recover \\(M\\), running a cube root naturally yields the plaintext \\(m\\). Obviously, the common modulus attack does not hold here, and we can also assume that the moduli are pairwise coprime, or else decomposing the modulus using the non-coprime modulus attack will work. So does Eve have a way to compute \\(M\\)? The answer is yes.
\n
In fact, the equivalent problem for solving \\(M\\) here is: Is there an efficient algorithm for solving a number that has known remainders of the Euclidean division by several integers, under the condition that the divisors are pairwise coprime? This efficient algorithm is Chinese Remainder Theorem!
\n
The Chinese remainder theorem gives the criterion that a system of one-element linear congruence equations has a solution and the method to solve it. For the following system of one-element linear congruence equations (be careful not to confuse it with the mathematical notation used to describe the attack scenario above):
Suppose that the integers \\(m_1,m_2,\\ldots,m_n\\) are pairwise coprime, then the system of equations \\((S)\\) has a solution for any integer \\(a_1,a_2,\\ldots,a_n\\) and the general solution can be constructed in four steps as follows:
The last line above, Eq. (4) gives the formula of the general solution. In the sense of modulus \\(M\\), the unique solution is \\(\\sum_{i=1}^n a_i t_i M_i \\bmod M\\).
\n
Try to solve the things whose number is unknown problem at the beginning of this article by using the Chinese remainder theorem
\n\n
First, correspond the variable symbols to the values: \\[m_1=3,a_1=2;\\quad m_2=5,a_2=3;\\quad m_3=7,a_3=2\\] Then calculate \\(M=3\\times5\\times7=105\\), which in turn leads to the derivation of： \\[\\begin{align}\nM_1 &=M/m_1=105/3=35,\\quad t_1=35^{-1}\\bmod 3 = 2\\\\\nM_2 &=M/m_2=105/5=21,\\quad t_2=21^{-1}\\bmod 5 = 1\\\\\nM_3 &=M/m_3=105/7=15,\\quad t_3=15^{-1}\\bmod 7 = 1\\\\\n\\end{align}\\] Finally, take these into the general solution formula: \\[x=k⋅105+(2⋅35⋅2+3⋅21⋅1+2⋅15⋅1)=k⋅105+233\\] So the smallest positive integer solution concerning modulus 105 is \\(233\\bmod 105=23\\)。
\n
In his mathematical text \"Suanfa Tongzong\", Cheng Dawei, a mathematician of the Ming Dynasty in the 16th century, compiled the solutions recorded by the mathematician Qin Jiushao of the Song Dynasty in the \"Mathematical Treatise in Nine Sections\" into a catchy \"Sun Tzu's Song\":
\n
\n
Three friends set out with seventy rare \nTwenty-one blossoms on five trees of plums \nSeven men reunited at the half-month \nAll be known once divided by one hundred and five
\n
\n
Here we must admire the wisdom of the ancient Chinese who, in the absence of a modern mathematical symbol system, were able to derive and summarize such an ingenious solution, contributing an important mathematical theorem to mankind.
\n\n\n
So Eve just applies the solution of the Chinese Remainder Theorem, computes \\(M\\), and then finds its cube root to get the plaintext \\(m\\), and the attack succeeds. More generally, setting the number of receivers to \\(k\\), if all receivers use the same \\(e\\), then this broadcast attack is feasible as long as \\(k\\ge e\\).
\n
Håstad further proves that even if padding is used to prevent broadcast attacks, if the messages generated by the padding scheme are linearly related to each other, such as using the formula \\(m_i=i2^b+m\\) (\\(b\\) is the number of bits of \\(m\\)) to generate the message sent to the receiver \\(i\\), then the broadcast attack can still recover the plaintext \\(m\\) as long as \\(k>e\\). The broadcast attack in this case is still based on the Chinese remainder theorem, but the specific cracking method depends on the information of the linear relationship.
\n
To summarize the above analysis, to prevent the broadcast attack, we must use a higher public exponent \\(e\\) and apply random padding at the same time. Nowadays, the common public key exponent \\(e\\) is $65537 (\\(2^{16}+1\\)), which can balance the efficiency and security of message encryption or signature verification operations.
\n
Last, Python routines for simulating broadcast attacks are given as follows:
\n
defsolve_crt(ai: list, mi: list): '''mi and ai are the list of modulus and remainders. The precondition of the function is that the modulus in the mi list are pairwise coprime.''' M = reduce(lambda x, y: x * y, mi) ti = [a * (M//m) * int(gmpy2.invert(M//m, m)) for (m, a) inzip(mi, ai)] return reduce(lambda x, y: x + y, ti) % M
defuint_to_bytes(x: int) -> bytes: '''convert unsigned integer to byte array''' if x == 0: returnbytes(1) return x.to_bytes((x.bit_length() + 7) // 8, 'big')
quote = b'The cosmos is within us. We are made of star stuff. - Carl Sagan' bob = RSA(1024, 3) carol = RSA(1024, 3) dave = RSA(1024, 3) cipher_list = [bob.encrypt(quote), carol.encrypt(quote), dave.encrypt(quote)] modulus_list = [bob.n, carol.n, dave.n]
This code uses two methods to simulate the broadcast attack. One calls the generic Chinese remainder theorem solver function solve_crt() and then gets the cube root of the result; the other calls the special broadcast attack function rsa_broadcast_attack() for the public key index \\(e=3\\), which directly outputs the cracked plaintext value. The internal implementation of these two functions is based on the generalized formula of the Chinese remainder theorem, and the output results should be identical. The cracked plaintext value is then input to the uint_to_bytes() function, which is converted into a byte array to compare with the original quote. Note that the program uses objects generated by the RSA class to simulate the receivers Bob, Carroll, and Dave, and the implementation of the RSA class is omitted here given the limitation of space.
\n
\n
To be continued, please look forward to the next part: RSA: Attack and Defense (2)
\n
\n\n\n\n
American computer scientist and security expert Gary McGraw has a famous piece of advice for software developers - \"never roll your own cryptography\"↩︎
\n
The original RSA paper (Part IX, Section C) did mention Miller's algorithm for factoring \\(N\\) with a known \\(d\\). This algorithm also applies to \\(d\\) generated by the Carmichael function \\(\\lambda(N)\\).↩︎
\n
gmpy2 is a Python extension module written in C that supports multi-precision arithmetic.↩︎
\n
On some special occasions, blinding can be used for effective privacy protection. For example, in cryptographic election systems and digital cash applications, the signer and the message author can be different.↩︎
\n
Johan Håstad, a Swedish theoretical computer scientist, a professor at the KTH Royal Institute of Technology, and a Fellow of the American Mathematical Society (AMS) and an Association for Computing Machinery (ACM) fellow.↩︎
\n\n\n","categories":["Technical Know-how"],"tags":["Cryptography","Network Security","Python Programming"]},{"title":"Please Stop Using TLS 1.0 and TLS 1.1 Now!","url":"/en/2022/11/10/Stop-TLS1-0-TLS1-1/","content":"
In March 2021, the Internet Engineering Task Force (IETF) released RFC 8996, classified as a current best practice, officially announcing the deprecation of the TLS 1.0 and TLS 1.1 protocols. If your applications and web services are still using these protocols, please stop immediately and update to TLS 1.2 or TLS 1.3 protocol versions as soon as possible to eliminate any possible security risks.
\n
One single vulnerability is all an attacker needs. — Window Snyder (American computer security expert, former Senior Security Strategist at Microsoft, and has been a top security officer at Apple, Intel and other companies)
\n
\n
RFC Interpretation
\n
The document title of RFC 8996 is quite straightforward, \"Deprecating TLS 1.0 and TLS 1.1\". So what is the rationale it gives? Here is a simple interpretation.
\n
First, take a look at its abstract:
\n
\n
This document formally deprecates Transport Layer Security (TLS) versions 1.0 (RFC 2246) and 1.1 (RFC 4346). Accordingly, those documents have been moved to Historic status. These versions lack support for current and recommended cryptographic algorithms and mechanisms, and various government and industry profiles of applications using TLS now mandate avoiding these old TLS versions. TLS version 1.2 became the recommended version for IETF protocols in 2008 (subsequently being obsoleted by TLS version 1.3 in 2018), providing sufficient time to transition away from older versions. Removing support for older versions from implementations reduces the attack surface, reduces opportunity for misconfiguration, and streamlines library and product maintenance.
\n
This document also deprecates Datagram TLS (DTLS) version 1.0 (RFC 4347) but not DTLS version 1.2, and there is no DTLS version 1.1.
\n
This document updates many RFCs that normatively refer to TLS version 1.0 or TLS version 1.1, as described herein. This document also updates the best practices for TLS usage in RFC 7525; hence, it is part of BCP 195.
\n
\n
The information given here is clear, the reasons for deprecating them are purely technical. TLS 1.0 and TLS 1.1 cannot support stronger encryption algorithms and mechanisms, and cannot meet the high-security requirements of various network applications in the new era. TLS is TCP-based. Corresponding to the UDP-based DTLS protocol, RFC 8996 also announced the deprecation of the DTLS 1.0 protocol.
\n
The Introduction section lists some details of the technical reasons:
\n\n
They require the implementation of older cipher suites that are no longer desirable for cryptographic reasons, e.g., TLS 1.0 makes TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA mandatory to implement.
The integrity of the handshake depends on SHA-1 hash.
\n
The authentication of the peers depends on SHA-1 signatures.
\n
Support for four TLS protocol versions increases the likelihood of misconfiguration.
\n
At least one widely used library has plans to drop TLS 1.1 and TLS 1.0 support in upcoming releases.
\n\n
Clauses 5 and 6 above are clear and need no further explanation.
\n
For 3DES mentioned in Clause 1, although it uses three independent keys with a total length of 168 bits, considering the possible meet-in-the-middle_attack attack, its effective key strength is only 112 bits. Also, the 3DES encryption block length is still 64 bits, which makes it extremely vulnerable to birthday attack (see Sweet32). NIST stipulates that a single 3DES key group can only be used for encrypting \\(2^{20}\\) data blocks (ie 8MB). This was of course too small, and eventually, NIST decided in 2017 to deprecate 3DES in the IPSec and TLS protocols.
\n
3DES is just one example, another category that has been phased out earlier is cipher suites that use RC4 stream ciphers, see RFC 7465 for details. In addition, there are various problems in the implementation of block cipher CBC mode, which are often exploited by attackers to crack TLS sessions. A summary of various attacks and countermeasures of TLS 1.0 and TLS 1.1 is described in detail in NIST800-52r2 and RFC7457. These two reference documents provide the key rationale for deprecation. Obviously, any protocol that mandates the implementation of insecure cipher suites should be on the list to be eliminated.
\n
In the second section of the document, the content in Section 1.1 \"The History of TLS\" of NIST800-52r2 is directly quoted (abbreviated as shown in the following table):
Improved initialization vector selection and padding error processing to address weaknesses discovered on the CBC mode of operation defined in TLS 1.0.
Enhanced encryption algorithms, particularly in the area of hash functions, can support SHA-2 series algorithms for hashing, MAC, and pseudorandom function computations, also added AEAD cipher suite.
A significant change to TLS that aims to address threats that have arisen over the years. Among the changes are a new handshake protocol, a new key derivation process that uses the HMAC-based Extract-and-Expand Key Derivation Function (HKDF), and the removal of cipher suites that use RSA key transport or static Diffie-Hellman key exchanges, the CBC mode of operation, or SHA-1.
\n
\n\n
\n
AEAD is an encryption mode that can guarantee the confidentiality, integrity, and authenticity of data at the same time, typically such as CCM and GCM. TLS 1.2 introduced a range of AEAD cipher suites, and its high security made it the exclusive choice for TLS 1.3. These annotate Clause 2 of technical reasons.
\n
Clauses 3 and 4 of technical reasons call out SHA-1, so what is the problem with SHA-1? Section 3 of the document cites a paper by two French researchers, Karthikeyan Bhargavan and Gaetan Leurent .
\n
As a cryptographic hash function, SHA-1 was designed by the National Security Agency (NSA) and then published as a Federal Information Processing Standard (FIPS) by the National Institute of Standards and Technology (NIST). SHA-1 can process a message up to \\(2^{64}\\) bits and generate a 160-bit (20-byte) hash value known as the message digest. Therefore, the complexity of brute force cracking based on birthday attack is \\(2^{80}\\) operations. In 2005, Chinese cryptographer Wang Xiaoyun and her research team made a breakthrough in this field. The high-efficiency SHA-1 attack method they published can be used to find a hash collision within a computational complexity of \\(2^{63}\\). This has brought a huge impact on the security of SHA-1, but it does not mean that the cracking method can enter the practical stage.
\n
Network security protocols (such as TLS, IKE, and SSH, etc.) rely on the second preimage resistance of cryptographic hash functions, that is, it is computationally impossible to find any secondary input value that has the same output as a specific input value. For example, for a cryptographic hash function \\(h(x)\\) and given input \\(x\\), it is difficult to find a sub-preimage \\(x^′ ≠ x\\) that is satisfying \\(h(x) = h(x^′)\\). Because finding a hash collision does not mean that a sub-preimage can be located, in practice, it was once thought that continuing to use SHA-1 is not a problem.
\n
However, in 2016, Bhargavan and Leurent (who implemented the aforementioned Sweet32 attack against 64-bit block ciphers) discovered a new class of methods to attack key exchange protocols that shattered this perception. These methods are based on the principle of the chosen prefix collision attack. That is, given two different prefixes \\(p_1\\) and \\(p_2\\), the attack finds two appendages \\(m_1\\) and \\(m_2\\) such that \\(h(p_1 ∥ m_1) = hash(p_2 ∥ m_2)\\). Using this approach, they demonstrated a man-in-the-middle attack against TLS clients and servers to steal sensitive data, and also showed that the attack could be used to masquerade and downgrade during TLS 1.1, IKEv2, and SSH-2 session handshakes. In particular, they proved that with only \\(2^{77}\\) operations the handshake protocol using SHA-1 or MD5 and SHA-1 concatenated hash values could be cracked.
\n
Since neither TLS 1.0 nor TLS 1.1 allows the peers to choose a stronger cryptographic hash function for signatures in the ServerKeyExchange or CertificateVerify messages, the IETF confirmed that using a newer protocol version is the only upgrade path.
\n
Sections 4 and 5 of the document again clarify that TLS 1.0 and TLS 1.1 must not be used, and negotiation to TLS 1.0 or TLS 1.1 from any TLS version is not allowed. This means that ClientHello.client_version and ServerHello.server_version issued by the TLS client and server, respectively, must not be {03,01} (TLS 1.0) or {03,02} (TLS 1.1). If the protocol version number in the Hello message sent by the other party is {03,01} or {03,02}, the local must respond with a \"protocol_version\" alert message and close the connection.
\n
It is worth noting that due to historical reasons, the TLS specification does not specify the value of the record layer version number (TLSPlaintext.version) when the client sends the ClientHello message. So to maximize interoperability, TLS servers MUST accept any value {03,XX} (including {03,00}) as the record layer version number for ClientHello messages, but they MUST NOT negotiate TLS 1.0 or 1.1.
\n
Section 6 of the document declares a textual revision to the previously published RFC 7525 (Recommendations for the Secure Use of TLS and DTLS). Three places in this RFC change implementation-time negotiations of TLS 1.0, TLS 1.1, and DTLS 1.0 from \"SHOULD NOT\" to \"MUST NOT\". The last section is a summary of standard RFC operations and security considerations.
\n
Industry Responses
\n
In the industry of large public online services, GitHub was the first to act. They started disabling TLS 1.0 and TLS 1.1 in all HTTPS connections back in February 2018, while also phasing out insecure diffie-hellman-group1-sha1 and diffie-hellman-group14-sha1 key exchange algorithms in the SSH connection service. In August 2018, Eric Rescorla, CTO of Mozilla Firefox, published the TLS 1.3 technical specification RFC 8446. Two months later, Mozilla issued a statement together with the three giants of Apple, Google, and Microsoft, and put the deprecation of TLS 1.0 and TLS 1.1 on the agenda.
\n
The following is a brief summary of the actions of several related well-known companies:
\n
\n
Microsoft: For Office 365 services, TLS 1.0 and 1.1 disabling for commercial customers was temporarily suspended due to COVID-19. The mandatory rollout of TLS 1.2 was restarted on October 15, 2020. Users of SharePoint and OneDrive will need to update and configure .NET to support TLS 1.2. Users of Teams Rooms recommend upgrading the app to version 4.0.64.0. The Surface Hub released support for TLS 1.2 in May 2019. The Edge browser version 84 does not use TLS 1.0/1.1 by default, while the Azure cloud computing service will permanently obsolete TLS 1.0/1.1 from March 31, 2022.
\n
Google: As early as 2018, TLS 1.3 was added to Chrome 70. Starting with Chrome 84, support for TLS 1.0 and TLS 1.1 is completely removed. After running TLS 1.3 in Search Engine, Gmail, YouTube, and various other Google services for some time, TLS 1.3 was officially rolled out in 2020 as the default configuration for all new and existing Cloud CDN and Global Load Balancing customers.
\n
Apple: Announced in September 2021 that TLS 1.0 and TLS 1.1 will be deprecated in iOS 15, iPadOS 15, macOS 12, watchOS 8, and tvOS 15, and support for them be completely removed in future releases. If the user's application activates the App Transport Security (ATS) feature on all connections, no changes are required. Users are also notified to ensure that the web server supports newer TLS versions and to remove the following deprecated Security.framework symbols from the app\n
\n
tls_protocol_version_t.TLSv10
\n
tls_protocol_version_t.TLSv11
\n
tls_protocol_version_t.DTLSv10
\n
\n
Mozilla: Starting with Firefox version 78, the minimum TLS version configured by default is TLS 1.2. In early 2020, Mozilla briefly removed TLS 1.0 and TLS 1.1 from Firefox completely, but this caused many users to be unable to open some COVID-19 outbreak public information sites, so the related functionality had to be restored. Following this, Mozilla provides helpful information on its technical support page, instructing users to modify the minimum TLS version number in the default configuration as needed.
\n
Cisco: The Cisco Umbrella (renamed from OpenDNS) service discontinued support for all versions of TLS prior to 1.2 on March 31, 2020. After this, only TLS 1.2 compliant clients will be able to connect. In the router and switch product lines, web management has basically been implemented around 2020 to allow only TLS 1.2 or subsequent versions.\n
\n
The CAPWAP connection between Cisco's Wireless Access Point (AP) and Wireless LAN Controller (WLC) is established over DTLS. All 802.11ac Wave 2 and 802.11ax APs from 2015 to the most recent release support DTLS 1.2. The AireOS WLC added DTLS 1.2 functionality in version 8.3.11x.0, and the next-generation C9800 WLC running IOS-XE supports DTLS 1.2 from the start. Note that because of the large number of existing network deployments using older equipment and software versions, DTLS 1.0 support cannot be removed immediately from APs and WLCs at this time to protect user investments. However, DTLS 1.2 is already the default optimal choice for APs and WLCs.
\n
\n
\n
Protocol Test
\n
Both TLS/DTLS clients and servers need to be tested to verify that their implementations follow the current best practices of RFC 8996.
\n
SSL Lab Test
\n
Qualys originated as a non-commercial SSL Labs Projects. They offer a free and simple client and server testing service, as well as a monitoring panel reporting TLS/SSL security scan statistics for the most popular Internet sites. Below is the most recent chart of protocol support statistics for November 2022.
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n\n
\n
Protocol Version
\n
Security
\n
Supporting Sites (Oct. 2022)
\n
Supporting Site (Nov. 2022)
\n
% Change
\n
\n\n\n
\n
SSL 2.0
\n
Insecure
\n
316（0.2%）
\n
303（0.2%)
\n
-0.0%
\n
\n
\n
SSL 3.0
\n
Insecure
\n
3,015（2.2%）
\n
2,930（2.2%)
\n
-0.0%
\n
\n
\n
TLS 1.0
\n
Deprecated
\n
47,450（34.9%）
\n
46,691（34.4)
\n
-0.5%
\n
\n
\n
TLS 1.1
\n
Deprecated
\n
51,674（38.1%）
\n
50,816（37.5%)
\n
-0.6%
\n
\n
\n
TLS 1.2
\n
Depending on the Cipher Suite and the Client
\n
135,557（99.8）
\n
135,445（99.9)
\n
+0.1%
\n
\n
\n
TLS 1.3
\n
Secure
\n
78,479（57.8%）
\n
79,163（58.4%)
\n
+0.6%
\n
\n\n
\n
As you can see, almost 100% of sites are running TLS 1.2, and the percentage of TLS 1.3 support is close to 60%. This is very encouraging data. While very few sites are still running SSL 2.0/3.0 and TLS 1.0/1.1 are both still supported at around 35%, overall their percentages are continuing to decline and this good trend should continue.
\n
This blog site is served by GitHub Page, enter the URL to SSL Server Test page and submit it to get a summary of the test results as follows.
\n
\n
The site achieved the highest overall security rating of A+. It got a perfect score for certificate and protocol support, and a 90 for both key exchange and password strength. This shows that GitHub fulfills its security promises to users and deserves the trust of programmers.
\n
The configuration section of the report gives details of the test results for protocol support and cipher suites as follows.
\n
\n
This further confirms that the GitHub Page only supports TLS 1.2/1.3, as required by RFC 8996. It can also be seen that under the \"Cipher Suites\" subheading, TLS 1.3 shows two GCMs and one ChaCha20-Poly1305, which are all cipher suites based on the AEAD algorithms. Three cipher suites of the same type are the preferred TLS 1.2 cipher suites for the server as well. This is exactly the current commonly adopted configuration of secure cryptographic algorithms.
\n
User Selftest
\n
If you suspect that a private server is still using the outdated TLS/SSL protocol, you can do a simple test with the command line tool curl, an example of which is as follows.
Here enter the command line option -tls-max 1.1 to set the highest protocol version 1.1 and connect to the Cisco home page. The output shows that the connection failed and that a \"protocol version\" alert message was received. This indicates that the server has rejected the TLS 1.1 connection request, and the response is exactly what is required by RFC 8996.
\n
The openssl command line tool provided by the general purpose open source cryptography and secure communication toolkit OpenSSL can also do the same test. To test whether the server supports the TLS 1.2 protocol, use the option s_client to emulate a TLS/SSL client and also enter -tls1_2 to specify that only TLS 1.2 is used. The command line runs as follows.
\n
❯ openssl s_client -connect www.cisco.com:443 -tls1_2 CONNECTED(00000005) depth=2 C = US, O = IdenTrust, CN = IdenTrust Commercial Root CA 1 verify return:1 depth=1 C = US, O = IdenTrust, OU = HydrantID Trusted Certificate Service, CN = HydrantID Server CA O1 verify return:1 depth=0 CN = www.cisco.com, O = Cisco Systems Inc., L = San Jose, ST = California, C = US verify return:1 --- Certificate chain 0 s:/CN=www.cisco.com/O=Cisco Systems Inc./L=San Jose/ST=California/C=US i:/C=US/O=IdenTrust/OU=HydrantID Trusted Certificate Service/CN=HydrantID Server CA O1 1 s:/C=US/O=IdenTrust/OU=HydrantID Trusted Certificate Service/CN=HydrantID Server CA O1 i:/C=US/O=IdenTrust/CN=IdenTrust Commercial Root CA 1 2 s:/C=US/O=IdenTrust/CN=IdenTrust Commercial Root CA 1 i:/C=US/O=IdenTrust/CN=IdenTrust Commercial Root CA 1 --- Server certificate -----BEGIN CERTIFICATE----- MIIHrzCCBpegAwIBAgIQQAF9KqwAKOKNhDf17h+WazANBgkqhkiG9w0BAQsFADBy ... 4TY7 -----END CERTIFICATE----- subject=/CN=www.cisco.com/O=Cisco Systems Inc./L=San Jose/ST=California/C=US issuer=/C=US/O=IdenTrust/OU=HydrantID Trusted Certificate Service/CN=HydrantID Server CA O1 --- No client certificate CA names sent Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 5765 bytes and written 322 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES128-GCM-SHA256 Session-ID: 1656D7D14447C1D5E68943F614A697455E60A036957D8D8C18F3B198DF42969F Session-ID-ctx: Master-Key: BB1209155344C55792077A4337964661FCA4F3F5BBF3185112F5E235BD07AD63838D24F5CF97161E696CB57398CAF478 TLS session ticket lifetime hint: 83100 (seconds) TLS session ticket: 0000 - 00 00 0b 33 d4 56 15 3d-64 e8 fa 1d cf c1 1c 04 ...3.V.=d....... ... 0090 - 1b 96 9c 25 82 70 a8 ed-24 1d 70 c9 28 56 84 59 ...%.p..$.p.(V.Y
This record is very detailed and the format is very readable. From the output, it can be understood that the digital certificate of the Cisco home page server is digitally signed and certified by the root certificate authority IdenTrust. The client-server session is built on the TLS 1.2 protocol, and the selected cipher suite is ECDHE-RSA-AES128-GCM-SHA256 of type AEAD, which is identical to the preferences provided by the GitHub Page.
\n
Browser Test
\n
If you are not sure about the security of your browser and want to test whether it still supports the pre-TLS 1.2 protocols, you can enter the following URL in your browser's address bar.
After connecting to the second URL with the default configuration of Firefox, the page shows the following
\n
\n
Secure Connection Failed
\n
An error occurred during a connection to tls-v1-1.badssl.com:1011. Peer using unsupported version of security protocol.
\n
Error code: SSL_ERROR_UNSUPPORTED_VERSION
\n
\n
The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
\n
Please contact the website owners to inform them of this problem.
\n
\n
This website might not support the TLS 1.2 protocol, which is the minimum version supported by Firefox.
\n
\n
This error message clearly indicates that Firefox is running a minimum TLS protocol version of 1.2 in this configuration, and since the other side is only running TLS 1.1, the two sides cannot establish a connection.
\n
So what is the result of the connection when the browser does still retain TLS 1.0/1.1 functionality?
\n
For testing purposes, you can first change the default TLS preference value of Firefox to 1.1 by following the steps below (refer to the figure below).
\n\n
Open a new tab, type about:config in the address bar, and press Enter/Return.
\n
The page prompts \"Proceed with Caution\", click the Accept the Risk and Continue button.
\n
In the search box at the top of the page, type TLS to display the filtered list.
\n
Find the security.tls.version.min preference option and click the Edit icon to change the minimum TLS version.\n
This bold red page tells you that the browser you are currently using does not have TLS 1.1 disabled and is a security risk, so try not to use it if you can.
\n
After testing, don't forget to restore the default TLS minimum version setting (3) for Firefox.
\n
References
\n
\n
Disclosure: This blog site is reader-supported. When you buy through the affiliate links below, as an Amazon Associate, I earn a tiny commission from qualifying purchases. Thank you.
\n
\n
Besides NIST and RFC documents, For an in-depth study of the TLS protocol specification, system implementation, and application deployment, a careful reading of the following three books is recommended.
\n
\n\n
SSL and TLS: Theory and Practice, Second Edition (2016) - This book provides a comprehensive discussion of the SSL, TLS, and DTLS protocols. It has complete details on the theory and practice of the protocols, offering readers a solid understanding of their design principles and modes of operation. The book also presents the advantages and disadvantages of the protocols compared to other Internet security protocols and provides the details necessary to correctly implement the protocols while saving time on the security practitioner’s side.
\n
Implementing SSL/TLS Using Cryptography and PKI (2011) - For a network professional who knows C programming, this book is a hands-on, practical guide to implementing SSL and TLS protocols for Internet security. Focused on how to implement SSL and TLS, it walks you through all the necessary steps, whether or not you have a working knowledge of cryptography. The book covers TLS 1.2, including implementations of the relevant cryptographic protocols, secure hashing, certificate parsing, certificate generation, and more.
\n
Bulletproof TLS and PKI, Second Edition: Understanding and Deploying SSL/TLS and PKI to Secure Servers and Web (2022) - This book is a complete guide to using TLS encryption and PKI to deploy secure servers and web applications. Written by Ivan Ristić, founder of the popular SSL Labs website, it will teach you everything you need to know to protect your systems from eavesdropping and impersonation attacks. You can also find just the right mix of theory, protocol detail, vulnerability and weakness information, and deployment advice to get the work done.
\n\n","categories":["Technology Review"],"tags":["Cryptography","Network Security"]},{"title":"TLS 1.3 and the Coming NIST Mandate","url":"/en/2023/08/21/TLS1-3-intro/","content":"
TLS (Transport Layer Security) is a cryptographic protocol to secure network communication. TLS 1.3 is the latest version of the TLS protocol, succeeding TLS 1.2. TLS 1.3 aims to provide more robust security, higher privacy protection, as well as better performance than previous versions. Here is a brief introduction to TLS 1.3. Also, we discuss NIST's requirement for TLS 1.3 readiness and give examples of enabling TLS 1.3 in some commonly used web servers.
\n
It takes 20 years to build a reputation and a few minutes of cyber-incident to ruin it. — Stéphane Nappo (Vice President and Global Chief Information Security Officer of Groupe SEB, France, 2018 Global CISO of the year)
\n
\n
Introduction to TLS 1.3
\n
TLS 1.3 is the latest recommended cryptographic protocol for protecting a wide variety of network communications, including web browsing, email, online trading, instant messaging, mobile payments, and many other applications. By using TLS 1.3, more secure and reliable communication connections can be established, ensuring confidentiality, authenticity, and data integrity. It was standardized by the Internet Engineering Task Force (IETF) in August 2018, and published as RFC 8446.
\n
TLS 1.3 introduces some important improvements over TLS 1.2. The table below presents a quick comparison of the two:
\n
\n
\n
\n
\n
\n
\n\n
\n
Aspect
\n
TLS 1.2
\n
TLS 1.3
\n
\n\n\n
\n
Protocol Design
\n
Request-response model
\n
Reduced round trips
\n
\n
\n
Handshake
\n
Multiple round trips
\n
Single round trip
\n
\n
\n
Cipher Suites
\n
Supports wide range, including insecure ones
\n
Focuses on stronger algorithms
\n
\n
\n
Security
\n
Known vulnerabilities, e.g., CBC vulnerabilities
\n
Addresses previous issues, stronger security
\n
\n
\n
Performance
\n
Higher latency due to more round trips
\n
Faster connection establishment
\n
\n
\n
Resilience to Attacks
\n
Vulnerable to downgrade attacks and padding oracle attacks
\n
Additional protections against attacks
\n
\n
\n
Compatibility
\n
Widely supported across platforms
\n
Increasing support, may not be available on older systems
\n
\n
\n
Implementation Supports
\n
Available in many cryptographic libraries
\n
Supported in various libraries
\n
\n\n
\n
It can be seen that enhanced security and performance improvements are the most notable features of TLS 1.3, and we can explore more into these in the following sections.
\n
Security Hardening
\n
Cipher Suites
\n
The protocol design principle of TLS 1.3 has enhanced security as its primary goal. As a result, TLS 1.3 drastically reduces the number of supported cipher suites. It removes insecure and weak cipher suites, leaving only more secure and modern cipher suites. This helps to increase the security of communications and avoids the use of outdated or vulnerable cipher suites.
\n
Specifically, TLS 1.3 removes various cipher suites that use static RSA key transport, static Diffie-Hellman key exchange, CBC mode of operation, or SHA-1. It adopts only a limited number of Authenticated Encryption with Associated Data (AEAD) cipher suites. AEAD can guarantee the confidentiality, integrity, and authenticity of data at the same time, and its high security makes it the exclusive choice for TLS 1.3.
\n
On the other hand, the name string of the cipher suite used in previous TLS versions included all algorithms for key exchange, digital signatures, encryption, and message authentication. Each cipher suite is assigned a 2-byte code point in the TLS Cipher Suites registry managed by the Internet Assigned Numbers Authority (IANA). Every time a new cryptographic algorithm is introduced, a series of new combinations need to be added to the list. This has led to an explosion of code points representing every valid choice of these parameters. This situation also makes the selection of cipher suites complicated and confusing.
\n
The design of TLS 1.3 changed the concept of the cipher suite. It separates the authentication and key exchange mechanisms from the record protection algorithm (including secret key length) and a hash to be used with both the key derivation function and handshake message authentication code (MAC). The new cipher suite naming convention is TLS_<AEAD>_<Hash>, where the hash algorithm is used for the newly defined key derivation function HKDF of TLS 1.3 and the MAC generation in the handshake phase. The cipher suites defined by the TLS 1.3 protocol are:
This simplified cipher suite definition and greatly reduced set of negotiation parameters also speed up TLS 1.3 handshake, improving overall performance.
\n
Key Exchange
\n
TLS 1.3 emphasizes forward secrecy, ensuring that the confidentiality of communications is protected even if long-term secrets used in the session key exchange are compromised. It only allows key exchange based on ephemeral Diffie-Hellman key exchange (DHE) or ephemeral elliptic curve Diffie-Hellman key exchange (ECDHE). Both have the property of forward secrecy. Also, the protocol explicitly restricts the use of secure elliptic curve groups and finite field groups for key exchange:
/* Finite Field Groups (DHE) */ ffdhe2048(0x0100), ffdhe3072(0x0101), ffdhe4096(0x0102), ffdhe6144(0x0103), ffdhe8192(0x0104),
\n
The above elliptic curve groups for ECDHE are specified by RFC 8422. The first three are defined by the FIPS.186-4 specification and the corresponding NIST names are P-256/P-384/P-512, while the next two (x25519/x448) are recommended by ANSI.X9-62.2005. RFC 7919 specifies four finite field groups (ffdhe####) for DHE. The primes in these finite field groups are all safe primes.
\n
In number theory, a prime number \\(p\\) is a safe prime if \\((p-1)/2\\) is also prime.
\n
\n
Signature Verification
\n
For signature verification in the key exchange phase, TLS 1.3 introduces more signature algorithms to meet different security requirements:
\n
\n
RSA signature algorithm: TLS 1.3 still supports RSA-based signature algorithms, including RSA-PKCS1-SHA256, RSA-PKCS1-SHA384, etc. These algorithms use RSA keys for digital signatures.
\n
ECDSA signature algorithm: TLS 1.3 introduces more signature algorithms based on elliptic curve cryptography (ECC), such as ECDSA-SHA256, ECDSA-SHA384, etc. These algorithms use elliptic curve keys for digital signatures and are generally superior to RSA in terms of security and performance.
\n
EdDSA signature algorithm: TLS 1.3 also introduces the EdDSA (Edwards-curve Digital Signature Algorithm) signature algorithm based on the Edwards curve. It features efficient performance and strong security for mobile devices and resource-constrained environments.
\n
RSASSA-PSS signature algorithm: In addition to the traditional RSA-PKCS1 signature algorithm, TLS 1.3 also introduces the RSASSA-PSS signature algorithm, which is a more secure signature method based on RSA and has better attack resistance.
\n
PSK signature algorithm: TLS 1.3 supports the signature algorithm based on the pre-shared key (PSK), which applies to the PSK handshake mode. This approach does not involve a digital certificate but uses a pre-shared key for verification.
\n
\n
TLS 1.3 stops using the DSA (Digital Signature Algorithm) signature algorithm. This is also a notable difference from TLS 1.2. DSA has some security and performance limitations and is rarely used in practice, so TLS 1.3 removed support for DSA certificates.
\n
Other Reinforcements
\n
Additionally, TLS 1.3 includes the following improvements to enhance security
\n
\n
TLS 1.3 does not allow data compression. The data compression feature in earlier versions of TLS could lead to security issues such as CRIME attacks. To avoid this risk, TLS 1.3 removed support for data compression entirely.
\n
Unlike earlier versions of TLS, TLS 1.3 prohibits renegotiation after the connection has been established. This helps reduce security risk and complexity. Renegotiation may introduce new security holes, and frequent negotiations during the connection process may also cause performance problems.
\n
All handshake messages following the ServerHello message during the TLS 1.3 handshake are now encrypted. The newly introduced EncryptedExtensions message enables encryption protection of various extensions previously sent in plain text.
\n
TLS 1.3 adds asymmetric cryptographic protection of the Certificate messages sent from the server to the client. This encryption prevents threats such as man-in-the-middle attacks, information leakage, and certificate forgery, further fortifying the security and privacy of the connection.
\n
\n
Performance Boosting
\n
Simplified Handshake
\n
The general trend towards high-speed mobile Internet requires the use of HTTPS/TLS to protect the privacy of all traffic as much as possible. The downside of this is that new connections can become a bit slower. For the client and web server to agree on a shared key, both parties need to exchange security attributes and related parameters through the TLS \"handshake process\". In TLS 1.2 and all protocols before it, the initial handshake process required at least two round-trip message transfers. Compared to pure HTTP, the extra latency introduced by the TLS handshake process of HTTPS can be very detrimental to performance-conscious applications.
\n
TLS 1.3 greatly simplifies the handshake process, requiring only one round trip in most cases, resulting in faster connection establishment and lower latency. Every TLS 1.3 connection will use (EC)DHE-based key exchange, and the parameters supported by the server may be easy to guess (such as ECDHE + x25519 or P-256). Since the options are limited, the client can directly send the (EC)DHE key share information in the first message without waiting for the server to confirm which key exchange it is willing to support. This way, the server can derive the shared secret one round in advance and send encrypted data.
\n
The following diagram compares the message sequences of the handshake process of TLS 1.2 and TLS 1.3. Both operate with public key-based authentication. The TLS 1.3 handshake shown below uses the symbols borrowed from the RFC 8446 specification: '+' indicates a noteworthy extension; '*' indicates an optional message or extension; '[]', '()', and '{}' represent encrypted messages, where the keys used for encryption are different.
\n\nTLS 1.2 handshake (left) vs. TLS 1.3 handshake (right)\n\n
This figure illustrates the following points:
\n
\n
TLS 1.3 removes several messages used by TLS 1.2: ServerHelloDone, ChangeCipherSpec, ServerKeyExchange, and ClientKeyExchange. The contents of TLS 1.2's ServerKeyExchange and ClientKeyExchange messages vary depending on the authentication and key-sharing method being negotiated. In TLS 1.3, this information was moved to the extensions of ClientHello and ServerHello messages. TLS 1.3 completely deprecates ServerHelloDone and ChangeCipherSpec messages, there is no replacement.
\n
For TLS 1.3 the public key-based authentication mode is probably the most important. It always uses (EC)DHE to achieve forward secrecy. The figure shows that the ClientHello message carries four extensions that are must-haves in this mode: key_share, signature_algorithms, supported_groups, and support_versions.
\n
During the TLS 1.2 handshake, the exchange of control data requires multiple round trips between the client and server. TLS 1.2's ClientKeyExchange and ChangeCipherSpec messages are carried in separate packets, and the Finished message is the first (and only) encrypted handshake message. The whole process needs to transmit 5-7 data packets.
\n
During the TLS 1.3 handshake, encrypted Application Data is already sent by the client after the first round trip. As mentioned earlier, the EncryptedExtension message provides privacy protection for ServerHello extensions in earlier versions of TLS. If mutual authentication is required (which is common in IoT deployments), the server will send a CertificateRequest message.
\n
The Certificate, CertificateVerify, and Finished messages in TLS 1.3 retain the semantics of earlier TLS versions, but they are all asymmetrically encrypted now. Echoing the description in the last section, by encrypting Certificate and CertificateVerify messages, TLS 1.3 better protects against man-in-the-middle and certificate forgery attacks while enhancing the privacy of connections. This is also an important security feature in the design of TLS 1.3.
\n
\n
In rare cases, when the server does not support a certain key-sharing method sent by the client, the server can send a new HelloRetryRequest message letting the client know which groups it supports. As the group list has shrunk significantly, this is not expected to happen very often.
\n
0-RTT Session Resumption
\n
0-RTT (Zero Round Trip Time) in TLS 1.3 is a special handshake mode. It allows clients to send encrypted data during the handshake phase, reducing the number of round trips required for connection establishment and enabling faster session resumption. The following is a brief explanation of the 0-RTT working mode:
\n\n
Store session tickets: During the normal TLS 1.3 handshake, the client and server generate a data structure called a \"session ticket\" during the handshake. Session tickets contain information about the connection, including key parameters and cipher suites. The server stores the session ticket provided by the client.
\n
0-RTT handshake: When the client reconnects to the server, it includes the previously saved session ticket in the early_data extension of the ClientHello message, along with encrypted Application Data. The client encrypts 0-RTT data using a pre-shared key (PSK) obtained from a previous connection.
\n
Server Response: After the server receives this message, if it supports 0-RTT mode and can recognize and verify the session ticket, it sends an EncryptedExtensions message, and then confirms the connection in the Finished message. This way, the server can quickly establish a secure connection with 0 round trips. It can also immediately send data to the client to achieve 0-RTT data transmission.
\n\n
The message sequence of the 0-RTT session resumption and data transmission process of TLS 1.3 is as follows:
\n\nTLS 1.3 0-RTT\n\n
FAQ
\n
\n
Does the TLS 1.3 protocol allow the use of RSA digital certificates?
\n
A common misconception is that \"TLS 1.3 is not compatible with RSA digital certificates\". The description in the \"Signature Verification\" section above shows that this is wrong. TLS 1.3 still supports the use of RSA for key exchange and authentication. However, considering the limitations of RSA, it is recommended that when building and deploying new TLS 1.3 applications, ECDHE key exchange algorithms and ECC digital certificates are preferred to achieve higher security and performance.
\n
During the TLS 1.3 handshake, how does the server request the client to provide a certificate?
\n
In some scenarios, the server also needs to verify the identity of the client to ensure that only legitimate clients can access server resources. This is the case with mTLS (mutual TLS). During the TLS 1.3 handshake, the server can specify that the client is required to provide a certificate by sending a special CertificateRequest extension. When the server decides to ask the client for a certificate, it sends a CertificateRequest extension message after the ServerHello message. This extended message contains some necessary parameters, such as a list of supported certificate types, a list of acceptable certificate authorities, and so on. When the client receives it, it knows that the server asked it for a certificate, and it can optionally respond to the request. If the client is also configured to support mTLS and decides to provide a certificate, it provides its certificate chain by sending a Certificate message.
\n
Is 0-RTT vulnerable to replay attacks?
\n
TLS 1.3's 0-RTT session resumption mode is non-interactive and does risk replay attacks in some cases. An attacker may repeat previously sent data to simulate a legitimate request. To avoid and reduce the risk of replay attacks to the greatest extent, TLS 1.3 provides some protection measures and suggestions:
\n\n
The simplest anti-replay method is that the server only allows each session ticket to be used once. For example, the server may maintain a database of all valid tickets that have not been used, deleting each ticket from the database as it is used. If an unknown ticket is received, the server falls back to a full handshake.
\n
The server may limit the time window in which session tickets are accepted, that is, the time range in which 0-RTT data is allowed to be valid. This reduces the chance of an attacker successfully replaying.
\n
Clients and servers should also use 0-RTT data only for stateless requests, that is, requests that do not affect the state of the server such as HTTP GET. For requests that need to modify the state of the server or have an impact, restrict the use of normal handshake patterns only.
\n
Another way to prevent replay is to store the unique value (usually a random value or a PSK bundled value) derived from the ClientHello message, and reject duplicates. Logging all ClientHellos would cause the state to grow without bound, but combined with #2 above, the server can log ClientHellos within a given time window and use obfuscated_ticket_age to ensure that tickets are not duplicated outside the window use.
\n
\n
If the client does not know whether the server supports TLS 1.3, how could it negotiate the TLS version via handshake?
\n
The TLS protocol provides a built-in mechanism for negotiating the running version between endpoints. TLS 1.3 continues this tradition. RFC 8446 Appendix D.1 \"Negotiating with an Older Server\" gives specific instructions:
\n
\n
A TLS 1.3 client who wishes to negotiate with servers that do not support TLS 1.3 will send a normal TLS 1.3 ClientHello containing 0x0303 (TLS 1.2) in ClientHello.legacy_version but with the correct version(s) in the \"supported_versions\" extension. If the server does not support TLS 1.3, it will respond with a ServerHello containing an older version number. If the client agrees to use this version, the negotiation will proceed as appropriate for the negotiated protocol.
\n
\n
The following screenshot of a TLS 1.3 ClientHello message decode demonstrates this. The version number of the handshake message displayed on the left is \"Version: TLS 1.2 (0x0303)\". At the same time, it can be seen that the cipher suite section first lists 3 TLS 1.3 AEAD cipher suites, followed by 14 TLS 1.2 regular cipher suites. On the right, there are 4 extensions - key_share, signature_algorithms, supported_groups, and support_versions. The support_versions extension includes both TLS 1.3 and TLS 1.2 version numbers. This is the TLS version list for the server to choose from. Additionally, the key_share extension includes the client's preferred key-sharing method as x25519 and secp256r1(i.e. NIST P-256)
\n
\n
Does the TLS 1.3 protocol work with UDP and EAP?
\n
TLS was originally designed for TCP connections, and a variant DTLS (Datagram Transport Layer Security) for UDP was introduced later. Based on TLS 1.3, IETF has released the corresponding upgraded version of the DTLS 1.3 protocol RFC 9147. The design goal of DTLS 1.3 is to provide \"equivalent security guarantees with the exception of order protection / non-replayability\". This protocol was released in April 2022, and currently, there are not many software libraries supporting it.
\n
TLS can also be used as an authentication and encryption protocol in various EAP types, such as EAP-TLS, EAP-FAST, and PEAP. Corresponding to TLS 1.3, IETF also published two technical standard documents:
\n
\n
RFC 9190: EAP-TLS 1.3: Using the Extensible Authentication Protocol with TLS 1.3 (Feb. 2022)
\n
RFC 9427: TLS-Based Extensible Authentication Protocol (EAP) Types for Use with TLS 1.3 (Jun. 2023)
\n
\n
Both protocols are also quite new, and the software library updates supporting them are still some time away.
\n
\n
NIST Mandate
\n
TLS 1.3 brings new security features and a faster TLS handshake. Since its release in 2018, many Internet services have migrated to this latest version. Nevertheless, widespread adoption across websites takes time. The non-commercial SSL Labs Projects has a dashboard called SSL Pulse that reports TLS/SSL security scan statistics for the most popular Internet sites. Below is the most recent chart of protocol support statistics by July 2023.
\n\nSource: SSL Pulse - 07/03/2023\n\n
As can be seen, of all 135,000+ probed sites the percentage of TLS 1.3 support is about 63.5%. That means there are still close to 50 thousand sites that do not leverage the security and performance benefits of TLS 1.3. Why? The decision to migrate a website to a new protocol version like TLS 1.3 can be complex and influenced by various factors. The top 3 common reasons hindering TLS 1.3 migration are
\n
\n
Compatibility Concerns: Some websites might have users who are still using outdated browsers or operating systems that do not support TLS 1.3. These websites need to maintain backward compatibility to ensure that all users can access their content securely.
\n
Resource Constraints: Migration involves technical updates, configuration changes, and testing. Smaller websites or those with limited resources might face challenges in allocating the necessary time and effort to make these changes.
\n
Third-Party Dependencies: Many websites rely on third-party services, content delivery networks, or other components. If these services do not yet support TLS 1.3, the website might delay migration to avoid disruptions or compatibility issues with these dependencies.
\n
\n
However, for network hardware/software vendors who want their products on the procurement list of any US public sector organization, there is a coming NIST mandate to make TLS 1.3 available by January 2024. This is stipulated in the National Institute of Standards and Technology Special Publication (NIST SP) 800-52 Rev. 2: Guidelines for the Selection, Configuration, and Use of Transport Layer Security (TLS) Implementations. Quoted from NIST SP 800-52 Rev. 2
\n
\n
3.1 Protocol Version Support
\n
Servers that support government-only applications shall be configured to use TLS 1.2 and should be configured to use TLS 1.3 as well. ...
\n
Servers that support citizen or business-facing applications (i.e., the client may not be part of a government IT system)10 shall be configured to negotiate TLS 1.2 and should be configured to negotiate TLS 1.3. ...
\n
Agencies shall support TLS 1.3 by January 1, 2024. After this date, servers shall support TLS 1.3 for both government-only and citizen or business-facing applications. In general, servers that support TLS 1.3 should be configured to use TLS 1.2 as well. However, TLS 1.2 may be disabled on servers that support TLS 1.3 if it has been determined that TLS 1.2 is not needed for interoperability.
\n
\n
As in the RFC documents, \"shall\" above is a strong keyword that means that the definition is an absolute requirement of the specification. So this NIST publication requires all servers owned by the US government agencies to be able to support TLS 1.3 by 01/01/2024. They must run a minimum TLS version 1.2 by default and can be configured to do TLS 1.3 only if desired.
\n
It is worth pointing out that this is not an official FIPS requirement, so not mandatory for the FIPS 140-3 certification at present. Besides, this NIPS document has a clear scope statement: \"The scope is further limited to TLS when used in conjunction with TCP/IP. For example, Datagram TLS (DTLS), which operates over datagram protocols, is outside the scope of these guidelines. NIST may issue separate guidelines for DTLS at a later date.\" Based on this, we can infer that DTLS and EAP are out of consideration for this mandate.
\n
Enabling TLS 1.3
\n
The enhanced security and optimized performance of TLS 1.3 make it the first choice for securing communication of various network applications. Now we demonstrate how to enable TLS 1.3 function in three commonly used web server software Apache, Nginx, and Lighttpd.
\n
NOTE: The implementation of many secure network communication applications relies on third-party SSL/TLS software libraries, such as wolfSSL, GnuTLS, NSS, and OpenSSL. Therefore, to enable the TLS 1.3 function of these applications, you need to ensure that the libraries they link with support TLS 1.3. For example, in September 2018, the popular OpenSSL project released version 1.1.1 of the library, with support for TLS 1.3 as its \"top new feature\".
\n
\n
Apache HTTP Server
\n
The Apache HTTP Server is an open-source web server software from the Apache Software Foundation. Apache HTTP server is widely used and is one of the most popular web server software due to its cross-platform and security. Apache supports a variety of features, many of which extend core functionality through compiled modules, such as authentication schemes, proxy servers, URL rewriting, SSL/TLS support, and compiling interpreters such as Perl/Python into the server.
\n
Apache HTTP Server has built-in support for TLS 1.3 since version 2.4.36, no need to install any additional modules or patches. The following command can be used to verify the version of the server
\n
$ apache2ctl -v Server version: Apache/2.4.41 (Ubuntu) Server built: 2020-04-13T17:19:17
\n
Once the version is verified, the SSLProtocol line of the configuration file can be updated. The following will enable the Apache HTTP server to only support the TLS 1.3 protocol
\n
/etc/apache2/mods-available/ssl.conf
# Only enable TLS 1.3 SSLProtocol -all +TLSv1.3
\n
If the server needs to be compatible with clients that support TLS 1.2, you can add +TLSv1.2. After updating the configuration, restart the service
\n
$ sudo service apache2 restart
\n
Nginx Web Server
\n
Nginx is a high-performance web server based on an asynchronous framework and modular design. It can also be used for reverse proxy, load balancer, and HTTP caching applications. It is free and open-source software released under the terms of a BSD-like license. Nginx uses an asynchronous event-driven approach to request processing, which can provide more predictable performance under high load. The current market share of Nginx is almost equal to that of the Apache HTTP server.
\n
Nginx supports TLS 1.3 from version 1.13.0. The following command can be used to verify its version
\n
$ nginx -v nginx version: nginx/1.17.10 (Ubuntu)
\n
In the Nginx configuration file, find the server block and modify the ssl_protocols line to enable TLS 1.3:
\n
/etc/nginx/nginx.conf
server { listen443 ssl http2; listen [::]:443 ssl http2; server_name example.com; root /var/www/example.com/public; ssl_certificate /path/to/your/certificate.crt; ssl_certificate_key /path/to/your/private-key.key; # support TLS 1.2 and TLS 1.3 ssl_protocols TLSv1.2 TLSv1.3; ... }
\n
If you don't need to continue to support TLS 1.2, delete the TLSv1.2 there. After the modification is complete, you can run the following command to test the configuration of Nginx, and then restart the service
\n
$ sudo nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful
$ sudo service nginx restart
\n
Lighttpd Web Server
\n
Lighttpd is a lightweight open-source web server software. It focuses on high performance, low memory footprint, and fast responsiveness. Lighttpd is suitable for serving web applications and static content of all sizes. Its design goal is to provide an efficient, flexible, and scalable web server, especially suitable for high-load and resource-constrained (such as embedded systems) environments.
\n
The first Lighttpd release to support TLS 1.3 is version 1.4.56. Starting with this version, the minimum version of TLS that Lighttpd supports by default is TLS 1.2. That is to say, Lighttpd supports TLS 1.2 and TLS 1.3 if no corresponding configuration file modification is made.
\n
To limit the use of Lighttpd to only the TLS 1.3 feature, first make sure the mod_openssl module is loaded. Then in the configuration file lighttpd.conf, find the server.modules section, and add the following ssl.openssl.ssl-conf-cmd line:
This will set the minimum version supported by Lighttpd to be TLS 1.3. Finally, save and reload the Lighttpd configuration for the changes to take effect:
By chance, I came across a picoCTF RSA challenge called Sum-O-Primes. This problem is not difficult, you can do it by knowing the basics of the RSA algorithm. In addition, if you are familiar with the history of the evolution of the RSA algorithm, you can find a second ingenious fast solution.
\n
picoCTF Project
\n
picoCTF is a free computer security education program created by security and privacy experts at Carnegie Mellon University. It uses original content built on the CTF (Capture the Flag) framework to provide a variety of challenges. It provides participants with valuable opportunities to systematically learn cybersecurity knowledge and gain practical experience.
\n
The collection of practice questions for picoCTF is called picoGym. The general problem solution is to search or decipher a string in the format \"picoCTF{...}\" from the given information, that is, the flag to be captured. As shown in the figure below, picoGym currently contains 271 cybersecurity challenge exercises, covering general skills, cryptography, reverse engineering, forensics, and other fields.
\n
\n
Sum-O-Primes Challenge
\n
There are 50 cryptography-related challenges in picoGym, one of which is Sum-O-Primes. The task of this challenge is simple and explained as follows:
\n
\n
We have so much faith in RSA we give you not just the product of the primes, but their sum as well!
That is, we not only give the product of the two prime numbers used by RSA but also tell you their sum. How are these given? You need to discover by yourself from the rest of the information. After clicking the two links and downloading the file, open the first Python file:
\n
gen.py
#!/usr/bin/python
from binascii import hexlify from gmpy2 import mpz_urandomb, next_prime, random_state import math import os import sys
FLAG = open('flag.txt').read().strip() FLAG = int(hexlify(FLAG.encode()), 16) SEED = int(hexlify(os.urandom(32)).decode(), 16) STATE = random_state(SEED)
If you have basic Python programming skills and understand the principles of the RSA algorithm, you should be able to read the above program quickly. What it does is:
\n\n
Open the file flag.txt to read the content. Then use the hexlify and int functions to convert it to an integer and store the result in a variable FLAG.
\n
Call the function get_prime to generate two prime numbers, store their sum in x and their product in n. Then assign 65537 to e and calculate the RSA private exponent d.
\n
Use standard pow functions to perform modular exponentiation, which implements RSA encryption to encrypt plaintext FLAG into ciphertext c.
\n
Print out x, n, and c.
\n\n
Open the second file, which is apparently the output of the first program in Python:
\n
output.txt
x = 154ee809a4dc337290e6a4996e0717dd938160d6abfb651736d9f5d524812a659b310ad1f221196ee8ab187fa746a1b488a4079cddfc5db08e78be0d96c83c01e9bb42420b40d6f0ad9f220633459a6dc058bb01c517386bfbd2d4811c9b08558b0e05534768581a74884758d15e15b4ef0dbd6a338bf1f52eed4f137957737d2 n = 6ce91e471f1df651b0d275d6d5522703feecdd77e7821a2caf9514104c059781c1b2e64772d9220addd657ecbd4e6cb8b5941608f6ab54bd5760074a5cd5854920439422192d2ee8912f1ebcc0d97714f209ee2a22e2da60e071541cb7e0772373cfea71831673378ee6432e63abfd14db0d4aa601928923253f9edd419ce96f4d68ce0aa3e6d6b530cd46eefbdac93038ce949c9dd2e573a47471cf8223f88b96e00a92f4d47fd277c42c4075b5e99b41a9f279f442bc0d533b9ddc50592e369e7026b3f7afaa8edf8972f0c3055f4de67a0eea963f099a32e1539de1d1727abadd9235f66371998ec883d1f89b8d907270842818cae49cd5c7f906c4752e81 c = 48b89662b9718fb391c96527272bf74c27810edaca09b63e694af9d11608010b1db9aedd1c867849371121941a1ccac610f7b28b92fa2f981babe816e6d3ecfab83514ed7e18e2b23fc3b96c7002ff47da897e9f2a9cb1b4e245396589e0b72affb73568a2016031555d2a46557919e44a15cd43fe9e1881d40dce1d1e36625e63b1472d3c317898102943072e06d79688c96b6ee2e584002c66497a9cdc48c38aa0548a7bc4fed9b4c23fcd493f38ece68788ef37a559b7f20c6941fcf8e567d9f50807259a7f11fa7a01d3125a1f7609cd94781f224ec8351605354b11c6b078fe015826342c3271ee3af4b99bb0a538b1e6b845594ee6546be8abd22ef2bd
\n
Once you understand the meaning of the question, you can make a judgment immediately —— if you can decrypt the ciphertext c and retrieve the plaintext FLAG, you can get the original content of flag.txt, that is, capture the flag.
\n
Conventional Solution
\n
RSA decryption requires a private key exponent d. Referring to the steps of the RSA algorithm below, it is obvious that this demands integer factorization for large prime numbers p and q first.
\n\n
Choose two large prime numbers \\(p\\) and \\(q\\), compute \\(n=pq\\)
\n
Compute Carmichael function\\(\\lambda(n)=\\operatorname{lcm}(p − 1, q − 1)\\) the product, \\(\\operatorname{lcm}\\) is a function to find the least common multiple
\n
Choose any number \\(e\\) that is less than and coprime to \\(\\lambda(n)\\), then compute \\(d\\), the modular multiplicative inverse of \\(e\\) regarding \\(\\lambda(n)\\), \\(d\\equiv e^{-1}\\pmod {\\lambda(n)}\\)
\n
\\((n,e)\\) is the RSA public key, \\((n,d)\\) the RSA private key
\n
Use the public key to encrypt the plaintext \\(m\\), the formula is \\(c\\equiv m^e\\pmod n\\)
\n
Use the private key to decrypt the ciphertext \\(c\\), the formula is \\(m\\equiv c^d\\pmod n\\)
\n\n
From here, the challenge becomes a problem that, knowing the sum and product of two large prime numbers known, find these two large prime numbers. That is, to solve a system of quadratic linear equations
We can get \\(p\\) and \\(q\\). The rest of the work is easy. The code to compute \\(d\\) from \\(p\\) and \\(q\\) can be copied directly from lines 28, 30, and 31 in gen.py. The final complete Python problem-solving code is as follows:
x = int((Lines[0].split())[2], 16) # x = p + q n = int((Lines[1].split())[2], 16) # n = p * q c = int((Lines[2].split())[2], 16) # Ciphertext
defsolve_rsa_primes(s: int, m: int) -> tuple: ''' Solve RSA prime numbers (p, q) from the quadratic equation p^2 - s * p + m = 0 with the formula p = s/2 +/- sqrt((s/2)^2 - m) Input: s - sum of primes, m - product of primes Output: (p, q) ''' half_s = s >> 1 tmp = math.isqrt(half_s ** 2 - m) returnint(half_s + tmp), int(half_s - tmp);
# Now run with the real input p, q = solve_rsa_primes(x, n) m = math.lcm(p - 1, q - 1) e = 65537 d = pow(e, -1, m) FLAG = pow(c, d, n) print(FLAG.to_bytes((FLAG.bit_length() + 7) // 8, 'big'))
\n
The above program defines a general function solve_rsa_primes to solve two large prime numbers. After it gets d, the same pow function is called to decrypt, and finally the plaintext is converted from a large integer to a byte sequence and printed out. The result of running this program is
Note: The function solve_rsa_primes calls math.isqrt to compute the integer square root of the given integer. This is indispensable! If it is written incorrectly with math.sqrt, the following overflow error will occur
\n
>>> =============== RESTART: /Users/zixi/Downloads/Sum-O-Primes.py ============== Traceback (most recent call last): File "/Users/zixi/Downloads/Sum-O-Primes.py", line 35, in <module> p, q = solve_rsa_primes(x, n) File "/Users/zixi/Downloads/Sum-O-Primes.py", line 31, in solve_rsa_primes tmp = math.sqrt(int(half_s ** 2 - m)) OverflowError: int too large to convert to float
\n
This error happens because math.sqrt uses floating-point arithmetic but fails to convert large integers to floating-point numbers.
\n
\n
Quick Solution
\n
The conventional solution to this problem has to solve a quadratic equation, so the integer square root operation is essential. Is there a solution that doesn't need a square root operation? The answer is yes.
\n
In the original RSA paper, the public exponent \\(e\\) and the private exponent \\(d\\) have the relationship as the following equation
\n
\\[d⋅e≡1\\pmod{\\varphi(n)}\\]
\n
Here the modular is the Euler's totient function \\(\\varphi(n)=(p-1)(q-1)\\). Since \\(\\varphi(N)\\) is always divisible by \\(\\lambda(n)\\), any d satisfying the above also satisfies \\(d⋅e≡1\\pmod{\\lambda(n)}\\), thus the private exponent is not unique. Although the calculated \\(d>\\lambda(n)\\), the square root operation can be avoided when applied to the Sum-O-Primes problem. This is because \\[\n\\begin{aligned}\n\\varphi(n)&=(p-1)(q-1)\\\\\n&=pq-(p+q)+1\\\\\n&=n-x+1\n\\end{aligned}\n\\]
\n
Hereby the formula for computing the private exponent becomes
Now that \\(n\\) and \\(x\\) are readily available, this method does not require finding \\(p\\) and \\(q\\) first, and naturally, there is no need for a square root operation. The Python code for this new solution is very concise
\n
d1 = pow(e, -1, n - x + 1) FLAG = pow(c, d1, n) print(FLAG.to_bytes((FLAG.bit_length() + 7) // 8, 'big'))
As shown above, this solution also succeeds in capturing the flag. The \\(d\\) value (d1) calculated by the new solution is more than 7 times that of the conventional solution.
\n","categories":["Technical Know-how"],"tags":["Cryptography","Python Programming","CTF"]},{"title":"Notes on Using uClibc Standard Library in Embedded Linux System","url":"/en/2023/03/10/uClibc-tips/","content":"
uClibc is a small and exquisite C standard library for embedded Linux systems. It is widely used in the development of low-end embedded systems and Internet of Things devices. Here are some recent experiences to provide convenience for engineers who need to solve similar problems or meet corresponding requirements.
\n
Low-level programming is good for the programmer's soul. — John Carmack (American computer programmer and video game developer, co-founder of the video game company id Software)
\n
\n
Introduction to uClibc
\n
uClibc (sometimes written as μClibc) is a small C standard library designed to provide support for embedded systems and mobile devices using operating systems based on the Linux kernel. uClibc was originally developed to support μClinux, a version of Linux not requiring a memory management unit thus especially suited for microcontroller systems. The \"uC\" in its name is the abbreviation of microcontroller in English, where \"u\" is a Latin script typographical approximation of the Greek letter μ that stands for \"micro\".
\n
uClibc is a free and open-source software licensed under the GNU Lesser GPL, and its library functions encapsulate the system calls of the Linux kernel. It can run on standard or MMU-less Linux systems and supports many processors such as i386, x86-64, ARM, MIPS, and PowerPC. Development of uClibc started in 1999 and was written mostly from scratch, but also absorbed code from glibc and other projects. uClibc is much smaller than glibc. While glibc aims to fully support all relevant C standards on a wide range of hardware and kernel platforms, uClibc focuses on embedded Linux systems. It also allows developers to enable or disable some features according to the memory space design requirements.
\n
The following records show the list of C standard library files in two similar embedded systems. The first uses glibc-2.23 version, and the second integrates uClibc-0.9.33.2 version. The total size of glibc library files is more than 2MB, while the uClibc library files add up to less than 1MB. It can be seen that using uClibc does save a lot of storage space.
\n
STM1:/# find . -name "*lib*2.23*" | xargs ls -alh -rwxr-xr-x 1 root root 9.6K Jan 1 1970 ./lib/libanl-2.23.so -rwxr-xr-x 1 root root 1.1M Jan 1 1970 ./lib/libc-2.23.so -rwxr-xr-x 1 root root 177.5K Jan 1 1970 ./lib/libcidn-2.23.so -rwxr-xr-x 1 root root 29.5K Jan 1 1970 ./lib/libcrypt-2.23.so -rwxr-xr-x 1 root root 9.5K Jan 1 1970 ./lib/libdl-2.23.so -rwxr-xr-x 1 root root 429.4K Jan 1 1970 ./lib/libm-2.23.so -rwxr-xr-x 1 root root 65.8K Jan 1 1970 ./lib/libnsl-2.23.so -rwxr-xr-x 1 root root 17.5K Jan 1 1970 ./lib/libnss_dns-2.23.so -rwxr-xr-x 1 root root 33.6K Jan 1 1970 ./lib/libnss_files-2.23.so -rwxr-xr-x 1 root root 90.5K Jan 1 1970 ./lib/libpthread-2.23.so -rwxr-xr-x 1 root root 65.7K Jan 1 1970 ./lib/libresolv-2.23.so -rwxr-xr-x 1 root root 25.9K Jan 1 1970 ./lib/librt-2.23.so -rwxr-xr-x 1 root root 9.5K Jan 1 1970 ./lib/libutil-2.23.so
STM2:/# find . -name "*lib*0.9.33*" | xargs ls -alh -rwxr-xr-x 1 root root 28.0K Jan 1 1970 ./lib/ld-uClibc-0.9.33.2.so -rwxr-xr-x 1 root root 36.1K Jan 1 1970 ./lib/libcrypt-0.9.33.2.so -rwxr-xr-x 1 root root 16.2K Jan 1 1970 ./lib/libdl-0.9.33.2.so -rwxr-xr-x 1 root root 72.1K Jan 1 1970 ./lib/libm-0.9.33.2.so -rwxr-xr-x 1 root root 116.4K Jan 1 1970 ./lib/libpthread-0.9.33.2.so -rwxr-xr-x 1 root root 16.2K Jan 1 1970 ./lib/librt-0.9.33.2.so -rwxr-xr-x 1 root root 28.3K Jan 1 1970 ./lib/libthread_db-0.9.33.2.so -rwxr-xr-x 1 root root 621.4K Jan 1 1970 ./lib/libuClibc-0.9.33.2.so -rwxr-xr-x 1 root root 8.1K Jan 1 1970 ./lib/libubacktrace-0.9.33.2.so -rwxr-xr-x 1 root root 4.1K Jan 1 1970 ./lib/libutil-0.9.33.2.so
\n
IPv6 and Interface API
\n
With the steady growth of IPv6 deployment, adding IPv6 protocol stack support for embedded systems has become necessary. In a software project that adds IPv4/IPv6 dual-stack function to devices using uClibc, it is found that there is an application link error - undefined reference to getifaddrs. getifaddrs() is a very useful function, we can call it to get the address information of all the network interfaces of the system. Query the Linux programming manual:
intgetifaddrs(struct ifaddrs **ifap); ... \t DESCRIPTION The getifaddrs() function creates a linked list of structures describing the network interfaces of the local system, and stores the address of the first item of the list in *ifap. ... VERSIONS The getifaddrs() function first appeared in glibc 2.3, but before glibc 2.3.3, the implementation supported only IPv4 addresses; IPv6 support was added in glibc 2.3.3. Support of address families other than IPv4 is available only on kernels that support netlink. ...
\n
The last sentence above is key: only kernels supporting netlink can support address families other than IPv4. The Linux kernel version running on this system is 3.x, which supports netlink. So, could there be a problem with uClibc's support for netlink that causes getifaddrs() not to get compiled?
\n
With this question in mind, search the source code directory of uClibc and find the C file that implements the function getifaddrs():
\n
libc/inet/ifaddrs.c
... #if __ASSUME_NETLINK_SUPPORT #ifdef __UCLIBC_SUPPORT_AI_ADDRCONFIG__ /* struct to hold the data for one ifaddrs entry, so we can allocate everything at once. */ structifaddrs_storage { structifaddrsifa; union { /* Save space for the biggest of the four used sockaddr types and avoid a lot of casts. */ structsockaddrsa; structsockaddr_llsl; structsockaddr_ins4; #ifdef __UCLIBC_HAS_IPV6__ structsockaddr_in6s6; #endif } addr, netmask, broadaddr; char name[IF_NAMESIZE + 1]; }; #endif/* __UCLIBC_SUPPORT_AI_ADDRCONFIG__ */ ... #ifdef __UCLIBC_SUPPORT_AI_ADDRCONFIG__ ... int getifaddrs(struct ifaddrs **ifap) ... #endif/* __UCLIBC_SUPPORT_AI_ADDRCONFIG__ */ ... #endif/* __ASSUME_NETLINK_SUPPORT */
\n
Just as expected! The implementation of the entire function and the definition of the associated data structure ifaddrs_storageare are placed under three nested conditional compilation directives with macros defined as
\n\n
__ASSUME_NETLINK_SUPPORT
\n
__UCLIBC_SUPPORT_AI_ADDRCONFIG__
\n
__UCLIBC_HAS_IPV6__
\n\n
Therefore, as long as their corresponding configuration lines are opened, the problem should be solved. After changing the configuration file of uClibc as follows, rebuild the dynamic link library of uClibc, then the application can be made successfully:
\n
--- a/toolchain/uClibc/config-0.9.33.2/common +++ b/toolchain/uClibc/config-0.9.33.2/common @@ -147,7 +147,8 @@ UCLIBC_HAS_RPC=y UCLIBC_HAS_FULL_RPC=y -# UCLIBC_HAS_IPV6 is not set +UCLIBC_HAS_IPV6=y -# UCLIBC_USE_NETLINK is not set +UCLIBC_USE_NETLINK=y +UCLIBC_SUPPORT_AI_ADDRCONFIG=y UCLIBC_HAS_BSD_RES_CLOSE=y
\n
SHA-2 Hash Function
\n
Embedded systems often need to provide remote SSH login services for system administrators, which requires the creation of system users and their passwords. Linux saves the user name and the hashed password in the /etc/shadow file. The storage format of the hash value follows a de facto standard called the Modular Crypt Format (MCF for short), and its format is as follows:
id: indicates the identifier of the hash algorithm (eg 1 for MD5, 5 for SHA-256, 6 for SHA-512)
\n
param=value: Hash complexity parameters (such as the number of rounds/iterations) and their values
\n
salt: radix-64 (charset [+/a-zA-Z0-9]) encoded salt
\n
hash: the radix-64 encoded hash result of the password and salt
\n
\n
With the rapid increase of computing power following Moore's Law, the previously commonly used MD5-based hashing scheme has become obsolete because it is too vulnerable to attack. Newly designed systems are now switched to the SHA-512 hashing scheme, corresponding to $6$ seen in the /etc/shadow file.
\n
Both generation and verification of user password hash values can be implemented with the POSIX C library function named crypt. This function is defined as follows:
\n
char *crypt(constchar *key, constchar *salt)
\n
The input parameter key points to the string containing the user's password, and salt points to a string in the format $<id>$<salt> indicating the hash algorithm and salt to be used. Most Linux distributions use the crypt function provided by the glibc library. The following figure summarizes the augmented crypt function in Glibc:
\n
\n
In an embedded Linux system integrating uClibc, uClibc provides support for the crypt function. But the test found that it returned a null pointer for the correct \\(6\\) input! What's going on here?
\n
The answer lies in the uClibc's implementation of the crypt function. Find the corresponding C source code:
if (salt[0] == '$') { if (salt[1] && salt[2] == '$') { /* no blowfish '2X' here ATM */ if (*++salt == '1') return __md5_crypt(ukey, usalt); #ifdef __UCLIBC_HAS_SHA256_CRYPT_IMPL__ elseif (*salt == '5') return __sha256_crypt(ukey, usalt); #endif #ifdef __UCLIBC_HAS_SHA512_CRYPT_IMPL__ elseif (*salt == '6') return __sha512_crypt(ukey, usalt); #endif } /* __set_errno(EINVAL);*//* ENOSYS might be misleading */ returnNULL; } return __des_crypt(ukey, usalt); }
\n
Aha! It turns out that it only does MD5 hashing by default, and the codes of SHA-256 and SHA-512 need their own conditional compilation macro definitions. This is easy to handle, just edit the configuration file of uClibc and open the latter two.
\n
--- a/toolchain/uClibc/config-0.9.33.2/common +++ b/toolchain/uClibc/config-0.9.33.2/common @@ -151,8 +151,8 @@ UCLIBC_HAS_REGEX_OLD=y UCLIBC_HAS_RESOLVER_SUPPORT=y -# UCLIBC_HAS_SHA256_CRYPT_IMPL is not set -# UCLIBC_HAS_SHA512_CRYPT_IMPL is not set +UCLIBC_HAS_SHA256_CRYPT_IMPL=y +UCLIBC_HAS_SHA512_CRYPT_IMPL=y UCLIBC_HAS_SHADOW=y
\n
Finally, take a look at the program that comes with uClibc to test the SHA-512 hash algorithm. It clearly lists the data structures defined by the test code, including the salt, the input password, and the expected output, as well as several test vectors:
\n
test/crypt/sha512c-test.c
staticconststruct { constchar *salt; constchar *input; constchar *expected; } tests[] = { { "$6$saltstring", "Hello world!", "$6$saltstring$svn8UoSVapNtMuq1ukKS4tPQd8iKwSMHWjl/O817G3uBnIFNjnQJu" "esI68u4OTLiBFdcbYEdFCoEOfaS35inz1" }, { "$6$rounds=10000$saltstringsaltstring", "Hello world!", "$6$rounds=10000$saltstringsaltst$OW1/O6BYHV6BcXZu8QVeXbDWra3Oeqh0sb" "HbbMCVNSnCM/UrjmM0Dp8vOuZeHBy/YTBmSK6H9qs/y3RnOaw5v." }, ... { "$6$rounds=10$roundstoolow", "the minimum number is still observed", "$6$rounds=1000$roundstoolow$kUMsbe306n21p9R.FRkW3IGn.S9NPN0x50YhH1x" "hLsPuWGsUSklZt58jaTfF4ZEQpyUNGc0dqbpBYYBaHHrsX." }, };
\n
It can be seen that the last test case defines the round value 10 ($6$rounds=10$roundstoolow), while the output shows that the round is 1000 (rounds=1000). This confirms that the crypt function implementation of uClibc matches the augmented function of Glibc - in order to ensure security, if the input specified round is too small, crypt will automatically set to the minimum round of 1000.
\n
DNS Security Patch
\n
In early May 2022, Nozomi Networks, a company focused on providing security solutions for industrial and critical infrastructure environments, released a newly discovered uClibc security vulnerability CVE-2022-30295. This vulnerability exists in the Domain Name System (DNS) implementation of all versions of uClibc and its fork uClibc-ng (prior to version 1.0.41). Since the implementation uses predictable transaction IDs when making DNS requests, there is a risk of DNS cache poisoning attacks.
\n
Specifically, applications often call gethostbyname library functions to resolve a network address for a given hostname. uClibc/uClibc-ng internally implements a __dns_lookup function for the actual DNS domain name request and response processing. Taking the last version 0.9.33.2 of uClibc as an example, the screenshot below shows the problematic code in the function __dns_lookup:
\n
\n
Referring to line 1308, at the first DNS request, the variable local_id is initialized to the transaction ID value of the last DNS request (stored in a static variable last_id). Line 1319 is the actual culprit, it simply updates the old local_id value by incrementing it by 1. This new value is stored back into the variable last_id, as shown on line 1322. Finally, on line 1334, the value of local_id is copied into the structure variable h, which represents the actual content of the DNS request header. This code works pretty much in all available versions of uClibc and uClibc-ng prior to version 1.0.41.
\n
This implementation makes the transaction ID in the DNS request predictable, because the attacker can estimate the value of the transaction ID in the next request as long as he/she detects the current transaction ID. By exploiting this vulnerability, an attacker can disrupt/poison the host's DNS cache by crafting a DNS response containing the correct source port and winning the competition with the legitimate response returned by the DNS server, making the network data of the application in the host system be directed to a trap site set by the attacker.
\n
The maintainers of uClibc-ng responded quickly to the announcement of this security vulnerability. They submitted a fix in mid-May 2022, and released version 1.0.41 including this patch at the end of that month. For uClibc, since this C standard library has stopped releasing any new versions since 2012, it is currently in an unmaintained state, so system R&D engineers need to come up with their repair. The following uClibc patches are available for reference:
\n
diff --git a/libc/inet/resolv.c b/libc/inet/resolv.c index 31e63810b..c2a8e2be4 100644 --- a/libc/inet/resolv.c +++ b/libc/inet/resolv.c @@ -315,6 +315,7 @@ Domain name in a message can be represented as either: #include <sys/utsname.h> #include <sys/un.h> #include <sys/stat.h> +#include <fcntl.h> #include <sys/param.h> #include <bits/uClibc_mutex.h> #include "internal/parse_config.h" @@ -1212,6 +1213,20 @@ static int __decode_answer(const unsigned char *message, /* packet */ return i + RRFIXEDSZ + a->rdlength; }
+uint16_t dnsrand_next(int urand_fd, int def_value) { + if (urand_fd == -1) return def_value; + uint16_t val; + if(read(urand_fd, &val, 2) != 2) return def_value; + return val; +} + +int dnsrand_setup(int *urand_fd, int def_value) { + if (*urand_fd > 0) return dnsrand_next(*urand_fd, def_value); + *urand_fd = open("/dev/urandom", O_RDONLY); + if (*urand_fd == -1) return def_value; + return dnsrand_next(*urand_fd, def_value); +} + /* On entry: * a.buf(len) = auxiliary buffer for IP addresses after first one * a.add_count = how many additional addresses are there already @@ -1237,6 +1252,7 @@ int __dns_lookup(const char *name, /* Protected by __resolv_lock: */ static int last_ns_num = 0; static uint16_t last_id = 1; + static int urand_fd = -1;
int i, j, fd, rc; int packet_len; @@ -1305,7 +1321,7 @@ int __dns_lookup(const char *name, } /* first time? pick starting server etc */ if (local_ns_num < 0) { - local_id = last_id; + local_id = dnsrand_setup(&urand_fd, last_id); /*TODO: implement /etc/resolv.conf's "options rotate" (a.k.a. RES_ROTATE bit in _res.options) local_ns_num = 0; @@ -1316,8 +1332,9 @@ int __dns_lookup(const char *name, retries_left--; if (local_ns_num >= __nameservers) local_ns_num = 0; - local_id++; + local_id = dnsrand_next(urand_fd, local_id++); local_id &= 0xffff; + DPRINTF("local_id:0x%hx\\n", local_id); /* write new values back while still under lock */ last_id = local_id; last_ns_num = local_ns_num;
\n
This uClibc patch is a simplified version of the uClibc-ng official patch. Its core is to read a double-byte random number from the system /dev/urandom file, and then use it to set the original local_id, the transaction ID of the DNS request. /dev/urandom is a special device file of the Linux system. It can be used as a non-blocking random number generator, which will reuse the data in the entropy pool to generate pseudo-random data.
\n
Note that in the above patch, the function dnsrand_setup must first check urand_fd whether it is positive, and only open /dev/urandom when it is not true. Otherwise, the file will be reopened every time the application does a DNS lookup, the system will quickly hit the maximum number of file descriptors allowed, and the system will crash because it cannot open any more files.
\n
Finally, a comparison of an embedded system using uClibc before and after adding DNS security patches is given. The following are the DNS packets intercepted by two sniffers. In the first unpatched system, the transaction ID of the DNS request is incremented in sequence, which is an obvious security hole; the second is after the patch is added, the transaction ID of each DNS request is a random value, and the loophole has been filled.
\n
\n","categories":["Technical Know-how"],"tags":["C/C++ Programming","System Programming","Computer Communications","Cryptography","TCP/IP"]}]
\ No newline at end of file
+[{"title":"AddressSanitizer - A Tool for Programmers to Detect Memory Access Errors","url":"/en/2022/04/22/ASAN-intro/","content":"
Memory access errors are the most common software errors that often cause program crashes. The AddressSanitizer tool, developed by Google engineers in 2012, has become the first choice of C/C++ programmers for its wide coverage, high efficiency, and low overhead. Here is a brief introduction to its principle and usage.
\n
One man's \"magic\" is another man's engineering. \"Supernatural\" is a null word. — Robert Anson Heinlein (American science fiction author, aeronautical engineer, and naval officer)
\n
\n
Tool Overview
\n
The C/C++ language allows programmers to have low-level control over memory, and this direct memory management has made it possible to write efficient application software. However, this has also made memory access errors, including buffer overflows, accesses to freed memory, and memory leaks, a serious problem that must be coped with in program design and implementation. While there are tools and software that provide the ability to detect such errors, their operational efficiency, and functional coverage are often less than ideal.
\n
In 2012, Google engineer Konstantin Serebryany and team members released an open-source memory access error detector for C/C++ programs called AddressSanitizer1. AddressSanitizer (ASan) applies new memory allocation, mapping, and code stubbing techniques to detect almost all memory access errors efficiently. Using the SPEC 2006 benchmark analysis package, ASan runs with an average slowdown of less than 2 and memory consumption of about 2.4 times. In comparison, another well-known detection tool Valgrind has an average slowdown of 20, which makes it almost impossible to put into practice.
\n
The following table summarizes the types of memory access errors that ASan can detect for C/C++ programs:
ASan itself cannot detect heap memory leaks. But when ASan is integrated into the compiler, as it replaces the memory allocation/free functions, the original leak detection feature of the compiler tool is consolidated with ASan. So, adding the ASan option to the compilation command line also turns on the leak detection feature by default.
\n
\n
This covers all common memory access errors except for \"uninitialized memory reads\" (UMR). ASan detects them with a false positive rate of 0, which is quite impressive. In addition, ASan detects several C++-specific memory access errors such as
\n
\n
Initialization Order Fiasco: When two static objects are defined in different source files and the constructor of one object calls the method of the other object, a program crash will occur if the former compilation unit is initialized first.
\n
Container Overflow: Given libc++/libstdc++ container, access [container.end(), container.begin() + container.capacity())], which crosses the [container.begin(), container.end()] range but still within the dynamically allocated memory area.
\n
Delete Mismatch: For the array object created by new foo[n], should not call delete foo for deletion, use delete [] foo instead.
\n
\n
ASan's high reliability and performance have made it the preferred choice of compiler and IDE developers since its introduction. Today ASan is integrated into all four major compilation toolsets:
ASan's developers first used the Chromium open-source browser for routine testing and found more than 300 memory access errors over 10 months. After integration into mainstream compilation tools, it reported long-hidden bugs in numerous popular open-source software, such as Mozilla Firefox, Perl, Vim, PHP, and MySQL. Interestingly, ASan also identified some memory access errors in the LLVM and GCC compilers' code. Now, many software companies have added ASan run to their mandatory quality control processes.
\n
Working Principle
\n
The USENIX conference paper 3, published by Serebryany in 2012, comprehensively describes the design principles, algorithmic ideas, and programming implementation of ASan. In terms of the overall structure, ASan consists of two parts.
\n\n
Compiler instrumentation - modifies the code to verify the shadow memory state at each memory access and creates poisoned red zones at the edges of global and stack objects to detect overflows or underflows.
\n
Runtime library replacement - replaces malloc/free and its related functions to create poisoned red zones at the edge of dynamically allocated heap memory regions, delay the reuse of memory regions after release, and generate error reports.
\n\n
Here shadow memory, compiler instrumentation, and memory allocation function replacement are all previously available techniques, so how has ASan innovatively applied them for efficient error detection? Let's take a look at the details.
\n
Shadow Memory
\n
Many inspection tools use separated shadow memory to record metadata about program memory, and then apply instrumentation to check the shadow memory during memory accesses to confirm that reads and writes are safe. The difference is that ASan uses a more efficient direct mapping shadow memory.
\n
The designers of ASan noted that typically the malloc function returns a memory address that is at least 8-byte aligned. For example, a request for 20 bytes of memory would divide 24 bytes of memory, with the last 3 bits of the actual return pointer being all zeros. in addition, any aligned 8-byte sequence would only have 9 different states: the first \\(k\\,(0\\leq k \\leq 8)\\) bytes are accessible, and the last \\(8-k\\) are not. From this, they came up with a more compact shadow memory mapping and usage scheme:
\n
\n
Reserve one-eighth of the virtual address space for shadow memory
\n
Directly map application memory to shadow memory using a formula that divides by 8 plus an offset\n
Each byte of shadow memory records one of the 9 states of the corresponding 8-byte memory block\n
\n
0 means all 8 bytes are addressable
\n
Any negative value indicates that the entire 8-byte word is unaddressable (poisoned )
\n
k (1 ≤ k ≤ 7) means that the first k bytes are addressable
\n
\n
\n
The following figure shows the address space layout and mapping relationship of ASan. Pay attention to the Bad area in the middle, which is the address segment after the shadow memory itself is mapped. Because shadow memory is not visible to the application, ASan uses a page protection mechanism to make it inaccessible.
\n
\n
Compiler Instrumentation
\n
Once the shadow memory design is determined, the implementation of compiler instrumentation to detect dynamic memory access errors is easy. For memory accesses of 8 bytes, the shadow memory bytes are checked by inserting instructions before the original read/write code, and an error is reported if they are not zero. For memory accesses of less than 8 bytes, the instrumentation is a bit more complicated, where the shadow memory byte values are compared with the last three bits of the read/write address. This situation is also known as the \"slow path\" and the sample code is as follows.
\n
// Check the cases where we access first k bytes of the qword // and these k bytes are unpoisoned. boolSlowPathCheck(shadow_value, address, kAccessSize) { last_accessed_byte = (address & 7) + kAccessSize - 1; return (last_accessed_byte >= shadow_value); } ...
For global and stack (local) objects, ASan has designed different instrumentation to detect their out-of-bounds access errors. The red zone around a global object is added by the compiler at compile time and its address is passed to the runtime library at application startup, where the runtime library function then poisons the red zone and writes down the address needed in error reporting. The stack object is created at function call time, and accordingly, its red zone is created and poisoned at runtime. In addition, because the stack object is deleted when the function returns, the instrumentation code must also zero out the shadow memory it is mapped to.
\n
In practice, the ASan compiler instrumentation process is placed at the end of the compiler optimization pipeline so that instrumentation only applies to the remaining memory access instructions after variable and loop optimization. In the latest GCC distribution, the ASan compiler stubbing code is located in two files in the gcc subdirectory gcc/asan.[ch].
\n
Runtime Library Replacement
\n
The runtime library needs to include code to manage shadow memory. The address segment to which shadow memory itself is mapped is to be initialized at application startup to disable access to shadow memory by other parts of the program. The runtime library replaces the old memory allocation and free functions and also adds some error reporting functions such as __asan_report_load8.
\n
The newly replaced memory allocation function malloc will allocate additional storage as a red zone before and after the requested memory block and set the red zone to be non-addressable. This is called the poisoning process. In practice, because the memory allocator maintains a list of available memory corresponding to different object sizes, if the list of a certain object is empty, the OS will allocate a large set of memory blocks and their red zones at once. As a result, the red zones of the preceding and following memory blocks will be connected, as shown in the following figure, where \\(n\\) memory blocks require only \\(n+1\\) red zones to be allocated.
\n
\n
The new free function needs to poison the entire storage area and place it in a quarantine queue after the memory is freed. This prevents the memory region from being allocated any time soon. Otherwise, if the memory region is reused immediately, there is no way to detect incorrect accesses to the recently freed memory. The size of the quarantine queue determines how long the memory region is in quarantine, and the larger it is the better its capability of detecting UAF errors!
\n
By default, both the malloc and free functions log their call stacks to provide more detailed information in the error reports. The call stack for malloc is kept in the red zone to the left of the allocated memory, so a large red zone can retain more call stack frames. The call stack for free is stored at the beginning of the allocated memory region itself.
\n
Integrated into the GCC compiler, the source code for the ASan runtime library replacement is located in the libsanitizer subdirectory libsanitizer/asan/*, and the resulting runtime library is compiled as libasan.so.
\n
Application Examples
\n
ASan is very easy to use. The following is an example of an Ubuntu Linux 20.4 + GCC 9.3.0 system running on an x86_64 virtual machine to demonstrate the ability to detect various memory access errors.
\n
Test Cases
\n
As shown below, the test program writes seven functions, each introducing a different error type. The function names are cross-referenced with the error types one by one:
intstack_buffer_overflow() { int s[10]; s[0] = 10; return s[10]; }
int *gp;
voidstack_use_after_return() { int r[10]; r[0] = 10; gp = &r[0]; return; }
voidstack_use_after_scope() { { int c = 0; gp = &c; } *gp = 10; return; }
\n
The test program calls the getopt library function to support a single-letter command line option that allows the user to select the type of error to be tested. The command line option usage information is as follows.
\n
\b$ ./asan-test
Test AddressSanitizer usage: asan-test [ -bfloprs ]
-b\theap buffer overflow -f\theap use after free -l\theap memory leak -o\tglobal buffer overflow -p\tstack use after scope -r\tstack use after return -s\tstack buffer overflow
\n
The GCC compile command for the test program is simple, just add two compile options
\n
\n
-fsanitize=address: activates the ASan tool
\n
-g: enable debugging and keep debugging information
\n
\n
OOB Test
\n
For Heap OOB error, the run result is
\n
$ ./asan-test -b ================================================================= ==57360==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x604000000038 at pc 0x55bf46fd64ed bp 0x7ffced908dc0 sp 0x7ffced908db0 READ of size 4 at 0x604000000038 thread T0 #0 0x55bf46fd64ec in heap_buffer_overflow /home/zixi/coding/asan-test.c:34 #1 0x55bf46fd6a3f in main /home/zixi/coding/asan-test.c:88 #2 0x7fd16f6560b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) #3 0x55bf46fd628d in _start (/home/zixi/coding/asan-test+0x128d)
0x604000000038 is located 0 bytes to the right of 40-byte region [0x604000000010,0x604000000038) allocated by thread T0 here: #0 0x7fd16f92ebc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8) #1 0x55bf46fd646c in heap_buffer_overflow /home/zixi/coding/asan-test.c:32 #2 0x55bf46fd6a3f in main /home/zixi/coding/asan-test.c:88 #3 0x7fd16f6560b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
SUMMARY: AddressSanitizer: heap-buffer-overflow /home/zixi/coding/asan-test.c:34 in heap_buffer_overflow Shadow bytes around the buggy address: 0x0c087fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c087fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c087fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c087fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c087fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0c087fff8000: fa fa 00 00 00 00 00[fa]fa fa fa fa fa fa fa fa 0x0c087fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c087fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c087fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c087fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c087fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd ... ==57360==ABORTING
\n
Referring to the heap-buffer-overflow function implementation, you can see that it requests 40 bytes of memory to hold 10 32-bit integers. However, on the return of the function, the code overruns to read the data after the allocated memory. As the above run log shows, the program detects a Heap OOB error and aborts immediately. ASan reports the name of the source file and line number asan-test.c:34 where the error occurred, and also accurately lists the original allocation function call stack for dynamically allocated memory. The \"SUMMARY\" section of the report also prints the shadow memory data corresponding to the address in question (observe the lines marked by =>). The address to be read is 0x604000000038, whose mapped shadow memory address 0x0c087fff8007 holds the negative value 0xfa (poisoned and not addressable). Because of this, ASan reports an error and aborts the program.
\n
The Stack OOB test case is shown below. ASan reports an out-of-bounds read error for a local object. Since the local variables are located in the stack space, the starting line number asan-test.c:37 of the function stack_buffr_overflow is listed. Unlike the Heap OOB report, the shadow memory poisoning values for the front and back redzone of the local variable are different, with the previous Stack left redzone being 0xf1 and the later Stack right redzone being 0xf3. Using different poisoning values (both negative after 0x80) helps to quickly distinguish between the different error types.
\n
$ ./asan-test -s ================================================================= ==57370==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f1cf5044058 at pc 0x55d8b7e9d601 bp 0x7ffc830c29e0 sp 0x7ffc830c29d0 READ of size 4 at 0x7f1cf5044058 thread T0 #0 0x55d8b7e9d600 in stack_buffer_overflow /home/zixi/coding/asan-test.c:40 #1 0x55d8b7e9daec in main /home/zixi/coding/asan-test.c:108 #2 0x7f1cf87760b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) #3 0x55d8b7e9d28d in _start (/home/zixi/coding/asan-test+0x128d)
Address 0x7f1cf5044058 is located in stack of thread T0 at offset 88 in frame #0 0x55d8b7e9d505 in stack_buffer_overflow /home/zixi/coding/asan-test.c:37
This frame has 1 object(s): [48, 88) 's' (line 38) <== Memory access at offset 88 overflows this variable HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork (longjmp and C++ exceptions *are* supported) SUMMARY: AddressSanitizer: stack-buffer-overflow /home/zixi/coding/asan-test.c:40 in stack_buffer_overflow Shadow bytes around the buggy address: 0x0fe41ea007b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe41ea007c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe41ea007d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe41ea007e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe41ea007f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0fe41ea00800: f1 f1 f1 f1 f1 f1 00 00 00 00 00[f3]f3 f3 f3 f3 0x0fe41ea00810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe41ea00820: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe41ea00830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe41ea00840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe41ea00850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 ... ==57370==ABORTING
\n
The following Global OOB test result also clearly shows the error line asan-test.c:16, the global variable name ga and its definition code location asan-test.c:13:5, and you can also see that the global object has a red zone poisoning value of 0xf9.
\n
$ ./asan-test -o ================================================================= ==57367==ERROR: AddressSanitizer: global-buffer-overflow on address 0x564363ea4048 at pc 0x564363ea1383 bp 0x7ffc0d6085d0 sp 0x7ffc0d6085c0 READ of size 4 at 0x564363ea4048 thread T0 #0 0x564363ea1382 in global_buffer_overflow /home/zixi/coding/asan-test.c:16 #1 0x564363ea1a6c in main /home/zixi/coding/asan-test.c:98 #2 0x7f8cb43890b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) #3 0x564363ea128d in _start (/home/zixi/coding/asan-test+0x128d)
Surprisingly, ASan does not report the obvious Global OOB error here. Why?
\n
The reason has to do with the way GCC treats global variables. The compiler treats functions and initialized variables as Strong symbols, while uninitialized variables are Weak symbols by default. Since the definition of weak symbols may vary from source file to source file, the size of the space required is unknown. The compiler cannot allocate space for weak symbols in the BSS segment, so it uses the COMMON block mechanism so that all weak symbols share a COMMON memory region, thus ASan cannot insert the red zone. During the linking process, after the linker reads all the input target files, it can determine the size of the weak symbols and allocate space for them in the BSS segment of the final output file.
\n
Fortunately, GCC's -fno-common option turns off the COMMON block mechanism, allowing the compiler to add all uninitialized global variables directly to the BSS segment of the target file, also allowing ASan to work properly. This option also disables the linker from merging weak symbols, so the linker reports an error directly when it finds a compiled unit with duplicate global variables defined in the target file.
\n
This is confirmed by a real test. Modify the GCC command line for the previous code segment
then compile, link, and run. ASan successfully reported the Global OOB error.
\n
UAF Test
\n
The following is a running record of UAF error detection. Not only is the information about the code that went wrong reported here, but also the call stack of the original allocation and free functions of the dynamic memory is given. The log shows that the memory was allocated by asan-test.c:25, freed at asan-test.c:27, and yet read at asan-test.c:28. The shadow memory data printed later indicates that the data filled is negative 0xfd, which is also the result of the poisoning of the memory after it is freed.
\n
$ \u0007./asan-test -\bf ================================================================= ==57363==ERROR: AddressSanitizer: heap-use-after-free on address 0x604000000034 at pc 0x558b4a45444e bp 0x7ffccf4ca790 sp 0x7ffccf4ca780 READ of size 4 at 0x604000000034 thread T0 #0 0x558b4a45444d in heap_use_after_free /home/zixi/coding/asan-test.c:28 #1 0x558b4a454a4e in main /home/zixi/coding/asan-test.c:91 #2 0x7fc7cc98b0b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) #3 0x558b4a45428d in _start (/home/zixi/coding/asan-test+0x128d)
0x604000000034 is located 36 bytes inside of 40-byte region [0x604000000010,0x604000000038) freed by thread T0 here: #0 0x7fc7ccc637cf in __interceptor_free (/lib/x86_64-linux-gnu/libasan.so.5+0x10d7cf) #1 0x558b4a454412 in heap_use_after_free /home/zixi/coding/asan-test.c:27 #2 0x558b4a454a4e in main /home/zixi/coding/asan-test.c:91 #3 0x7fc7cc98b0b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
previously allocated by thread T0 here: #0 0x7fc7ccc63bc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8) #1 0x558b4a4543bd in heap_use_after_free /home/zixi/coding/asan-test.c:25 #2 0x558b4a454a4e in main /home/zixi/coding/asan-test.c:91 #3 0x7fc7cc98b0b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
SUMMARY: AddressSanitizer: heap-use-after-free /home/zixi/coding/asan-test.c:28 in heap_use_after_free Shadow bytes around the buggy address: 0x0c087fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c087fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c087fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c087fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c087fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0c087fff8000: fa fa fd fd fd fd[fd]fa fa fa fa fa fa fa fa fa 0x0c087fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c087fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c087fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c087fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c087fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd ... ==57363==ABORTING
\n
HML Test
\n
The results of the memory leak test are as follows. Unlike the other test cases, ABORTING is not printed at the end of the output record. This is because, by default, ASan only generates a memory leak report when the program terminates (process ends). If you want to check for leaks on the fly, you can call ASan's library function __lsan_do_recoverable_leak_check, whose definition is located in the header file sanitizer/lsan_interface.h.
Direct leak of 40 byte(s) in 1 object(s) allocated from: #0 0x7f06b85b1bc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8) #1 0x5574a8bcd3a0 in heap_leak /home/zixi/coding/asan-test.c:20 #2 0x5574a8bcda5d in main /home/zixi/coding/asan-test.c:94 #3 0x7f06b82d90b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
SUMMARY: AddressSanitizer: 40 byte(s) leaked in 1 allocation(s).
\n
UAS Test
\n
See the stack_use_after_scope function code, where the memory unit holding the local variable c is written outside of its scope. The test log accurately reports the line number line 54 where the variable is defined and the location of the incorrect writing code asan-test.c:57:
\n
./asan-test -\bp ================================================================= ==57368==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7f06f0a9b020 at pc 0x56121a7548d9 bp 0x7ffd1de0d050 sp 0x7ffd1de0d040 WRITE of size 4 at 0x7f06f0a9b020 thread T0 #0 0x56121a7548d8 in stack_use_after_scope /home/zixi/coding/asan-test.c:57 #1 0x56121a754a7b in main /home/zixi/coding/asan-test.c:101 #2 0x7f06f42cd0b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) #3 0x56121a75428d in _start (/home/zixi/coding/asan-test+0x128d)
Address 0x7f06f0a9b020 is located in stack of thread T0 at offset 32 in frame #0 0x56121a7547d0 in stack_use_after_scope /home/zixi/coding/asan-test.c:52
This frame has 1 object(s): [32, 36) 'c' (line 54) <== Memory access at offset 32 is inside this variable HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork (longjmp and C++ exceptions *are* supported) SUMMARY: AddressSanitizer: stack-use-after-scope /home/zixi/coding/asan-test.c:57 in stack_use_after_scope Shadow bytes around the buggy address: 0x0fe15e14b5b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe15e14b5c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe15e14b5d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe15e14b5e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe15e14b5f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0fe15e14b600: f1 f1 f1 f1[f8]f3 f3 f3 00 00 00 00 00 00 00 00 0x0fe15e14b610: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe15e14b620: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe15e14b630: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe15e14b640: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fe15e14b650: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 ... ==57368==ABORTING
\n
UAR Test
\n
The UAR test has its peculiarities. Because the stack memory of a function is reused immediately after it returns, to detect local object access errors after return, a \"pseudo-stack\" of dynamic memory allocation must be set up, for details check the relevant Wiki page of ASan4. Since this algorithm change has some performance impact, ASan does not detect UAR errors by default. If you really need to, you can set the environment variable ASAN_OPTIONS to detect_stack_use_after_return=1 before running. The corresponding test logs are as follows.
\n
$ export ASAN_OPTIONS=detect_stack_use_after_return=1 $ env | grep ASAN ASAN_OPTIONS=detect_stack_use_after_return=1 $ ./asan-test -\br ================================================================= ==57369==ERROR: AddressSanitizer: stack-use-after-return on address 0x7f5493e93030 at pc 0x55a356890ac9 bp 0x7ffd22c5cf30 sp 0x7ffd22c5cf20 READ of size 4 at 0x7f5493e93030 thread T0 #0 0x55a356890ac8 in main /home/zixi/coding/asan-test.c:105 #1 0x7f54975c50b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) #2 0x55a35689028d in _start (/home/zixi/coding/asan-test+0x128d)
Address 0x7f5493e93030 is located in stack of thread T0 at offset 48 in frame #0 0x55a356890682 in stack_use_after_return /home/zixi/coding/asan-test.c:45
ASan supports many other compiler flags and runtime environment variable options to control and tune the functionality and scope of the tests. For those interested please refer to the ASan flags Wiki page5.
\n
A zip archive of the complete test program is available for download here: asan-test.c.gz
\n\n\n","categories":["Tool Guide"],"tags":["C/C++ Programming","System Programming"]},{"title":"Does Diffie-Hellman Key Exchange Use a Technology Similar to RSA?","url":"/en/2022/11/21/DH-and-RSA/","content":"
Recently, at a WPA3 technology introduction meeting within the R&D team, the speaker mentioned that the OWE technology for encrypted wireless open networks is based on Diffie-Hellman key exchange, and casually said that Diffie-Hellman key exchange is using technology similar to RSA. This statement is wrong! Although Diffie-Hellman key exchange and RSA encryption algorithms belong to public key cryptography, their working mechanisms and application scenarios are different. As a research and development engineer and technician supporting network security, it is necessary to clearly understand the working mechanism and mathematical principles of the two, as well as the differences and connections between them.
\n
A cryptographic system should be secure even if everything about the system, except the key, is public knowledge. — Auguste Kerckhoffs (Dutch linguist and cryptographer, best known for his “Kerckhoffs's principle” of cryptography)
\n
\n
Diffie-Hellman Key Exchange
\n
Diffie-Hellman key exchange (DH for short) is a secure communication protocol that allows two communicating parties to exchange messages over an insecure public channel to create a shared secret without any foreknowledge. This secret can be used to generate keys for subsequent communications between the two parties using symmetric encryption techniques (e.g. AES).
\n
The idea of this kind of public key distribution to achieve shared secrets was first proposed by Ralph Merkle, a doctoral student of Stanford University professor Martin Hellman, and then Professor Hellman's research assistant Whitfield Diffie and Professor Herman jointly invented a practical key exchange protocol. In 1976, Diffie and Hellman were invited to publish their paper \"New Directions in Cryptography\" in IEEE Transactions on Information Theory, which laid the foundation for the public key cryptography system and officially announced the birth of the new Diffie-Herman key exchange technology.
Alice chooses a prime number \\(p=71\\), and then a primitive root \\(g=7\\) of the multiplicative group of integers modulo \\(p\\)
\n
Alice chooses a random number \\(a=17\\) that is less than \\(p\\), calculate \\(A=g^a\\bmod\\;p=7^{17}\\bmod\\;71 = 62\\)
\n
Alice sends all \\((p,g,A)\\) to Bob
\n
Bob also chooses a random number \\(b=39\\) that is less than \\(p\\), calculate \\(B=g^b\\bmod\\;p=7^{39}\\bmod\\;71 = 13\\)
\n
Bob sends \\(B\\) back to Alice
\n
Alice calculates \\(s=B^a\\bmod\\;p=13^{17}\\bmod\\;71 = 42\\)
\n
Bob calculate \\(s=A^b\\bmod\\;p=62^{39}\\bmod\\;71 = 42\\)
\n\n
Is it troublesome calculating \\(\\color{#93F}{\\bf62^{39}\\bmod\\;71}\\)? It is actually very easy……
\n\n
Remember that modular arithmetic has the property of preserving primitive operations: \\[(a⋅b)\\bmod\\;m = [(a\\bmod\\;m)⋅(b\\bmod\\;m)]\\bmod\\;m\\] Combining with the principle of Exponentiation by Squaring, and applying the right-to-left binary method to do fast calculation: \\[\\begin{align}\n62^{39}\\bmod\\;71 & = (62^{2^0}⋅62^{2^1}⋅62^{2^2}⋅62^{2^5})\\bmod\\;71\\\\\n& = (62⋅10⋅(62^{2^1}⋅62^{2^1})⋅(62^{2^4}⋅62^{2^4}))\\bmod\\;71\\\\\n& = (62⋅10⋅(10⋅10)⋅(62^{2^3}⋅62^{2^3}⋅62^{2^4}))\\bmod\\;71\\\\\n& = (62⋅10⋅29⋅(29⋅29⋅62^{2^3}⋅62^{2^4}))\\bmod\\;71\\\\\n& = (62⋅10⋅29⋅(60⋅60⋅62^{2^4}))\\bmod\\;71\\\\\n& = (62⋅10⋅29⋅(50⋅50))\\bmod\\;71\\\\\n& = (62⋅10⋅29⋅15)\\bmod\\;71\\\\\n& = 42\n\\end{align}\\]
\n\n\n
As if by magic, both Alice and Bob get the same \\(s\\) value of \\(42\\). This is the shared secret of two people! After this, Alice and Bob can use the hash value of \\(s\\) as a symmetric key for encrypted communication, which is unknown to any third party.
\n
Why? Because of the nature of the modular exponentiation of the multiplicative group, \\(g^{ab}\\) and \\(g^{ba}\\) are equal with the modulo \\(p\\):
So calculated \\(s\\) values must be the same. Of course, real applications would use much larger \\(p\\), otherwise the attacker can exhaust all the remainder to try to crack the ciphertext encrypted by the symmetric key.
\n
Notice \\((p,g,A,B)\\) is public and \\((a,b,s)\\) is secret. Now suppose an eavesdropper Eve can see all the messages between Alice and Bob, can she deduce \\(s\\)? The answer is that this is only practically possible if the values of \\((p,a,b)\\) are very small. Eve must first invert \\((a,b)\\) from what she knows about \\((p,g,A,B)\\):
This is the famous discrete logarithm problem. It is a recognized computational challenge and no polynomial-time efficient algorithm is currently found to compute the discrete logarithm. So this protocol is considered eavesdropping-safe as long as the appropriate \\((p,a,b)\\) is chosen. RFC 3526 recommends 6 Modular Exponential (MODP) DH groups of large prime numbers for practical applications, the smallest of which has 1536 bits!
\n
It should also be emphasized that Diffie-Hellman key exchange itself does not require authentication of both communicating parties, so it is vulnerable to man-in-the-middle attacks. If an attacker can tamper with the messages sent and received by both sides in the middle of the channel, he can complete Diffie-Hellman key exchange twice by pretending to be an identity. The attacker can then decrypt the entire message. Therefore, usually practical applications need to incorporate authentication mechanisms to prevent such attacks.
\n
Diffie-Hellman key exchange technique is a crucial contribution to modern cryptography. In 2015, 39 years after the announcement of this invention, Diffie and Hellman jointly won the ACM Turing Award, known as the \"Nobel Prize of Computing\". The ACM award poster directly stated that they \"invented public key cryptography\".
\n
\n
RSA Encryption Algorithm
\n
RSA is a public key encryption algorithm. The public key encryption system with the same name as the core technology is widely used in secure data transmission. Today, the comprehensive development of the Internet has provided great convenience to the public in all aspects of society. Whether you are surfing, gaming, entertaining, shopping, instant messaging with friends and family, managing a bank account, investing in financial securities, or simply sending and receiving email, RSA is working behind the scenes to protect your privacy and data security.
\n
RSA is actually an acronym for the last names of three people: American cryptographer Ronald Rivest, Israeli cryptographer Adi Shamir, and American computer scientist Leonard Max Adleman. In 1977, Levister, Shamir, and Adleman collaborated at the Massachusetts Institute of Technology (MIT) to invent the RSA encryption algorithm. The algorithm was first published in a public technical report at MIT, and later compiled and published in the February 1978 issue of ACM Communications under the title \"A Method for Obtaining Digital Signatures and Public Key Cryptosystems\".
\n
The basic idea of RSA is that the user creates a key pair consisting of a public key and a private key. The public key is freely distributed and the private key must be kept secret. Anyone can encrypt a message with the public key, and the resulting ciphertext can only be deciphered by the private key holder. On the other hand, any message encrypted with the private key can be decrypted by the public key. Since we assume that the private key can only be held by a specific object, encrypting with the private key is equivalent to generating a digital signature, and decrypting with the public key is equivalent to verifying the signature.
\n
The RSA encryption algorithm consists of a four-step operational process: key generation, key distribution, encryption, and decryption. A simple and concrete example is also given below to illustrate.
\n\n
Alice randomly chooses two prime numbers \\(p=127\\) and \\(q=5867\\), computes \\(N=pq=745109\\)
When \\(p\\) and \\(q\\) are both primes, \\(\\lambda(pq)=\\mathrm{lcm}(p − 1, q − 1)\\)
\n
\\(\\mathrm{lcm}\\) represents the function for the least common multiple, which may be calculated through the Euclidean algorithm
\n
\\(\\mathrm{lcm}(126,5866)=52794\\)
\n
\n
Alice chooses an integer \\(e=5\\) less than \\(\\lambda(N)\\) but also coprime with \\(\\lambda(N)\\), and calculates the modular multiplicative inverse of \\(e\\) modulo \\(\\lambda(N)\\). That is \\(d\\equiv e^{-1}\\pmod {\\lambda(N)}\\), \\(d=10559\\)\n
\n
The definition of modular multiplicative inverse is, determine \\(d\\) such that \\((d⋅e)\\;\\bmod\\;\\lambda(N)=1\\)
\n
\\(d=10559\\equiv 5^{-1}\\pmod {52794}\\)
\n
\n
\\(\\pmb{(N,e)}\\)is Alice's public key，\\(\\pmb{(N,d)}\\)is her private key\n
\n
Alice sends her public key \\((745109,5)\\) to Bob
\n
Alice saves her private key \\((745109,10559)\\) in a secret place
\n
Alice distroies all records of \\(p,q,\\lambda(N)\\)
\n
\n
When Bob wants to send Alice a message \\(M\\), according to the encoding format agreed upon by both parties, he first translates \\(M\\) to one or more positive integers \\(m\\) that are all less than \\(N\\), and then uses Alice's public key to compute the ciphertext \\(c\\) one by one. The calculation formula is \\(\\pmb{c\\equiv m^e\\pmod N}\\)\n
\n
Assume \\(M\\) is \"CACC 9678\", and the encoding scheme is 0 for spaces, 1-26 for a-z/A-Z (ignoring case), and 27-36 for 0-9
\n
Encoding yields the positive integer string \"030103 030036 333435\". Note that each integer is less than 745109
\n
After encryption, it becomes ciphertext integer string \"184539 741303 358095\"\n
\n
\\(184539 \\equiv 30103^5\\pmod {745109}\\)
\n
\\(741303 \\equiv 30036^5\\pmod {745109}\\)
\n
\\(358095 \\equiv 333435^5\\pmod {745109}\\)
\n
\n
\n
After Alice receives the ciphertext integer string, she uses her private key to compute the plaintext one by one, the calculation formula is \\(\\pmb{m\\equiv c^d\\pmod N}\\)\n
The third step above works out \\(d\\) from \\(\\color{#93F}{\\bf(d\\cdot 5)\\;mod\\;52794=1}\\), here's how
\n\n
The modular multiplicative invers can be solved quickly by applying the Extended Euclidean algorithm. Referring to this Wiki page, with the precondition of coprime, the following equation can be written (\\(gcd\\) is the function for the greatest common divisor function):
\n
\\[52794s+5t=\\mathrm{gcd}(5, 52794)=1\\]
\n
The goal is to find the smallest positive integer \\(t\\) that satisfies the above equation. The following table shows the iterative process of the algorithm:
\n
\n\n
\n
Index \\(i\\)
\n
Quotient \\(q_{i-1}\\)
\n
Remainder \\(r_i\\)
\n
\\(s_i\\)
\n
\\(t_i\\)
\n
\n\n\n
\n
0
\n
\n
\\(52794\\)
\n
\\(1\\)
\n
\\(0\\)
\n
\n
\n
1
\n
\n
\\(5\\)
\n
\\(0\\)
\n
\\(1\\)
\n
\n
\n
2
\n
\\(52794 \\div5 = 10558\\)
\n
\\(4\\)
\n
\\(1 - 10558\\times 0 = 1\\)
\n
\\(0 - 10558\\times 1 = -10558\\)
\n
\n
\n
3
\n
\\(5 \\div4 = 1\\)
\n
\\(1\\)
\n
\\(0-1\\times1 = -1\\)
\n
\\(1 - 1\\times (-10558) = \\bf10559\\)
\n
\n\n
\n
It only takes two iterations to get the remainder \\(1\\) and the algorithm ends. The final \\(t\\) is the \\(5^{-1}\\pmod {52794}\\) we want.
\n\n\n
String together after decoding to get the same information \"CACC 9678\". Why does Alice's decrypted message match exactly the one sent by Bob? The reason lies in the modular exponentiation operation. First of all, because \\(c\\equiv m^e\\pmod N\\), we can get \\(c^d\\equiv (m^e)^d \\equiv m^{ed} \\pmod N\\). Since \\((d⋅e)\\;mod\\;\\lambda(N)=1\\), it is deduced that \\(ed = 1 + h\\lambda(N)\\) (\\(h\\) is a non-negative integer为非负整数). Combine these two
The penultimate congruence above (symbol \\(\\equiv\\)) is based on Euler's theorem). This proves the correctness of the decryption formula \\({m\\equiv c^d\\pmod N}\\)! You can also see that the order of \\(e\\) and \\(d\\) is irrelevant for the result of \\(m^{ed}\\pmod N\\), so the message that Alice encrypted with her private key can be decrypted by Bob with Alice's public key. This also proves the feasibility of digital signatures.
\n
In terms of security, if a third party can derive \\(d\\) from Alice's public key \\((N,e)\\), then the algorithm is broken. But the prerequisite for cracking is to first identify \\(p\\) and \\(q\\) from \\(N\\), which is very difficult when \\(N\\) is big. In fact, this is the famous problem of factoring large numbers, another recognized computational challenge. So far, \"the best-known algorithms are faster than exponential order of magnitude times and slower than polynomial order of magnitude times.\" The latest record, published on the RSA Factoring Challenge website, is the February 2020 crack of RSA-250, a large number of 829 bits. This development indicates that the security of 1024-bit \\(N\\)-valued public keys is already in jeopardy. In view of this, National Institute of Standards and Technology (NIST) recommends that RSA keys be at least 2048 bits in length for real-world applications.
\n
On the other hand, although the public key does not need to be transmitted confidentially, it is required to be reliably distributed. Otherwise, Eve could pretend to be Alice and send her own public key to Bob. If Bob believes it, Eve can intercept all messages passed from Bob to Alice and decrypt them with her own private key. Eve will then encrypt this message with Alice's public key and pass it to her. Alice and Bob cannot detect such a man-in-the-middle attack. The solution to this problem is to establish a trusted third-party authority to issue certificates to ensure the reliability of public keys. This is the origin of the Public Key Infrastructure (PKI).
\n
The RSA public key encryption algorithm is the genius creation of three cryptographers and computer scientists. Its invention is a new milestone in public key cryptography and has become the cornerstone of modern secure Internet communication. The outstanding contribution of Levister, Shamir, and Adelman earned them the ACM Turing Award in 2002, a full 13 years before Diffie and Herman!
\n
\n
Difference and Connection
\n
The following table summarizes the comparison of Diffie-Hellman key exchange and RSA public key encryption algorithm:
Modular exponentiation, exponentiation by squaring
\n
Modular exponentiation, exponentiation by squaring, extended Euclidean algorithms
\n
\n
\n
Public Key
\n
\\((p,g,A,B)\\)
\n
\\((N,e)\\)
\n
\n
\n
Private Key
\n
\\((a,b,s)\\)
\n
\\((N,d)\\)
\n
\n
\n
Security
\n
Discrete logarithm problem
\n
Large number prime factorization problem
\n
\n
\n
Typical Applications
\n
Key Exchange
\n
Encryption/Decryption, Digital Signature
\n
\n
\n
Key Kength
\n
\\(\\ge2048\\) bits
\n
\\(\\ge2048\\) bits
\n
\n
\n
Authentication
\n
Requires external support
\n
Requires PKI support for public key distribution
\n
\n
\n
Forward Secrecy
\n
Support
\n
Not support
\n
\n\n
\n
As can be seen, both are asymmetric public key techniques, and both have a public and private key pair. They both use Modular exponentiation and exponentiation by squaring mathematical operations, and the RSA public-key encryption algorithm also requires the application of the extended Euclidean algorithm to solve the modular multiplicative inverse. Despite these similarities, the mathematical principles underlying them are different, and the computational challenges corresponding to their security are different in nature. These characteristics determine that the Diffie-Hellman key exchange can be used for key exchange, but not for encryption/decryption, while the RSA public key encryption algorithm can not only encrypt/decrypt but also support digital signatures. Therefore, the argument that the two use similar technologies cannot be established in general.
\n
ElGamal encryption based on the evolution of the Diffie-Hellman key exchange can be used to encrypt/decrypt messages, but due to some historical reasons and the great commercial success of the RSA public key encryption algorithm, ElGamal encryption is not popular.
\n
\n
In modern cryptography, key length is defined as the number of bits of a key used by an encryption algorithm. Theoretically, since all algorithms may be cracked by brute force, the key length determines an upper limit on the security of an encryption algorithm. Cryptanalytic study shows that the key strengths of Diffie-Hellman key exchange and RSA public key encryption algorithm are about the same. The computational intensities for breaking discrete logarithms and factoring large numbers are comparable. Therefore, the recommended key length for both cryptographic technologies in practical applications is at least 2048 bits.
\n
For authentication, Diffie-Hellman key exchange requires external support, otherwise it is not resistant to man-in-the-middle attacks. RSA public key encryption algorithm can be used to verify digital signatures, but only if there is a PKI supporting reliable public key distribution. The current system of PKI is quite mature, and there is a special Certificate Authority (CA) that undertakes the responsibility of public key legitimacy checking in the public key system, as well as issues and manages public key digital certificates in X.509 format.
\n
One problem with the RSA public key encryption algorithm in practice is that it does not have Forward Secrecy. Forward Secrecy, sometimes referred to as Perfect Forward Secrecy, is a security property of confidential communication protocols, meaning that the leakage of the long-term used master key does not result in the leakage of past session information. If the system has forward secrecy, it can protect the historical communication records in case of private key leakage. Imagine a situation where, although Eve cannot decrypt the RSA-encrypted messages between Alice and Bob, Eve can archive the entire past message ciphertext. One day in the future, Alice's private key for some reason was leaked, then Eve can decrypt all the message records.
\n
The solution to this problem is Diffie-Hellman key exchange! Remember that the \\((A,B)\\) in the public key of the Diffie-Hellman key exchange is generated by both parties from their respective private keys \\((a,b)\\), so if a random \\((a,b)\\) value is generated at each session, future key leaks will not crack the previous session key. This shows that Diffie-Hellman key exchange supports forward secrecy! If we combine the forward secrecy of Diffie-Hellman key exchange with the digital signature feature of the RSA public key encryption algorithm, we can implement a key exchange with authentication protection. This process can be simplified by the following example.
\n\n
Alice and Bob exchange authenticated RSA public key certificates
\n
Alice and Bob each generate a random \\((a,b)\\) value and compute \\((A,B)\\) using the shared Diffie-Hellman \\((p,g)\\).
\n
Alice encrypts \\(A\\) with her RSA private key to generate a digital signature, which she sends to Bob along with \\(A\\)
\n
Bob encrypts \\(B\\) with his own RSA private key to generate a digital signature and sends it to Alice along with \\(B\\).
\n
Alice verifies the signature with Bob's RSA public key, confirms that \\(B\\) came from Bob, and computes \\(s\\) using \\((p,a,B)\\). 6.
\n
Bob verifies the signature with Alice's RSA public key, confirms that \\(A\\) came from Alice, and computes \\(s\\) using \\((p,b,A)\\)
\n
Alice and Bob agree to share a secret and generate a subsequent symmetric encryption (AES) session key for confidential communication
\n\n
Here the RSA digital signature safeguards the key exchange from man-in-the-middle attacks. Also in the second step above, if a new random number is generated for each session, then even if Alice's or Bob's RSA private keys are leaked one day, it does not threaten the security of previous sessions because the eavesdropper still has to solve the discrete logarithm puzzle. We have also achieved forward secrecy. In fact, this is the working mechanism of the DHE-RSA cipher suite as defined by the ubiquitous Transport Layer Security (TLS) protocol.
\n
DHE-RSA Cipher Suite
\n
Transport Layer Security (TLS) and its predecessor Secure Sockets Layer (SSL) is a security protocol that provides security and data integrity for Internet communications. TLS is widely used in applications such as browsers, email, instant messaging, VoIP, and virtual private networks (VPNs), and has become the de facto industry standard for secure Internet communications. Currently, TLS 1.2 is the commonly supported version of the protocol, supporting secure connections over TCP. Datagram Transport Layer Security (DTLS) protocol is also defined for UDP applications. DTLS is much the same as TLS, with some extensions for connectionless UDP transport in terms of reliability and security. DTLS 1.2 matches the functionality of TLS 1.2.
\n
The TLS protocol uses a client-server architectural model. It works by using X.509 authentication and asymmetric encryption algorithms to authenticate the communicating parties, after which keys are exchanged to generate a symmetric encryption session key. This session key is then used to encrypt the data exchanged between the two communicating parties, ensuring the confidentiality and reliability of the information without fear of attack or eavesdropping by third parties. For identification purposes, the TLS 1.2 protocol combines the authentication, key exchange, bulk encryption, and message authentication code algorithms used into the Cipher Suite name. Each Cipher Suite is given a double-byte encoding. The TLS Cipher Suite Registry provides a reference table of all registered Cipher Suite names, sorted by encoding value from small to large.
\n
Since the computation intensity of asymmetric encryption algorithms (RSA, etc.) is much higher than that of symmetric encryption algorithms (AES, etc.), practical applications almost always use symmetric encryption algorithms to encrypt messages in batches in terms of performance.
\n
\n
TLS 1.2 protocol supports a series of cipher suites that combine the Diffie-Hellman key exchange with the RSA public key encryption algorithm. They all start with TLS_DH_RSA or TLS_DHE_RSA`. The \"E\" in DHE stands for \"Ephemeral\", which means that a random \\((a,b)\\) value is required to be generated for each session. So TLS_DHE_RSA cipher suite can provide forward secrecy, while TLS_DH_RSA cannot, and the former should be preferred in practical applications.
\n
Here we take a typical TLS_DHE_RSA_WITH_AES_128_CBC_SHA (encoding 0x00,0x33) cipher suite as an example to explain the process of Diffie-Hellman working with RSA to establish a DTLS session. First, explain the composition of the cipher suite.
\n
\n
DHE: ephemeral DH to implement key exchange
\n
RSA: public key for signing and certifying the DHE
Referring to the packet file dtls-dhe-rsa.pcap captured from the network port, the following handshake protocol message sequence chart can be obtained
\n
\nsequenceDiagram\n\nautonumber\nparticipant C as Client\nparticipant S as Server\nNote over C,S: Handshake Protocol\nrect rgb(230, 250, 255)\nC->>S: Client Hello (Cr, Cipher Suites))\nS-->>C: Hello Verify Request (Cookie)\nC->>S: Client Hello (Cr, Cookie, Cipher Suites)\nS-->>C: Server Hello (Sr, Cipher Suite), Certificate (Sn, Se)\nS-->>C: Server Key Exchange (p,g,A,Ss)\nS-->>C: Certificate Request, Server Hello Done\nC->>S: Certificate (Cn, Ce)\nC->>S: Client Key Exchange (B)\nC->>S: Certificate Verify (Cs)\nend\nNote over C,S: Establish Secure Channel\nrect rgb(239, 252, 202)\nC->>S: Change Cipher Spec, Encrypted Handshake Message\nS-->>C: Change Cipher Spec, Encrypted Handshake Message\nC->>S: Application Data\nS-->>C: Application Data\nend\n \n
\n
Below is the analysis with regard to the data package numbers in the message sequence chart:
\n
\n
Packets \\(\\require{enclose}\\enclose{circle}{1}-\\enclose{circle}{3}\\) present the initial handshake message exchange.\n
\n
The client first sends a Hello message containing a random number \\(C_r\\) and a list of supported cipher suites
\n
The server responds with a Hello Verify Request message containing a block of information (cookie)
\n
The client receives the Hello Verify Request and resends the Hello message with the entire contents of the previous message plus a copy of the cookie
\n
\n
\n
Hello verification is specific to DTLS to prevent denial of service attacks. The protocol stipulates that the server will not continue to serve the client until it receives a hello message containing the copied cookie.
\n
\n
\n
Packets \\(\\require{enclose}\\enclose{circle}{4}-\\enclose{circle}{6}\\) shows the server enters verification and key exchange stage:\n
\n
The server responds with a Hello message first, which contains the random number \\(S_r\\) and the selected cipher suite\n
\n
As shown below, the server selects TLS_DHE_RSA_WITH_AES_128_CBC_SHA!
\n
\n
The same packet also contains the Server Certificate message, which is typically large and divided into multiple fragments
\n
The server certificate provides the RSA public key \\((S_N,\\;S_e)\\) that verifies its signature
\n
Next, the server sends a Key Exchange message containing its DH public key \\((p,g,A)\\) and signature \\(Ss\\)\n
\n
The length of \\(p\\) in the figure below is 256 bytes, which means that the key length is 2048 bits and \\(Pubkey\\) is \\(A\\).
\n
You can also see in the figure that the algorithms chosen for the signature are SHA512 and RSA.
\n
The operation is to first compute \\(\\operatorname{SHA512}(Cr,Sr,p,g,A)\\) and then encrypt it with the server RSA private key
\n
\n
After that, the server sends a Certificate Request message and a Hello Done message\n
\n
The server requests the client to send an RSA public key certificate that verifies its signature
\n
\n
\n
\n
Note: If DH-RSA cipher suite is used, the server-side DH public key parameters \\((p,g,A)\\) are unchanged and will be included directly in its certificate message. At this time, the server will not issue a Key Exchange message \\(\\require{enclose}\\enclose{circle}{5}\\). For DHE-RSA, the \\(A\\) value is different for each session.
\n
\n
\n
Packets \\(\\require{enclose}\\enclose{circle}{7}-\\enclose{circle}{9}\\) shows the client enters verification and key echange stage:\n
\n
The client first sends a Certificate message, which contains the RSA public key \\((C_N,\\;C_e)\\) and also splits into multiple fragments
\n
The client then sends a Key Exchange message, which contains its DH public key \\(B\\)\n
\n
The \\(Pubkey\\) in the following figure is \\(B\\)
\n
\n
The client finally sends a Certificate Verify message, which contains the signature \\(Cs\\)\n
\n
The signature covers all previous messages except for the initial Client Hello \\(\\require{enclose}\\enclose{circle}{1}\\) and the Hello Verify Request \\(\\require{enclose}\\enclose{circle}{2}\\)
\n
The signature operation also computes SHA512 and encrypts it with the client's RSA private key
\n
\n
\n
Packets \\(\\require{enclose}\\enclose{circle}{10}-\\enclose{circle}{11}\\) completes handshake and establishs the secure channel:\n
\n
Each side first verifies the signature sent by the other side
\n
After successful verification, DH algorithm is run to generate the same premaster key
\n
Both parties call pseudo-random function (PRF) to generate a 48-byte master key from the premaster key \\[master\\_secret = \\operatorname{PRF}(pre\\_master\\_secret,\\unicode{x201C}master\\;secret\\unicode{x201D},Cr+Sr)[0..47]\\]
\n
Both parties call PRF again to generate a 72-byte key block from the master key \\[key\\_block = \\operatorname{PRF}(master\\_secret,\\unicode{x201C}key\\;expansion\\unicode{x201D},Sr+Cr)[0..71]\\]
\n
Key blocks are assigned to HMAC-SHA1 and AES_128_CBC function blocks.\n
Server Write Message Authentication Code (MAC) key: 20 bytes
\n
Client Write Encryption Key: 16 bytes
\n
Server write encryption key: 16 bytes
\n
\nNote that TLS/DTLS 1.2 specifies that this cipher suite uses an explicit initial vector (IV) and does not require the allocation of a key block
\n
The client generates a Change Cipher Spec message indicating the start of the encryption and MAC modules
\n
The client invokes PRF a third time to generate the 12-byte end-of-handshake authentication code used for master key and handshake message authentication, which is packaged into an end-of-handshake message and entered into the encryption and MAC modules \\[\\operatorname{PRF}(master\\_secret,finished\\_label,\\operatorname{SHA256}(handshake\\_messages))[0..11]\\]
\n
The client sends the Change Cipher Spec message and the encrypted end-of-handshake message to the server
\n
The server verifies the received client end-of-handshake message and repeats the above three steps to generate its own Change Cipher Spec message and encrypted an end-of-handshake message, then send them to the client
\n
The client completes the handshake by verifying the received server end-of-handshake message. Now the encrypted secure channel is established
\n
\n
Packets \\(\\require{enclose}\\enclose{circle}{12}-\\enclose{circle}{13}\\) shows that the encrypted application data exchange has officially started
\n
\n
This is the complete process of establishing a secure message channel using the TLS_DHE_RSA_WITH_AES_128_CBC_SHA (encoding 0x00,0x33) cipher suite, where DHE implements a key exchange with forward secrecy protection and RSA digital signature provides authentication for DHE, creating a solution for secure communication. With a clear understanding of this, we will better grasp the working mechanism of Diffie-Hellman and RSA, effectively apply them in practice and avoid unnecessary mistakes.
The problem of Endianness is essentially a question about how computers store large numbers.
\n
I do not fear computers. I fear lack of them. — Isaac Asimov (American writer and professor of biochemistry, best known for his hard science fiction)
\n
\n
We know that one basic memory unit can hold one byte, and each memory unit has its address. For an integer larger than decimal 255 (0xff in hexadecimal), more than one memory unit is required. For example, 4660 is 0x1234 in hexadecimal and requires two bytes. Different computer systems use different methods to store these two bytes. In our common PC, the least-significant byte 0x34 is stored in the low address memory unit and the most-significant byte 0x12 is stored in the high address memory unit. While in Sun workstations, the opposite is true, with 0x34 in the high address memory unit and 0x12 in the low address memory unit. The former is called Little Endian and the latter is Big Endian.
\n
How can I remember these two data storing modes? It is quite simple. First, remember that the addresses of the memory units we are talking about are always arranged from low to high. For a multi-byte number, if the first byte in the low address you see is the least-significant byte, the system is Little Endian, where Little matches low. On the contrary is Big Endian, where Big corresponds to \"high\".
\n
Program Example
\n
To deepen our understanding of Endianness, let's look at the following example of a C program:
\n
char a = 1; \t \t \t char b = 2; short c = 255;\t/* 0x00ff */ long d = 0x44332211;
\n
On Intel 80x86 based systems, the memory content corresponding to variables a, b, c, and d are shown in the following table：
\n
\n\n
\n
Address Offset
\n
Memory Content
\n
\n\n\n
\n
0x0000
\n
01 02 FF 00
\n
\n
\n
0x0004
\n
11 22 33 44
\n
\n\n
\n
We can immediately tell that this system is Little Endian. For a 16-bit integer short c, we see the least-significant byte 0xff first, and the next one is 0x00. Similarly for a 32-bit integer long d, the least-significant byte 0x11 is stored at the lowest address 0x0004. If this is in a Big Endian computer, memory content would be 01 02 00 FF 44 33 22 11.
\n
At the run time all computer processors must choose between these two Endians. The following is a shortlist of processor types with supported Endian modes:
\n
\n
Pure Big Endian: Sun SPARC, Motorola 68000, Java Virtual Machine
\n
Bi-Endian running Big Endian mode: MIPS with IRIX, PA-RISC, most Power and PowerPC systems
\n
Bi-Endian running Little Endian mode: ARM, MIPS with Ultrix, most DEC Alpha, IA-64 with Linux
\n
Little Endian: Intel x86, AMD64, DEC VAX
\n
\n
How to detect the Endianess of local system in the program? The following function can be called for a quick check. If the return value is 1, it is Little Endian, else Big Endian：
\n
inttest_endian() { int x = 1; return *((char *)&x); }
\n
Network Order
\n
Endianness is also important for computer communications. Imagine that when a Little Endian system communicates with a Big Endian system, the receiver and sender will interpret the data completely differently if not handled properly. For example, for the variable d in the C program segment above, the Little Endian sender sends 11 22 33 44 four bytes, which the Big Endian receiver converts to the value 0x11223344. This is very different from the original value. To solve this problem, the TCP/IP protocol specifies a special \"network byte order\" (referred to as \"network order\"), which means that regardless of the Endian supported by the computer system, the most-significant byte is always sent first while transmitting data. From the definition, we can see that the network order corresponds to the Big Endian.
\n
To avoid communication problems caused by Endianness and to facilitate software developers to write portable programs, some C preprocessing macros are defined for conversion between network bytes and local byte order. htons() and htonl() are used to convert local byte order to network byte order, the former works with 16-bit unsigned numbers and the latter for 32-bit unsigned numbers. ntohs() and ntohl() implement the conversion in the opposite direction. The prototype definitions of these four macros can be found as follows (available in the netinet/in.h file on Linux systems).
About the IP packet header checksum algorithm, simply put, it is 16-bit ones' complement of the ones' complement sum of all 16-bit words in the header. However, not many sources show exactly how this is done. The same checksum algorithm is used by TCP segment and UDP datagram, but the data involved in the checksum computing is different from that in the IP header. In addition, the checksum operation of the IPv6 packet is different from that of IPv4. Therefore, it is necessary to make a comprehensive analysis of the checksum algorithm of IP packets.
\n
Nothing in life is to be feared, it is only to be understood. — Marie Curie (Polish and naturalized-French physicist and chemist, twice Nobel Prize winner)
Here the 16-bit Header Checksum field is used for error-checking of the IPv4 header. While computing the IPv4 header checksum, the sender first clears the checksum field to zero, then calculates the sum of each 16-bit value within the header. The sum is saved in a 32-bit value. If the total number of bytes is odd, the last byte is added separately.
\n
After all additions, the higher 16 bits saving the carry is added to the lower 16 bits. Repeat this till all higher 16 bits are zeros. Finally, the sender takes the ones' complement of the lower 16 bits of the result and writes it to the IP header checksum field.
\n
The following demonstrates the entire calculation process using actual captured IPv4 packets.
At the beginning of the above 16-bit hex dump is the Ethernet frame header. The IP packet header starts from offset 0x000e, with the first byte 0x45 and the last byte 0xe9. Based on the previous description of the algorithm, we can make the following calculations:
Notice at step (1) we replace the checksum field with 0x0000. As can be seen, the calculated header checksum 0x598f is the same as the value in the captured packet. This calculating process is only used for the sender to generate the initial checksum. In practice, for the intermediate forwarding router and the final receiver, they can just sum up all header fields of the received IP packet by the same algorithm. If the result is 0xffff, the checksum verification passes.
\n
C Program Implementation
\n
How to program IPv4 header checksum computing? RFC 1071 (Computing the Internet Checksum) shows a reference \"C\" language implementation:
\n
{ /* Compute Internet Checksum for "count" bytes * beginning at location "addr". */ registerlong sum = 0;
while( count > 1 ) { /* This is the inner loop */ sum += * (unsignedshort *) addr++; count -= 2; }
/* Add left-over byte, if any */ if ( count > 0 ) sum += * (unsignedchar *) addr;
/* Fold 32-bit sum to 16 bits */ while (sum>>16) sum = (sum & 0xffff) + (sum >> 16);
checksum = ~sum; }
\n
In a real network connection, the source device can call the above code to generate the initial IPv4 header checksum. This checksum is then updated at each step of the routing hop because the router must decrement the Time To Live (TTL) field. RFC 1141 (Incremental Updating of the Internet Checksum) gives a reference implementation of fast checksum update：
For TCP segment and UDP datagram, both have 16-bit header checksum fields used for error-checking by the destination host. The checksum computing algorithm is the same as the IP header, except for the difference of covered data. Here the checksum is calculated over the whole TCP/UDP header and the payload, plus a pseudo-header that mimics the IPv4 header as shown below:
It consists of the source and destination IP addresses, the protocol number (TCP:6/UDP:17), and the total length of the TCP/UDP header and payload (in bytes). The purpose of including the pseudo-header in the checksum computing is to confirm the packet reaches the expected destination and avoid IP spoofing attacks. Besides, for IPv4 UDP header checksum is optional, it carries all-zeros if unused.
\n
IPv6 Difference
\n
IPv6 is IP protocol version 6, and its main design goal was to resolve the problem of IPv4 address exhaustion. Of course, it provides many benefits in other aspects. Although IPv6 usage is growing slowly, the trend is unstoppable. The latest IPv6 standard is published in RFC 8200(Internet Protocol, Version 6 (IPv6) Specification).
Notice that the IPv6 header does not include a checksum field, a significant difference from IPv4. The absence of a checksum in the IPv6 header furthers the end-to-end principle of Internet design, to simplify router processing and speed up the packet transmission. Protection for data integrity can be accomplished by error detection at the link layer or the higher-layer protocols between endpoints (such as TCP/UDP on the transport layer). This is why IPv6 forces the UDP layer to set the header checksum.
\n
For IPv6 TCP segment and UDP datagram header checksum computing, the pseudo-header that mimics the IPv6 header is shown below
In actual IPv6 network applications, UDP-Lite (Lightweight UDP) can be used to balance error detection and transmission efficiency. UDP-Lite has its own protocol number 136, and its standard is described in RFC 3828 (The Lightweight User Datagram Protocol (UDP-Lite)).
\n
Referring to the following header format, UDP-Lite uses the same set of port number values assigned by the IANA for use by UDP. But it redefines the Length field in the UDP header to a Checksum Coverage, which allows the application layer to control the length of checksummed data. This is useful for the application that can be tolerant of the potentially lossy transmission of the uncovered portion of the data.
UDP-Lite protocol defines the values of \"Checksum Coverage\" (in bytes) as shown in the following table:
\n
\n
\n
\n
\n
\n
\n\n
\n
Checksum Coverage
\n
Coverage Area
\n
Description
\n
\n\n\n
\n
0
\n
entire UDP-Lites datagram
\n
Calculation covers IP pseudo-header
\n
\n
\n
1-7
\n
(invalid)
\n
The receiver has to drop the datagram
\n
\n
\n
8
\n
UDP-Lites header
\n
Calculation covers IP pseudo-header
\n
\n
\n
> 8
\n
UDP-Lites header + portion of payload data
\n
Calculation covers IP pseudo-header
\n
\n
\n
> IP datagram length
\n
(invalid)
\n
The receiver has to drop the datagram
\n
\n\n
\n
For multimedia applications running VoIP or streaming video data transmission protocols, it'd better receive data with some degree of corruption than not receiving any data at all. Another example is the CAPWAP protocol used to connect Cisco wireless controller and access points. It specifies UDP-Lite as the default transport protocol for the CAPWAP Data channel, while the connection is established over the IPv6 network.
\n
At last, share a C program snippet to present how to initialize a Berkeley socket to establish an IPv6 UDP-Lite connection:
int udplite_conn = socket(AF_INET6, SOCK_DGRAM, IPPROTO_UDPLITE); int val = 8; /* checksum only covers 8-byte UDP-Lite header */ (void)setsockopt(udplite_conn, IPPROTO_UDPLITE, UDPLITE_SEND_CSCOV, &val, sizeof val); (void)setsockopt(udplite_conn, IPPROTO_UDPLITE, UDPLITE_RECV_CSCOV, &val, sizeof val);
\n
Here IPPROTO_UDPLITE is protocol number 136, which is used together with AF_INET6 address family parameter in socket() function call for IPv6 socket creation. The UDPLITE_SEND_CSCOV(10) and UDPLITE_RECV_CSCOV(11) are the control parameters of socket options configuration function setsockopt(), used for setting the Checksum Coverage value in the sender and the receiver respectively. Remember that both the sender and the receiver must set the same value, otherwise, the receiver will not be able to verify the checksum properly.
IPv6 supports multiple addresses, making address assignments more flexible and convenient. Unlike IPv4, which relied solely on the DHCP protocol for address assignment, IPv6 incorporates a native Stateless Address AutoConfiguration SLAAC) protocol. SLAAC can either work alone to provide IPv6 addresses to hosts, or it can work with DHCPv6 to generate new assignment schemes. Here is a comprehensive analysis of the dynamic address allocation mechanism for IPv6.
\n
Who the hell knew how much address space we needed? — Vint Cerf (American Internet pioneer and one of \"the fathers of the Internet\")
\n
\n
IPv6 Address Overview
\n
Address Formats
\n
The most significant difference between IPv6 and IPv4 is its large address space. IPv4 has 32 bits (4 bytes) and allows for approximately 4.29 (232) billion addresses. IPv6, on the other hand, defines 128 bits (16 bytes) and supports approximately 340 x 1036 addresses. This is a pretty impressive number, and there will be no address depletion for the foreseeable future. A typical IPv6 address can be divided into two parts. As shown in the figure below, the first 64 bits are used to represent the network, and the next 64 bits are used as the interface identifier.
\n
The interface identifier can be generated in several ways:
Automatically established randomly or cryptographically
\n
\n
IETF recommends a canonical textual representation format for ease of writing. It includes leading zeros suppression and compression of consecutive all-zero fields. With the network prefix length at the end, the above address can be shortened to 2001:db8:130f::7000:0:140b/64.
Unicast: A network address corresponds to a single network node, point-to-point connection.
\n
Anycast: The target address corresponds to a group of receiving nodes, but only the \"nearest\" one receives.
\n
Multicast: The target address corresponds to a group of nodes that can receive replicated messages.
\n\n
Note that there are no broadcast addresses in IPv6, their function being superseded by multicast addresses. Anycast addresses are syntactically indistinguishable from unicast addresses and have very limited applications. A typical application for anycast is to set up a DNS root server to allow hosts to look up domain names in close proximity. For unicast and multicast addresses, they can be identified by different network prefixes:
\n
\n
\n
\n
\n
\n
\n
\n\n
\n
Address Type
\n
Binary Form
\n
Hexadecimal Form
\n
Application
\n
\n\n\n
\n
Link-local address (unicast)
\n
1111 1110 10
\n
fe80::/10
\n
Use on a single link, non-routable
\n
\n
\n
Unique local address (unicast)
\n
1111 1101
\n
fd00::/8
\n
Analogous to IPv4 private network addressing
\n
\n
\n
Global unicast address
\n
001
\n
2000::/3
\n
Internet communications
\n
\n
\n
Multicast address
\n
1111 1111
\n
ff00::/8
\n
Group communications, video streaming
\n
\n\n
\n
Each interface of a host must have a link-local address. Additionally, it can be manually or dynamically autoconfigured to obtain a unique local address and a global unicast address. Thus, IPv6 interfaces naturally have multiple unicast addresses. Unique local addresses are managed by the local network administrator, while the global unicast addresses are allocated by the IANA-designated regional registry. Referring to the following diagram, all current global unicast addresses are assigned from the 2000::/3 address block, with the first 48 bits of the address identifying the service provider's global routing network and the next 16 bits identifying the enterprise or campus internal subnet: Because an IPv6 multicast address can only be used as a destination address, its bit definition is different from that of unicast. Referring to RFC 4291, a multicast address containing 4 bits of the feature flags, 4 bits of the group scope, and the last 112 bits of the group identifier: Furthermore the same protocol specifies a few pre-defined IPv6 multicast addresses, the most important of which are
\n
\n
All Nodes Addresses on the local link — ff02::1
\n
All Routers Addresses on the local link — ff02::2
\n
Solicited-Node Address on local link — ff02::1:ffxx:xxxx
\n
\n
Dynamic Allocation Schemes
\n
NDP Protocol
\n
IPv6 dynamic address assignment depends on Neighbor Discovery Protocol (NDP). NDP acts at the data link layer and is responsible for discovering other nodes and corresponding IPv6 addresses on the link and determining available routes and maintaining information reachability to other active nodes. It provides the IPv6 network with the equivalent of the Address Resolution Protocol (ARP) and ICMP router discovery and redirection protocols in IPv4 networks. However, NDP adds many improvements and new features. NDP defines five ICMPv6 message types:
\n\n
Router Solicitation (RS)
\n
Router Advertisement (RA)
\n
Neighbor Solicitation (NS)
\n
Neighbor Advertisement (NA)
\n
Redirect
\n\n
The first two message types here, RS and RA, are the keys to implementing dynamic IPv6 address assignment. The host sends an RS message to the multicast address ff02::2 of all routers in the local network segment to request routing information. When the router receives the RS from the network node, it sends an immediate RA in response. The message format of the RA is as follows
It defines two special bits, M and O, with the following meaning:
\n
\n
M — \"Managed address configuration\" flag, set to 1 when the address is obtained from DHCPv6.
\n
O — \"Other configuration\" flag, set to 1 to indicate that other configuration information is available via DHCPv6
\n
\n
The RA message ends with the Options section, which originally had three possible options: Source Link-Layer Address, MTU, and Prefix Information. Later, RFC 8106 (which replaced RFC 6106) added the Recursive DNS Server (RDNSS) and DNS Search List (DNSSL) options. The Prefix Information option directly provide hosts with on-link prefixes and prefixes for Address Autoconfiguration, and it has the following format
Here the Prefix Length and the Prefix jointly determine the network prefix of the IPv6 address. In addition, the Prefix Information option also defines two special bits, L and A:
\n
\n
L — on-link flag. When set, indicates that this prefix can be used for on-link determination.
\n
A — autonomous address-configuration flag. When set, indicates that this prefix can be used for SLAAC.
\n
\n
Similar to the IPv4 subnet mask feature, the purpose of the \"on-link\" determination is to allow the host to determine which networks an interface can access. By default, the host only considers the network where the link-local address is located as \"on-link\". If the \"on-link\" status of a destination address cannot be determined, the host forwards the IPv6 datagram to the default gateway (or default router) by default. When the host receives an RA message, if the \"on-link\" flag for a prefix information option is set to 1 and the Valid Lifetime is also a non-zero value, the host creates a new prefix network entry for it in the prefix list. All unexpired prefix network entries are \"on-link\".
\n
Message Sequence
\n
After understanding the NDP protocol and the information conveyed by the RA messages, let's see how they guide the network nodes to achieve dynamic address assignment.
\n
Routers in the network periodically send RA messages to the multicast addresses (ff02::1) of all nodes in the local subnet. However, to avoid latency, the host sends one or more RS messages to all routers in the local subnet as soon as it has finished booting. The protocol requires the routers to respond to the RA messages within 0.5 seconds. Then, based on the values of the M/O/A bits in the received RA messages, the host decides how to dynamically configure the unique local and global unicast addresses of the interface and how to obtain other configuration information. With certain combinations of bit fetch values, the host needs to run DHCPv6 client software to connect to the server to obtain address assignment and/or other configuration information. The entire process is shown in the following message sequence diagram.
\n
\nsequenceDiagram\n\nparticipant R as Router\nparticipant H as Host\nparticipant S as DHCPv6 Server\nNote over R,H: Router Request\nrect rgb(239, 252, 202)\nH->>R: Router Solicitation\nR-->>H: Router Advertisement\nend\nNote over H,S: Address Request\nrect rgb(230, 250, 255)\nH->>S: DHCPv6 Solicit\nS-->>H: DHCPv6 Advertise\nH->>S: DHCPv6 Request\nS-->>H: DHCPv6 Reply\nend\nNote over H,S: Other Information Request\nrect rgb(230, 250, 255)\nH->>S: DHCPv6 Information-request\nS-->>H: DHCPv6 Reply\nend\n\n
\n
Note: Unlike the IPv4 DHCP protocol, DHCPv6 clients use UDP port 546 and servers use UDP port 547.
\n
\n
Next explain in detail three dynamic allocation schemes determined by the combination of the M/O/A-bit values:
\n
\n
SLAAC
\n
SLAAC + Stateless DHCPv6
\n
Stateful DHCPv6
\n
\n
SLAAC
\n
SLAAC is the simplest automatic IPv6 address assignment scheme and does not require any server. It works by sending an RS message request after the host starts up and the router sends back RA messages to all nodes in the local network segment. If the RA message contains the following configuration
\n
\n
M-bit and O-bit all clear in the message header
\n
L-bit and A-bit all set in Prefix Information option
\n
\n
Then the host receives this RA message and performs the following operations to implement SLAAC:
\n\n
Combine the network prefix with the local interface identifier to generate a unique local address or global unicast address.
\n
Install the default gateway (or default route) to point to the router address (source address of the RA message).
\n
Set this interface as the \"on-link\" corresponding to the network prefix, which is also the next-hop interface of the default gateway above.
\n
If the RDNSS and/or DNSSL options are included, install the name servers and domain name suffixes.
\n\n
This way, the host gets one or more IPv6 unique local addresses or global unicast addresses, plus the default gateway and domain name service information to complete various Internet connections.
\n
The following is an example of the SLAAC configuration on a Cisco Catalyst 9300 Multilayer Access Switch:
\n
ipv6 unicast-routing interface Vlan10 ipv6 enable ipv6 address 2001:ABCD:1000::1/64 ipv6 nd ra dns server 2001:4860:4860::8888 infinite ipv6 nd ra dns search-list example.com
\n
The Layer 3 interface of the Cisco Multilayer Switch provides routing functionality. As you can see, when IPv6 is activated on the Layer 3 interface in VLAN 10, its default address auto-assignment scheme is SLAAC. the control bits of RA messages from this interface are all set according to the SLAAC scheme, and the network prefixes for each IPv6 address it configures are automatically added to the RA prefix information options list. Of course, the network administrator can also exclude certain network prefixes with a separate interface configuration command. The last two lines of the example configuration command specify RDNSS and DNSSL, which are also added to the RA message options.
\n
If a host connects to a port in VLAN 10, it immediately gets a global unicast address with the network prefix of 2001:ABCD:1000::/64, and its default gateway address is set to 2001:ABCD:1000::1. Open a browser and enter a URL, and it will send a message to the specified domain name server 2001:4860:4860::8888 (Google's public name server address) to obtain the IPv6 address of the destination URL to establish a connection.
\n
SLAAC + Stateless DHCPv6
\n
SLAAC automatic address assignment is fast and easy, providing a plug-and-play IPv6 deployment solution for small and medium-sized network deployments. However, if a network node needs access to additional configuration information, such as NTP/SNTP server, TFTP server, and SIP server addresses, or if its functionality relies on certain Vendor-specific Information Options, it must choose SLAAC + stateless DHCPv6 scheme.
\n
This scenario still uses SLAAC automatic address assignment, but the router instructs the host to connect to a DHCPv6 server for additional configuration information. At this point, the RA message sent back by the router has
\n
\n
M-bit clear and O-bit set in the message header
\n
L-bit and A-bit all set in Prefix Information option
\n
\n
After receiving this RA message, the host performs the following actions:
\n\n
Combine the network prefix with the local interface identifier to generate a unique local address or global unicast address.
\n
Install a default gateway (or default route) pointing to the router address (source address of the RA message).
\n
Set this interface as the \"on-link\" corresponding to the network prefix, which is also the next-hop interface of the default gateway above.
\n
If the RDNSS and/or DNSSL options are included, install the name servers and domain name suffixes.
\n
Start the DHCPv6 client and connect to the DHCPv6 server to request additional configuration information.
\n
Save the additional configuration information replied by the DHCPv6 server.
\n\n
As you can see, SLAAC + stateless DHCPv6 is not different from SLAAC in terms of address assignment. DHCPv6 only provides additional configuration information and does not assign IPv6 addresses. So the DHCPv6 server does not track the address assignment status of network nodes, which is what \"stateless\" means.
\n
The corresponding configuration commands on the Catalyst 9300 switch are as follows.
The difference with the SLAAC example is that the VLAN 10 interface configuration command ipv6 nd other-config-flag explicitly specifies to set the O-bit of the RA message. Its next command, ipv6 dhcp server vlan-10-clients, activates the DHCPv6 server response feature of the interface, corresponding to the server's pool name of vlan-10-clients. The DHCPv6 server is configured above the interface configuration, starting at ipv6 dhcp pool vlan-10-clients, and contains the DNS server address, DNS domain name, and SNTP server address.
\n
If you are using a separate DHCPv6 server located on a network segment, you can remove the ipv6 dhcp server command and enable the ipv6 dhcp relay destination command on the next line of the example to specify the address to forward DHCPv6 requests to the external server.
\n
Stateful DHCPv6
\n
Many large enterprises use DHCP to manage the IPv4 addresses of their devices, so deploying DHCPv6 to centrally assign and manage IPv6 addresses is a natural preference. This is where Stateful DHCPv6 comes into play. This scenario also requires RA messages sent by the router but does not rely solely on network prefixes for automatic address assignment. The control bits of the RA messages are configured to
\n
\n
M-bit set in the message header, O-bit does not matter
\n
L-bit and A-bit can be set or clear as desired in Prefix Information option
\n
\n
Upon receiving this RA message, the host performs the following actions:
\n\n
Generate a unique local address or a global unicast address if there is a Prefix Information option with the A-bit set.
\n
Install a default gateway (or default route) pointing to the router address (source address of the RA message).
\n
If there is a Prefix Information option with the L-bit set, set this interface to \"on-link\" with the corresponding network prefix.
\n
If the RDNSS and/or DNSSL options are included, install the name servers and domain suffixes.
\n
Start the DHCPv6 client and connect to the server to request addresses and other configuration information.
\n
Set the address assigned by the DHCPv6 server to this interface.
\n
Save additional configuration information from the DHCPv6 server response.
\n\n
An example of the Stateful DHCPv6 configuration command on a Catalyst 9300 switch is as follows.
Compared to SLAAC + Stateless DHCPv6, the interface configuration here removes the ipv6 nd other-config-flag and replaces it with the ipv6 nd managed-config-flag command. This corresponds to setting the M-bit of the RA message header. The DHCPv6 server configuration adds two address prefix commands to set the network prefix. Also, the ipv6 nd prefix 2001:ABCD:1:1::/64 no-advertise configured for the interface specifies that the router does not include the 2001:ABCD:1:1::/64 prefix information option into the RA. So, this example host interface will not generate SLAAC addresses, but only two addresses from DHPCv6: a unique local address with the network prefix FD09:9:5:90::/64, and a global unicast address with the network prefix 2001:9:5:90::/64. The interface identifier for each of these two addresses is also specified by DHPCv6.
\n
How to distinguish the source of dynamically assigned addresses for host interfaces? The method is simple. One thing to remember is that DHPCv6 does not send the network prefix length to the requestor, so the network prefix length of the addresses received from DHPCv6 is 128, while the network prefix length of the addresses generated by SLAAC will not be 128. See the following example of the wired0 interface on a Linux host:
We can immediately determine that the interface is using Stateful DHCPv6 address assignment, but also generates the SLAAC address with the same network prefix 2001:20::/64 received.
\n
\n
2001:20::53c7:1364:a4d8:fd91/128 — DHCPv6 address, random interface identifer
\n
2001:20::a2ec:f9ff:fe6c:d930/64 — SLAAC addeess, interface identifer is MAC in EUI-64 format
\n
fe80::a2ec:f9ff:fe6c:d930/64 — Link-local address, interface identifer is MAC in EUI-64 format
\n
\n
Note: DHPCv6 server also does not provide any IPv6 default gateway information. The host needs to be informed of the dynamic default gateway from the RA message.
\n
\n
Summary and Comparison
\n
The following table shows the control bit combinations of RA messages concerning different address allocation and other configuration acquisition methods.
\n
\n\n
\n
M-bit
\n
O-bit
\n
A-bit
\n
Host Address
\n
Other Configuration
\n
\n\n\n
\n
0
\n
0
\n
0
\n
Static Settings
\n
Manual Configuration
\n
\n
\n
0
\n
0
\n
1
\n
Prefix specified by RA, automatically generated
\n
manually configured
\n
\n
\n
0
\n
1
\n
0
\n
Static Settings
\n
DHCPv6
\n
\n
\n
0
\n
1
\n
1
\n
Prefix specified by RA, automatically generated
\n
DHCPv6
\n
\n
\n
1
\n
0
\n
0
\n
Stateful DHCPv6
\n
DHCPv6
\n
\n
\n
1
\n
0
\n
1
\n
Stateful DHCPv6 and/or automatically generated
\n
DHCPv6
\n
\n
\n
1
\n
1
\n
0
\n
Stateful DHCPv6
\n
DHCPv6
\n
\n
\n
1
\n
1
\n
1
\n
Stateful DHCPv6 and/or automatically generated
\n
DHCPv6
\n
\n\n
\n
Summarize three dynamic allocation schemes:
\n
\n
\n
\n
\n
\n
\n\n
\n
Allocation Scheme
\n
Features
\n
Appiccation Scenarios
\n
\n\n\n
\n
SLAAC
\n
Simple and practical, fast deployment
\n
SMB, Consumer Product Networking, Internet of Things (IoT)
\n
\n
\n
SLAAC + Stateless DHCPv6
\n
Auto Configuration, Extended Services
\n
SMBs need additional network services
\n
\n
\n
Stateful DHCPv6
\n
Centralized management and control
\n
Large enterprises, institutions, and campus networks
\n
\n\n
\n
Note: Since IPv6 network interfaces can have multiple addresses (a link-local address, plus one or more unique local addresses and/or global unicast addresses), it becomes important how the source address is selected when establishing an external connection. RFC 6724 gives detailed IPv6 source address selection rules. In the development of embedded systems, the control plane and the data plane connected to the same remote device are often implemented by different functional components. For example, the control plane directly calls a Linux userspace socket to establish the connection, and the IPv6 source address used for the connection is selected by the TCP/IP stack, while the data plane directly implements data encapsulation processing and transmission in kernel space. In this case, the IPv6 source address selected by the control plane has to be synchronized to the data plane in time, otherwise, the user data might not be delivered to the same destination.
\n
\n
Troubleshooting Guide
\n
The common IPv6 dynamic address assignment debugging and troubleshooting commands on Cisco routers and switches are listed in the following table.
\n
\n
\n
\n
\n
\n\n
\n
Command
\n
Description
\n
\n\n\n
\n
show ipv6 interface brief
\n
Displays a short summary of IPv6 status and configuration for each interface
\n
\n
\n
show ipv6 interface [type] [num]
\n
Displays IPv6 and NDP usability status information for single interface
\n
\n
\n
show ipv6 interface [type] [num] prefix
\n
Displays IPv6 network prefix information for single interface
\n
\n
\n
show ipv6 dhcp pool
\n
Display DHCPv6 configuration pool information
\n
\n
\n
show ipv6 dhcp binding
\n
Displays all automatic client bindings from the DHCPv6 server binding table
\n
\n
\n
show ipv6 dhcp interface [type] [num]
\n
Display DHCPv6 interface information
\n
\n
\n
debug ipv6 nd
\n
Debug IPv6 NDP protocol
\n
\n
\n
debug ipv6 dhcp
\n
Debug DHCPv6 server
\n
\n\n
\n
The following console NDP protocol debug log shows that the router received an RS message from host FE80::5850:6D61:1FB:EF3A and responded with an RA message to the multicast address FF02::1 of all nodes in this network:
\n
Router# debug ipv6 nd ICMP Neighbor Discovery events debugging is on Router# show logging | include RS ICMPv6-ND: Received RS on GigabitEthernet0/0/0 from FE80::5850:6D61:1FB:EF3A Router# show logging | include RA ICMPv6-ND: Sending solicited RA on GigabitEthernet0/0/0 ICMPv6-ND: Request to send RA for FE80::C801:EFFF:FE5A:8 ICMPv6-ND: Setup RA from FE80::C801:EFFF:FE5A:8 to FF02::1 on GigabitEthernet0/0/0
\n
And the next log shows an example of Stateless DHCPv6 observed after entering the debug ipv6 dhcp debug command. Host FE80::5850:6D61:1FB:EF3A sends an INFORMATION-REQUEST message to the DHCPv6 server, which selects the source address FE80::C801:B9FF:FEF0:8 and sends a response message.
\n
Router#debug ipv6 dhcp IPv6 DHCP debugging is on
IPv6 DHCP: Received INFORMATION-REQUEST from FE80::5850:6D61:1FB:EF3A on FastEthernet0/0 IPv6 DHCP: Option VENDOR-CLASS(16) is not processed IPv6 DHCP: Using interface pool LAN_POOL IPv6 DHCP: Source Address from SAS FE80::C801:B9FF:FEF0:8 IPv6 DHCP: Sending REPLY to FE80::5850:6D61:1FB:EF3A on FastEthernet0/0
\n
The following debug log of Stateful DHCPv6 shows the complete process of two message exchanges (SOLICIT/ADVERTISE, REQUEST/REPLY) on lines 1, 15, 16, and 26.
\n
IPv6 DHCP: Received SOLICIT from FE80::5850:6D61:1FB:EF3A on FastEthernet0/0 IPv6 DHCP: Option UNKNOWN(39) is not processed IPv6 DHCP: Option VENDOR-CLASS(16) is not processed IPv6 DHCP: Using interface pool LAN_POOL IPv6 DHCP: Creating binding for FE80::5850:6D61:1FB:EF3A in pool LAN_POOL IPv6 DHCP: Binding for IA_NA 0E000C29 not found IPv6 DHCP: Allocating IA_NA 0E000C29 in binding for FE80::5850:6D61:1FB:EF3A IPv6 DHCP: Looking up pool 2001:ABCD::/64 entry with username '000100011F3E8772000C29806CCC0E000C29' IPv6 DHCP: Poolentry for the user not found IPv6 DHCP: Allocated new address 2001:ABCD::D9F7:61C:D803:DCF1 IPv6 DHCP: Allocating address 2001:ABCD::D9F7:61C:D803:DCF1 in binding for FE80::5850:6D61:1FB:EF3A, IAID 0E000C29 IPv6 DHCP: Updating binding address entry for address 2001:ABCD::D9F7:61C:D803:DCF1 IPv6 DHCP: Setting timer on 2001:ABCD::D9F7:61C:D803:DCF1 for 60 seconds IPv6 DHCP: Source Address from SAS FE80::C801:B9FF:FEF0:8 IPv6 DHCP: Sending ADVERTISE to FE80::5850:6D61:1FB:EF3A on FastEthernet0/0 IPv6 DHCP: Received REQUEST from FE80::5850:6D61:1FB:EF3A on FastEthernet0/0 IPv6 DHCP: Option UNKNOWN(39) is not processed IPv6 DHCP: Option VENDOR-CLASS(16) is not processed IPv6 DHCP: Using interface pool LAN_POOL IPv6 DHCP: Looking up pool 2001:ABCD::/64 entry with username '000100011F3E8772000C29806CCC0E000C29' IPv6 DHCP: Poolentry for user found IPv6 DHCP: Found address 2001:ABCD::D9F7:61C:D803:DCF1 in binding for FE80::5850:6D61:1FB:EF3A, IAID 0E000C29 IPv6 DHCP: Updating binding address entry for address 2001:ABCD::D9F7:61C:D803:DCF1 IPv6 DHCP: Setting timer on 2001:ABCD::D9F7:61C:D803:DCF1 for 172800 seconds IPv6 DHCP: Source Address from SAS FE80::C801:B9FF:FEF0:8 IPv6 DHCP: Sending REPLY to FE80::5850:6D61:1FB:EF3A on FastEthernet0/0
\n
For complex cases where it is difficult to identify whether the problem is with the host, router, or DHCPv6 server, we recommend using the free open-source network packet analysis software Wireshark to capture packets of the entire process for analysis. While analyzing packets with Wireshark, you can apply the keyword filtering function.
\n
\n\n
\n
Filter String
\n
Only Show
\n
\n\n\n
\n
icmpv6.type=133
\n
ICMPv6 RS
\n
\n
\n
icmpv6.nd.ra.flag
\n
ICMPv6 RA
\n
\n
\n
dhcpv6
\n
DHCPv6 packets
\n
\n\n
\n
We can either run Wireshark directly on the host side, or we can use the Switched Port Analyzer (SPAN) provided with the switch. Running on the network side, SPAN can collectively redirect packets from a given port to the monitor port running Wireshark for capturing. Cisco Catalyst 9300 Series switches also directly integrate with Wireshark software to intercept and analyze filtered packets online, making it very easy to use.
Accurate and effective testing of IPv6 products is key to ensuring high interoperability, security, and reliability of IPv6 infrastructure deployments. The IPv6 Ready logo is an IPv6 testing and certification program created by the IPv6 Forum. Its goals are to define IPv6 conformance and interoperability test specifications, provide a self-testing toolset, establish Global IPv6 Test Centers and provide product validation services, and finally, issue IPv6 Ready logo.
IPv6 Core Protocols Test Specification (Conformance)
\n
IPv6 Core Protocols Interoperability Test Specification (Interoperability)
\n
\n
Along with these two new test specifications, the project team also affirmed two permanent changes:
\n\n
Testing must be done in an IPv6-only environment, without any IPv4 being used for the device to function.
\n
The device under test must have IPv6 on and enabled on all IP interfaces by default.
\n\n
Not surprisingly, the new version 5.0 core protocols test specification has a section dedicated to defining SLAAC test cases to validate this core IPv6 protocol.
\n
IPv6 Core Protocol RFC List
\n
In the list below, the RFCs shown in bold are directly covered by the IPv6 Ready Version 5.0 Core Protocol Test Specification:
\n
\n
RFC 4191 Default Router Preferences and More-Specific Routes
\n
RFC 4193 Unique Local IPv6 Unicast Addresses
\n
RFC 4291 IP Version 6 Addressing Architecture
\n
RFC 4443 Internet Control Message Protocol (ICMPv6) for the Internet Protocol Version 6 (IPv6) Specification
\n
RFC 4861 Neighbor Discovery for IP version 6 (IPv6)
\n
RFC 4862 IPv6 Stateless Address Autoconfiguration
\n
RFC 4941 Privacy Extensions for Stateless Address Autoconfiguration in IPv6
\n
RFC 5095 Deprecation of Type 0 Routing Headers in IPv6
\n
RFC 6724 Default Address Selection for Internet Protocol Version 6 (IPv6)
\n
RFC 6980 Security Implications of IPv6 Fragmentation with IPv6 Neighbor Discovery
\n
RFC 7217 A Method for Generating Semantically Opaque Interface Identifiers with IPv6 Stateless Address Autoconfiguration (SLAAC)
\n
RFC 8064 Recommendation on Stable IPv6 Interface Identifiers
\n
RFC 8106 IPv6 Router Advertisement Options for DNS Configuration
\n
RFC 8200 Internet Protocol, Version 6 (IPv6) Specification
\n
RFC 8201 Path MTU Discovery for IP version 6
\n
RFC 8415 Dynamic Host Configuration Protocol for IPv6 (DHCPv6)
\n
\n","categories":["Study Notes"],"tags":["TCP/IP","Cisco Technology"]},{"title":"Implement Textbook RSA in Python","url":"/en/2022/01/22/Python-Textbook-RSA/","content":"
RSA encryption algorithm is one of the core technologies of modern public-key cryptography and is widely used on the Internet. As a classical algorithm of public-key cryptography, the programming implementation of textbook RSA can help us quickly grasp its mathematical mechanism and design ideas, and accumulate important experience in the software implementation of cryptography. Here is a detailed example of textbook RSA implementation in Python 3.8 programming environment.
\n
Random numbers should not be generated with a method chosen at random. — Donald Knuth（American computer scientist, mathematician, and professor emeritus at Stanford University, the 1974 recipient of the ACM Turing Award, often called the \"father of the analysis of algorithms\"）
\n
\n
Generating Large Primes
\n
The security of the RSA encryption algorithm is built on the mathematical challenge of factoring the product of two large prime numbers. The first step in constructing the RSA encryption system is to generate two large prime numbers \\(p\\) and \\(q\\), and calculate the modulus \\(N=pq\\). \\(N\\) is the length of the RSA key, the larger the more secure. Nowadays, practical systems require the key length to be no less than 2048 bits, with corresponding \\(p\\) and \\(q\\) about 1024 bits each.
\n
A general effectiveness method for generating such large random prime numbers is a probability-based randomization algorithm, which proceeds as follows:
In this software implementation, the first step can generate odd numbers directly. Also for demonstration purposes, the second step uses the first 50 prime numbers greater than 2 for the basic primality test. The whole process is shown in the following flowchart.
\n
\n
For the first step, Python function programming requires importing the library function randrange() from the random library. The function uses the input number of bits n in the exponents of 2, which specify the start and end values of randrange(). It also sets the step size to 2 to ensure that only n-bit random odd values are returned.
\n
from random import randrange
defgenerate_n_bit_odd(n: int): '''Generate a random odd number in the range [2**(n-1)+1, 2**n-1]''' assert n > 1 return randrange(2 ** (n - 1) + 1, 2 ** n, 2)
\n
The code for the second step is simple. It defines an array with elements of 50 prime numbers after 2, then uses a double loop in the function to implement the basic primality test. The inner for loop runs the test with the elements of the prime array one by one. It aborts back to the outer loop immediately upon failure, from there it calls the function in the first step to generate the next candidate odd number and test again.
\n
defget_lowlevel_prime(n): """Generate a prime candidate not divisible by first primes""" whileTrue: # Obtain a random odd number c = generate_n_bit_odd(n)
# Test divisibility by pre-generated primes for divisor in first_50_primes: if c % divisor == 0and divisor ** 2 <= c: break else: # The for loop did not encounter a break statement, # so it passes the low-level primality test. return c
\n
The Miller-Rabin primality test1 in the third step is a widely used method for testing prime numbers. It uses a probabilistic algorithm to determine whether a given number is a composite or possibly a prime number. Although also based on Fermat's little theorem, the Miller-Rabin primality test is much more efficient than the Fermat primality test. Before showing the Python implementation of the Miller-Rabin prime test, a brief description of how it works is given here.
\n
By Fermat's little theorem, for a prime \\(n\\), if the integer \\(a\\) is not a multiple of \\(n\\), then we have \\(a^{n-1}\\equiv 1\\pmod n\\). Therefore if \\(n>2\\), \\(n-1\\) is an even number and must be expressed in the form \\(2^{s}*d\\), where both \\(s\\) and \\(d\\) are positive integers and \\(d\\) is odd. This yields \\[a^{2^{s}*d}\\equiv 1\\pmod n\\] If we keep taking the square root of the left side of the above equation and then modulo it, we will always get \\(1\\) or \\(-1\\)2. If we get \\(1\\), it means that the following equation ② holds; if we never get \\(1\\), then equation ① holds: \\[a^{d}\\equiv 1{\\pmod {n}}{\\text{ ①}}\\]\\[a^{2^{r}d}\\equiv -1{\\pmod {n}}{\\text{ ②}}\\] where \\(r\\) is some integer that lies in the interval \\([0, s-1]\\). So, if \\(n\\) is a prime number greater than \\(2\\), there must be either ① or ② that holds. The conditional statement of this law is also true, i.e.** if we can find a \\(\\pmb{a}\\) such that for any \\(\\pmb{0\\leq r\\leq s-1}\\) the following two equations are satisfied: \\[\\pmb{a^{d}\\not \\equiv 1\\pmod n}\\]\\[\\pmb{a^{2^{r}d}\\not \\equiv -1\\pmod n}\\] Then \\(\\pmb{n}\\) must not be a prime number**. This is the mathematical concept of the Miller-Rabin primality test. For the number \\(n\\) to be tested, after calculating the values of \\(s\\) and \\(d\\), the base \\(a\\) is chosen randomly and the above two equations are tested iteratively. If neither holds, \\(n\\) is a composite number, otherwise, \\(n\\) may be a prime number. Repeating this process, the probability of \\(n\\) being a true prime gets larger and larger. Calculations show that after \\(k\\) rounds of testing, the maximum error rate of the Miller-Rabin primality test does not exceed \\(4^{-k}\\).
\n
The Miller-Rabin primality test function implemented in Python is as follows, with the variables n,s,d,k in the code corresponding to the above description.
\n
defmiller_rabin_primality_check(n, k=20): '''Miller-Rabin Primality Test with a specified round of test Input: n - n > 3, an odd integer to be tested for primality k - the number of rounds of testing to perform Output: True - passed (n is a strong probable prime) False - failed (n is a composite)''' # For a given odd integer n > 3, write n as (2^s)*d+1, # where s and d are positive integers and d is odd. assert n > 3 if n % 2 == 0: returnFalse s, d = 0, n - 1 while d % 2 == 0: d >>= 1 s += 1
for _ inrange(k): a = randrange(2, n - 1) x = pow(a, d, n) if x == 1or x == n - 1: continue for _ inrange(s): x = pow(x, 2, n) if x == n - 1: break else: # The for loop did not encounter a break statement, # so it fails the test, it must be a composite returnFalse
# Passed the test, it is a strong probable prime returnTrue
\n
Putting all of the above together, the whole process can be wrapped into the following function, where the input of the function is the number of bits and the output is a presumed random large prime number.
\n
defget_random_prime(num_bits): whileTrue: pp = get_lowlevel_prime(num_bits) if miller_rabin_primality_check(pp): return pp
\n
Utility Functions
\n\n
Greatest Common Divisor (GCD) gcd(a,b) and Least Common Multiple lcm(a,b): \nThe RSA encryption algorithm needs to calculate the Carmichael function \\(\\lambda(N)\\) of modulus \\(N\\), with the formula \\(\\lambda(pq)= \\operatorname{lcm}(p - 1, q - 1)\\), where the least common multiple function is used. The relationship between the least common multiple and the greatest common divisor is: \\[\\operatorname{lcm}(a,b)={\\frac{(a\\cdot b)}{\\gcd(a,b)}}\\] There is an efficient Euclidean algorithm for finding the greatest common divisor, which is based on the principle that the greatest common divisor of two integers is equal to the greatest common divisor of the smaller number and the remainder of the division of the two numbers. The specific implementation of Euclid's algorithm can be done iteratively or recursively. The iterative implementation of the maximum convention function is applied here, and the Python code for the two functions is as follows:
\n
defgcd(a, b): '''Computes the Great Common Divisor using the Euclid's algorithm''' while b: a, b = b, a % b return a
deflcm(a, b): """Computes the Lowest Common Multiple using the GCD method.""" return a // gcd(a, b) * b
\n
Extended Euclidean Algorithm exgcd(a,b) and Modular Multiplicative Inverse invmod(e,m): \nThe RSA key pair satisfies the equation \\((d⋅e)\\bmod \\lambda(N)=1\\), i.e., the two are mutually modular multiplicative inverses with respect to the modulus \\(\\lambda(N)\\). The extended Euclidean algorithm can be applied to solve the modular multiplicative inverse \\(d\\) of the public key exponent \\(e\\) quickly. The principle of the algorithm is that given integers \\(a,b\\), it is possible to find integers \\(x,y\\) (one of which is likely to be negative) while finding the greatest common divisor of \\(a,b\\) such that they satisfy Bézout's identity: \\[a⋅x+b⋅y=\\gcd(a, b)\\] substituted into the parameters \\(a=e\\) and \\(b=m=\\lambda(N)\\) of the RSA encryption algorithm, and since \\(e\\) and \\(\\lambda(N)\\) are coprime, we can get: \\[e⋅x+m⋅y=1\\] the solved \\(x\\) is the modulo multiplicative inverse \\(d\\) of \\(e\\). The Python implementations of these two functions are given below:
\n
defexgcd(a, b): """Extended Euclidean Algorithm that can give back all gcd, s, t such that they can make Bézout's identity: gcd(a,b) = a*s + b*t Return: (gcd, s, t) as tuple""" old_s, s = 1, 0 old_t, t = 0, 1 while b: q = a // b s, old_s = old_s - q * s, s t, old_t = old_t - q * t, t a, b = b, a % b return a, old_s, old_t
definvmod(e, m): """Find out the modular multiplicative inverse x of the input integer e with respect to the modulus m. Return the minimum positive x""" g, x, y = exgcd(e, m) assert g == 1
# Now we have e*x + m*y = g = 1, so e*x ≡ 1 (mod m). # The modular multiplicative inverse of e is x. if x < 0: x += m return x
Similarly, an iterative approach is applied here to implement the extended Euclidean algorithm, with the modular inverse multiplicative function calling the former.
\n\n
Implementing RSA Class
\n
Note: Textbook RSA has inherent security vulnerabilities. The reference implementation in the Python language given here is for learning and demonstration purposes only, by no means to be used in actual applications. Otherwise, it may cause serious information security incidents. Keep this in mind!
\n
\n
Based on the object-oriented programming idea, it can be designed to encapsulate the RSA keys and all corresponding operations into a Python class. The decryption and signature generation of the RSA class are each implemented in two ways, regular and fast. The fast method is based on the Chinese Remainder Theorem and Fermat's Little Theorem. The following describes the implementation details of the RSA class.
\n\n
Object Initialization Method \nInitialization method __init__() has the user-defined paramaters with default values shown as below：
\n
\n
Key bit-length (\\(N\\))：2048
\n
Public exponent (\\(e\\))：65537
\n
Fast decryption or signature generation：False
\n
\n
This method internally calls the get_random_prime() function to generate two large random prime numbers \\(p\\) and \\(q\\) that are about half the bit-length of the key. It then calculates their Carmichael function and verifies that the result and \\(e\\) are coprime. If not, it repeats the process till found. Thereafter it computes the modulus \\(N\\) and uses the modular multiplicative inverse function invmod() to determine the private exponent \\(d\\). If a fast decryption or signature generation function is required, three additional values are computed as follows: \\[\\begin{align}\nd_P&=d\\bmod (p-1)\\\\\nd_Q&=d\\bmod (q-1)\\\\\nq_{\\text{inv}}&=q^{-1}\\pmod {p}\n\\end{align}\\]
Encryption and Decryption Methods \nRSA encryption and regular decryption formulas are \\[\\begin{align}\nc\\equiv m^e\\pmod N\\\\\nm\\equiv c^d\\pmod N\n\\end{align}\\] Python built-in pow() function supports modular exponentiation. The above two can be achieved by simply doing the corresponding integer to byte sequence conversions and then calling pow() with the public or private key exponent:
For fast descryption, a few extra steps are needed: \\[\\begin{align}\nm_1&=c^{d_P}\\pmod {p}\\tag{1}\\label{eq1}\\\\\nm_2&=c^{d_Q}\\pmod {q}\\tag{2}\\label{eq2}\\\\\nh&=q_{\\text{inv}}(m_1-m_2)\\pmod {p}\\tag{3}\\label{eq3}\\\\\nm&=m_{2}+hq\\pmod {pq}\\tag{4}\\label{eq4}\n\\end{align}\\] In practice, if \\(m_1-m_2<0\\) in the step \\((3)\\), \\(p\\) needs to be added to adjust to a positive number. It can also be seen that the acceleration ratio would theoretically be close to \\(4\\) because the fast decryption method decreases the modulus and exponent by roughly half the order. Considering the additional computational steps, the actual speedup ratio estimate is subtracted by a correction \\(\\varepsilon\\), noted as \\(4-\\varepsilon\\). The code of the fast decryption function is as follows:
\n
defdecrypt_fast(self, encrypted_int_data: int): # Use Chinese Remaider Theorem + Fermat's Little Theorem to # do fast RSA description assert self.fast == True m1 = pow(encrypted_int_data, self.d_P, self.p) m2 = pow(encrypted_int_data, self.d_Q, self.q) t = m1 - m2 if t < 0: t += self.p h = (self.q_Inv * t) % self.p m = (m2 + h * self.q) % self.n return uint_to_bytes(m)
\n
Signature Generation and Verification Methods \nThe RSA digital signature generation and verification methods are very similar to encryption and regular decryption functions, except that the public and private exponents are used in reverse. The signature generation uses the private exponent, while the verification method uses the public key exponent. The implementation of fast signature generation is the same as the fast decryption steps, but the input and output data are converted and adjusted accordingly. The specific implementations are presented below：
\n
defgenerate_signature(self, encoded_msg_digest: bytes): """Use RSA private key to generate Digital Signature for given encoded message digest""" int_data = uint_from_bytes(encoded_msg_digest) returnpow(int_data, self.d, self.n) \t defgenerate_signature_fast(self, encoded_msg_digest: bytes): # Use Chinese Remaider Theorem + Fermat's Little Theorem to # do fast RSA signature generation assert self.fast == True int_data = uint_from_bytes(encoded_msg_digest) s1 = pow(int_data, self.d_P, self.p) s2 = pow(int_data, self.d_Q, self.q) t = s1 - s2 if t < 0: t += self.p h = (self.q_Inv * t) % self.p s = (s2 + h * self.q) % self.n return s
defverify_signature(self, digital_signature: int): """Use RSA public key to decrypt given Digital Signature""" int_data = pow(digital_signature, self.e, self.n) return uint_to_bytes(int_data)
\n\n
Functional Tests
\n
Once the RSA class is completed, it is ready for testing. To test the basic encryption and decryption functions, first initialize an RSA object with the following parameters
\n
\n
Key length (modulo \\(N\\)): 512 bits
\n
Public exponent (\\(e\\)): 3
\n
Fast decryption or signature generation: True
\n
\n
Next, we can call the encryption method encrypt() of the RSA object instance to encrypt the input message, and then feed the ciphertext to the decryption method decrypt() and the fast decryption method decrypt_fast() respectively. We use the assert statement to compare the result with the original message. The code snippet is as follows.
\n
# ---- Test RSA class ---- alice = RSA(512, 3, True) msg = b'Textbook RSA in Python' ctxt = alice.encrypt(msg) assert alice.decrypt(ctxt) == msg assert alice.decrypt_fast(ctxt) == msg print("RSA message encryption/decryption test passes!")
\n
Likewise, we can also test the signature methods. In this case, we need to add the following import statement to the beginning of the file
\n
from hashlib import sha1
\n
This allows us to generate the message digest with the library function sha1() and then call the generate_signature() and generate_signature_fast() methods of the RSA object instance to generate the signature, respectively. Both signatures are fed to the verify_signature()` function and the result should be consistent with the original message digest. This test code is shown below.
If no AssertionError is seen, we would get the following output, indicating that both the encryption and signature tests passed.
\n
RSA message encryption/decryption test passes! RSA signature generation/verification test passes!
\n
Performance Tests
\n
Once the functional tests are passed, it is time to see how the performance of fast decryption is. We are interested in what speedup ratio we can achieve, which requires timing the execution of the code. For time measurements in Python programming, we have to import the functions urandom() and timeit() from the Python built-in libraries os and timeit, respectively:
\n
from os import urandom from timeit import timeit
\n
urandom() is for generaring random bype sequence, while timeit() can time the execution of a given code segment. For the sake of convenience, the RSA decryption methods to be timed are first packed into two functions:
\n
\n
decrypt_norm() - Regular decryption method
\n
decrypt_fast() - Fast descryption method
\n
\n
Both use the assert statement to check the result, as shown in the code below:
The outer loop iterates over different key lengths klen, from 512 bits to 4096 bits in 5 levels, and the corresponding RSA object obj is initialized with:
\n
\n
Key length (modular \\(N\\)): klen
\n
Public exponent (\\(e\\)): 65537
\n
Fast decryption or signature generation: True
\n
\n
The variable rpt is also set in the outer loop to be the square root of the key length, and the timing variables t_n and t_f are cleared to zeros.
\n
The inner layer also loops 5 times, each time executing the following operations:
\n
\n
Call urandom() to generate a random sequence of bytes mg with bits half the length of the key
\n
Call obj.encrypt() to generate the ciphertext ct
\n
call timeit() and enter the packing functions decrypt_norm() and decrypt_fast() with the decryption-related parameters obj, ct and mg, respectively, and set the number of executions to rpt
\n
The return values of the timeit() function are stored cumulatively in t_n and t_f
\n
\n
\n
At the end of each inner loop, the current key length, the mean value of the timing statistics, and the calculated speedup ratio t_n/t_f are printed. The actual program segment is printed below:
Start RSA fast decryption profiling... Key size 512 => norm 0.0008s, fast 0.0003s Speedup: 2.43 Key size 1024 => norm 0.0043s, fast 0.0015s Speedup: 2.88 Key size 2048 => norm 0.0273s, fast 0.0085s Speedup: 3.19 Key size 3072 => norm 0.0835s, fast 0.0240s Speedup: 3.48 Key size 4096 => norm 0.1919s, fast 0.0543s Speedup: 3.53
\n
The test results confirm the effectiveness of the fast decryption method. As the key length increases, the computational intensity gradually increases and the running timeshare of the core decryption operation becomes more prominent, so the speedup ratio grows correspondingly. However, the final speedup ratio tends to a stable value of about 3.5, which is consistent with the upper bound of the theoretical estimate (\\(4-\\varepsilon\\)).
\n
The Python code implementation of the textbook RSA helps reinforce the basic number theory knowledge we have learned and also benefits us with an in-depth understanding of the RSA encryption algorithm. On this basis, we can also extend to experiment some RSA elementary attack and defense techniques to further master this key technology of public-key cryptography. For the complete program click here to download: textbook-rsa.py.gz
\n\n\n\n
Gary Lee Miller, a professor of computer science at Carnegie Mellon University, first proposed a deterministic algorithm based on the unproven generalized Riemann hypothesis. Later Professor Michael O. Rabin of the Hebrew University of Jerusalem, Israel, modified it to obtain an unconditional probabilistic algorithm.↩︎
\n
This is because it follows from \\(x^2\\equiv 1\\pmod n\\) that \\((x-1)(x+1)=x^{2}-1\\equiv 0\\pmod n\\). Since \\(n\\) is a prime number, by Euclid's Lemma, it must divide either \\(x- 1\\) or \\(x+1\\), so \\(x\\bmod n\\) must be \\(1\\) or \\(-1\\).↩︎
\n\n\n","categories":["Technical Know-how"],"tags":["Cryptography","Python Programming"]},{"title":"Build an Awesome Raspberry Pi NAS for Home Media Streaming","url":"/en/2021/12/29/RPi-NAS-Plex/","content":"
Network Attached Storage (NAS) provides data access to a heterogeneous group of clients over computer networks. As hard drive prices continue to drop, NAS devices have made their way into the homes of the masses. Leading brands in the SMB and home NAS market, such as Synology, have their products range in price from as low as $300 to $700 for the high models. But if you are a Raspberry Pi player, you can build a very nice home NAS and streaming service for only about half the cost of the lowest price.
\n
Knowledge obtained on the papers always feels shallow, must know this thing to practice. — LU You (Chinese historian and poet of the Southern Song Dynasty)
\n
\n
This blog records the whole process of building a Raspberry Pi NAS and home media server, including project planning, system implementation, and performance review. It also covers some important experiences and lessons that could hopefully benefit anyone interested in this DIY project.
\n
Project Planning
\n
Raspberry Pi 4B features an upgraded 1.8GHz Broadcom BCM2711（quad-core Cortex-A72）processor and onboard RAM up to 8GB. It includes two new USB 3.0 ports and a full-speed Gigabit Ethernet interface. The power supply is also updated to a USB-C connector. All these greatly improve system throughput and overall comprehensive performance, and we can use them to create a full-featured home NAS.
\n
For NAS system software, OpenMediaVault (OMV) is a complete NAS solution based on Debian Linux. It is a Linux rewrite of the well-known free and open-source NAS server system FreeNAS (based on FreeBSD). The salient features of OMV are
\n
\n
Simple and easy-to-use out-of-the-box solution, no need for expert-level knowledge of computer networking and storage systems
\n
Available for x86-64 and ARM platforms with a full Web Administration interface
\n
Supports a variety of different protocols (SFTP、SMB/CIFS, NFS, etc.) for file storage access
\n
Can be controlled via SSH (if enabled), and provides Access Right Management for users and groups
\n
\n
While primarily designed for home environments or small home offices, OMV's use is not limited to those scenarios. The system is built on a modular design. It can be easily extended with available plugins right after the installation of the base system. OMV is the NAS server system software we are looking for.
\n
The NAS system with media playback services provides an excellent audio/video-on-demand experience in a home network environment. Plex Media Server software integrates Internet media services (YouTube, Vimeo, TED, etc.) and local multimedia libraries to provide streaming media playback on users' various devices. The features of Plex for managing local libraries are
\n
\n
Centralized management and easy sharing of a single library
\n
Web interface with media resource navigation, streaming playback
\n
Real-time saving and resuming of playback progress
\n
Multi-user support and hierarchical playback rights settings
\n
\n
The Plex Media Server software itself is free and supports a wide range of operating systems, making it ideal for integration with home NAS.
\n
These cover all the software needed for our NAS project, but they are not enough for a complete NAS system. We also need a preferred case, otherwise, the Raspberry Pi NAS will only run bare metal. Although there are many cases available in the market for Raspberry Pi 4B, as a NAS system we need a case kit that can accommodate at least 1-2 internal SSD/HDD and must also have a good heat dissipation design.
\n
After some review and comparison, we chose Geekworm's NASPi Raspberry Pi 4B NAS storage kit. NASPi is a NUC (Next Unit of Computing) style NAS storage kit designed for the latest Raspberry Pi 4B. It consists of three components:
\n\n
X823 shield board, which provides storage function for 2.5-inch SDD/HDD
\n
X-C1 adapter board, which adjusts all Raspberry Pi 4B interfaces to the back of the case and provides power management and safe shutdown function
\n
Temperature-controlled PWM (Pulse-Width Modulation) fan as the cooling system
\n\n
All these components are packed into a case made of aluminum alloy with an anodized surface.
\n
Thereon our NAS project can be planned with the following subsystems:
\n
\n
Hardware System:\n
\n
Raspberry Pi 4B 8GB RAM
\n
32GB microSD for OS storage
\n
NASPi NAS storage kit
\n
15-20W USB-C power adaptor
\n
500GB internal SSD（USB 3.0）
\n
2TB external HDD（USB 3.0）
\n
\n
Software System:\n
\n
Raspberry Pi OS Lite（with no desktop environment）
\n
OMV for NAS file server
\n
Plex media server providing streaming service
\n
\n
\n
It is important to note that NAS servers are generally headless systems without a keyboard, mouse, or monitor. This poses some challenges for the installation, configuration, and tuning of hardware and software systems. In practice, as described in the next section, we run an SSH terminal connection to complete the basic project implementation process.
\n
System Implementation
\n
The execution of this project was divided into four stages, which are described in detail as follows.
\n
Prepare Raspberry Pi 4B
\n
In the first stage, we need to prepare the Raspberry Pi OS and do some basic unit tests. This is important, if we delay the OS test until the entire NSAPi kit is assembled, it will be troublesome to find problems with the Raspberry Pi then.
\n
Bake Raspberry Pi OS
\n
First, insert the microSD card into the USB adapter and connect it to the macOS computer, then go to the Raspberry Pi website and download the Raspberry Pi Imager software to run. From the application screen, click CHOOSE OS > Raspberry Pi OS (other) > Raspberry Pi OS Lite (32-bit) step by step. This selects the lightweight Raspberry Pi OS that does not require a desktop environment, and then click CHOOSE STORAGE to pick the microSD card.
\n
Next is a trick - hit the ctrl-shift-x key combination and the following advanced options dialog box will pop up Here is exactly the option we need to enable SSH on boot up - Enable SSH. It also allows the user to pre-set a password for the default username pi (default is raspberry). Once set up, click SAVE to return to the main page and then click WRITE to start formatting the microSD card and writing OS to it. When finished, remove the microSD card and insert it into the Raspberry Pi, connect the Ethernet cable then power it up.
\n
Probe IP Address
\n
At this point we encountered a problem: since the installed system does not have a desktop environment, it cannot connect to the keyboard, mouse, and monitor, so how do we find its IP address? There are two ways:
\n\n
connect to the home router's management WebUI and find the address for the hostname 'raspberry'.
\n
run the Nmap tool to scan the target subnet and check the changes before and after the Raspberry Pi boots up
\n\n
The log of the Nmap tool run can be seen below. Notice that a new IP address 192.168.2.4 is showing up in the scan report. Rerunning Nmap against this address alone, we saw that TCP port 22 was open. We could roughly determine that this might be our newly online Raspberry Pi:
\n
❯ nmap -sn 192.168.2.0/24 Starting Nmap 7.92 ( https://nmap.org ) at 2021-11-28 21:07 PST Nmap scan report for router.sx.com (192.168.2.1) Host is up (0.0050s latency). Nmap scan report for 192.168.2.3 Host is up (0.0048s latency). Nmap scan report for 192.168.2.4 ## New IP after Raspberry Pi boots up Host is up (0.0057s latency). Nmap done: 256 IP addresses (3 hosts up) scanned in 15.31 seconds
❯ nmap 192.168.2.4 Nmap scan report for 192.168.2.4 Host is up (0.0066s latency). Not shown: 999 closed tcp ports (conn-refused) PORT STATE SERVICE 22/tcp open ssh
\n
System Update and Upgrade
\n
Next, try SSH connection
\n
❯ ssh pi@192.168.2.4 pi@192.168.2.4's password: Linux raspberrypi 5.10.63-v7l+ #1488 SMP Thu Nov 18 16:15:28 GMT 2021 armv7l The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Fri Dec 24 19:46:15 2021 from 192.168.2.3 pi@raspberrypi:~ $
\n
Once confirmed, we executed the following commands in the Raspberry Pi to update and upgrade the system:
This stage concluded with the stability test of the Raspberry Pi 4B system Ethernet connection. The test was executed on a macOS computer using the simple ping command, setting the -i 0.05 option to specify 20 packets per second and the -t 3600 option for one hour run
\n
❯ sudo ping -i 0.05 192.168.2.4 -t 3600
\n
There should be no more than 1% packet loss or timeout on a subnet with no wireless connectivity, otherwise, it should be checked for troubleshooting. As a matter of fact, in our test, it was happening that nearly 10% of ping packets got lost and the SSH connection dropped intermittently. Searching the Internet, we found that there have been quite a few reports of similar issues with the Raspberry Pi 4B Ethernet connection. The analysis and suggestions given by people on the relevant forums focus on the following
\n\n
Unstable power supply accounts for packet loss, and needs to be replaced with a reliable USB-C power adapter of 15W or more.
\n
Energy-efficient Ethernet (Energy-Efficient Ethernet) malfunction, can be fixed by disabling it.
\n
The full-speed Gigabit Ethernet connection function is faulty and has to be downgraded to 100Mbit/s for stable use.
\n\n
Practically, we tried all of the above with little success. Later, we found that the home router connected to the Raspberry Pi 4B was a Belkin N750 DB made in 2011. Although it provides Wi-Fi dual-band 802.11n and 4 Gigabit Ethernet ports, the manufacturing date is too long ago, which makes people doubt its interoperability. Also points 2 and 3 of the above report are essentially interoperability issues. Thinking of these, we immediately ordered the TP-Link TL-SG105 5-port Gigabit Ethernet switch. After receiving it, we extended the Gigabit Ethernet port of N750 with TL-SG105, connected Raspberry Pi 4B to TL-SG105, and retested it. Sure enough, this time the ping packet loss rate was less than 0.1% and the SSH connection became solid.
\n
The conclusion is that the Raspberry Pi 4B Gigabit Ethernet interface may have compatibility issues with some older devices, which can be solved by inserting a desktop switch with good interoperability between the two.
\n
NSAPi Kit Assembly
\n
In the second stage, we assembled the NSAPi storage kit, intending to finish all hardware installation and complete the standalone NAS body.
\n
Prepare Internal SSD
\n
The NSAPi supports either an internal SSD or HDD. The project picked a Samsung 870 EVO 500GB internal SSD, here we ought to first make sure the SSD works properly on its own, otherwise, we would have to disassemble the NASPi to replace it. The SSD can be hooked up to Windows for file systems and basic read/write operation checks. In the case of a newly purchased SSD, the following steps can be done on Windows to quickly format it:
\n\n
Click on Start or the Windows button, select Control Panel > System and Security
\n
Select Administrative Tools > Computer Management > Disk management
\n
Choose the disk to be formatted, right-click then select Format
\n
Check the following in the Dialog box pop-up\n
\n
File System → NTFS
\n
Allocation Unit Size → Default
\n
Volume Label → (enter volume name)
\n
Perform a quick format
\n
\n
Click the OK button to start a fast format for the SSD
\n\n
⚠️Note: Here the chosen file system is NTFS. OMV supports NTFS mounting and reads/writes.
\n
PWM Fan Control
\n
Before the actual hardware assembly, a special software provided by Geekworm - PWM fan control script - must be installed. PWM fan speed adjustment to temperature change is a major feature that lets NASPi stand out from other hardware solutions. So this step is critical.
\n
Referring to Geekworm's X-C1 software wiki page, the installation command sequence on the SSH session connected to the Raspberry Pi 4B system is as follows
If you can't do git clone directly on Raspberry Pi 4B, you can first download the X-C1 software on the SSH client, then transfer it to Raspberry Pi 4B using scp. After that, continue to execute the subsequent commands
\n
❯ scp -r x-c1 pi@192.168.2.4:/home/pi/
\n
How does X-C1 software control PWM fan?
\n\n
The core of X-C1 software is a Python script named fan.py, which is presented below
Its logic is quite simple. With the pigpio module imported, it first initializes a PWM control object and then starts a while loop with a 1-second sleep cycle inside. The CPU temperature is read at each cycle, and the duty cycle of PWM is set according to the temperature level to control the fan speed. The duty cycle is 0 when it is lower than 30℃, and the fan stops; when it is higher than 75℃, the duty cycle is 100, and the fan spins at full speed. Users can modify the temperature threshold and duty cycle parameters in the program to customize the PWM fan control.
\n\n\n
In addition, the following pi-temp.sh script, which reads out the GPU and CPU temperatures, is also useful
\n
pi@raspberrypi:~ $ cat ./pi-temp.sh #!/bin/bash # Script: pi-temp.sh # Purpose: Display the ARM CPU and GPU temperature of Raspberry Pi # ------------------------------------------------------- cpu=$(</sys/class/thermal/thermal_zone0/temp) echo"$(date) @ $(hostname)" echo"-------------------------------------------" echo"GPU => $(vcgencmd measure_temp)" echo"CPU => temp=$((cpu/1000))’C"
pi@raspberrypi:~ $ ./pi-temp.sh Mon 29 Nov 06:59:17 GMT 2021 @ raspberrypi ------------------------------------------- GPU => temp=33.1'C CPU => temp=32’C
\n
Hardware Assembly Process
\n
Below is a snapshot of the Geekworm NASPi parts out of the box (except for the Raspberry Pi 4B on the far right of the second row and the screwdriver in the lower right corner)
\n
The three key components in the second row, from left to right, are
\n
\n
X-C1 V1.3 adapter board provides power management, interface adaptation, and security shutdown functions
\n
X823 V1.5 shield board provides a 2.5-inch SSD/HDD storage function (UASP supported)
\n
4010 PWM fan and metal fan bracket
\n
\n
The assembly process was done step-by-step mainly by referring to NASPi installation video on Youtube, and the steps are generalized as follows.
\n\n
Insert the SSD into the SATA III connector of X823, flip it to the other side, and fix it with screws.
\n
Install the Raspberry Pi 4B after fixing the spacers on this side, and place the 7-pin cable between the two
\n
Install the PWM fan on top of the Raspberry Pi 4B with the additional spacers
\n
Connect X-C1 and Raspberry Pi 4B, insert a 7-pin connector right to the X-C1 GPIO port and a 3-pin connector to the X-C1 FAN port
\n
Align and insert the 2x7-pin daughterboard to the GPIO port of the Raspberry Pi 4B and fix it with screws
\n
Plug in the USB 3.0 connector to connect the X823 USB 3.0 port to the corresponding Raspberry Pi 4B USB 3.0
\n\n
Now the installation of the internal accessories has been completed, we have a view of this
\n
\n
At this point, we added the USB-C power and pressed the front button to start the system, we could see the PWM fan started to spin. It was also observed that the fan spin rate was not constant, which demonstrated that the temperature controller PWM fan was working properly.
\n
The front button switch with embedded blue LED decides the whole system's on/off state and can be tested below
\n
\n
Press the switch after power-on, and the system starts
\n
Press and hold the switch for 1-2 seconds while running, then the system restarts
\n
Press and hold the switch for 3 seconds during operation to shut down the system safely.
\n
Press and hold the switch for 7-8 seconds during operation to force shutdown
\n
\n
Running the off command on the SSH connection can also trigger a safe shutdown. Be cautious that we should not use the Linux shutdown command, as that would not power down the X-C1 board.
\n
After the button switch test, we now unplugged the USB 3.0 connector and inserted the entire module into the case. Next was to add the back panel and tighten the screws, then re-insert the USB 3.0 connector. This completed the whole NASPi storage kit assembly process. Below are the front and rear views of the final system provided by Geekworm (all interfaces and vents are marked).
\n
\n
OMV Installation and Configuration
\n
The third stage is for installing and configuring the key software package of the NAS system - PMV. The goal is to bring up the basic network file access service. Before restarting the NAS, we plugged a Seagate 2TB external HDD into the remaining USB 3.0 port. After booting, connected SSH to NASPi from macOS and performed the following process.
\n
Install OMV Package
\n
Installing OMV is as simple as running the following command line directly from a terminal with an SSH connection.
Due to the large size of the entire OMV package, this installation process can take a long time. After the installation, the IP address of the system may change and you will need to reconnect to SSH at this time.
\n
(Reading database ... 51781 files and directories currently installed.) Purging configuration files for dhcpcd5 (1:8.1.2-1+rpt3) ... Purging configuration files for raspberrypi-net-mods (1.3.2) ... Enable and start systemd-resolved ... Unblocking wifi with rfkill ... Adding eth0 to openmedivault database ... IP address may change and you could lose connection if running this script via ssh. client_loop: send disconnect: Broken pipe\t
\n
After reconnecting, you can use dpkg to view the OMV packages. As you can see, the latest version of OMV installed here is 6.0.5.
\n
pi@raspberrypi:~ $ dpkg -l | grep openme ii openmediavault 6.0.5-1 all openmediavault - The open network attached storage solution ii openmediavault-flashmemory 6.0.2 all folder2ram plugin for openmediavault ii openmediavault-keyring 1.0 all GnuPG archive keys of the OpenMediaVault archive ii openmediavault-omvextrasorg 6.0.4 all OMV-Extras.org Package Repositories for OpenMediaVault
\n
OMV Management UI
\n
At this point OMV's workbench is live. Launching a browser on a macOS computer and typing in the IP address will open the beautiful login screen (click on the 🌍 icon in the upper right corner to select the user interface language): After logging in with the default username and password shown above, you will see the Workbench screen. The first thing you should do at this point is to click the ⚙️ icon in the top right corner to bring up the settings menu and click \"Change Password\". You can also change the language here Clicking on \"Dashboard\" in the settings menu allows you to select the relevant components to be enabled. The menu on the left side provides task navigation for administrators and can be hidden when not needed. The complete OMV administration manual can be found in the online documentation
\n
Configure File Services
\n
Next is the key process for configuring the NAS, which consists of the following 5 steps.
\n\n
Scan for mounted disk drives
\n
Click Storage > Disks from the sidebar menu to enter the hard drive management page. If there is an external USB storage device just plugged in, you can click 🔍 here to scan it out. The scan results for this system are as follows. The internal Samsung 500GB SSD and external Seagate 2TB HDD are detected, and the 32GB microSD that contains the entire software system is listed at the top:
\n
On the SSH terminal, we could see the information for the same set of mounted drivers
Click Storage > File Systems from the sidebar menu to enter the file system management page. If the storage device does not have a file system yet, click ⨁ to Create or Mount the file system. OMV can create/mount ext4, ext3, JFS, and xfs file systems, but only mounts are supported for the NTFS file system. The following figure shows that OMV correctly mounts NTFS file systems for SSDs and HDDs:
\n
Set Shared Folders
\n
From the sidebar menu, click Storage > File Systems to access the shared folder management page. Here, click ⨁ to create a shared folder. When creating it, specify the name, corresponding file system, and relative path, and you can also add comments. Select the created folder and click the pencil icon again to edit the related information. This system sets the relative paths of shared folders Zixi-Primary and Zixi-Secondary for SSD and HDD respectively Notice the orange alert at the top of the figure above, which alerts the administrator that the configurations have changed and must click on the ✔️ icon to take effect.
\n
Add shared folder access users
\n
Click User Management > Users from the sidebar menu to enter the user management page. The system's default user pi has root privileges and cannot be used for file-sharing access due to security concerns. So you need to add a new user separately. On this page, click ⨁ to Create or Import user, only user name and password are required when creating a new user, others are optional. Once created, select this user and click the third folder+key icon (prompting \"Shared folder privileges\") to enter the following privileges settings page As shown in the figure, for this new user zixi, the administrator can set the read and write access permissions for each shared folder.
\n
Start file share services
\n
If you expand the \"Services\" item in the navigation menu, you can see that OMV manages five services: FTP, NFS, Rsync, SMB/CIFS, and SSH. SSH is enabled at the beginning of the system OS image preparation. NFS and SMB/CIFS are the most common network file-sharing protocols, and both are supported by macOS. Take SMB/CIFS as an example here. Click Services > SMB/CIFS from the sidebar menu to enter the management page. The page contains two buttons: Settings and Shares. Click \"Settings\" first to activate the SMB/CIFS service and configure the workgroup name on the new page, other options can be left as default. After saving, it returns to the SMB/CIFS administration page. Then enter \"Shares\", click ⨁ to Create shared folders Zixi-Primary and Zixi-Secondary on the new page then save. After that, click the ✔️ icon in the orange warning bar to make all configuration updates take effect, and you will end up with the following result
\n\n
Now our Raspberry Pi NAS system is ready for file sharing and the SMB/CIFS service is started. After checking the relevant components to turn on, our dashboard live monitoring looks like this
\n
Set Up Client Device
\n
Once the server side is ready, we need to add the network share folder on the client side as follows.
\n
\n
Windows PC client\n
\n
Open File Explore, click “This PC”
\n
Right-click on the blank area at the right pane, select \"Add a network location” on the popup menu
\n
Enter “\\\\<IP-address>\\” in the “Internet or network address\" input box
\n
Enter username and password when prompted
\n
\n
MacBook client (screenshot below)\n
\n
Open Finder, click the menu item Go
\n
Click “Connect to Server...”
\n
Enter URL “smb://<IP-address>/”, then click Connect
\n
Enter username and password when prompted \n
\n
\n
\n
Once the client side is set up, users can perform various operations on the network share folder as if it were a local directory, such as previewing, creating new, opening or copying files, creating new subdirectories, or deleting existing subdirectories.
\n
Plex Installation and Configuration
\n
The last stage is to install and configure the Plex Media Server, and then start a network streaming service.
\n
Install Media Server
\n
The process of installing Plex Media Server requires HTTPS transport support, so we must first install the https-transport package. SSH to our Raspberry Pi NAS and execute the install command
\n
sudo apt-get install apt-transport-https
\n
Next add the Plex repository to the system, which requires downloading the Plex sign key first. Here are the related commands and run logs
\n
pi@raspberrypi:~ $ curl https://downloads.plex.tv/plex-keys/PlexSign.key | sudo apt-key add - % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 100 3072 100 3072 0 0 10039 0 --:--:-- --:--:-- --:--:-- 10039 OK
\n
Use the same apt-key command to check the newly added Plex sign key
\n
pi@raspberrypi:~ $ apt-key list Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). /etc/apt/trusted.gpg ... pub rsa4096 2015-03-22 [SC] CD66 5CBA 0E2F 88B7 373F 7CB9 9720 3C7B 3ADC A79D uid [ unknown] Plex Inc. sub rsa4096 2015-03-22 [E] ...
\n
You can see that Plex uses 4096-bit RSA keys. For the warning message \"apt-key is deprecated...\" in the above log, you can ignore it for now. Go to read some discussion on the askubuntu forum if you are interested.
\n
The next step is to add the Plex repository to the system repository list, and then update the packages
echo deb https://downloads.plex.tv/repo/deb public main | sudo tee /etc/apt/sources.list.d/plexmediaserver.list sudo apt-get update
Now we can start the actual Plex Media Server installation with the following installation commands
\n
pi@raspberrypi:~ $ sudo apt install plexmediaserver Reading package lists... Done Building dependency tree... Done Reading state information... Done The following NEW packages will be installed: plexmediaserver 0 upgraded, 1 newly installed, 0 to remove and 20 not upgraded. Need to get 66.1 MB of archives. After this operation, 146 MB of additional disk space will be used. Get:1 https://downloads.plex.tv/repo/deb public/main armhf plexmediaserver armhf 1.25.0.5282-2edd3c44d [66.1 MB] Fetched 66.1 MB in 28s (2392 kB/s) Selecting previously unselected package plexmediaserver. (Reading database ... 51783 files and directories currently installed.) Preparing to unpack .../plexmediaserver_1.25.0.5282-2edd3c44d_armhf.deb ... PlexMediaServer install: Pre-installation Validation. PlexMediaServer install: Pre-installation Validation complete. Unpacking plexmediaserver (1.25.0.5282-2edd3c44d) ... Setting up plexmediaserver (1.25.0.5282-2edd3c44d) ...
Configuration file '/etc/apt/sources.list.d/plexmediaserver.list' ==> File on system created by you or by a script. ==> File also in package provided by package maintainer. What would you like to do about it ? Your options are: Y or I : install the package maintainer's version N or O : keep your currently-installed version D : show the differences between the versions Z : start a shell to examine the situation The default action is to keep your current version. *** plexmediaserver.list (Y/I/N/O/D/Z) [default=N] ? PlexMediaServer install: PlexMediaServer-1.25.0.5282-2edd3c44d - Installation starting. PlexMediaServer install: PlexMediaServer install: Now installing based on: PlexMediaServer install: Installation Type: New PlexMediaServer install: Process Control: systemd PlexMediaServer install: Plex User: plex PlexMediaServer install: Plex Group: plex PlexMediaServer install: Video Group: video PlexMediaServer install: Metadata Dir: /var/lib/plexmediaserver/Library/Application Support PlexMediaServer install: Temp Directory: /tmp PlexMediaServer install: Lang Encoding: en_US.UTF-8 PlexMediaServer install: Nvidia GPU card: Not Found PlexMediaServer install: PlexMediaServer install: Completing final configuration. Created symlink /etc/systemd/system/multi-user.target.wants/plexmediaserver.service → /lib/systemd/system/plexmediaserver.service. PlexMediaServer install: PlexMediaServer-1.25.0.5282-2edd3c44d - Installation successful. Errors: 0, Warnings: 0
\n
The log shows a question is asked about the Plex media server list (plexmediaserver.list), just choose the default N. When we see \"Installation successful\", we know that the installation was successful. At this point, the Plex streaming service is up and running. Invoking the Nmap scan again from the macOS side, we find that TCP port 32400 for Plex service is open.
\n
❯ nmap -p1-65535 192.168.2.4 | grep open 22/tcp open ssh 80/tcp open http 111/tcp open rpcbind 139/tcp open netbios-ssn 445/tcp open microsoft-ds 2049/tcp open nfs 5357/tcp open wsdapi 32400/tcp open plex
\n
Configure Media Server
\n
The configuration of the Plex Media Server has been done on the web GUI. Launch a browser on the macOS computer and type in the URL http://<IP-address>:32400/web, now we can see the following page if no surprise We can sign in with a Google, Facebook, or Apple account, or we can enter an email to create a new account. Follow the instructions on the page step by step, no need for any payment, soon we reach the Server Setup page. Here we can configure the server name and add libraries. Normally we don't need to access our home media server from outside, so remember to uncheck the \"Allow me to access my media outside my home\" box in this step. To add a library, first select the type of library (movies, TV episodes, music, photos, etc.), then click the \"BROWSE FOR MEDIA FOLDER\" button to browse and select the corresponding folder. Once the library is added, the included media files will immediately appear in the local service directory, as shown in the screenshot below Here we have a local server named ZIXI-RPI-NAS for our Raspberry Pi NAS, the movie directory in the library shows The Matrix trilogy and is playing the first one The Matrix. Move your mouse over the server name and ➕ icon will appear to the right, click on it to continue adding new media libraries.
\n
Once the Plex Media Server is configured, we can open a browser from any device on our home network to do streaming on-demand, without the need to download additional applications. The whole experience is just like our own proprietary home Netflix service. This is awesome!
\n
Performance Review
\n
By connecting a macOS laptop to one of the remaining ports of the TL-SG105, we could perform some simple same-subnet tests to evaluate the performance of this NAS system fully.
\n
System Stress Test
\n
Referring to Geekworm NASPi Stress Test Wiki page, we executed the following command over SSH connection, which cloned the test script from GitHub and ran the stress test:
\n
git clone https://github.com/geekworm-com/rpi-cpu-stress cd rpi-cpu-stress chmod +x stress.sh sudo ./stress.sh
\n
Simultaneously we established a second SSH session and ran htop to monitor system status. The screenshot below was taken while close to the 5-minute mark (left is the htop real-time display, and right is the stress test output) Dividing the temp value on the right side by 1000 gave the CPU temperature. All 4 CPU cores reached 100% full load during the test, while the maximum temperature did not exceed 70°C. At this moment, there was no obvious heat sensation when touching the case. Typing ctrl-c to stop the stress test, and then executing the temperature measurement script again
\n
pi@raspberrypi:~ $ ./pi-temp.sh Fri Dec 24 15:59:21 PST 2021 @ raspberrypi ------------------------------------------- GPU => temp=39.9'C CPU => temp=40'C
\n
The system temperature returned to a low range value. This test result assures the system meets the design goal.
\n
File Transfer Speed Test
\n
The file transfer speed can be roughly measured with the secure remote copy tool SCP. First, create a 1GB size file by running the mkfile command on the macOS client, then copy it to the user directory of the remote NAS system with the scp command
After the copy was done, it would print the time spent and the deduced speed. Running the command with the source and the destination reversed would give us the speed of receiving a file from the NAS system.
As can be seen, the speed of remote write is around 20MB/s, while the speed of remote file read can reach over 60MB/s. Considering that scp-related encryption and decryption are implemented in software on general-purpose Raspberry Pi systems, this result should be considered passable.
\n
Disk Access Speed Test
\n
The real test of the NAS's performance is the network drive read/write speed test. For this, we downloaded the AmorphousDiskMark app from Apple's App Store. This is an easy and efficient drive speed test that measures the read/write performance of a storage device in terms of MB/s and IOPS (input/output operations per second). It has four types of tests:
\n\n
sequential read/write, 1MB block, queue depth 8
\n
sequential read/write, 1MB block, queue depth 1
\n
random read/write, 4KB block, queue depth 64
\n
random read/write, 4KB block, queue depth 1
\n\n
The above queue depths are the default values, but other values are also available. In addition, users can also modify the test file size and duration.
\n
Run the application on the macOS client and select the remote SMB folders Zixi-Primary (Samsung SSD) and Zixi-Secondary (Seagate HDD) respectively at the top, then click the All button in the upper left corner to start the NAS drive speed test process. A side-by-side comparison of the two test results is shown below
\n
\n
This gives a few observations:
\n
\n
Reads are faster than writes for NAS drives, and the difference under random access is huge.
\n
SSD outperforms HDD for both sequential and random accesses.
\n
Large queue depth speeds up reads, especially for random accesses, but there is little impact on writes.
\n
For both SSDs and HDDs, sequential reads/writes are significantly more efficient than random reads/writes.
\n
For both SSDs and HDDs, sequential reads/writes reach their highest speeds at large queue depths.
\n
\n
These are not surprising and are consistent with the test results on macOS laptops with direct external SSDs and HDDs, only with the lower numbers. With this NAS system, both the SSD and HDD are connected via the USB 3.0 interface. USB 3.0 supports transfer speeds of up to 5Gbit/s, so the performance bottleneck of the system is the network interface bandwidth and processor power.
\n
That being said, for both SSDs and HDDs, the transfer speeds have been more than 900Mbit/s at 1MB sequential read and queue depth 8, close to the upper bandwidth limit of the Gigabit Ethernet interface. This read speed can support a single 1080p60 video stream at a frame rate of 60fps or 2 parallel 1080i50 video streams at a frame rate of 25fps, which is sufficient for home streaming services. In another media service test, the NAS system performs satisfactorily with three computers playing HD video on demand and one phone playing MP3 music without any lag.
\n
Project Summary
\n
This completes our Raspberry Pi home NAS project. Now we can move our NAS to a more permanent location to provide network file and streaming services for the whole family.
\n
\n
Economically, our home NAS has the cost summarized in the table below (excluding SSD/HDD)
\n
\n
\n
\n
\n
\n
\n\n
\n
Devices
\n
Functions
\n
Cost（$）
\n
\n\n\n
\n
Raspberry Pi 4B 2/4/8GB RAM
\n
Primary hardware system
\n
45/55/75
\n
\n
\n
Samsung 32GB EVO+ Class-10 Micro SDHC
\n
OS storage
\n
10
\n
\n
\n
Geekworm NASPi Raspberry Pi 4B NAS Storage Kit
\n
Case, extending board and PWM fan
\n
60
\n
\n
\n
Geekworm 20W 5V 4A USB-C Power Adaptor
\n
Power supply
\n
15
\n
\n
\n
TP-Link TL-SG105 5-Port Gigabit Ethernet Switch
\n
Desktop switch
\n
15
\n
\n\n
\n
Even with the choice of 8GB RAM Raspberry Pi 4B, the whole cost is only $175, a little more than half of the price of the low-end brand NAS sold in the market. Unless there are a lot of client devices that need streaming services, the memory consumption is usually under 2GB, so the 2GB Raspberry Pi 4B should be able to work in most home scenarios. That cuts the cost down to $145, less than half the MSRP.
\n
On the other hand, this DIY project was a very good exercise of hands-on practice, helping us gain valuable intuitive experience in building network connections, configuring system hardware and software, and tuning and testing application layer services. To sum up, the home NAS system built with Raspberry Pi 4B and OMV, combined with a Plex media server, provides a cost-effective solution for file backup and streaming media services in the home network.
\n
Appendix: List of related devices and Amazon links
\n
\n
Disclosure: This blog site is reader-supported. When you buy through the affiliate links below, as an Amazon Associate, I earn a tiny commission from qualifying purchases. Thank you.
\n","categories":["DIY Projects"],"tags":["Raspberry Pi","NAS"]},{"title":"RSA: Attack and Defense (1)","url":"/en/2023/03/16/RSA-attack-defense/","content":"
RSA is a public-key cryptosystem built on top of an asymmetric encryption algorithm, which was jointly invented by three cryptographers and computer scientists at the Massachusetts Institute of Technology in 1977. The RSA public-key encryption algorithm and cryptosystem provide data confidentiality and signature verification functions widely used on the Internet. Since its birth, RSA has become a major research object of modern cryptography. Many cryptanalysts and information security experts have been studying its possible theoretical flaws and technical loopholes to ensure security and reliability in practical applications.
\n
There are certain things whose number is unknown. If we count them by threes, we have two left over; by fives, we have three left over; and by sevens, two are left over. How many things are there? — Sunzi Suanjing, Volume 2.26
\n
\n
Fortunately, after more than 40 years of extensive research and practical application tests, although many sophisticated attack methods have been discovered, RSA is generally safe. These attack methods all take advantage of the improper use of RSA or the vulnerability of software and hardware implementations, and cannot shake the security foundation of its encryption algorithm. On the other hand, the research on these attack methods shows that implementing a safe and robust RSA application is not a simple task. A consensus in cryptography and network security hardware and software engineering practice is: never roll your own cryptography!1 The appropriate solution is to use an existing, well-tested, and reliably maintained library or API to implement the RSA algorithm and protocol application.
\n
Here is a brief survey of the common means of attacking RSA, the mathematical mechanism on which the attack is based, and the corresponding protective measures. Referring to the previous article, let’s start by reviewing the working mechanism and process of RSA:
\n\n
Choose two large prime numbers \\(p\\) and \\(q\\), compute \\(N=pq\\)
When both \\(p\\) and \\(q\\) are prime, \\(\\lambda(pq)=\\operatorname {lcm}(p − 1, q − 1)\\)
\n
\\(\\operatorname{lcm}\\) is a function to find the least common multiple, which can be calculated by the Euclidean algorithm
\n
\n
Choose a number \\(e\\) that is less than \\(\\lambda(N)\\) and also coprime with it, then calculate the modular multiplicative inverse of \\(e\\) modulo \\(\\lambda(N)\\). That is \\(d\\equiv e^{-1}\\pmod {\\lambda(N)}\\)\n
\n
Per the definition of modular multiplicative inverse, find \\(d\\) such that \\((d⋅e)\\bmod\\lambda(N)=1\\)
\\(\\pmb{(N,e)}\\) is the public key，\\(\\pmb{(N,d)}\\) is the private key\n
\n
The public key can be known by everyone, but the private key must be kept secret
\n
The records of \\(p,q,\\lambda(N)\\) can all be discarded
\n
\n
The sender first converts the message into a positive integer less than \\(N\\) according to the agreed encoding format, then uses the receiver's public key to compute the ciphertext with the formula \\(\\pmb{c\\equiv m^e\\pmod N}\\)
\n
After receiving the ciphertext, the receiver uses its private key to compute the plaintext \\(m\\) with the formula \\(\\pmb{m\\equiv c^d\\pmod N}\\), then decodes it into the original message
\n
A message encrypted with the private key can also be decrypted by the public key, i.e. if \\(\\pmb{s\\equiv m^d\\pmod N}\\), \\(\\pmb{m\\equiv s^e\\pmod N}\\). This is the supported digital signature feature
\n\n
Note that the second and third steps in the original RSA paper use Euler's totient function\\(\\varphi(N)\\) instead of \\(\\lambda(N)\\). The relationship between these two functions is: \\[\\varphi(N)=\\lambda(N)⋅\\operatorname{gcd}(p-1,q-1)\\] Here \\(\\operatorname{gcd}\\) is the greatest common divisor function. Using \\(\\lambda(N)\\) can yield the minimum workable private exponent \\(d\\), which is conducive to efficient decryption and signature operations. Implementations that follow the above procedure, whether using Euler's or Carmichael's functions, are often referred to as \"textbook RSA \".
\n
Textbook RSA is insecure, and there are many simple and effective means of attack. Before discussing the security holes of the textbook RSA in detail, it is necessary to review the first known attack method - integer factorization!
\n
Integer Factorization
\n
The theoretical cornerstone of the security of the RSA encryption algorithm is the problem of factoring large numbers. If we can separate \\(p\\) and \\(q\\) from the known \\(N\\), we can immediately derive the private exponent \\(d\\) and thus completely crack RSA. Factoring large numbers is a presumed difficult computational problem. The best-known asymptotic running time algorithm is General Number Field Sieve, and its time complexity is \\({\\displaystyle \\exp \\left(\\left(c+o(1)\\right)(\\ln N)^{\\frac {1}{3}}(\\ln \\ln N)^{\\frac {2}{3}}\\right)}\\), where the constant \\(c = 4/\\sqrt[3]{9}\\)，\\(\\displaystyle \\exp\\) and \\(\\displaystyle \\exp\\) is the exponential function of Euler's number (2.718).
\n
For a given large number, it is difficult to accurately estimate the actual complexity of applying the GNFS algorithm. However, based on the heuristic complexity empirical estimation, we can roughly see the increasing trend of computational time complexity:
\n
\n
For a large number of 1024 bits, there are two prime factors of about 500 bits each, and the decomposition requires basic arithmetic operations of order \\(2^{70}\\)
\n
For a large number of 2048 bits, there are two prime factors of about 1000 bits each, and the decomposition requires basic arithmetic operations of order \\(2^{90}\\), a million times slower than the 1024-bit number
\n
\n
The rapid development of computer software and hardware technology has made many tasks that seemed impossible in the past become a reality. Check the latest record released by the RSA Factoring Challenge website. In February 2020, a team led by French computational mathematician Paul Zimmermann successfully decomposed the large number RSA-250 with 250 decimal digits (829 bits):
According to the announcement of the factorization released by Zimmerman, using a 2.1GHz Intel Xeon Gold 6130 processor, the total computing time to complete this task is about 2700 CPU core-years. This number may seem large, but in today's era of cluster computing, grid computing, and cloud computing for the masses, it's not a stretch to think that organizations with strong financial backing can reduce computing time to hours or even minutes. As an example, go to the online tool website of the free open-source mathematical software system SageMath and enter the following first 5 lines of Sage Python code:
The result was obtained within minutes, and a large number of 72 decimal digits (240 bits) was decomposed. You know, in the 1977 RSA paper, it is mentioned that it takes about 104 days to decompose a 75-digit decimal number. The technological progress of mankind is so amazing!
\n
As the attacker's spear becomes sharper and sharper, the defender's shield must become thicker and thicker. Therefore, 1024-bit RSA is no longer secure, and applications should not use public key \\(N\\) values that are less than 2048 bits. And when high security is required, choose 4096-bit RSA.
\n
Elementary Attacks
\n
Although the decomposition of large numbers is an attack method known to everyone, the security vulnerabilities caused by some low-level errors commonly found in RSA applications make it possible to use simple attacks to succeed, and some typical ones are explained below.
\n
\n
In the early development of RSA, finding large prime numbers took quite a bit of time based on the backward computing power of the time. Therefore, some system implementations tried to share the modulus \\(N\\). The idea was to generate only one set \\((p,q)\\), and then all users would use the same \\(N=pq\\) values, with a central authority that everyone trusted assigning key pairs \\((e_i,d_i)\\) to each user \\(i\\), and nothing would go wrong as long as the respective private keys \\(d_i\\) were kept. Unfortunately, this is a catastrophic mistake! This implementation has two huge security holes：
\n\n
The user \\(i\\) can decompose \\(N\\) using his own key pair \\((e_i,d_i)\\). Whether \\(d\\) is generated using the Euler function \\(\\varphi(N)\\) or the Carmichael function \\(\\lambda(N)\\), there are algorithms that quickly derive the prime factors \\(p\\) and \\(q\\) from a given \\(d\\)2. And once \\(p\\) and \\(q\\) are known, user \\(i\\) can compute any other user's private key \\(d_j\\) with one's public key \\((N,e_j)\\). At this point, the other users have no secrets from user \\(i\\).
\n
Even if all users do not have the knowledge and skill to decompose \\(N\\), or are \"nice\" enough not to know the other users' private keys, a hacker can still perform a common modulus attack to break the users' messages. If the public keys of two users, Alice and Bob, are \\(e_1\\) and \\(e_2\\), and \\(e_1\\) and \\(e_2\\) happen to be mutually prime (which is very likely), then by Bézout's identity, the eavesdropper Eve can find that \\(s\\) and \\(t\\) satisfy: \\[e_{1}s+e_{2}t=gcd(e_1,e_2)=1\\] At this point, if someone sends the same message \\(m\\) to Alice and Bob, Eve can decrypt \\(m\\) after recording the two ciphertexts \\(c_1\\) and \\(c_2\\) and performing the following operation: \\[c_1^s⋅c_2^t\\equiv(m^{e _1})^s⋅(m^{e_2})^t\\equiv m^{e_{1}s+e_{2}t}\\equiv m\\pmod N\\] The corresponding Python function code is shown below.
\n
defcommon_modulus(e1, e2, N, c1, c2): # Call the extended Euclidean algorithm function g, s, t = gymp2.gcdext(e1, e2) assert g == 1 if s < 0: # Find c1's modular multiplicative inverse\t\t re = int(gmpy2.invert(c1, N)) c1 = pow(re, s*(-1), N) c2 = pow(c2, t, N) else: # t is negative, find c2's modular multiplicative inverse re = int(gmpy2.invert(c2, N)) c2 = pow(re, t*(-1), N) c1 = pow(c1, a, N) return (c1*c2) % N
Two library functions of gmpy23 are called here: gcdext() to implement the extended Euclidean algorithm, and invert() to find the modular multiplicative inverse element. Note that Python's exponential function pow() supports modular exponentiation, but the exponent must not be negative. Since one of \\(s\\) or \\(t\\) must be negative, you have to first call invert() to convert \\(c_1\\) or \\(c_2\\) to the corresponding modular multiplicative inverse, then invert the negative number to calculate the modular exponent. For example, lines 7 and 8 above implement \\(c_1^s=(c_1^{-1})^{-s}\\bmod N\\).
\n
\n
Is it possible to reuse only \\(p\\) or \\(q\\) since the shared modulus \\(N\\) is proven to be insecure? This seems to avoid the common-modulus attack and ensure that each user's public key \\(N\\) value is unique. Big mistake! This is an even worse idea! The attacker gets the public \\(N\\) values of all users and simply combines \\((N_1,N_2)\\) pairwise to solve Euclid's algorithm for the great common divisor, and a successful solution gives a prime factor \\(p\\), and a simple division gives the other prime factor \\(q\\). With \\(p\\) and \\(q\\), the attacker can immediately compute the user's private key \\(d\\). This is the non-coprime modulus attack.
\n
When applying textbook RSA, if both the public exponent \\(e\\) and the plaintext \\(m\\) are small, such that \\(c=m^e<N\\), the plaintext \\(m\\) can be obtained by directly calculating the \\(e\\)th root of the ciphertext \\(c\\). Even if \\(m^e>N\\) but not large enough, then since \\(m^e=c+k⋅N\\), you can loop through the small \\(k\\) values to perform brute-force root extraction cracking. Here is the Python routine:
\n
defcrack_small(c, e, N, repeat) times = 0 msg = 0 for k inrange(repeat): m, is_exact = gmpy2.iroot(c + times, e) if is_exact andpow(m, e, N) == c: msg = int(m) break times += N return msg
Here the gmpy2 library function iroot() is called to find the \\(e\\)th root.
\n
Textbook RSA is deterministic, meaning that the same plaintext \\(m\\) always generates the same ciphertext \\(c\\). This makes codebook attack possible: the attacker precomputes all or part of the \\(m\\to c\\) mapping table and saves, then simply searches the intercepted ciphertext for a match. Determinism also means that textbook RSA is not semantically secure and that the ciphertext can reveal some information about the plaintext. Repeated occurrences of the ciphertext indicate that the sender is sending the same message over and over again.
\n
Textbook RSA is malleable, where a particular form of algebraic operation is performed on the ciphertext and the result is reflected in the decrypted plaintext. For example, if there are two plaintexts \\(m_1\\) and \\(m_2\\), and encryption yields \\(c_1=m_1^e\\bmod N\\) and \\(c_2=m_2^e\\bmod N\\), what does \\((c_1⋅c_2)\\) decryption yield? Look at the following equation: \\[(c_1⋅c_2)^d\\equiv m_1^{ed}⋅m_2^{ed}\\equiv m_1⋅m_2\\pmod N\\] So the plaintext obtained after decrypting the product of the two ciphertexts is equal to the product of the two plaintexts. This feature is detrimental to RSA encryption systems in general and provides an opportunity for chosen-ciphertext attack. The following are two examples of attack scenarios:
\n\n
Imagine that there is an RSA decryption machine that can decrypt messages with an internally saved private key \\((N,d)\\). For security reasons, the decryptor will reject repeated input of the same ciphertext. An attacker, Marvin, finds a piece of ciphertext \\(c\\) that is rejected by the decryptor when he enters it directly because the ciphertext \\(c\\) has been decrypted before. Marvin finds a way to crack it. He prepares a plaintext \\(r\\) himself, encrypts it with the public key \\((N,e)\\) to generate a new ciphertext \\(c'={r^e}c\\bmod N\\), and then feeds the ciphertext \\(c'\\) to the decryptor. The decryption machine has not decrypted this new ciphertext, so it will not reject it. The result of the decryption is \\[m'\\equiv (c')^d\\equiv r^{ed}c^d\\equiv rm\\pmod N\\] Now that Marvin has \\(m'\\), he can calculate \\(m\\) using the formula \\(m\\equiv m'r^{-1}\\pmod N\\).
\n
Suppose Marvin wants Bob to sign a message \\(m\\), but Bob refuses to do so after reading the message content. Marvin can achieve his goal by using an attack called blinding4. He picks a random message \\(r\\), generates \\(m'={r^e}m\\bmod N\\), and then takes \\(m'\\) to Bob to sign. Bob probably thinks \\(m'\\) is irrelevant and signs it. The result of Bob's signature is \\(s'=(m')^d\\bmod N\\). Now Marvin has Bob's signature on the original message \\(m\\) using the formula \\(s=s'r^{-1}\\bmod N\\). Why? The reason is that \\[s^e\\equiv (s')^er^{-e}\\equiv (m')^{ed}r^{-e}\\equiv m'r^{-e}\\equiv m\\pmod N\\]
\n
\n
\n
The above is by no means a complete list of elementary attack methods, but they are illustrative. In practical RSA applications, we must be very careful and should do the following:
\n
\n
generate a unique public key modulus \\(N\\) for each user individually to prevent common-mode attacks
\n
not reuse the prime factor to generate the public key modulus \\(N\\), to eliminate the non-coprime modulus attack
\n
\n
For the textbook RSA deterministic and malleable flaws, and possible brute-force root extraction cracking vulnerabilities, the padding with random elements method can be used to protect against them, and the protection is valid due to the following:
\n
\n
Padding ensures that the number of bits in the encrypted message is close to \\(N\\), while not using small \\(e\\) values, making possible brute-force root extraction cracking ineffective
\n
Random padding makes the same plaintext produce different ciphertexts, guaranteeing semantic security and making ciphertext attacks impossible
\n
Strictly format-defined padding destroys malleability and reduces the possibility of ciphertext selection attacks. For example, if the first few bytes after padding must be a given value, the decrypted data will most likely not conform to the predefined format after the algebraic operation on the corresponding ciphertext, which disrupts the ciphertext selection attack.
\n
\n
Low Public Exponent Attacks
\n
Using low public exponent is dangerous, and there are advanced attacks in the case of non-padding or improper padding, even if brute-force root extraction cracking does not succeed.
\n
Broadcast Attack
\n
Discovered by Swedish theoretical computer scientist Johan Håstad 5, hence the name Håstad's Broadcast Attack. Consider this simplified scenario, assuming that Alice needs to send the same message \\(m\\) to Bob, Carol, and Dave. The public keys of the three recipients are \\((N_1,3)\\), \\((N_2,3)\\), and \\((N_3,3)\\), i.e., the public exponent is all 3 and the public key modulus is different for each. The messages are not padded and Alice directly encrypts and sends three ciphertexts \\(c_1,c_2,c_3\\) using the public keys of the other three:
At this point Eve secretly writes down the three ciphertexts, marking \\(M=m^3\\), and if she can recover \\(M\\), running a cube root naturally yields the plaintext \\(m\\). Obviously, the common modulus attack does not hold here, and we can also assume that the moduli are pairwise coprime, or else decomposing the modulus using the non-coprime modulus attack will work. So does Eve have a way to compute \\(M\\)? The answer is yes.
\n
In fact, the equivalent problem for solving \\(M\\) here is: Is there an efficient algorithm for solving a number that has known remainders of the Euclidean division by several integers, under the condition that the divisors are pairwise coprime? This efficient algorithm is Chinese Remainder Theorem!
\n
The Chinese remainder theorem gives the criterion that a system of one-element linear congruence equations has a solution and the method to solve it. For the following system of one-element linear congruence equations (be careful not to confuse it with the mathematical notation used to describe the attack scenario above):
Suppose that the integers \\(m_1,m_2,\\ldots,m_n\\) are pairwise coprime, then the system of equations \\((S)\\) has a solution for any integer \\(a_1,a_2,\\ldots,a_n\\) and the general solution can be constructed in four steps as follows:
The last line above, Eq. (4) gives the formula of the general solution. In the sense of modulus \\(M\\), the unique solution is \\(\\sum_{i=1}^n a_i t_i M_i \\bmod M\\).
\n
Try to solve the things whose number is unknown problem at the beginning of this article by using the Chinese remainder theorem
\n\n
First, correspond the variable symbols to the values: \\[m_1=3,a_1=2;\\quad m_2=5,a_2=3;\\quad m_3=7,a_3=2\\] Then calculate \\(M=3\\times5\\times7=105\\), which in turn leads to the derivation of： \\[\\begin{align}\nM_1 &=M/m_1=105/3=35,\\quad t_1=35^{-1}\\bmod 3 = 2\\\\\nM_2 &=M/m_2=105/5=21,\\quad t_2=21^{-1}\\bmod 5 = 1\\\\\nM_3 &=M/m_3=105/7=15,\\quad t_3=15^{-1}\\bmod 7 = 1\\\\\n\\end{align}\\] Finally, take these into the general solution formula: \\[x=k⋅105+(2⋅35⋅2+3⋅21⋅1+2⋅15⋅1)=k⋅105+233\\] So the smallest positive integer solution concerning modulus 105 is \\(233\\bmod 105=23\\)。
\n
In his mathematical text \"Suanfa Tongzong\", Cheng Dawei, a mathematician of the Ming Dynasty in the 16th century, compiled the solutions recorded by the mathematician Qin Jiushao of the Song Dynasty in the \"Mathematical Treatise in Nine Sections\" into a catchy \"Sun Tzu's Song\":
\n
\n
Three friends set out with seventy rare \nTwenty-one blossoms on five trees of plums \nSeven men reunited at the half-month \nAll be known once divided by one hundred and five
\n
\n
Here we must admire the wisdom of the ancient Chinese who, in the absence of a modern mathematical symbol system, were able to derive and summarize such an ingenious solution, contributing an important mathematical theorem to mankind.
\n\n\n
So Eve just applies the solution of the Chinese Remainder Theorem, computes \\(M\\), and then finds its cube root to get the plaintext \\(m\\), and the attack succeeds. More generally, setting the number of receivers to \\(k\\), if all receivers use the same \\(e\\), then this broadcast attack is feasible as long as \\(k\\ge e\\).
\n
Håstad further proves that even if padding is used to prevent broadcast attacks, if the messages generated by the padding scheme are linearly related to each other, such as using the formula \\(m_i=i2^b+m\\) (\\(b\\) is the number of bits of \\(m\\)) to generate the message sent to the receiver \\(i\\), then the broadcast attack can still recover the plaintext \\(m\\) as long as \\(k>e\\). The broadcast attack in this case is still based on the Chinese remainder theorem, but the specific cracking method depends on the information of the linear relationship.
\n
To summarize the above analysis, to prevent the broadcast attack, we must use a higher public exponent \\(e\\) and apply random padding at the same time. Nowadays, the common public key exponent \\(e\\) is $65537 (\\(2^{16}+1\\)), which can balance the efficiency and security of message encryption or signature verification operations.
\n
Last, Python routines for simulating broadcast attacks are given as follows:
\n
defsolve_crt(ai: list, mi: list): '''mi and ai are the list of modulus and remainders. The precondition of the function is that the modulus in the mi list are pairwise coprime.''' M = reduce(lambda x, y: x * y, mi) ti = [a * (M//m) * int(gmpy2.invert(M//m, m)) for (m, a) inzip(mi, ai)] return reduce(lambda x, y: x + y, ti) % M
defuint_to_bytes(x: int) -> bytes: '''convert unsigned integer to byte array''' if x == 0: returnbytes(1) return x.to_bytes((x.bit_length() + 7) // 8, 'big')
quote = b'The cosmos is within us. We are made of star stuff. - Carl Sagan' bob = RSA(1024, 3) carol = RSA(1024, 3) dave = RSA(1024, 3) cipher_list = [bob.encrypt(quote), carol.encrypt(quote), dave.encrypt(quote)] modulus_list = [bob.n, carol.n, dave.n]
This code uses two methods to simulate the broadcast attack. One calls the generic Chinese remainder theorem solver function solve_crt() and then gets the cube root of the result; the other calls the special broadcast attack function rsa_broadcast_attack() for the public key index \\(e=3\\), which directly outputs the cracked plaintext value. The internal implementation of these two functions is based on the generalized formula of the Chinese remainder theorem, and the output results should be identical. The cracked plaintext value is then input to the uint_to_bytes() function, which is converted into a byte array to compare with the original quote. Note that the program uses objects generated by the RSA class to simulate the receivers Bob, Carroll, and Dave, and the implementation of the RSA class is omitted here given the limitation of space.
\n
\n
To be continued, please look forward to the next part: RSA: Attack and Defense (2)
\n
\n\n\n\n
American computer scientist and security expert Gary McGraw has a famous piece of advice for software developers - \"never roll your own cryptography\"↩︎
\n
The original RSA paper (Part IX, Section C) did mention Miller's algorithm for factoring \\(N\\) with a known \\(d\\). This algorithm also applies to \\(d\\) generated by the Carmichael function \\(\\lambda(N)\\).↩︎
\n
gmpy2 is a Python extension module written in C that supports multi-precision arithmetic.↩︎
\n
On some special occasions, blinding can be used for effective privacy protection. For example, in cryptographic election systems and digital cash applications, the signer and the message author can be different.↩︎
\n
Johan Håstad, a Swedish theoretical computer scientist, a professor at the KTH Royal Institute of Technology, and a Fellow of the American Mathematical Society (AMS) and an Association for Computing Machinery (ACM) fellow.↩︎
\n\n\n","categories":["Technical Know-how"],"tags":["Cryptography","Network Security","Python Programming"]},{"title":"Please Stop Using TLS 1.0 and TLS 1.1 Now!","url":"/en/2022/11/10/Stop-TLS1-0-TLS1-1/","content":"
In March 2021, the Internet Engineering Task Force (IETF) released RFC 8996, classified as a current best practice, officially announcing the deprecation of the TLS 1.0 and TLS 1.1 protocols. If your applications and web services are still using these protocols, please stop immediately and update to TLS 1.2 or TLS 1.3 protocol versions as soon as possible to eliminate any possible security risks.
\n
One single vulnerability is all an attacker needs. — Window Snyder (American computer security expert, former Senior Security Strategist at Microsoft, and has been a top security officer at Apple, Intel and other companies)
\n
\n
RFC Interpretation
\n
The document title of RFC 8996 is quite straightforward, \"Deprecating TLS 1.0 and TLS 1.1\". So what is the rationale it gives? Here is a simple interpretation.
\n
First, take a look at its abstract:
\n
\n
This document formally deprecates Transport Layer Security (TLS) versions 1.0 (RFC 2246) and 1.1 (RFC 4346). Accordingly, those documents have been moved to Historic status. These versions lack support for current and recommended cryptographic algorithms and mechanisms, and various government and industry profiles of applications using TLS now mandate avoiding these old TLS versions. TLS version 1.2 became the recommended version for IETF protocols in 2008 (subsequently being obsoleted by TLS version 1.3 in 2018), providing sufficient time to transition away from older versions. Removing support for older versions from implementations reduces the attack surface, reduces opportunity for misconfiguration, and streamlines library and product maintenance.
\n
This document also deprecates Datagram TLS (DTLS) version 1.0 (RFC 4347) but not DTLS version 1.2, and there is no DTLS version 1.1.
\n
This document updates many RFCs that normatively refer to TLS version 1.0 or TLS version 1.1, as described herein. This document also updates the best practices for TLS usage in RFC 7525; hence, it is part of BCP 195.
\n
\n
The information given here is clear, the reasons for deprecating them are purely technical. TLS 1.0 and TLS 1.1 cannot support stronger encryption algorithms and mechanisms, and cannot meet the high-security requirements of various network applications in the new era. TLS is TCP-based. Corresponding to the UDP-based DTLS protocol, RFC 8996 also announced the deprecation of the DTLS 1.0 protocol.
\n
The Introduction section lists some details of the technical reasons:
\n\n
They require the implementation of older cipher suites that are no longer desirable for cryptographic reasons, e.g., TLS 1.0 makes TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA mandatory to implement.
The integrity of the handshake depends on SHA-1 hash.
\n
The authentication of the peers depends on SHA-1 signatures.
\n
Support for four TLS protocol versions increases the likelihood of misconfiguration.
\n
At least one widely used library has plans to drop TLS 1.1 and TLS 1.0 support in upcoming releases.
\n\n
Clauses 5 and 6 above are clear and need no further explanation.
\n
For 3DES mentioned in Clause 1, although it uses three independent keys with a total length of 168 bits, considering the possible meet-in-the-middle_attack attack, its effective key strength is only 112 bits. Also, the 3DES encryption block length is still 64 bits, which makes it extremely vulnerable to birthday attack (see Sweet32). NIST stipulates that a single 3DES key group can only be used for encrypting \\(2^{20}\\) data blocks (ie 8MB). This was of course too small, and eventually, NIST decided in 2017 to deprecate 3DES in the IPSec and TLS protocols.
\n
3DES is just one example, another category that has been phased out earlier is cipher suites that use RC4 stream ciphers, see RFC 7465 for details. In addition, there are various problems in the implementation of block cipher CBC mode, which are often exploited by attackers to crack TLS sessions. A summary of various attacks and countermeasures of TLS 1.0 and TLS 1.1 is described in detail in NIST800-52r2 and RFC7457. These two reference documents provide the key rationale for deprecation. Obviously, any protocol that mandates the implementation of insecure cipher suites should be on the list to be eliminated.
\n
In the second section of the document, the content in Section 1.1 \"The History of TLS\" of NIST800-52r2 is directly quoted (abbreviated as shown in the following table):
Improved initialization vector selection and padding error processing to address weaknesses discovered on the CBC mode of operation defined in TLS 1.0.
Enhanced encryption algorithms, particularly in the area of hash functions, can support SHA-2 series algorithms for hashing, MAC, and pseudorandom function computations, also added AEAD cipher suite.
A significant change to TLS that aims to address threats that have arisen over the years. Among the changes are a new handshake protocol, a new key derivation process that uses the HMAC-based Extract-and-Expand Key Derivation Function (HKDF), and the removal of cipher suites that use RSA key transport or static Diffie-Hellman key exchanges, the CBC mode of operation, or SHA-1.
\n
\n\n
\n
AEAD is an encryption mode that can guarantee the confidentiality, integrity, and authenticity of data at the same time, typically such as CCM and GCM. TLS 1.2 introduced a range of AEAD cipher suites, and its high security made it the exclusive choice for TLS 1.3. These annotate Clause 2 of technical reasons.
\n
Clauses 3 and 4 of technical reasons call out SHA-1, so what is the problem with SHA-1? Section 3 of the document cites a paper by two French researchers, Karthikeyan Bhargavan and Gaetan Leurent .
\n
As a cryptographic hash function, SHA-1 was designed by the National Security Agency (NSA) and then published as a Federal Information Processing Standard (FIPS) by the National Institute of Standards and Technology (NIST). SHA-1 can process a message up to \\(2^{64}\\) bits and generate a 160-bit (20-byte) hash value known as the message digest. Therefore, the complexity of brute force cracking based on birthday attack is \\(2^{80}\\) operations. In 2005, Chinese cryptographer Wang Xiaoyun and her research team made a breakthrough in this field. The high-efficiency SHA-1 attack method they published can be used to find a hash collision within a computational complexity of \\(2^{63}\\). This has brought a huge impact on the security of SHA-1, but it does not mean that the cracking method can enter the practical stage.
\n
Network security protocols (such as TLS, IKE, and SSH, etc.) rely on the second preimage resistance of cryptographic hash functions, that is, it is computationally impossible to find any secondary input value that has the same output as a specific input value. For example, for a cryptographic hash function \\(h(x)\\) and given input \\(x\\), it is difficult to find a sub-preimage \\(x^′ ≠ x\\) that is satisfying \\(h(x) = h(x^′)\\). Because finding a hash collision does not mean that a sub-preimage can be located, in practice, it was once thought that continuing to use SHA-1 is not a problem.
\n
However, in 2016, Bhargavan and Leurent (who implemented the aforementioned Sweet32 attack against 64-bit block ciphers) discovered a new class of methods to attack key exchange protocols that shattered this perception. These methods are based on the principle of the chosen prefix collision attack. That is, given two different prefixes \\(p_1\\) and \\(p_2\\), the attack finds two appendages \\(m_1\\) and \\(m_2\\) such that \\(h(p_1 ∥ m_1) = hash(p_2 ∥ m_2)\\). Using this approach, they demonstrated a man-in-the-middle attack against TLS clients and servers to steal sensitive data, and also showed that the attack could be used to masquerade and downgrade during TLS 1.1, IKEv2, and SSH-2 session handshakes. In particular, they proved that with only \\(2^{77}\\) operations the handshake protocol using SHA-1 or MD5 and SHA-1 concatenated hash values could be cracked.
\n
Since neither TLS 1.0 nor TLS 1.1 allows the peers to choose a stronger cryptographic hash function for signatures in the ServerKeyExchange or CertificateVerify messages, the IETF confirmed that using a newer protocol version is the only upgrade path.
\n
Sections 4 and 5 of the document again clarify that TLS 1.0 and TLS 1.1 must not be used, and negotiation to TLS 1.0 or TLS 1.1 from any TLS version is not allowed. This means that ClientHello.client_version and ServerHello.server_version issued by the TLS client and server, respectively, must not be {03,01} (TLS 1.0) or {03,02} (TLS 1.1). If the protocol version number in the Hello message sent by the other party is {03,01} or {03,02}, the local must respond with a \"protocol_version\" alert message and close the connection.
\n
It is worth noting that due to historical reasons, the TLS specification does not specify the value of the record layer version number (TLSPlaintext.version) when the client sends the ClientHello message. So to maximize interoperability, TLS servers MUST accept any value {03,XX} (including {03,00}) as the record layer version number for ClientHello messages, but they MUST NOT negotiate TLS 1.0 or 1.1.
\n
Section 6 of the document declares a textual revision to the previously published RFC 7525 (Recommendations for the Secure Use of TLS and DTLS). Three places in this RFC change implementation-time negotiations of TLS 1.0, TLS 1.1, and DTLS 1.0 from \"SHOULD NOT\" to \"MUST NOT\". The last section is a summary of standard RFC operations and security considerations.
\n
Industry Responses
\n
In the industry of large public online services, GitHub was the first to act. They started disabling TLS 1.0 and TLS 1.1 in all HTTPS connections back in February 2018, while also phasing out insecure diffie-hellman-group1-sha1 and diffie-hellman-group14-sha1 key exchange algorithms in the SSH connection service. In August 2018, Eric Rescorla, CTO of Mozilla Firefox, published the TLS 1.3 technical specification RFC 8446. Two months later, Mozilla issued a statement together with the three giants of Apple, Google, and Microsoft, and put the deprecation of TLS 1.0 and TLS 1.1 on the agenda.
\n
The following is a brief summary of the actions of several related well-known companies:
\n
\n
Microsoft: For Office 365 services, TLS 1.0 and 1.1 disabling for commercial customers was temporarily suspended due to COVID-19. The mandatory rollout of TLS 1.2 was restarted on October 15, 2020. Users of SharePoint and OneDrive will need to update and configure .NET to support TLS 1.2. Users of Teams Rooms recommend upgrading the app to version 4.0.64.0. The Surface Hub released support for TLS 1.2 in May 2019. The Edge browser version 84 does not use TLS 1.0/1.1 by default, while the Azure cloud computing service will permanently obsolete TLS 1.0/1.1 from March 31, 2022.
\n
Google: As early as 2018, TLS 1.3 was added to Chrome 70. Starting with Chrome 84, support for TLS 1.0 and TLS 1.1 is completely removed. After running TLS 1.3 in Search Engine, Gmail, YouTube, and various other Google services for some time, TLS 1.3 was officially rolled out in 2020 as the default configuration for all new and existing Cloud CDN and Global Load Balancing customers.
\n
Apple: Announced in September 2021 that TLS 1.0 and TLS 1.1 will be deprecated in iOS 15, iPadOS 15, macOS 12, watchOS 8, and tvOS 15, and support for them be completely removed in future releases. If the user's application activates the App Transport Security (ATS) feature on all connections, no changes are required. Users are also notified to ensure that the web server supports newer TLS versions and to remove the following deprecated Security.framework symbols from the app\n
\n
tls_protocol_version_t.TLSv10
\n
tls_protocol_version_t.TLSv11
\n
tls_protocol_version_t.DTLSv10
\n
\n
Mozilla: Starting with Firefox version 78, the minimum TLS version configured by default is TLS 1.2. In early 2020, Mozilla briefly removed TLS 1.0 and TLS 1.1 from Firefox completely, but this caused many users to be unable to open some COVID-19 outbreak public information sites, so the related functionality had to be restored. Following this, Mozilla provides helpful information on its technical support page, instructing users to modify the minimum TLS version number in the default configuration as needed.
\n
Cisco: The Cisco Umbrella (renamed from OpenDNS) service discontinued support for all versions of TLS prior to 1.2 on March 31, 2020. After this, only TLS 1.2 compliant clients will be able to connect. In the router and switch product lines, web management has basically been implemented around 2020 to allow only TLS 1.2 or subsequent versions.\n
\n
The CAPWAP connection between Cisco's Wireless Access Point (AP) and Wireless LAN Controller (WLC) is established over DTLS. All 802.11ac Wave 2 and 802.11ax APs from 2015 to the most recent release support DTLS 1.2. The AireOS WLC added DTLS 1.2 functionality in version 8.3.11x.0, and the next-generation C9800 WLC running IOS-XE supports DTLS 1.2 from the start. Note that because of the large number of existing network deployments using older equipment and software versions, DTLS 1.0 support cannot be removed immediately from APs and WLCs at this time to protect user investments. However, DTLS 1.2 is already the default optimal choice for APs and WLCs.
\n
\n
\n
Protocol Test
\n
Both TLS/DTLS clients and servers need to be tested to verify that their implementations follow the current best practices of RFC 8996.
\n
SSL Lab Test
\n
Qualys originated as a non-commercial SSL Labs Projects. They offer a free and simple client and server testing service, as well as a monitoring panel reporting TLS/SSL security scan statistics for the most popular Internet sites. Below is the most recent chart of protocol support statistics for November 2022.
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n\n
\n
Protocol Version
\n
Security
\n
Supporting Sites (Oct. 2022)
\n
Supporting Site (Nov. 2022)
\n
% Change
\n
\n\n\n
\n
SSL 2.0
\n
Insecure
\n
316（0.2%）
\n
303（0.2%)
\n
-0.0%
\n
\n
\n
SSL 3.0
\n
Insecure
\n
3,015（2.2%）
\n
2,930（2.2%)
\n
-0.0%
\n
\n
\n
TLS 1.0
\n
Deprecated
\n
47,450（34.9%）
\n
46,691（34.4)
\n
-0.5%
\n
\n
\n
TLS 1.1
\n
Deprecated
\n
51,674（38.1%）
\n
50,816（37.5%)
\n
-0.6%
\n
\n
\n
TLS 1.2
\n
Depending on the Cipher Suite and the Client
\n
135,557（99.8）
\n
135,445（99.9)
\n
+0.1%
\n
\n
\n
TLS 1.3
\n
Secure
\n
78,479（57.8%）
\n
79,163（58.4%)
\n
+0.6%
\n
\n\n
\n
As you can see, almost 100% of sites are running TLS 1.2, and the percentage of TLS 1.3 support is close to 60%. This is very encouraging data. While very few sites are still running SSL 2.0/3.0 and TLS 1.0/1.1 are both still supported at around 35%, overall their percentages are continuing to decline and this good trend should continue.
\n
This blog site is served by GitHub Page, enter the URL to SSL Server Test page and submit it to get a summary of the test results as follows.
\n
\n
The site achieved the highest overall security rating of A+. It got a perfect score for certificate and protocol support, and a 90 for both key exchange and password strength. This shows that GitHub fulfills its security promises to users and deserves the trust of programmers.
\n
The configuration section of the report gives details of the test results for protocol support and cipher suites as follows.
\n
\n
This further confirms that the GitHub Page only supports TLS 1.2/1.3, as required by RFC 8996. It can also be seen that under the \"Cipher Suites\" subheading, TLS 1.3 shows two GCMs and one ChaCha20-Poly1305, which are all cipher suites based on the AEAD algorithms. Three cipher suites of the same type are the preferred TLS 1.2 cipher suites for the server as well. This is exactly the current commonly adopted configuration of secure cryptographic algorithms.
\n
User Selftest
\n
If you suspect that a private server is still using the outdated TLS/SSL protocol, you can do a simple test with the command line tool curl, an example of which is as follows.
Here enter the command line option -tls-max 1.1 to set the highest protocol version 1.1 and connect to the Cisco home page. The output shows that the connection failed and that a \"protocol version\" alert message was received. This indicates that the server has rejected the TLS 1.1 connection request, and the response is exactly what is required by RFC 8996.
\n
The openssl command line tool provided by the general purpose open source cryptography and secure communication toolkit OpenSSL can also do the same test. To test whether the server supports the TLS 1.2 protocol, use the option s_client to emulate a TLS/SSL client and also enter -tls1_2 to specify that only TLS 1.2 is used. The command line runs as follows.
\n
❯ openssl s_client -connect www.cisco.com:443 -tls1_2 CONNECTED(00000005) depth=2 C = US, O = IdenTrust, CN = IdenTrust Commercial Root CA 1 verify return:1 depth=1 C = US, O = IdenTrust, OU = HydrantID Trusted Certificate Service, CN = HydrantID Server CA O1 verify return:1 depth=0 CN = www.cisco.com, O = Cisco Systems Inc., L = San Jose, ST = California, C = US verify return:1 --- Certificate chain 0 s:/CN=www.cisco.com/O=Cisco Systems Inc./L=San Jose/ST=California/C=US i:/C=US/O=IdenTrust/OU=HydrantID Trusted Certificate Service/CN=HydrantID Server CA O1 1 s:/C=US/O=IdenTrust/OU=HydrantID Trusted Certificate Service/CN=HydrantID Server CA O1 i:/C=US/O=IdenTrust/CN=IdenTrust Commercial Root CA 1 2 s:/C=US/O=IdenTrust/CN=IdenTrust Commercial Root CA 1 i:/C=US/O=IdenTrust/CN=IdenTrust Commercial Root CA 1 --- Server certificate -----BEGIN CERTIFICATE----- MIIHrzCCBpegAwIBAgIQQAF9KqwAKOKNhDf17h+WazANBgkqhkiG9w0BAQsFADBy ... 4TY7 -----END CERTIFICATE----- subject=/CN=www.cisco.com/O=Cisco Systems Inc./L=San Jose/ST=California/C=US issuer=/C=US/O=IdenTrust/OU=HydrantID Trusted Certificate Service/CN=HydrantID Server CA O1 --- No client certificate CA names sent Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 5765 bytes and written 322 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES128-GCM-SHA256 Session-ID: 1656D7D14447C1D5E68943F614A697455E60A036957D8D8C18F3B198DF42969F Session-ID-ctx: Master-Key: BB1209155344C55792077A4337964661FCA4F3F5BBF3185112F5E235BD07AD63838D24F5CF97161E696CB57398CAF478 TLS session ticket lifetime hint: 83100 (seconds) TLS session ticket: 0000 - 00 00 0b 33 d4 56 15 3d-64 e8 fa 1d cf c1 1c 04 ...3.V.=d....... ... 0090 - 1b 96 9c 25 82 70 a8 ed-24 1d 70 c9 28 56 84 59 ...%.p..$.p.(V.Y
This record is very detailed and the format is very readable. From the output, it can be understood that the digital certificate of the Cisco home page server is digitally signed and certified by the root certificate authority IdenTrust. The client-server session is built on the TLS 1.2 protocol, and the selected cipher suite is ECDHE-RSA-AES128-GCM-SHA256 of type AEAD, which is identical to the preferences provided by the GitHub Page.
\n
Browser Test
\n
If you are not sure about the security of your browser and want to test whether it still supports the pre-TLS 1.2 protocols, you can enter the following URL in your browser's address bar.
After connecting to the second URL with the default configuration of Firefox, the page shows the following
\n
\n
Secure Connection Failed
\n
An error occurred during a connection to tls-v1-1.badssl.com:1011. Peer using unsupported version of security protocol.
\n
Error code: SSL_ERROR_UNSUPPORTED_VERSION
\n
\n
The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
\n
Please contact the website owners to inform them of this problem.
\n
\n
This website might not support the TLS 1.2 protocol, which is the minimum version supported by Firefox.
\n
\n
This error message clearly indicates that Firefox is running a minimum TLS protocol version of 1.2 in this configuration, and since the other side is only running TLS 1.1, the two sides cannot establish a connection.
\n
So what is the result of the connection when the browser does still retain TLS 1.0/1.1 functionality?
\n
For testing purposes, you can first change the default TLS preference value of Firefox to 1.1 by following the steps below (refer to the figure below).
\n\n
Open a new tab, type about:config in the address bar, and press Enter/Return.
\n
The page prompts \"Proceed with Caution\", click the Accept the Risk and Continue button.
\n
In the search box at the top of the page, type TLS to display the filtered list.
\n
Find the security.tls.version.min preference option and click the Edit icon to change the minimum TLS version.\n
This bold red page tells you that the browser you are currently using does not have TLS 1.1 disabled and is a security risk, so try not to use it if you can.
\n
After testing, don't forget to restore the default TLS minimum version setting (3) for Firefox.
\n
References
\n
\n
Disclosure: This blog site is reader-supported. When you buy through the affiliate links below, as an Amazon Associate, I earn a tiny commission from qualifying purchases. Thank you.
\n
\n
Besides NIST and RFC documents, For an in-depth study of the TLS protocol specification, system implementation, and application deployment, a careful reading of the following three books is recommended.
\n
\n\n
SSL and TLS: Theory and Practice, Second Edition (2016) - This book provides a comprehensive discussion of the SSL, TLS, and DTLS protocols. It has complete details on the theory and practice of the protocols, offering readers a solid understanding of their design principles and modes of operation. The book also presents the advantages and disadvantages of the protocols compared to other Internet security protocols and provides the details necessary to correctly implement the protocols while saving time on the security practitioner’s side.
\n
Implementing SSL/TLS Using Cryptography and PKI (2011) - For a network professional who knows C programming, this book is a hands-on, practical guide to implementing SSL and TLS protocols for Internet security. Focused on how to implement SSL and TLS, it walks you through all the necessary steps, whether or not you have a working knowledge of cryptography. The book covers TLS 1.2, including implementations of the relevant cryptographic protocols, secure hashing, certificate parsing, certificate generation, and more.
\n
Bulletproof TLS and PKI, Second Edition: Understanding and Deploying SSL/TLS and PKI to Secure Servers and Web (2022) - This book is a complete guide to using TLS encryption and PKI to deploy secure servers and web applications. Written by Ivan Ristić, founder of the popular SSL Labs website, it will teach you everything you need to know to protect your systems from eavesdropping and impersonation attacks. You can also find just the right mix of theory, protocol detail, vulnerability and weakness information, and deployment advice to get the work done.
By chance, I came across a picoCTF RSA challenge called Sum-O-Primes. This problem is not difficult, you can do it by knowing the basics of the RSA algorithm. In addition, if you are familiar with the history of the evolution of the RSA algorithm, you can find a second ingenious fast solution.
\n
picoCTF Project
\n
picoCTF is a free computer security education program created by security and privacy experts at Carnegie Mellon University. It uses original content built on the CTF (Capture the Flag) framework to provide a variety of challenges. It provides participants with valuable opportunities to systematically learn cybersecurity knowledge and gain practical experience.
\n
The collection of practice questions for picoCTF is called picoGym. The general problem solution is to search or decipher a string in the format \"picoCTF{...}\" from the given information, that is, the flag to be captured. As shown in the figure below, picoGym currently contains 271 cybersecurity challenge exercises, covering general skills, cryptography, reverse engineering, forensics, and other fields.
\n
\n
Sum-O-Primes Challenge
\n
There are 50 cryptography-related challenges in picoGym, one of which is Sum-O-Primes. The task of this challenge is simple and explained as follows:
\n
\n
We have so much faith in RSA we give you not just the product of the primes, but their sum as well!
That is, we not only give the product of the two prime numbers used by RSA but also tell you their sum. How are these given? You need to discover by yourself from the rest of the information. After clicking the two links and downloading the file, open the first Python file:
\n
gen.py
#!/usr/bin/python
from binascii import hexlify from gmpy2 import mpz_urandomb, next_prime, random_state import math import os import sys
FLAG = open('flag.txt').read().strip() FLAG = int(hexlify(FLAG.encode()), 16) SEED = int(hexlify(os.urandom(32)).decode(), 16) STATE = random_state(SEED)
If you have basic Python programming skills and understand the principles of the RSA algorithm, you should be able to read the above program quickly. What it does is:
\n\n
Open the file flag.txt to read the content. Then use the hexlify and int functions to convert it to an integer and store the result in a variable FLAG.
\n
Call the function get_prime to generate two prime numbers, store their sum in x and their product in n. Then assign 65537 to e and calculate the RSA private exponent d.
\n
Use standard pow functions to perform modular exponentiation, which implements RSA encryption to encrypt plaintext FLAG into ciphertext c.
\n
Print out x, n, and c.
\n\n
Open the second file, which is apparently the output of the first program in Python:
\n
output.txt
x = 154ee809a4dc337290e6a4996e0717dd938160d6abfb651736d9f5d524812a659b310ad1f221196ee8ab187fa746a1b488a4079cddfc5db08e78be0d96c83c01e9bb42420b40d6f0ad9f220633459a6dc058bb01c517386bfbd2d4811c9b08558b0e05534768581a74884758d15e15b4ef0dbd6a338bf1f52eed4f137957737d2 n = 6ce91e471f1df651b0d275d6d5522703feecdd77e7821a2caf9514104c059781c1b2e64772d9220addd657ecbd4e6cb8b5941608f6ab54bd5760074a5cd5854920439422192d2ee8912f1ebcc0d97714f209ee2a22e2da60e071541cb7e0772373cfea71831673378ee6432e63abfd14db0d4aa601928923253f9edd419ce96f4d68ce0aa3e6d6b530cd46eefbdac93038ce949c9dd2e573a47471cf8223f88b96e00a92f4d47fd277c42c4075b5e99b41a9f279f442bc0d533b9ddc50592e369e7026b3f7afaa8edf8972f0c3055f4de67a0eea963f099a32e1539de1d1727abadd9235f66371998ec883d1f89b8d907270842818cae49cd5c7f906c4752e81 c = 48b89662b9718fb391c96527272bf74c27810edaca09b63e694af9d11608010b1db9aedd1c867849371121941a1ccac610f7b28b92fa2f981babe816e6d3ecfab83514ed7e18e2b23fc3b96c7002ff47da897e9f2a9cb1b4e245396589e0b72affb73568a2016031555d2a46557919e44a15cd43fe9e1881d40dce1d1e36625e63b1472d3c317898102943072e06d79688c96b6ee2e584002c66497a9cdc48c38aa0548a7bc4fed9b4c23fcd493f38ece68788ef37a559b7f20c6941fcf8e567d9f50807259a7f11fa7a01d3125a1f7609cd94781f224ec8351605354b11c6b078fe015826342c3271ee3af4b99bb0a538b1e6b845594ee6546be8abd22ef2bd
\n
Once you understand the meaning of the question, you can make a judgment immediately —— if you can decrypt the ciphertext c and retrieve the plaintext FLAG, you can get the original content of flag.txt, that is, capture the flag.
\n
Conventional Solution
\n
RSA decryption requires a private key exponent d. Referring to the steps of the RSA algorithm below, it is obvious that this demands integer factorization for large prime numbers p and q first.
\n\n
Choose two large prime numbers \\(p\\) and \\(q\\), compute \\(n=pq\\)
\n
Compute Carmichael function\\(\\lambda(n)=\\operatorname{lcm}(p − 1, q − 1)\\) the product, \\(\\operatorname{lcm}\\) is a function to find the least common multiple
\n
Choose any number \\(e\\) that is less than and coprime to \\(\\lambda(n)\\), then compute \\(d\\), the modular multiplicative inverse of \\(e\\) regarding \\(\\lambda(n)\\), \\(d\\equiv e^{-1}\\pmod {\\lambda(n)}\\)
\n
\\((n,e)\\) is the RSA public key, \\((n,d)\\) the RSA private key
\n
Use the public key to encrypt the plaintext \\(m\\), the formula is \\(c\\equiv m^e\\pmod n\\)
\n
Use the private key to decrypt the ciphertext \\(c\\), the formula is \\(m\\equiv c^d\\pmod n\\)
\n\n
From here, the challenge becomes a problem that, knowing the sum and product of two large prime numbers known, find these two large prime numbers. That is, to solve a system of quadratic linear equations
We can get \\(p\\) and \\(q\\). The rest of the work is easy. The code to compute \\(d\\) from \\(p\\) and \\(q\\) can be copied directly from lines 28, 30, and 31 in gen.py. The final complete Python problem-solving code is as follows:
x = int((Lines[0].split())[2], 16) # x = p + q n = int((Lines[1].split())[2], 16) # n = p * q c = int((Lines[2].split())[2], 16) # Ciphertext
defsolve_rsa_primes(s: int, m: int) -> tuple: ''' Solve RSA prime numbers (p, q) from the quadratic equation p^2 - s * p + m = 0 with the formula p = s/2 +/- sqrt((s/2)^2 - m) Input: s - sum of primes, m - product of primes Output: (p, q) ''' half_s = s >> 1 tmp = math.isqrt(half_s ** 2 - m) returnint(half_s + tmp), int(half_s - tmp);
# Now run with the real input p, q = solve_rsa_primes(x, n) m = math.lcm(p - 1, q - 1) e = 65537 d = pow(e, -1, m) FLAG = pow(c, d, n) print(FLAG.to_bytes((FLAG.bit_length() + 7) // 8, 'big'))
\n
The above program defines a general function solve_rsa_primes to solve two large prime numbers. After it gets d, the same pow function is called to decrypt, and finally the plaintext is converted from a large integer to a byte sequence and printed out. The result of running this program is
Note: The function solve_rsa_primes calls math.isqrt to compute the integer square root of the given integer. This is indispensable! If it is written incorrectly with math.sqrt, the following overflow error will occur
\n
>>> =============== RESTART: /Users/zixi/Downloads/Sum-O-Primes.py ============== Traceback (most recent call last): File "/Users/zixi/Downloads/Sum-O-Primes.py", line 35, in <module> p, q = solve_rsa_primes(x, n) File "/Users/zixi/Downloads/Sum-O-Primes.py", line 31, in solve_rsa_primes tmp = math.sqrt(int(half_s ** 2 - m)) OverflowError: int too large to convert to float
\n
This error happens because math.sqrt uses floating-point arithmetic but fails to convert large integers to floating-point numbers.
\n
\n
Quick Solution
\n
The conventional solution to this problem has to solve a quadratic equation, so the integer square root operation is essential. Is there a solution that doesn't need a square root operation? The answer is yes.
\n
In the original RSA paper, the public exponent \\(e\\) and the private exponent \\(d\\) have the relationship as the following equation
\n
\\[d⋅e≡1\\pmod{\\varphi(n)}\\]
\n
Here the modular is the Euler's totient function \\(\\varphi(n)=(p-1)(q-1)\\). Since \\(\\varphi(N)\\) is always divisible by \\(\\lambda(n)\\), any d satisfying the above also satisfies \\(d⋅e≡1\\pmod{\\lambda(n)}\\), thus the private exponent is not unique. Although the calculated \\(d>\\lambda(n)\\), the square root operation can be avoided when applied to the Sum-O-Primes problem. This is because \\[\n\\begin{aligned}\n\\varphi(n)&=(p-1)(q-1)\\\\\n&=pq-(p+q)+1\\\\\n&=n-x+1\n\\end{aligned}\n\\]
\n
Hereby the formula for computing the private exponent becomes
Now that \\(n\\) and \\(x\\) are readily available, this method does not require finding \\(p\\) and \\(q\\) first, and naturally, there is no need for a square root operation. The Python code for this new solution is very concise
\n
d1 = pow(e, -1, n - x + 1) FLAG = pow(c, d1, n) print(FLAG.to_bytes((FLAG.bit_length() + 7) // 8, 'big'))
As shown above, this solution also succeeds in capturing the flag. The \\(d\\) value (d1) calculated by the new solution is more than 7 times that of the conventional solution.
\n","categories":["Technical Know-how"],"tags":["Cryptography","Python Programming","CTF"]},{"title":"TLS 1.3 and the Coming NIST Mandate","url":"/en/2023/08/21/TLS1-3-intro/","content":"
TLS (Transport Layer Security) is a cryptographic protocol to secure network communication. TLS 1.3 is the latest version of the TLS protocol, succeeding TLS 1.2. TLS 1.3 aims to provide more robust security, higher privacy protection, as well as better performance than previous versions. Here is a brief introduction to TLS 1.3. Also, we discuss NIST's requirement for TLS 1.3 readiness and give examples of enabling TLS 1.3 in some commonly used web servers.
\n
It takes 20 years to build a reputation and a few minutes of cyber-incident to ruin it. — Stéphane Nappo (Vice President and Global Chief Information Security Officer of Groupe SEB, France, 2018 Global CISO of the year)
\n
\n
Introduction to TLS 1.3
\n
TLS 1.3 is the latest recommended cryptographic protocol for protecting a wide variety of network communications, including web browsing, email, online trading, instant messaging, mobile payments, and many other applications. By using TLS 1.3, more secure and reliable communication connections can be established, ensuring confidentiality, authenticity, and data integrity. It was standardized by the Internet Engineering Task Force (IETF) in August 2018, and published as RFC 8446.
\n
TLS 1.3 introduces some important improvements over TLS 1.2. The table below presents a quick comparison of the two:
\n
\n
\n
\n
\n
\n
\n\n
\n
Aspect
\n
TLS 1.2
\n
TLS 1.3
\n
\n\n\n
\n
Protocol Design
\n
Request-response model
\n
Reduced round trips
\n
\n
\n
Handshake
\n
Multiple round trips
\n
Single round trip
\n
\n
\n
Cipher Suites
\n
Supports wide range, including insecure ones
\n
Focuses on stronger algorithms
\n
\n
\n
Security
\n
Known vulnerabilities, e.g., CBC vulnerabilities
\n
Addresses previous issues, stronger security
\n
\n
\n
Performance
\n
Higher latency due to more round trips
\n
Faster connection establishment
\n
\n
\n
Resilience to Attacks
\n
Vulnerable to downgrade attacks and padding oracle attacks
\n
Additional protections against attacks
\n
\n
\n
Compatibility
\n
Widely supported across platforms
\n
Increasing support, may not be available on older systems
\n
\n
\n
Implementation Supports
\n
Available in many cryptographic libraries
\n
Supported in various libraries
\n
\n\n
\n
It can be seen that enhanced security and performance improvements are the most notable features of TLS 1.3, and we can explore more into these in the following sections.
\n
Security Hardening
\n
Cipher Suites
\n
The protocol design principle of TLS 1.3 has enhanced security as its primary goal. As a result, TLS 1.3 drastically reduces the number of supported cipher suites. It removes insecure and weak cipher suites, leaving only more secure and modern cipher suites. This helps to increase the security of communications and avoids the use of outdated or vulnerable cipher suites.
\n
Specifically, TLS 1.3 removes various cipher suites that use static RSA key transport, static Diffie-Hellman key exchange, CBC mode of operation, or SHA-1. It adopts only a limited number of Authenticated Encryption with Associated Data (AEAD) cipher suites. AEAD can guarantee the confidentiality, integrity, and authenticity of data at the same time, and its high security makes it the exclusive choice for TLS 1.3.
\n
On the other hand, the name string of the cipher suite used in previous TLS versions included all algorithms for key exchange, digital signatures, encryption, and message authentication. Each cipher suite is assigned a 2-byte code point in the TLS Cipher Suites registry managed by the Internet Assigned Numbers Authority (IANA). Every time a new cryptographic algorithm is introduced, a series of new combinations need to be added to the list. This has led to an explosion of code points representing every valid choice of these parameters. This situation also makes the selection of cipher suites complicated and confusing.
\n
The design of TLS 1.3 changed the concept of the cipher suite. It separates the authentication and key exchange mechanisms from the record protection algorithm (including secret key length) and a hash to be used with both the key derivation function and handshake message authentication code (MAC). The new cipher suite naming convention is TLS_<AEAD>_<Hash>, where the hash algorithm is used for the newly defined key derivation function HKDF of TLS 1.3 and the MAC generation in the handshake phase. The cipher suites defined by the TLS 1.3 protocol are:
This simplified cipher suite definition and greatly reduced set of negotiation parameters also speed up TLS 1.3 handshake, improving overall performance.
\n
Key Exchange
\n
TLS 1.3 emphasizes forward secrecy, ensuring that the confidentiality of communications is protected even if long-term secrets used in the session key exchange are compromised. It only allows key exchange based on ephemeral Diffie-Hellman key exchange (DHE) or ephemeral elliptic curve Diffie-Hellman key exchange (ECDHE). Both have the property of forward secrecy. Also, the protocol explicitly restricts the use of secure elliptic curve groups and finite field groups for key exchange:
/* Finite Field Groups (DHE) */ ffdhe2048(0x0100), ffdhe3072(0x0101), ffdhe4096(0x0102), ffdhe6144(0x0103), ffdhe8192(0x0104),
\n
The above elliptic curve groups for ECDHE are specified by RFC 8422. The first three are defined by the FIPS.186-4 specification and the corresponding NIST names are P-256/P-384/P-512, while the next two (x25519/x448) are recommended by ANSI.X9-62.2005. RFC 7919 specifies four finite field groups (ffdhe####) for DHE. The primes in these finite field groups are all safe primes.
\n
In number theory, a prime number \\(p\\) is a safe prime if \\((p-1)/2\\) is also prime.
\n
\n
Signature Verification
\n
For signature verification in the key exchange phase, TLS 1.3 introduces more signature algorithms to meet different security requirements:
\n
\n
RSA signature algorithm: TLS 1.3 still supports RSA-based signature algorithms, including RSA-PKCS1-SHA256, RSA-PKCS1-SHA384, etc. These algorithms use RSA keys for digital signatures.
\n
ECDSA signature algorithm: TLS 1.3 introduces more signature algorithms based on elliptic curve cryptography (ECC), such as ECDSA-SHA256, ECDSA-SHA384, etc. These algorithms use elliptic curve keys for digital signatures and are generally superior to RSA in terms of security and performance.
\n
EdDSA signature algorithm: TLS 1.3 also introduces the EdDSA (Edwards-curve Digital Signature Algorithm) signature algorithm based on the Edwards curve. It features efficient performance and strong security for mobile devices and resource-constrained environments.
\n
RSASSA-PSS signature algorithm: In addition to the traditional RSA-PKCS1 signature algorithm, TLS 1.3 also introduces the RSASSA-PSS signature algorithm, which is a more secure signature method based on RSA and has better attack resistance.
\n
PSK signature algorithm: TLS 1.3 supports the signature algorithm based on the pre-shared key (PSK), which applies to the PSK handshake mode. This approach does not involve a digital certificate but uses a pre-shared key for verification.
\n
\n
TLS 1.3 stops using the DSA (Digital Signature Algorithm) signature algorithm. This is also a notable difference from TLS 1.2. DSA has some security and performance limitations and is rarely used in practice, so TLS 1.3 removed support for DSA certificates.
\n
Other Reinforcements
\n
Additionally, TLS 1.3 includes the following improvements to enhance security
\n
\n
TLS 1.3 does not allow data compression. The data compression feature in earlier versions of TLS could lead to security issues such as CRIME attacks. To avoid this risk, TLS 1.3 removed support for data compression entirely.
\n
Unlike earlier versions of TLS, TLS 1.3 prohibits renegotiation after the connection has been established. This helps reduce security risk and complexity. Renegotiation may introduce new security holes, and frequent negotiations during the connection process may also cause performance problems.
\n
All handshake messages following the ServerHello message during the TLS 1.3 handshake are now encrypted. The newly introduced EncryptedExtensions message enables encryption protection of various extensions previously sent in plain text.
\n
TLS 1.3 adds asymmetric cryptographic protection of the Certificate messages sent from the server to the client. This encryption prevents threats such as man-in-the-middle attacks, information leakage, and certificate forgery, further fortifying the security and privacy of the connection.
\n
\n
Performance Boosting
\n
Simplified Handshake
\n
The general trend towards high-speed mobile Internet requires the use of HTTPS/TLS to protect the privacy of all traffic as much as possible. The downside of this is that new connections can become a bit slower. For the client and web server to agree on a shared key, both parties need to exchange security attributes and related parameters through the TLS \"handshake process\". In TLS 1.2 and all protocols before it, the initial handshake process required at least two round-trip message transfers. Compared to pure HTTP, the extra latency introduced by the TLS handshake process of HTTPS can be very detrimental to performance-conscious applications.
\n
TLS 1.3 greatly simplifies the handshake process, requiring only one round trip in most cases, resulting in faster connection establishment and lower latency. Every TLS 1.3 connection will use (EC)DHE-based key exchange, and the parameters supported by the server may be easy to guess (such as ECDHE + x25519 or P-256). Since the options are limited, the client can directly send the (EC)DHE key share information in the first message without waiting for the server to confirm which key exchange it is willing to support. This way, the server can derive the shared secret one round in advance and send encrypted data.
\n
The following diagram compares the message sequences of the handshake process of TLS 1.2 and TLS 1.3. Both operate with public key-based authentication. The TLS 1.3 handshake shown below uses the symbols borrowed from the RFC 8446 specification: '+' indicates a noteworthy extension; '*' indicates an optional message or extension; '[]', '()', and '{}' represent encrypted messages, where the keys used for encryption are different.
\n\nTLS 1.2 handshake (left) vs. TLS 1.3 handshake (right)\n\n
This figure illustrates the following points:
\n
\n
TLS 1.3 removes several messages used by TLS 1.2: ServerHelloDone, ChangeCipherSpec, ServerKeyExchange, and ClientKeyExchange. The contents of TLS 1.2's ServerKeyExchange and ClientKeyExchange messages vary depending on the authentication and key-sharing method being negotiated. In TLS 1.3, this information was moved to the extensions of ClientHello and ServerHello messages. TLS 1.3 completely deprecates ServerHelloDone and ChangeCipherSpec messages, there is no replacement.
\n
For TLS 1.3 the public key-based authentication mode is probably the most important. It always uses (EC)DHE to achieve forward secrecy. The figure shows that the ClientHello message carries four extensions that are must-haves in this mode: key_share, signature_algorithms, supported_groups, and support_versions.
\n
During the TLS 1.2 handshake, the exchange of control data requires multiple round trips between the client and server. TLS 1.2's ClientKeyExchange and ChangeCipherSpec messages are carried in separate packets, and the Finished message is the first (and only) encrypted handshake message. The whole process needs to transmit 5-7 data packets.
\n
During the TLS 1.3 handshake, encrypted Application Data is already sent by the client after the first round trip. As mentioned earlier, the EncryptedExtension message provides privacy protection for ServerHello extensions in earlier versions of TLS. If mutual authentication is required (which is common in IoT deployments), the server will send a CertificateRequest message.
\n
The Certificate, CertificateVerify, and Finished messages in TLS 1.3 retain the semantics of earlier TLS versions, but they are all asymmetrically encrypted now. Echoing the description in the last section, by encrypting Certificate and CertificateVerify messages, TLS 1.3 better protects against man-in-the-middle and certificate forgery attacks while enhancing the privacy of connections. This is also an important security feature in the design of TLS 1.3.
\n
\n
In rare cases, when the server does not support a certain key-sharing method sent by the client, the server can send a new HelloRetryRequest message letting the client know which groups it supports. As the group list has shrunk significantly, this is not expected to happen very often.
\n
0-RTT Session Resumption
\n
0-RTT (Zero Round Trip Time) in TLS 1.3 is a special handshake mode. It allows clients to send encrypted data during the handshake phase, reducing the number of round trips required for connection establishment and enabling faster session resumption. The following is a brief explanation of the 0-RTT working mode:
\n\n
Store session tickets: During the normal TLS 1.3 handshake, the client and server generate a data structure called a \"session ticket\" during the handshake. Session tickets contain information about the connection, including key parameters and cipher suites. The server stores the session ticket provided by the client.
\n
0-RTT handshake: When the client reconnects to the server, it includes the previously saved session ticket in the early_data extension of the ClientHello message, along with encrypted Application Data. The client encrypts 0-RTT data using a pre-shared key (PSK) obtained from a previous connection.
\n
Server Response: After the server receives this message, if it supports 0-RTT mode and can recognize and verify the session ticket, it sends an EncryptedExtensions message, and then confirms the connection in the Finished message. This way, the server can quickly establish a secure connection with 0 round trips. It can also immediately send data to the client to achieve 0-RTT data transmission.
\n\n
The message sequence of the 0-RTT session resumption and data transmission process of TLS 1.3 is as follows:
\n\nTLS 1.3 0-RTT\n\n
FAQ
\n
\n
Does the TLS 1.3 protocol allow the use of RSA digital certificates?
\n
A common misconception is that \"TLS 1.3 is not compatible with RSA digital certificates\". The description in the \"Signature Verification\" section above shows that this is wrong. TLS 1.3 still supports the use of RSA for key exchange and authentication. However, considering the limitations of RSA, it is recommended that when building and deploying new TLS 1.3 applications, ECDHE key exchange algorithms and ECC digital certificates are preferred to achieve higher security and performance.
\n
During the TLS 1.3 handshake, how does the server request the client to provide a certificate?
\n
In some scenarios, the server also needs to verify the identity of the client to ensure that only legitimate clients can access server resources. This is the case with mTLS (mutual TLS). During the TLS 1.3 handshake, the server can specify that the client is required to provide a certificate by sending a special CertificateRequest extension. When the server decides to ask the client for a certificate, it sends a CertificateRequest extension message after the ServerHello message. This extended message contains some necessary parameters, such as a list of supported certificate types, a list of acceptable certificate authorities, and so on. When the client receives it, it knows that the server asked it for a certificate, and it can optionally respond to the request. If the client is also configured to support mTLS and decides to provide a certificate, it provides its certificate chain by sending a Certificate message.
\n
Is 0-RTT vulnerable to replay attacks?
\n
TLS 1.3's 0-RTT session resumption mode is non-interactive and does risk replay attacks in some cases. An attacker may repeat previously sent data to simulate a legitimate request. To avoid and reduce the risk of replay attacks to the greatest extent, TLS 1.3 provides some protection measures and suggestions:
\n\n
The simplest anti-replay method is that the server only allows each session ticket to be used once. For example, the server may maintain a database of all valid tickets that have not been used, deleting each ticket from the database as it is used. If an unknown ticket is received, the server falls back to a full handshake.
\n
The server may limit the time window in which session tickets are accepted, that is, the time range in which 0-RTT data is allowed to be valid. This reduces the chance of an attacker successfully replaying.
\n
Clients and servers should also use 0-RTT data only for stateless requests, that is, requests that do not affect the state of the server such as HTTP GET. For requests that need to modify the state of the server or have an impact, restrict the use of normal handshake patterns only.
\n
Another way to prevent replay is to store the unique value (usually a random value or a PSK bundled value) derived from the ClientHello message, and reject duplicates. Logging all ClientHellos would cause the state to grow without bound, but combined with #2 above, the server can log ClientHellos within a given time window and use obfuscated_ticket_age to ensure that tickets are not duplicated outside the window use.
\n
\n
If the client does not know whether the server supports TLS 1.3, how could it negotiate the TLS version via handshake?
\n
The TLS protocol provides a built-in mechanism for negotiating the running version between endpoints. TLS 1.3 continues this tradition. RFC 8446 Appendix D.1 \"Negotiating with an Older Server\" gives specific instructions:
\n
\n
A TLS 1.3 client who wishes to negotiate with servers that do not support TLS 1.3 will send a normal TLS 1.3 ClientHello containing 0x0303 (TLS 1.2) in ClientHello.legacy_version but with the correct version(s) in the \"supported_versions\" extension. If the server does not support TLS 1.3, it will respond with a ServerHello containing an older version number. If the client agrees to use this version, the negotiation will proceed as appropriate for the negotiated protocol.
\n
\n
The following screenshot of a TLS 1.3 ClientHello message decode demonstrates this. The version number of the handshake message displayed on the left is \"Version: TLS 1.2 (0x0303)\". At the same time, it can be seen that the cipher suite section first lists 3 TLS 1.3 AEAD cipher suites, followed by 14 TLS 1.2 regular cipher suites. On the right, there are 4 extensions - key_share, signature_algorithms, supported_groups, and support_versions. The support_versions extension includes both TLS 1.3 and TLS 1.2 version numbers. This is the TLS version list for the server to choose from. Additionally, the key_share extension includes the client's preferred key-sharing method as x25519 and secp256r1(i.e. NIST P-256)
\n
\n
Does the TLS 1.3 protocol work with UDP and EAP?
\n
TLS was originally designed for TCP connections, and a variant DTLS (Datagram Transport Layer Security) for UDP was introduced later. Based on TLS 1.3, IETF has released the corresponding upgraded version of the DTLS 1.3 protocol RFC 9147. The design goal of DTLS 1.3 is to provide \"equivalent security guarantees with the exception of order protection / non-replayability\". This protocol was released in April 2022, and currently, there are not many software libraries supporting it.
\n
TLS can also be used as an authentication and encryption protocol in various EAP types, such as EAP-TLS, EAP-FAST, and PEAP. Corresponding to TLS 1.3, IETF also published two technical standard documents:
\n
\n
RFC 9190: EAP-TLS 1.3: Using the Extensible Authentication Protocol with TLS 1.3 (Feb. 2022)
\n
RFC 9427: TLS-Based Extensible Authentication Protocol (EAP) Types for Use with TLS 1.3 (Jun. 2023)
\n
\n
Both protocols are also quite new, and the software library updates supporting them are still some time away.
\n
\n
NIST Mandate
\n
TLS 1.3 brings new security features and a faster TLS handshake. Since its release in 2018, many Internet services have migrated to this latest version. Nevertheless, widespread adoption across websites takes time. The non-commercial SSL Labs Projects has a dashboard called SSL Pulse that reports TLS/SSL security scan statistics for the most popular Internet sites. Below is the most recent chart of protocol support statistics by July 2023.
\n\nSource: SSL Pulse - 07/03/2023\n\n
As can be seen, of all 135,000+ probed sites the percentage of TLS 1.3 support is about 63.5%. That means there are still close to 50 thousand sites that do not leverage the security and performance benefits of TLS 1.3. Why? The decision to migrate a website to a new protocol version like TLS 1.3 can be complex and influenced by various factors. The top 3 common reasons hindering TLS 1.3 migration are
\n
\n
Compatibility Concerns: Some websites might have users who are still using outdated browsers or operating systems that do not support TLS 1.3. These websites need to maintain backward compatibility to ensure that all users can access their content securely.
\n
Resource Constraints: Migration involves technical updates, configuration changes, and testing. Smaller websites or those with limited resources might face challenges in allocating the necessary time and effort to make these changes.
\n
Third-Party Dependencies: Many websites rely on third-party services, content delivery networks, or other components. If these services do not yet support TLS 1.3, the website might delay migration to avoid disruptions or compatibility issues with these dependencies.
\n
\n
However, for network hardware/software vendors who want their products on the procurement list of any US public sector organization, there is a coming NIST mandate to make TLS 1.3 available by January 2024. This is stipulated in the National Institute of Standards and Technology Special Publication (NIST SP) 800-52 Rev. 2: Guidelines for the Selection, Configuration, and Use of Transport Layer Security (TLS) Implementations. Quoted from NIST SP 800-52 Rev. 2
\n
\n
3.1 Protocol Version Support
\n
Servers that support government-only applications shall be configured to use TLS 1.2 and should be configured to use TLS 1.3 as well. ...
\n
Servers that support citizen or business-facing applications (i.e., the client may not be part of a government IT system)10 shall be configured to negotiate TLS 1.2 and should be configured to negotiate TLS 1.3. ...
\n
Agencies shall support TLS 1.3 by January 1, 2024. After this date, servers shall support TLS 1.3 for both government-only and citizen or business-facing applications. In general, servers that support TLS 1.3 should be configured to use TLS 1.2 as well. However, TLS 1.2 may be disabled on servers that support TLS 1.3 if it has been determined that TLS 1.2 is not needed for interoperability.
\n
\n
As in the RFC documents, \"shall\" above is a strong keyword that means that the definition is an absolute requirement of the specification. So this NIST publication requires all servers owned by the US government agencies to be able to support TLS 1.3 by 01/01/2024. They must run a minimum TLS version 1.2 by default and can be configured to do TLS 1.3 only if desired.
\n
It is worth pointing out that this is not an official FIPS requirement, so not mandatory for the FIPS 140-3 certification at present. Besides, this NIPS document has a clear scope statement: \"The scope is further limited to TLS when used in conjunction with TCP/IP. For example, Datagram TLS (DTLS), which operates over datagram protocols, is outside the scope of these guidelines. NIST may issue separate guidelines for DTLS at a later date.\" Based on this, we can infer that DTLS and EAP are out of consideration for this mandate.
\n
Enabling TLS 1.3
\n
The enhanced security and optimized performance of TLS 1.3 make it the first choice for securing communication of various network applications. Now we demonstrate how to enable TLS 1.3 function in three commonly used web server software Apache, Nginx, and Lighttpd.
\n
NOTE: The implementation of many secure network communication applications relies on third-party SSL/TLS software libraries, such as wolfSSL, GnuTLS, NSS, and OpenSSL. Therefore, to enable the TLS 1.3 function of these applications, you need to ensure that the libraries they link with support TLS 1.3. For example, in September 2018, the popular OpenSSL project released version 1.1.1 of the library, with support for TLS 1.3 as its \"top new feature\".
\n
\n
Apache HTTP Server
\n
The Apache HTTP Server is an open-source web server software from the Apache Software Foundation. Apache HTTP server is widely used and is one of the most popular web server software due to its cross-platform and security. Apache supports a variety of features, many of which extend core functionality through compiled modules, such as authentication schemes, proxy servers, URL rewriting, SSL/TLS support, and compiling interpreters such as Perl/Python into the server.
\n
Apache HTTP Server has built-in support for TLS 1.3 since version 2.4.36, no need to install any additional modules or patches. The following command can be used to verify the version of the server
\n
$ apache2ctl -v Server version: Apache/2.4.41 (Ubuntu) Server built: 2020-04-13T17:19:17
\n
Once the version is verified, the SSLProtocol line of the configuration file can be updated. The following will enable the Apache HTTP server to only support the TLS 1.3 protocol
\n
/etc/apache2/mods-available/ssl.conf
# Only enable TLS 1.3 SSLProtocol -all +TLSv1.3
\n
If the server needs to be compatible with clients that support TLS 1.2, you can add +TLSv1.2. After updating the configuration, restart the service
\n
$ sudo service apache2 restart
\n
Nginx Web Server
\n
Nginx is a high-performance web server based on an asynchronous framework and modular design. It can also be used for reverse proxy, load balancer, and HTTP caching applications. It is free and open-source software released under the terms of a BSD-like license. Nginx uses an asynchronous event-driven approach to request processing, which can provide more predictable performance under high load. The current market share of Nginx is almost equal to that of the Apache HTTP server.
\n
Nginx supports TLS 1.3 from version 1.13.0. The following command can be used to verify its version
\n
$ nginx -v nginx version: nginx/1.17.10 (Ubuntu)
\n
In the Nginx configuration file, find the server block and modify the ssl_protocols line to enable TLS 1.3:
\n
/etc/nginx/nginx.conf
server { listen443 ssl http2; listen [::]:443 ssl http2; server_name example.com; root /var/www/example.com/public; ssl_certificate /path/to/your/certificate.crt; ssl_certificate_key /path/to/your/private-key.key; # support TLS 1.2 and TLS 1.3 ssl_protocols TLSv1.2 TLSv1.3; ... }
\n
If you don't need to continue to support TLS 1.2, delete the TLSv1.2 there. After the modification is complete, you can run the following command to test the configuration of Nginx, and then restart the service
\n
$ sudo nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful
$ sudo service nginx restart
\n
Lighttpd Web Server
\n
Lighttpd is a lightweight open-source web server software. It focuses on high performance, low memory footprint, and fast responsiveness. Lighttpd is suitable for serving web applications and static content of all sizes. Its design goal is to provide an efficient, flexible, and scalable web server, especially suitable for high-load and resource-constrained (such as embedded systems) environments.
\n
The first Lighttpd release to support TLS 1.3 is version 1.4.56. Starting with this version, the minimum version of TLS that Lighttpd supports by default is TLS 1.2. That is to say, Lighttpd supports TLS 1.2 and TLS 1.3 if no corresponding configuration file modification is made.
\n
To limit the use of Lighttpd to only the TLS 1.3 feature, first make sure the mod_openssl module is loaded. Then in the configuration file lighttpd.conf, find the server.modules section, and add the following ssl.openssl.ssl-conf-cmd line:
This will set the minimum version supported by Lighttpd to be TLS 1.3. Finally, save and reload the Lighttpd configuration for the changes to take effect:
\n","categories":["Technical Know-how"],"tags":["C/C++ Programming","Cryptography","Network Security"]},{"title":"Notes on Using uClibc Standard Library in Embedded Linux System","url":"/en/2023/03/10/uClibc-tips/","content":"
uClibc is a small and exquisite C standard library for embedded Linux systems. It is widely used in the development of low-end embedded systems and Internet of Things devices. Here are some recent experiences to provide convenience for engineers who need to solve similar problems or meet corresponding requirements.
\n
Low-level programming is good for the programmer's soul. — John Carmack (American computer programmer and video game developer, co-founder of the video game company id Software)
\n
\n
Introduction to uClibc
\n
uClibc (sometimes written as μClibc) is a small C standard library designed to provide support for embedded systems and mobile devices using operating systems based on the Linux kernel. uClibc was originally developed to support μClinux, a version of Linux not requiring a memory management unit thus especially suited for microcontroller systems. The \"uC\" in its name is the abbreviation of microcontroller in English, where \"u\" is a Latin script typographical approximation of the Greek letter μ that stands for \"micro\".
\n
uClibc is a free and open-source software licensed under the GNU Lesser GPL, and its library functions encapsulate the system calls of the Linux kernel. It can run on standard or MMU-less Linux systems and supports many processors such as i386, x86-64, ARM, MIPS, and PowerPC. Development of uClibc started in 1999 and was written mostly from scratch, but also absorbed code from glibc and other projects. uClibc is much smaller than glibc. While glibc aims to fully support all relevant C standards on a wide range of hardware and kernel platforms, uClibc focuses on embedded Linux systems. It also allows developers to enable or disable some features according to the memory space design requirements.
\n
The following records show the list of C standard library files in two similar embedded systems. The first uses glibc-2.23 version, and the second integrates uClibc-0.9.33.2 version. The total size of glibc library files is more than 2MB, while the uClibc library files add up to less than 1MB. It can be seen that using uClibc does save a lot of storage space.
\n
STM1:/# find . -name "*lib*2.23*" | xargs ls -alh -rwxr-xr-x 1 root root 9.6K Jan 1 1970 ./lib/libanl-2.23.so -rwxr-xr-x 1 root root 1.1M Jan 1 1970 ./lib/libc-2.23.so -rwxr-xr-x 1 root root 177.5K Jan 1 1970 ./lib/libcidn-2.23.so -rwxr-xr-x 1 root root 29.5K Jan 1 1970 ./lib/libcrypt-2.23.so -rwxr-xr-x 1 root root 9.5K Jan 1 1970 ./lib/libdl-2.23.so -rwxr-xr-x 1 root root 429.4K Jan 1 1970 ./lib/libm-2.23.so -rwxr-xr-x 1 root root 65.8K Jan 1 1970 ./lib/libnsl-2.23.so -rwxr-xr-x 1 root root 17.5K Jan 1 1970 ./lib/libnss_dns-2.23.so -rwxr-xr-x 1 root root 33.6K Jan 1 1970 ./lib/libnss_files-2.23.so -rwxr-xr-x 1 root root 90.5K Jan 1 1970 ./lib/libpthread-2.23.so -rwxr-xr-x 1 root root 65.7K Jan 1 1970 ./lib/libresolv-2.23.so -rwxr-xr-x 1 root root 25.9K Jan 1 1970 ./lib/librt-2.23.so -rwxr-xr-x 1 root root 9.5K Jan 1 1970 ./lib/libutil-2.23.so
STM2:/# find . -name "*lib*0.9.33*" | xargs ls -alh -rwxr-xr-x 1 root root 28.0K Jan 1 1970 ./lib/ld-uClibc-0.9.33.2.so -rwxr-xr-x 1 root root 36.1K Jan 1 1970 ./lib/libcrypt-0.9.33.2.so -rwxr-xr-x 1 root root 16.2K Jan 1 1970 ./lib/libdl-0.9.33.2.so -rwxr-xr-x 1 root root 72.1K Jan 1 1970 ./lib/libm-0.9.33.2.so -rwxr-xr-x 1 root root 116.4K Jan 1 1970 ./lib/libpthread-0.9.33.2.so -rwxr-xr-x 1 root root 16.2K Jan 1 1970 ./lib/librt-0.9.33.2.so -rwxr-xr-x 1 root root 28.3K Jan 1 1970 ./lib/libthread_db-0.9.33.2.so -rwxr-xr-x 1 root root 621.4K Jan 1 1970 ./lib/libuClibc-0.9.33.2.so -rwxr-xr-x 1 root root 8.1K Jan 1 1970 ./lib/libubacktrace-0.9.33.2.so -rwxr-xr-x 1 root root 4.1K Jan 1 1970 ./lib/libutil-0.9.33.2.so
\n
IPv6 and Interface API
\n
With the steady growth of IPv6 deployment, adding IPv6 protocol stack support for embedded systems has become necessary. In a software project that adds IPv4/IPv6 dual-stack function to devices using uClibc, it is found that there is an application link error - undefined reference to getifaddrs. getifaddrs() is a very useful function, we can call it to get the address information of all the network interfaces of the system. Query the Linux programming manual:
intgetifaddrs(struct ifaddrs **ifap); ... \t DESCRIPTION The getifaddrs() function creates a linked list of structures describing the network interfaces of the local system, and stores the address of the first item of the list in *ifap. ... VERSIONS The getifaddrs() function first appeared in glibc 2.3, but before glibc 2.3.3, the implementation supported only IPv4 addresses; IPv6 support was added in glibc 2.3.3. Support of address families other than IPv4 is available only on kernels that support netlink. ...
\n
The last sentence above is key: only kernels supporting netlink can support address families other than IPv4. The Linux kernel version running on this system is 3.x, which supports netlink. So, could there be a problem with uClibc's support for netlink that causes getifaddrs() not to get compiled?
\n
With this question in mind, search the source code directory of uClibc and find the C file that implements the function getifaddrs():
\n
libc/inet/ifaddrs.c
... #if __ASSUME_NETLINK_SUPPORT #ifdef __UCLIBC_SUPPORT_AI_ADDRCONFIG__ /* struct to hold the data for one ifaddrs entry, so we can allocate everything at once. */ structifaddrs_storage { structifaddrsifa; union { /* Save space for the biggest of the four used sockaddr types and avoid a lot of casts. */ structsockaddrsa; structsockaddr_llsl; structsockaddr_ins4; #ifdef __UCLIBC_HAS_IPV6__ structsockaddr_in6s6; #endif } addr, netmask, broadaddr; char name[IF_NAMESIZE + 1]; }; #endif/* __UCLIBC_SUPPORT_AI_ADDRCONFIG__ */ ... #ifdef __UCLIBC_SUPPORT_AI_ADDRCONFIG__ ... int getifaddrs(struct ifaddrs **ifap) ... #endif/* __UCLIBC_SUPPORT_AI_ADDRCONFIG__ */ ... #endif/* __ASSUME_NETLINK_SUPPORT */
\n
Just as expected! The implementation of the entire function and the definition of the associated data structure ifaddrs_storageare are placed under three nested conditional compilation directives with macros defined as
\n\n
__ASSUME_NETLINK_SUPPORT
\n
__UCLIBC_SUPPORT_AI_ADDRCONFIG__
\n
__UCLIBC_HAS_IPV6__
\n\n
Therefore, as long as their corresponding configuration lines are opened, the problem should be solved. After changing the configuration file of uClibc as follows, rebuild the dynamic link library of uClibc, then the application can be made successfully:
\n
--- a/toolchain/uClibc/config-0.9.33.2/common +++ b/toolchain/uClibc/config-0.9.33.2/common @@ -147,7 +147,8 @@ UCLIBC_HAS_RPC=y UCLIBC_HAS_FULL_RPC=y -# UCLIBC_HAS_IPV6 is not set +UCLIBC_HAS_IPV6=y -# UCLIBC_USE_NETLINK is not set +UCLIBC_USE_NETLINK=y +UCLIBC_SUPPORT_AI_ADDRCONFIG=y UCLIBC_HAS_BSD_RES_CLOSE=y
\n
SHA-2 Hash Function
\n
Embedded systems often need to provide remote SSH login services for system administrators, which requires the creation of system users and their passwords. Linux saves the user name and the hashed password in the /etc/shadow file. The storage format of the hash value follows a de facto standard called the Modular Crypt Format (MCF for short), and its format is as follows:
id: indicates the identifier of the hash algorithm (eg 1 for MD5, 5 for SHA-256, 6 for SHA-512)
\n
param=value: Hash complexity parameters (such as the number of rounds/iterations) and their values
\n
salt: radix-64 (charset [+/a-zA-Z0-9]) encoded salt
\n
hash: the radix-64 encoded hash result of the password and salt
\n
\n
With the rapid increase of computing power following Moore's Law, the previously commonly used MD5-based hashing scheme has become obsolete because it is too vulnerable to attack. Newly designed systems are now switched to the SHA-512 hashing scheme, corresponding to $6$ seen in the /etc/shadow file.
\n
Both generation and verification of user password hash values can be implemented with the POSIX C library function named crypt. This function is defined as follows:
\n
char *crypt(constchar *key, constchar *salt)
\n
The input parameter key points to the string containing the user's password, and salt points to a string in the format $<id>$<salt> indicating the hash algorithm and salt to be used. Most Linux distributions use the crypt function provided by the glibc library. The following figure summarizes the augmented crypt function in Glibc:
\n
\n
In an embedded Linux system integrating uClibc, uClibc provides support for the crypt function. But the test found that it returned a null pointer for the correct \\(6\\) input! What's going on here?
\n
The answer lies in the uClibc's implementation of the crypt function. Find the corresponding C source code:
if (salt[0] == '$') { if (salt[1] && salt[2] == '$') { /* no blowfish '2X' here ATM */ if (*++salt == '1') return __md5_crypt(ukey, usalt); #ifdef __UCLIBC_HAS_SHA256_CRYPT_IMPL__ elseif (*salt == '5') return __sha256_crypt(ukey, usalt); #endif #ifdef __UCLIBC_HAS_SHA512_CRYPT_IMPL__ elseif (*salt == '6') return __sha512_crypt(ukey, usalt); #endif } /* __set_errno(EINVAL);*//* ENOSYS might be misleading */ returnNULL; } return __des_crypt(ukey, usalt); }
\n
Aha! It turns out that it only does MD5 hashing by default, and the codes of SHA-256 and SHA-512 need their own conditional compilation macro definitions. This is easy to handle, just edit the configuration file of uClibc and open the latter two.
\n
--- a/toolchain/uClibc/config-0.9.33.2/common +++ b/toolchain/uClibc/config-0.9.33.2/common @@ -151,8 +151,8 @@ UCLIBC_HAS_REGEX_OLD=y UCLIBC_HAS_RESOLVER_SUPPORT=y -# UCLIBC_HAS_SHA256_CRYPT_IMPL is not set -# UCLIBC_HAS_SHA512_CRYPT_IMPL is not set +UCLIBC_HAS_SHA256_CRYPT_IMPL=y +UCLIBC_HAS_SHA512_CRYPT_IMPL=y UCLIBC_HAS_SHADOW=y
\n
Finally, take a look at the program that comes with uClibc to test the SHA-512 hash algorithm. It clearly lists the data structures defined by the test code, including the salt, the input password, and the expected output, as well as several test vectors:
\n
test/crypt/sha512c-test.c
staticconststruct { constchar *salt; constchar *input; constchar *expected; } tests[] = { { "$6$saltstring", "Hello world!", "$6$saltstring$svn8UoSVapNtMuq1ukKS4tPQd8iKwSMHWjl/O817G3uBnIFNjnQJu" "esI68u4OTLiBFdcbYEdFCoEOfaS35inz1" }, { "$6$rounds=10000$saltstringsaltstring", "Hello world!", "$6$rounds=10000$saltstringsaltst$OW1/O6BYHV6BcXZu8QVeXbDWra3Oeqh0sb" "HbbMCVNSnCM/UrjmM0Dp8vOuZeHBy/YTBmSK6H9qs/y3RnOaw5v." }, ... { "$6$rounds=10$roundstoolow", "the minimum number is still observed", "$6$rounds=1000$roundstoolow$kUMsbe306n21p9R.FRkW3IGn.S9NPN0x50YhH1x" "hLsPuWGsUSklZt58jaTfF4ZEQpyUNGc0dqbpBYYBaHHrsX." }, };
\n
It can be seen that the last test case defines the round value 10 ($6$rounds=10$roundstoolow), while the output shows that the round is 1000 (rounds=1000). This confirms that the crypt function implementation of uClibc matches the augmented function of Glibc - in order to ensure security, if the input specified round is too small, crypt will automatically set to the minimum round of 1000.
\n
DNS Security Patch
\n
In early May 2022, Nozomi Networks, a company focused on providing security solutions for industrial and critical infrastructure environments, released a newly discovered uClibc security vulnerability CVE-2022-30295. This vulnerability exists in the Domain Name System (DNS) implementation of all versions of uClibc and its fork uClibc-ng (prior to version 1.0.41). Since the implementation uses predictable transaction IDs when making DNS requests, there is a risk of DNS cache poisoning attacks.
\n
Specifically, applications often call gethostbyname library functions to resolve a network address for a given hostname. uClibc/uClibc-ng internally implements a __dns_lookup function for the actual DNS domain name request and response processing. Taking the last version 0.9.33.2 of uClibc as an example, the screenshot below shows the problematic code in the function __dns_lookup:
\n
\n
Referring to line 1308, at the first DNS request, the variable local_id is initialized to the transaction ID value of the last DNS request (stored in a static variable last_id). Line 1319 is the actual culprit, it simply updates the old local_id value by incrementing it by 1. This new value is stored back into the variable last_id, as shown on line 1322. Finally, on line 1334, the value of local_id is copied into the structure variable h, which represents the actual content of the DNS request header. This code works pretty much in all available versions of uClibc and uClibc-ng prior to version 1.0.41.
\n
This implementation makes the transaction ID in the DNS request predictable, because the attacker can estimate the value of the transaction ID in the next request as long as he/she detects the current transaction ID. By exploiting this vulnerability, an attacker can disrupt/poison the host's DNS cache by crafting a DNS response containing the correct source port and winning the competition with the legitimate response returned by the DNS server, making the network data of the application in the host system be directed to a trap site set by the attacker.
\n
The maintainers of uClibc-ng responded quickly to the announcement of this security vulnerability. They submitted a fix in mid-May 2022, and released version 1.0.41 including this patch at the end of that month. For uClibc, since this C standard library has stopped releasing any new versions since 2012, it is currently in an unmaintained state, so system R&D engineers need to come up with their repair. The following uClibc patches are available for reference:
\n
diff --git a/libc/inet/resolv.c b/libc/inet/resolv.c index 31e63810b..c2a8e2be4 100644 --- a/libc/inet/resolv.c +++ b/libc/inet/resolv.c @@ -315,6 +315,7 @@ Domain name in a message can be represented as either: #include <sys/utsname.h> #include <sys/un.h> #include <sys/stat.h> +#include <fcntl.h> #include <sys/param.h> #include <bits/uClibc_mutex.h> #include "internal/parse_config.h" @@ -1212,6 +1213,20 @@ static int __decode_answer(const unsigned char *message, /* packet */ return i + RRFIXEDSZ + a->rdlength; }
+uint16_t dnsrand_next(int urand_fd, int def_value) { + if (urand_fd == -1) return def_value; + uint16_t val; + if(read(urand_fd, &val, 2) != 2) return def_value; + return val; +} + +int dnsrand_setup(int *urand_fd, int def_value) { + if (*urand_fd > 0) return dnsrand_next(*urand_fd, def_value); + *urand_fd = open("/dev/urandom", O_RDONLY); + if (*urand_fd == -1) return def_value; + return dnsrand_next(*urand_fd, def_value); +} + /* On entry: * a.buf(len) = auxiliary buffer for IP addresses after first one * a.add_count = how many additional addresses are there already @@ -1237,6 +1252,7 @@ int __dns_lookup(const char *name, /* Protected by __resolv_lock: */ static int last_ns_num = 0; static uint16_t last_id = 1; + static int urand_fd = -1;
int i, j, fd, rc; int packet_len; @@ -1305,7 +1321,7 @@ int __dns_lookup(const char *name, } /* first time? pick starting server etc */ if (local_ns_num < 0) { - local_id = last_id; + local_id = dnsrand_setup(&urand_fd, last_id); /*TODO: implement /etc/resolv.conf's "options rotate" (a.k.a. RES_ROTATE bit in _res.options) local_ns_num = 0; @@ -1316,8 +1332,9 @@ int __dns_lookup(const char *name, retries_left--; if (local_ns_num >= __nameservers) local_ns_num = 0; - local_id++; + local_id = dnsrand_next(urand_fd, local_id++); local_id &= 0xffff; + DPRINTF("local_id:0x%hx\\n", local_id); /* write new values back while still under lock */ last_id = local_id; last_ns_num = local_ns_num;
\n
This uClibc patch is a simplified version of the uClibc-ng official patch. Its core is to read a double-byte random number from the system /dev/urandom file, and then use it to set the original local_id, the transaction ID of the DNS request. /dev/urandom is a special device file of the Linux system. It can be used as a non-blocking random number generator, which will reuse the data in the entropy pool to generate pseudo-random data.
\n
Note that in the above patch, the function dnsrand_setup must first check urand_fd whether it is positive, and only open /dev/urandom when it is not true. Otherwise, the file will be reopened every time the application does a DNS lookup, the system will quickly hit the maximum number of file descriptors allowed, and the system will crash because it cannot open any more files.
\n
Finally, a comparison of an embedded system using uClibc before and after adding DNS security patches is given. The following are the DNS packets intercepted by two sniffers. In the first unpatched system, the transaction ID of the DNS request is incremented in sequence, which is an obvious security hole; the second is after the patch is added, the transaction ID of each DNS request is a random value, and the loophole has been filled.
The three key components in the second row, from left to right, are
Dividing the 
![](\"dtls-server-hello.png\")
\n![](\"dtls-server-key.png\")
\n![](\"dtls-client-key.png\")
\n![](\"OMV-Storage-Disks-en.png\")
![](\"OMV-Storage-FS-en.png\")
\n![](\"OMV-Storage-SharedFolders-en.png\")
Notice the orange alert at the top of the figure above, which alerts the administrator that the configurations have changed and must click on the ✔️ icon to take effect. \n![](\"OMV-User-authorize-en.png\")
As shown in the figure, for this new user zixi, the administrator can set the read and write access permissions for each shared folder. \n![](\"OMV-SMB-Shares-en.png\")
\n![](\"macOS-SMB.png\")
\n![](\"TLS1_3-clienthello.jpg\")
\n![](\"asan-mm.png\")
![](\"asan-redzone.png\")
![](\"acm-turing-2015.jpeg\")
![](\"acm-turing-2002.jpeg\")
![](\"ipv6-addr-format.png\")
![](\"ipv6-addr-alloc.png\")
Because an IPv6 multicast address can only be used as a destination address, its bit definition is different from that of unicast. Referring to RFC 4291, a multicast address containing 4 bits of the feature flags, 4 bits of the group scope, and the last 112 bits of the group identifier: ![](\"ipv6-multicast-addr.png\")
Furthermore the same protocol specifies a few pre-defined IPv6 multicast addresses, the most important of which are![](\"finding-prime-en.jpg\")
![](\"RPi-4B.png\")
![](\"RPi-Imager-advopt.png\")
Here is exactly the option we need to enable SSH on boot up - Enable SSH. It also allows the user to pre-set a password for the default username ![](\"NASPi-unbox.jpg\")
The three key components in the second row, from left to right, are![](\"NASPi-internal.jpg\")
![](\"NASPi-outside.jpg\")
![](\"OMV-login-default.png\")
After logging in with the default username and password shown above, you will see the Workbench screen. The first thing you should do at this point is to click the ⚙️ icon in the top right corner to bring up the settings menu and click \"Change Password\". You can also change the language here ![](\"OMV-start-en.png\")
Clicking on \"Dashboard\" in the settings menu allows you to select the relevant components to be enabled. The menu on the left side provides task navigation for administrators and can be hidden when not needed. The complete OMV administration manual can be found in the online documentation![](\"OMV-Dashboard-en.png\")
![](\"Plex-notice.png\")
We can sign in with a Google, Facebook, or Apple account, or we can enter an email to create a new account. Follow the instructions on the page step by step, no need for any payment, soon we reach the Server Setup page. Here we can configure the server name and add libraries. Normally we don't need to access our home media server from outside, so remember to uncheck the \"Allow me to access my media outside my home\" box in this step. To add a library, first select the type of library (movies, TV episodes, music, photos, etc.), then click the \"BROWSE FOR MEDIA FOLDER\" button to browse and select the corresponding folder. Once the library is added, the included media files will immediately appear in the local service directory, as shown in the screenshot below ![](\"Plex-movie-en.png\")
Here we have a local server named ZIXI-RPI-NAS for our Raspberry Pi NAS, the movie directory in the library shows The Matrix trilogy and is playing the first one The Matrix. Move your mouse over the server name and ➕ icon will appear to the right, click on it to continue adding new media libraries.![](\"RPi-stress-test.png\")
Dividing the ![](\"ADM-SSD.png\")
![](\"ADM-HDD.png\")
![](\"NASPi-final.jpg\")
![](\"SSL-Pulse-2022-11.png\")
![](\"SSL-Report-Sum.png\")
![](\"SSL-Report-Conf.png\")
![](\"Firefox-TLS-MIN-VER.png\")
![](\"TLS-V1-1-BADSSL.png\")
![](\"SSL-TLS.jpg\")
![](\"Implement-TLS.jpg\")
![](\"Bulletproof-TLS.jpg\")
![\"TLS](\"TLS1_2vs1_3-handshake.jpg\")
![\"TLS](\"TLS1_3-0-RTT.jpg\")
![\"Source:](\"TLS1_3-stats.png\")
![](\"picoGym.png\")
![](\"crypt-glibc-features.png\")
![](\"CVE-2022-30295.png\")
![](\"uClibc-DNS-cve.png\")
![](\"uClibc-DNS-fix.png\")
