Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge 8.7.1 #3

Merged
merged 510 commits into from
May 6, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
510 commits
Select commit Hold shift + click to select a range
e75a48d
ntml_wb: fix buffer type typo
icing Jan 31, 2024
8243ad6
asyn-thread: use wakeup_close to close the read descriptor
bagder Feb 1, 2024
5b50bf0
cmdline-docs/Makefile: avoid using a fixed temp file name
bagder Feb 1, 2024
26f7941
RELEASE-NOTES: synced
bagder Feb 1, 2024
a911f4f
docs: remove `mk-ca-bundle.1` from `man_MANS`
jamacku Feb 1, 2024
627c8c5
OS400: avoid using awk in the build scripts
monnerat Jan 31, 2024
1049195
ftp: treat a 226 arriving before data as a signal to read data
bagder Feb 1, 2024
b8c0038
sendf: ignore response body to HEAD
bagder Feb 1, 2024
a84ad94
connect.c: fix typo
fffaraz Feb 4, 2024
5413215
configure: add --disable-docs flag
arachsys Feb 3, 2024
7e2c53f
proxy1.0.md: fix example
verhovsky Feb 3, 2024
3ec8520
ALTSVC.md: correct a typo
larsks Feb 2, 2024
8911d86
cookie.md: provide an example sending a fixed cookie
bagder Feb 5, 2024
0dc0362
HTTP/2: write response directly
icing Jan 31, 2024
aefb0bd
docs: add missing slashes to SChannel client certificate documentation
JDepooter Feb 3, 2024
5cc2b01
md4: include strdup.h for the memdup proto
bagder Feb 5, 2024
75d79a4
tool_cb_hdr: only parse etag + content-disposition for 2xx
bagder Feb 5, 2024
d6825df
openssl-quic: check on Windows that socket conv to int is possible
icing Feb 5, 2024
a982d19
THANKS: add Dmitry Tretyakov
bagder Feb 5, 2024
577182a
configure: do not link with nghttp3 unless necessary
icing Feb 5, 2024
bf411cc
libcurl-security.md: Active FTP passes on the local IP address
bagder Feb 5, 2024
ad79612
RELEASE-NOTES: synced
bagder Feb 5, 2024
8094474
docs: fix the --disable-docs for autotools
bagder Feb 5, 2024
d94733b
docs: make curldown do angle brackets like markdown
bagder Feb 5, 2024
8050761
TODO: Support latest rustls
bagder Feb 6, 2024
174c89e
version: allow building with ancient libpsl
nafmo Feb 2, 2024
4f79455
docs: add necessary setup for nghttp3
gengjiawen Feb 4, 2024
c177e19
https-proxy: use IP address and cert with ip in alt names
icing Feb 1, 2024
755b31d
docs: add mk-ca-bundle.1 to dist
bagder Feb 6, 2024
a5c8620
header.md: remove backslash, make nicer markdown
bagder Feb 6, 2024
bdb0bf4
cmake: fix function description in comment [ci skip]
vszakats Feb 6, 2024
dc3eb67
badwords: use hostname, not host name
bagder Feb 6, 2024
911fc96
cmdline-docs: quote and angle bracket cleanup
bagder Feb 6, 2024
924e264
tool_operate: change precedence of server Retry-After time
bagder Feb 5, 2024
8f40b30
KNOWN_BUGS: unicode on Windows
bagder Feb 6, 2024
142ac25
lib: convert Curl_get_line to use dynbuf
bagder Feb 6, 2024
1d96828
CURLOPT_WRITEFUNCTION.md: typo fix
danielsz Feb 7, 2024
0f0edc2
curl: exit on config file parser errors
bagder Feb 5, 2024
7cf8414
tests: support setting/using blank content env variables
bagder Feb 5, 2024
ed09a99
vtls: revert "receive max buffer" + add test case
icing Feb 1, 2024
d1c9f38
lib582: remove code causing warning that is never run
bagder Feb 7, 2024
ef4bd8d
curl: when allocating variables, add the name into the struct
bagder Feb 7, 2024
505f43c
HTTP3.md: remove quiche word in Openssl 3.2
Karthikdasari0423 Feb 8, 2024
cf5f604
docs: make sure curl.1 is included in dist tarballs
bagder Feb 8, 2024
e52cddb
TODO: align the TOC with the header
bagder Feb 8, 2024
5a4b2f9
HTTP3.md: adjust the OpenSSL QUIC install instructions
Karthikdasari0423 Feb 8, 2024
6984aa3
ftp: tracing improvements
icing Feb 8, 2024
e7fd32b
ftp: do lineend conversions in client writer
icing Feb 6, 2024
c54d0ff
write-out.md: clarify error handling details
bagder Feb 8, 2024
476adfe
multi: add xfer_buf to multi handle
icing Jan 26, 2024
d0a851f
dist: make sure the http tests are in the tarball
bagder Feb 9, 2024
d8f01e0
ftp: fix socket wait activity in ftp_domore_getsock
icing Feb 9, 2024
07e5b3e
mbedtls: fix building when MBEDTLS_X509_REMOVE_INFO flag is defined
MAntoniak Feb 8, 2024
05104f8
configure: add warning for using TLS libraries without 1.3 support
bagder Feb 8, 2024
f8513ee
cmake: add warning for using TLS libraries without 1.3 support
vszakats Feb 8, 2024
d7332e3
Revert "CI: run Circle macOS builds on x86 for now"
dfandrich Jan 4, 2024
089b4e3
TODO: avoid nroff
bagder Feb 9, 2024
9b43dcd
TODO: use pkg-config to find libpsl
bagder Feb 9, 2024
d900bd2
RELEASE-NOTES: synced
bagder Feb 9, 2024
922091c
scripts: Fix cijobs.pl for Azure and GHA
dfandrich Feb 9, 2024
b08200d
test1165: improve pattern matching
Karlson2k Feb 8, 2024
8436406
CI: bump to actions/cache@v4 to avoid warning
dfandrich Feb 10, 2024
f8bd04e
mbedtls: use mbedtls_ssl_conf_{min|max}_tls_version
MAntoniak Feb 8, 2024
ed596eb
KNOWN_BUGS: FTP upload fails if remebered dir is deleted
bagder Feb 10, 2024
24d6c28
schannel: fix hang on unexpected server close
jay Feb 9, 2024
5691a6c
transfer: improve Windows SO_SNDBUF update limit
jay Feb 9, 2024
30f1cb2
CI: Bump the Circle CI base Ubuntu image to the latest 20.04
dfandrich Feb 16, 2024
e87751d
vtls: fix tls proxy peer verification
icing Feb 13, 2024
b2497a8
cmake: fix install for older CMake versions
levitte Feb 10, 2024
298c120
checksrc.pl: fix handling .checksrc with CRLF
Karlson2k Feb 18, 2024
ee92f3d
CURLOPT_POSTQUOTE.md: fix typo
lufriem Feb 11, 2024
6428311
HTTP3.md: always run nghttp3 submodule init
Karthikdasari0423 Feb 12, 2024
0e2ffa3
file: use xfer buf for file:// transfers
icing Feb 13, 2024
59e2c78
http_chunks: fix the accounting of consumed bytes
icing Feb 14, 2024
b4df2c9
RELEASE-NOTES: synced
bagder Feb 19, 2024
50f65b4
cd2nroff: remove backticks from titles
bagder Feb 19, 2024
96af350
libcurl-docs: cleanups
bagder Feb 19, 2024
8b1f3aa
spellcheck.yml: remove .1/.3 handling, clean all man page .md files
bagder Feb 19, 2024
33d3153
mk-ca-bundle.md: cleanups and polish
bagder Feb 19, 2024
5f48ba1
BINDINGS: add mcurl, the python binding
bagder Feb 19, 2024
36401d0
MANUAL.md: fix typo
ranemirusG Feb 19, 2024
bdff974
OpenSSL QUIC: adapt to v3.3.x
icing Feb 13, 2024
ab027d9
docs: dist curl*.1 and install without perl
bagder Feb 19, 2024
30a3880
curl_setup.h: add curl_uint64_t internal type
Karlson2k Feb 10, 2024
cbe41d1
SHA-512/256: implement hash algorithm
Karlson2k Feb 7, 2024
6d6113e
tests: add SHA-512/256 unit test
Karlson2k Feb 7, 2024
e3461bb
digest: support SHA-512/256
Karlson2k Feb 8, 2024
f0c446a
websocket: fix curl_ws_recv()
icing Feb 15, 2024
f7e5987
DoH: add trace configuration
icing Nov 27, 2023
d5b0fee
sha512_256: remove the cast macro, minor language/format edits
bagder Feb 20, 2024
cc04c73
CURLINFO_USED_PROXY: return bool whether the proxy was used
bagder Feb 22, 2024
ac208cc
write-out: add '%{proxy_used}'
bagder Feb 22, 2024
32234ff
RELEASE-NOTES: synced
bagder Feb 22, 2024
93d8e35
curlver: bump to 8.7.0 for next release
bagder Feb 22, 2024
f274fc5
multi: fix multi_sock handling of select_bits
icing Feb 22, 2024
8e83b6b
THANKS: add bug reporter from #740
bagder Feb 23, 2024
8dbc3c7
BUG-BOUNTY.md: clarify that the curl security team decides
bagder Feb 22, 2024
9b3f67e
configure.ac: find libpsl with pkg-config
ffontaine Feb 15, 2024
8d4ff40
gen: make `\>` in input to render as plain '>' in output
bagder Feb 23, 2024
d096809
cd2nroff: gen: make `\>` in input to render as plain '>' in output
bagder Feb 23, 2024
e3a3bb3
mprintf: fix format prefix I32/I64 for windows compilers
jay Feb 16, 2024
7448054
setopt: fix check for CURLOPT_PROXY_TLSAUTH_TYPE value
swt2c Feb 24, 2024
f47487c
libssh/libssh2: return error on too big range
bagder Feb 24, 2024
e2bd0c1
strtoofft: fix the overflow check
bagder Feb 25, 2024
e3a4273
rustls: make curl compile with 0.12.0
kpcyrd Feb 25, 2024
e26c362
lib: remove curl_mimepart object when CURL_DISABLE_MIME
MAntoniak Feb 9, 2024
5b41fac
sendf: Curl_client_write(), make passed in buf const
icing Feb 6, 2024
2abfa38
http: move headers collecting to writer
icing Feb 6, 2024
e925d0d
c-hyper: add header collection writer in hyper builds
icing Feb 6, 2024
2254551
urldata: move authneg bit from conn to Curl_easy
icing Feb 16, 2024
463472a
lib: move client writer into own source
icing Feb 7, 2024
57446b6
lib: initialize output pointers to NULL before calling strto[ff,l,ul]
Ne02ptzero Feb 26, 2024
9c8968e
http_chunks: remove unused 'endptr' variable
bagder Feb 26, 2024
b8ed0f8
RELEASE-NOTES: synced
bagder Feb 26, 2024
5929822
lib: send rework
icing Feb 14, 2024
f73cb3e
CURLOPT_SSL_CTX_FUNCTION.md: no promises of lifetime after return
bagder Feb 26, 2024
2097a09
docs: use present tense
bagder Feb 27, 2024
757dfdf
multi: make add_handle free any multi_easy
bagder Feb 26, 2024
3755153
lib: Curl_read/Curl_write clarifications
icing Feb 15, 2024
f0eacd9
fopen: fix narrowing conversion warning on 32-bit Android
kiefer-andreas Feb 26, 2024
17d302e
setopt: Fix disabling all protocols
danielgustafsson Feb 27, 2024
6e494a2
docs: more language cleanups
bagder Feb 27, 2024
f540e43
examples: use present tense in comments
bagder Feb 27, 2024
01b42aa
TODO: Select signature algorithms
bagder Feb 27, 2024
26bccd6
TODO: build HTTP/3 with OpenSSL and nghttp3 using cmake
bagder Feb 27, 2024
9342563
KNOWN_BUGS: HTTP/2 prior knowledge over proxy
bagder Feb 27, 2024
efda7ed
KNOWN_BUGS: Implicit FTPS upload timeout
bagder Feb 27, 2024
745b99e
KNOWN_BUGS: FTPS upload, FileZilla, GnuTLS and close_notify
bagder Feb 27, 2024
b8ad95b
KNOWN_BUGS: IMAPS connection fails with rustls error
bagder Feb 27, 2024
6540545
getparam: make --ftp-ssl work again
bagder Feb 28, 2024
8d67c61
curldown: Fix email address in Copyright
danielgustafsson Feb 28, 2024
9369c30
lib: Curl_read/Curl_write clarifications
icing Feb 15, 2024
32e0544
cmdline-opts/_VERSION: provide %VERSION correctly
bagder Feb 28, 2024
5083809
tests: add test1598 for POST with trailers
icing Feb 28, 2024
dcf3824
github/labeler: improve the match patterns
dfandrich Feb 29, 2024
89733e2
configure: build & install shell completions when enabled
dfandrich Feb 8, 2024
2cd78f5
misc: Fix typos in docs and lib
RainRat Feb 29, 2024
0f7aba8
configure: Don't make shell completions without perl
dfandrich Mar 1, 2024
57777a2
docs: Update minimal binary size in INSTALL.md
dfandrich Nov 10, 2023
dff74ae
appveyor: Properly skip if only CircleCI is changed
dfandrich Mar 1, 2024
ddb8716
ftp: Mark a const buffer as const
dfandrich Mar 1, 2024
9e2ee70
hyper: disable test1598 due to lack of trailer support
icing Feb 29, 2024
a0cbe4b
cmdline-opts/_EXITCODES: sync with libcurl-errors
jay Feb 29, 2024
ab173d1
configure: Don't build shell completions when disabled
dfandrich Mar 2, 2024
b1005d1
bufq: writing into a softlimit queue cannot be partial
icing Mar 1, 2024
ae7ad31
rustls: fix two warnings related to number types
kpcyrd Feb 29, 2024
46aea3d
RELEASE-NOTES: synced
bagder Mar 4, 2024
e3905de
lib: further send/upload handling polish
icing Feb 28, 2024
9454757
cookie: if psl fails, reject the cookie
bagder Mar 3, 2024
ee31f69
pytest: adapt to API change
icing Mar 4, 2024
36a95c5
build(deps): bump fsfe/reuse-action from 2 to 3
dependabot[bot] Mar 4, 2024
e455490
_VARIABLES.md: improve the description
bagder Mar 4, 2024
07b6675
smtp: free a temp resource
bagder Mar 4, 2024
6f685f0
CONTRIBUTE: update the section on documentation format
bagder Mar 4, 2024
eb9166d
gen.pl: make the "manpageification" faster
bagder Mar 4, 2024
065faf2
openssl-quic: fix unity build, casing, indentation
vszakats Mar 4, 2024
9c7768c
openssl-quic: fix BIO leak and Windows warning
vszakats Mar 4, 2024
14bcea0
lib: enhance client reader resume + rewind
icing Feb 29, 2024
a5dd943
smpt: fix starttls
Flakebi Mar 5, 2024
a54d0bd
KNOWN_BUGS: fix typo
vszakats Mar 5, 2024
df1fcb4
http: better error message for HTTP/1.x response without status line
mkauf Mar 4, 2024
c426277
TIMER_STARTTRANSFER: set the same for everyone
icing Mar 5, 2024
8e74164
cmake: add USE_OPENSSL_QUIC support
talregev Mar 3, 2024
4cea098
TODO: remove "build HTTP/3 with OpenSSL and nghttp3 using cmake"
bagder Mar 5, 2024
6c632b2
RELEASE-NOTES: synced
bagder Mar 5, 2024
0ba4714
mime: add client reader
icing Feb 29, 2024
db5c9f4
transfer.c: break receive loop in speed limited transfers
icing Mar 5, 2024
cc6f2f0
http2: fix push discard
icing Mar 6, 2024
2ca530d
http: fix dead code in setting post client reader
icing Mar 6, 2024
9978d40
lib: add `void *ctx` to reader/writer instances
icing Mar 6, 2024
f03c856
docs: ascii version of manpage without nroff
bagder Mar 4, 2024
62c08d5
mkhelp: simplify the generated hugehelp program
bagder Mar 5, 2024
48f5426
trace-config.md: remove the mutexed options list
jay Mar 2, 2024
2ea178a
KNOWN_BUGS: FTPS server compatibility on Windows with Schannel
jay Mar 3, 2024
ed97fe0
cmake: fix `CURL_WINDOWS_SSPI=ON` with Schannel disabled
vszakats Mar 6, 2024
923f7f8
paramhlp: fix CRLF-stripping files with "-d @file"
bagder Mar 6, 2024
6a13d4d
test463: HTTP with -d @file with file containing CR, LF and null byte
bagder Mar 6, 2024
296e855
cmake: fix libcurl.pc and curl-config library specifications
levitte Feb 13, 2024
1347cf2
GIT-INFO: convert to markdown
bagder Mar 7, 2024
deca803
http2: push headers better cleanup
icing Mar 6, 2024
9b81f1b
cmake: enable `ENABLE_CURL_MANUAL` by default
vszakats Mar 6, 2024
1e517e9
digest: add check for hashing error
Karlson2k Mar 7, 2024
05268cf
sha512_256: add support for GnuTLS and OpenSSL
Karlson2k Mar 6, 2024
a89be3c
docs/cmdline-opts/.gitignore: ignore curl.txt
icing Mar 7, 2024
8a9fbd6
ngtcp2: no recvbuf for stream
icing Mar 7, 2024
2c0f2e8
hyper: implement unpausing via client reader
icing Mar 7, 2024
835e4cb
asyn-ares: fix data race warning
RMMoreton Feb 26, 2024
a03b91e
RELEASE-NOTES: synced
bagder Mar 7, 2024
cf3b60e
KNOWN_BUGS: POP3 issue when reading small chunks
bagder Mar 7, 2024
e55db0c
mkhelp: rename variable to fix compiler warnings
vszakats Mar 7, 2024
5267bf5
http2: memory errors in the push callbacks are fatal
bagder Mar 7, 2024
aba98d2
buildconf.bat: remove outdated groff/nroff use
bagder Mar 7, 2024
b4d73e6
http2: minor tweaks to optimize two struct sizes
bagder Mar 7, 2024
1ab2efb
test1140/1173: extend wildcards to find curl.1
bagder Mar 7, 2024
7168137
docs/cmdline-opts: drop the curl.1 from the dist tarball
bagder Mar 7, 2024
800617f
lib1598: fix `CURLOPT_POSTFIELDSIZE` usage
MarcelRaad Mar 8, 2024
a586b8c
lib: client reader polish
icing Mar 7, 2024
39173f6
VULN-DISCLOSURE-POLICY.md: update detail about CVE requests
bagder Mar 8, 2024
1ccf1cd
scripts/managen: the new name and home for the manpage generator
bagder Mar 8, 2024
fcef00d
lib: keep conn IP information together
icing Mar 8, 2024
cd93765
RELEASE-NOTES: synced
bagder Mar 11, 2024
6aeb729
request: clarify message when request has been sent off
icing Mar 11, 2024
4e4e8af
lib: move 'done' parameter to SingleRequests
icing Mar 11, 2024
3ccce37
tool_getparam: handle non-existing (out of range) short-options
bagder Mar 11, 2024
942896f
mbedtls: properly cleanup the thread-shared entropy
jay Mar 7, 2024
451a7a7
managen: remove space before protocols
bagder Mar 12, 2024
e7219c2
cmdline-opts: language cleanups
bagder Mar 12, 2024
14d9afd
tidy-up: one comment and EOF newlines
vszakats Mar 12, 2024
aedbbdf
vquic-tls: return appropirate errors on wolfSSL errors
bagder Mar 11, 2024
181f5f3
vquic-tls: fix the error code returned for bad CA file
jay Mar 13, 2024
0f4520a
GOVERNANCE: document the core team
bagder Mar 13, 2024
fb3c251
SPONSORS: describe the basics
bagder Mar 13, 2024
09f3679
GHA/linux: add sysctl trick to work-around GitHub runner issue
bagder Mar 14, 2024
4094818
curl: make --libcurl output better CURLOPT_*SSLVERSION
bagder Mar 14, 2024
1356028
RELEASE-NOTES: synced
bagder Mar 14, 2024
c765b04
TLS: start shutdown only when peer did not already close
icing Mar 8, 2024
6ea7587
test 1541: verify getinfo values on first header callback
icing Mar 14, 2024
77b0571
http: revisit http_perhapsrewind()
icing Mar 13, 2024
79cdae4
ipv6.md: mention IPv4 mapped addresses
bagder Mar 15, 2024
3d0fd38
mbedtls: fix pytest for newer versions
icing Mar 15, 2024
80a3b83
http: expect 100 rework
icing Mar 11, 2024
b600638
curl_sha512_256: work around a NetBSD bug
Karlson2k Mar 15, 2024
6bd4ca0
tool_getparam: accept a blank -w ""
bagder Mar 18, 2024
522ea54
http: improve response header handling, save cpu cycles
icing Mar 18, 2024
fd1260d
CURLOPT_INTERFACE.md: remove spurious amp, add see-also
bagder Mar 19, 2024
e6af565
RELEASE-NOTES: corrected
Karlson2k Mar 20, 2024
33d9652
RELEASE-NOTES: synced
bagder Mar 21, 2024
0c82042
http2: remove the third (unused) argument from http2_data_done()
bagder Mar 20, 2024
0f08b43
http: separate response parsing from response action
icing Mar 11, 2024
98f67a6
http2, http3: only return CURLE_PARTIAL_FILE when bytes were received
icing Mar 19, 2024
b935fd4
docs: make each libcurl man specify protocol(s)
bagder Mar 21, 2024
3ff3c09
docs/libcurl: cleanups
bagder Mar 21, 2024
c577500
docs/libcurl: add TLS backend info for all TLS options
bagder Mar 21, 2024
7bc61bf
CURLMOPT_MAX*: mention what happens if changed mid-transfer
bagder Mar 22, 2024
e3fe020
docs/libcurl: generate PROTOCOLS from meta-data
bagder Mar 22, 2024
b564a5f
TODO: update 13.11 with more information
Man2Dev Mar 22, 2024
647e86a
curl-rustls.m4: add pkg-config support to rustls detection
Kangie Mar 24, 2024
fe9f68f
cmdline-opts: shorter help texts
bagder Mar 25, 2024
0aaea58
wolfSSL: do not call the stub function wolfSSL_BIO_set_init()
fabiankeil Mar 17, 2024
5765683
DISTROS: add document with distro pointers
bagder Mar 23, 2024
47da1f2
http: remove stale comment about rewindbeforesend
bagder Mar 25, 2024
1b29784
CURLOPT_POSTFIELDS.md: used for MQTT as well
bagder Mar 26, 2024
b447a77
THANKS: new contributors from the 8.7.0 release
bagder Mar 27, 2024
72cf468
RELEASE-NOTES: synced
bagder Mar 27, 2024
de7b3e8
RELEASE-PROCEDURE: remove old release dates, add new pending ones
bagder Mar 27, 2024
7428c99
Merge 8.7.1
perklet Apr 28, 2024
022da4d
WIP: save at 2024-04-28 22:15:55
perklet Apr 28, 2024
587dea8
WIP: save at 2024-04-29 14:02:17
perklet Apr 29, 2024
c29c35a
Fix bugs in getparam
perklet May 5, 2024
2062025
backport curl br encoding patch
perklet May 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
lib: Curl_read/Curl_write clarifications
- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to
  clarify when and at what level they operate
- send/recv of transfer related data is now done via
  `Curl_xfer_send()/Curl_xfer_recv()` which no longer has
  socket/socketindex as parameter. It decides on the transfer
  setup of `conn->sockfd` and `conn->writesockfd` on which
  connection filter chain to operate.
- send/recv on a specific connection filter chain is done via
  `Curl_conn_send()/Curl_conn_recv()` which get the socket index
  as parameter.
- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for
  naming consistency
- clarify that the special CURLE_AGAIN hangling to return
  `CURLE_OK` with length 0 only applies to `Curl_xfer_send()`
  and CURLE_AGAIN is returned by all other send() variants.
- fix a bug in websocket `curl_ws_recv()` that mixed up data
  when it arrived in more than a single chunk (to be made
  into a sperate PR, also)

Added as documented [in
CLIENT-READER.md](https://github.com/curl/curl/blob/5b1f31dfbab8aef467c419c68aa06dc738cb75d4/docs/CLIENT-READERS.md).

- old `Curl_buffer_send()` completely replaced by new `Curl_req_send()`
- old `Curl_fillreadbuffer()` replaced with `Curl_client_read()`
- HTTP chunked uploads are now formatted in a client reader added when
  needed.
- FTP line-end conversions are done in a client reader added when
  needed.
- when sending requests headers, remaining buffer space is filled with
  body data for sending in "one go". This is independent of the request
  body size. Resolves curl#12938 as now small and large requests have the
  same code path.

Changes done to test cases:

- test513: now fails before sending request headers as this initial
  "client read" triggers the setup fault. Behaves now the same as in
  hyper build
- test547, test555, test1620: fix the length check in the lib code to
  only fail for reads *smaller* than expected. This was a bug in the
  test code that never triggered in the old implementation.

Closes curl#12969
  • Loading branch information
icing authored and bagder committed Feb 28, 2024
commit 9369c30cd87c041cf983bcdfabd1570980abbaf6
93 changes: 93 additions & 0 deletions docs/CLIENT-READERS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# curl client readers

Client readers is a design in the internals of libcurl, not visible in its public API. They were started
in curl v8.7.0. This document describes the concepts, its high level implementation and the motivations.

## Naming

`libcurl` operates between clients and servers. A *client* is the application using libcurl, like the command line tool `curl` itself. Data to be uploaded to a server is **read** from the client and **sent** to the server, the servers response is **received** by `libcurl` and then **written** to the client.

With this naming established, client readers are concerned with providing data from the application to the server. Applications register callbacks via `CURLOPT_READFUNCTION`, data via `CURLOPT_POSTFIELDS` and other options to be used by `libcurl` when the request is send.

## Invoking

The transfer loop that sends and receives, is using `Curl_client_read()` to get more data to send for a transfer. If no specific reader has been installed yet, the default one that uses `CURLOPT_READFUNCTION` is added. The prototype is

```
CURLcode Curl_client_read(struct Curl_easy *data, char *buf, size_t blen,
size_t *nread, bool *eos);
```
The arguments are the transfer to read for, a buffer to hold the read data, its length, the actual number of bytes placed into the buffer and the `eos` (*end of stream*) flag indicating that no more data is available. The `eos` flag may be set for a read amount, if that amount was the last. That way curl can avoid to read an additional time.

The implementation of `Curl_client_read()` uses a chain of *client reader* instances to get the data. This is similar to the design of *client writers*. The chain of readers allows processing of the data to send.

The definition of a reader is:

```
struct Curl_crtype {
const char *name; /* writer name. */
CURLcode (*do_init)(struct Curl_easy *data, struct Curl_creader *writer);
CURLcode (*do_read)(struct Curl_easy *data, struct Curl_creader *reader,
char *buf, size_t blen, size_t *nread, bool *eos);
void (*do_close)(struct Curl_easy *data, struct Curl_creader *reader);
bool (*needs_rewind)(struct Curl_easy *data, struct Curl_creader *reader);
};

struct Curl_creader {
const struct Curl_crtype *crt; /* type implementation */
struct Curl_creader *next; /* Downstream reader. */
Curl_creader_phase phase; /* phase at which it operates */
};
```

`Curl_creader` is a reader instance with a `next` pointer to form the chain. It as a type `crt` which provides the implementation. The main callback is `do_read()` which provides the data to the caller. The others are for setup and tear down. `needs_rewind()` is explained further below.

## Phases and Ordering

Since client readers may transform the data being read through the chain, the order in which they are called is relevant for the outcome. When a reader is created, it gets the `phase` property in which it operates. Reader phases are defined like:

```
typedef enum {
CURL_CR_NET, /* data send to the network (connection filters) */
CURL_CR_TRANSFER_ENCODE, /* add transfer-encodings */
CURL_CR_PROTOCOL, /* before transfer, but after content decoding */
CURL_CR_CONTENT_ENCODE, /* add content-encodings */
CURL_CR_CLIENT /* data read from client */
} Curl_creader_phase;
```

If a reader for phase `PROTOCOL` is added to the chain, it is always added *after* any `NET` or `TRANSFER_ENCODE` readers and *before* and `CONTENT_ENCODE` and `CLIENT` readers. If there is already a reader for the same phase, the new reader is added before the existing one(s).

### Example: `chunked` reader

In `http_chunks.c` a client reader for chunked uploads is implemented. This one operates at phase `CURL_CR_TRANSFER_ENCODE`. Any data coming from the reader "below" has the HTTP/1.1 chunk handling applied and returned to the caller.

When this reader sees an `eos` from below, it generates the terminal chunk, adding trailers if provided by the application. When that last chunk is fully returned, it also sets `eos` to the caller.

### Example: `lineconv` reader

In `sendf.c` a client reader that does line-end conversions is implemented. It operates at `CURL_CR_CONTENT_ENCODE` and converts any "\n" to "\r\n". This is used for FTP ASCII uploads or when the general `crlf` options has been set.

### Example: `null` reader

Implemented in `sendf.c` for phase `CURL_CR_CLIENT`, this reader has the simple job of providing transfer bytes of length 0 to the caller, immediately indicating an `eos`. This reader is installed by HTTP for all GET/HEAD requests and when authentication is being negotiated.

### Example: `buf` reader

Implemented in `sendf.c` for phase `CURL_CR_CLIENT`, this reader get a buffer pointer and a length and provides exactly these bytes. This one is used in HTTP for sending `postfields` provided by the application.

## Request retries

Sometimes it is necessary to send a request with client data again. Transfer handling can inquire via `Curl_client_read_needs_rewind()` if a rewind (e.g. a reset of the client data) is necessary. This asks all installed readers if they need it and give `FALSE` of none does.

## Summary and Outlook

By adding the client reader interface, any protocol can control how/if it wants the curl transfer to send bytes for a request. The transfer loop becomes then blissfully ignorant of the specifics.

The protocols on the other hand no longer have to care to package data most efficiently. At any time, should more data be needed, it can be read from the client. This is used when sending HTTP requests headers to add as much request body data to the initial sending as there is room for.

Future enhancements based on the client readers:
* delegate the actual "rewinding" to the readers. The should know how it is done, eliminating the `readrewind.c` protocol specifics in `multi.c`.
* `expect-100` handling: place that into a HTTP specific reader at `CURL_CR_PROTOCOL` and eliminate the checks in the generic transfer parts.
* `eos` detection: `upload_done` is partly triggered now by comparing the number of bytes sent to a known size. This is no longer necessary since the core readers obey length restrictions.
* `eos forwarding`: transfer should forward an `eos` flag to the connection filters. Filters like HTTP/2 and HTTP/3 can make use of that, terminating streams early. This would also eliminate length checks in stream handling.
1 change: 1 addition & 0 deletions docs/Makefile.am
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ EXTRA_DIST = \
CODE_OF_CONDUCT.md \
CODE_REVIEW.md \
CODE_STYLE.md \
CLIENT-READERS.md \
CLIENT-WRITERS.md \
CONNECTION-FILTERS.md \
CONTRIBUTE.md \
Expand Down
21 changes: 21 additions & 0 deletions lib/bufq.c
Original file line number Diff line number Diff line change
Expand Up @@ -417,6 +417,17 @@ ssize_t Curl_bufq_write(struct bufq *q,
return nwritten;
}

CURLcode Curl_bufq_cwrite(struct bufq *q,
const char *buf, size_t len,
size_t *pnwritten)
{
ssize_t n;
CURLcode result;
n = Curl_bufq_write(q, (const unsigned char *)buf, len, &result);
*pnwritten = (n < 0)? 0 : (size_t)n;
return result;
}

ssize_t Curl_bufq_read(struct bufq *q, unsigned char *buf, size_t len,
CURLcode *err)
{
Expand All @@ -440,6 +451,16 @@ ssize_t Curl_bufq_read(struct bufq *q, unsigned char *buf, size_t len,
return nread;
}

CURLcode Curl_bufq_cread(struct bufq *q, char *buf, size_t len,
size_t *pnread)
{
ssize_t n;
CURLcode result;
n = Curl_bufq_read(q, (unsigned char *)buf, len, &result);
*pnread = (n < 0)? 0 : (size_t)n;
return result;
}

bool Curl_bufq_peek(struct bufq *q,
const unsigned char **pbuf, size_t *plen)
{
Expand Down
7 changes: 7 additions & 0 deletions lib/bufq.h
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,10 @@ ssize_t Curl_bufq_write(struct bufq *q,
const unsigned char *buf, size_t len,
CURLcode *err);

CURLcode Curl_bufq_cwrite(struct bufq *q,
const char *buf, size_t len,
size_t *pnwritten);

/**
* Read buf from the start of the buffer queue. The buf is copied
* and the amount of copied bytes is returned.
Expand All @@ -187,6 +191,9 @@ ssize_t Curl_bufq_write(struct bufq *q,
ssize_t Curl_bufq_read(struct bufq *q, unsigned char *buf, size_t len,
CURLcode *err);

CURLcode Curl_bufq_cread(struct bufq *q, char *buf, size_t len,
size_t *pnread);

/**
* Peek at the head chunk in the buffer queue. Returns a pointer to
* the chunk buf (at the current offset) and its length. Does not
Expand Down
102 changes: 28 additions & 74 deletions lib/c-hyper.c
Original file line number Diff line number Diff line change
Expand Up @@ -117,8 +117,6 @@ size_t Curl_hyper_send(void *userp, hyper_context *ctx,
DEBUGF(infof(data, "Curl_hyper_send(%zu)", buflen));
result = Curl_conn_send(data, io_ctx->sockindex,
(void *)buf, buflen, &nwrote);
if(!result && !nwrote)
result = CURLE_AGAIN;
if(result == CURLE_AGAIN) {
DEBUGF(infof(data, "Curl_hyper_send(%zu) -> EAGAIN", buflen));
/* would block, register interest */
Expand Down Expand Up @@ -659,48 +657,13 @@ static CURLcode request_target(struct Curl_easy *data,
return result;
}

static int uploadpostfields(void *userdata, hyper_context *ctx,
hyper_buf **chunk)
{
struct Curl_easy *data = (struct Curl_easy *)userdata;
(void)ctx;
if(data->req.exp100 > EXP100_SEND_DATA) {
if(data->req.exp100 == EXP100_FAILED)
return HYPER_POLL_ERROR;

/* still waiting confirmation */
if(data->hyp.exp100_waker)
hyper_waker_free(data->hyp.exp100_waker);
data->hyp.exp100_waker = hyper_context_waker(ctx);
return HYPER_POLL_PENDING;
}
if(data->req.upload_done)
*chunk = NULL; /* nothing more to deliver */
else {
/* send everything off in a single go */
hyper_buf *copy = hyper_buf_copy(data->set.postfields,
(size_t)data->req.p.http->postsize);
if(copy)
*chunk = copy;
else {
data->state.hresult = CURLE_OUT_OF_MEMORY;
return HYPER_POLL_ERROR;
}
/* increasing the writebytecount here is a little premature but we
don't know exactly when the body is sent */
data->req.writebytecount += (size_t)data->req.p.http->postsize;
Curl_pgrsSetUploadCounter(data, data->req.writebytecount);
data->req.upload_done = TRUE;
}
return HYPER_POLL_READY;
}

static int uploadstreamed(void *userdata, hyper_context *ctx,
hyper_buf **chunk)
{
size_t fillcount;
struct Curl_easy *data = (struct Curl_easy *)userdata;
CURLcode result;
bool eos;
(void)ctx;

if(data->req.exp100 > EXP100_SEND_DATA) {
Expand All @@ -714,32 +677,15 @@ static int uploadstreamed(void *userdata, hyper_context *ctx,
return HYPER_POLL_PENDING;
}

if(data->req.upload_chunky && data->req.authneg) {
fillcount = 0;
data->req.upload_chunky = FALSE;
result = CURLE_OK;
}
else {
result = Curl_fillreadbuffer(data, data->set.upload_buffer_size,
&fillcount);
}
result = Curl_client_read(data, data->state.ulbuf,
data->set.upload_buffer_size,
&fillcount, &eos);
if(result) {
data->state.hresult = result;
return HYPER_POLL_ERROR;
}
if(!fillcount) {
if((data->req.keepon & KEEP_SEND_PAUSE) != KEEP_SEND_PAUSE)
/* done! */
*chunk = NULL;
else {
/* paused, save a waker */
if(data->hyp.send_body_waker)
hyper_waker_free(data->hyp.send_body_waker);
data->hyp.send_body_waker = hyper_context_waker(ctx);
return HYPER_POLL_PENDING;
}
}
else {

if(fillcount) {
hyper_buf *copy = hyper_buf_copy((uint8_t *)data->state.ulbuf, fillcount);
if(copy)
*chunk = copy;
Expand All @@ -751,8 +697,19 @@ static int uploadstreamed(void *userdata, hyper_context *ctx,
don't know exactly when the body is sent */
data->req.writebytecount += fillcount;
Curl_pgrsSetUploadCounter(data, data->req.writebytecount);
return HYPER_POLL_READY;
}
else if(eos) {
*chunk = NULL;
return HYPER_POLL_READY;
}
else {
/* paused, save a waker */
if(data->hyp.send_body_waker)
hyper_waker_free(data->hyp.send_body_waker);
data->hyp.send_body_waker = hyper_context_waker(ctx);
return HYPER_POLL_PENDING;
}
return HYPER_POLL_READY;
}

/*
Expand All @@ -773,7 +730,7 @@ static CURLcode bodysend(struct Curl_easy *data,
else {
hyper_body *body;
Curl_dyn_init(&req, DYN_HTTP_REQUEST);
result = Curl_http_req_send(data, &req, httpreq);
result = Curl_http_req_complete(data, &req, httpreq);

if(!result)
result = Curl_hyper_header(data, headers, Curl_dyn_ptr(&req));
Expand All @@ -782,18 +739,15 @@ static CURLcode bodysend(struct Curl_easy *data,

body = hyper_body_new();
hyper_body_set_userdata(body, data);
if(data->set.postfields)
hyper_body_set_data_func(body, uploadpostfields);
else {
result = Curl_get_upload_buffer(data);
if(result) {
hyper_body_free(body);
return result;
}
/* init the "upload from here" pointer */
data->req.upload_fromhere = data->state.ulbuf;
hyper_body_set_data_func(body, uploadstreamed);
result = Curl_get_upload_buffer(data);
if(result) {
hyper_body_free(body);
return result;
}
/* init the "upload from here" pointer */
data->req.upload_fromhere = data->state.ulbuf;
hyper_body_set_data_func(body, uploadstreamed);

if(HYPERE_OK != hyper_request_set_body(hyperreq, body)) {
/* fail */
result = CURLE_OUT_OF_MEMORY;
Expand Down Expand Up @@ -888,7 +842,7 @@ CURLcode Curl_http(struct Curl_easy *data, bool *done)
may be parts of the request that is not yet sent, since we can deal with
the rest of the request in the PERFORM phase. */
*done = TRUE;
Curl_cw_reset(data);
Curl_client_reset(data);

/* Add collecting of headers written to client. For a new connection,
* we might have done that already, but reuse
Expand Down
2 changes: 1 addition & 1 deletion lib/cf-h1-proxy.c
Original file line number Diff line number Diff line change
Expand Up @@ -1013,7 +1013,7 @@ static CURLcode cf_h1_proxy_connect(struct Curl_cfilter *cf,
data->req.header = TRUE; /* assume header */
data->req.bytecount = 0;
data->req.ignorebody = FALSE;
Curl_cw_reset(data);
Curl_client_reset(data);
Curl_pgrsSetUploadCounter(data, 0);
Curl_pgrsSetDownloadCounter(data, 0);

Expand Down
12 changes: 3 additions & 9 deletions lib/file.c
Original file line number Diff line number Diff line change
Expand Up @@ -293,10 +293,10 @@ static CURLcode file_upload(struct Curl_easy *data)
CURLcode result = CURLE_OK;
char *xfer_buf;
size_t xfer_blen;
char *uphere_save;
curl_off_t bytecount = 0;
struct_stat file_stat;
const char *sendbuf;
bool eos = FALSE;

/*
* Since FILE: doesn't do the full init, we need to provide some extra
Expand Down Expand Up @@ -340,21 +340,16 @@ static CURLcode file_upload(struct Curl_easy *data)
data->state.resume_from = (curl_off_t)file_stat.st_size;
}

/* Yikes! Curl_fillreadbuffer uses data->req.upload_fromhere to READ
* client data to! Please, someone fix... */
uphere_save = data->req.upload_fromhere;

result = Curl_multi_xfer_buf_borrow(data, &xfer_buf, &xfer_blen);
if(result)
goto out;

while(!result) {
while(!result && !eos) {
size_t nread;
ssize_t nwrite;
size_t readcount;

data->req.upload_fromhere = xfer_buf;
result = Curl_fillreadbuffer(data, xfer_blen, &readcount);
result = Curl_client_read(data, xfer_buf, xfer_blen, &readcount, &eos);
if(result)
break;

Expand Down Expand Up @@ -401,7 +396,6 @@ static CURLcode file_upload(struct Curl_easy *data)
out:
close(fd);
Curl_multi_xfer_buf_release(data, xfer_buf);
data->req.upload_fromhere = uphere_save;

return result;
}
Expand Down
Loading