httr2 1.1.0

  2025/01/20

  httr2

  Hadley Wickham

We’re chuffed to announce the release of httr2 1.1.0. httr2 (pronounced “hitter2”) is a comprehensive HTTP client that provides a modern, pipeable API for working with web APIs. It builds on top of {curl} to provide features like explicit request objects, built-in rate limiting & retry tooling, comprehensive OAuth support, and secure handling of secrets and credentials.

In this blog post, we’ll dive into the new streaming interface built around req_perform_connection(), explore the new suite of URL manipulation tools, and highlight a few of the other biggest changes (including better support for AWS and enhancements to the caching system), and update you on the lifecycle changes.

This blog post includes the most important enhacenments in versions 1.0.1 through 1.0.7, where we’ve been iterating on various features and fixing numerous bugs. For a complete list of changes, you can check the GitHub release notes or the NEWS file.

Installation

Install httr2 from CRAN with:

install.packages("httr2")

Streaming data

The headline feature of this release is a better API for streaming responses, where the body is not available immediately but is streamed back over time. This is particularly important for interacting with LLMs, where it’s needed to make chat responses feel snappy. You can try it out in ellmer, our new package for chatting with LLMs from a variety of providers.

The most important new function is req_perform_connection(), which supersedes the older callback-based req_perform_stream(). Unlike its predecessor, req_perform_connection() returns a regular response object with a connection object for the body:

library(httr2)

req <- request(example_url()) |> req_template("/stream-bytes/:n", n = 10240)
resp <- req_perform_connection(req)
resp
#> <httr2_response>
#> GET http://127.0.0.1:49283/stream-bytes/10240
#> Status: 200 OK
#> Content-Type: application/octet-stream
#> Body: Streaming connection

Once you have a streaming connection you can repeatedly call a resp_stream_*() function to pull down data in chunks, using resp_stream_is_complete() to figure out when to stop.

while (!resp_stream_is_complete(resp)) {
  bytes <- resp_stream_raw(resp, kb = 2)
  cat("Downloaded ", length(bytes), " bytes\n", sep = "")
}
#> Downloaded 2048 bytes
#> Downloaded 2048 bytes
#> Downloaded 2048 bytes
#> Downloaded 2048 bytes
#> Downloaded 2048 bytes
#> Downloaded 0 bytes

As well as resp_stream_raw(), which returns a raw vector, you can use resp_stream_lines() to stream lines and resp_stream_sse() to stream server-sent events.

URL manipulation tools

Working with URLs got easier with three new functions: url_modify(), url_modify_query(), and url_modify_relative(). You can see how they work in the examples below:

# url_modify() modifies components of a URL
url_modify("https://example.com", hostname = "github.com")
#> [1] "https://github.com/"
url_modify("https://example.com", scheme = "http")
#> [1] "http://example.com/"
url_modify("https://example.com", path = "abc", query = list(foo = "bar"))
#> [1] "https://example.com/abc?foo=bar"

# url_modify_query() lets you modify individual query parameters
# modifying an existing parameter:
url_modify_query("http://example.com?a=1&b=2", a = 10)
#> [1] "http://example.com/?b=2&a=10"
# delete a parameter:
url_modify_query("http://example.com?a=1&b=2", b = NULL)
#> [1] "http://example.com/?a=1"
# add a new parameter:
url_modify_query("http://example.com?a=1&b=2", c = 3)
#> [1] "http://example.com/?a=1&b=2&c=3"

# url_modify_relative() navigates to a relative URL
url_modify_relative("https://example.com/a/b/c.html", "/d/e/f.html")
#> [1] "https://example.com/d/e/f.html"
url_modify_relative("https://example.com/a/b/c.html", "C.html")
#> [1] "https://example.com/a/b/C.html"
url_modify_relative("https://example.com/a/b/c.html", "../B.html")
#> [1] "https://example.com/a/B.html"

We also added req_url_relative() to make it easier to navigate to a relative URL for an existing request.

Other improvements

There are a handful of other improvements that are worth highlighting:

  • We’ve made it easier to talk to AWS web services with req_auth_aws_v4() for signing requests and resp_stream_aws() for streaming responses. Special thanks goes to the lifion-aws-event-stream project for providing a clear reference implementation.

  • We’ve run-down a long list of bugs that made req_cache() unreliable. This includes improving the handling of header-only changes, better cache pruning, and new debugging options. If you’re working with a web API that supports caching, we highly recommend that you try it out. The next release of { gh} will use a cache by default, and my use of the dev version suggests that it gives a pretty nice performance improvment.

  • is_online() provides an easy way to check internet connectivity.

  • req_perform_promise() allows you to execute requests in the background (thanks to @gergness) using an efficient approach that waits on curl socket activity (thanks to @shikokuchuo).

Breaking changes

As httr2 continues to mature, we’ve made some lifecycle changes:

Acknowledgements

A big thanks to all 76 folks who filed issues, created PRs and generally helped to make httr2 better! @Aariq, @AGeographer, @amael-ls, @anishjoni, @asadow, @atheriel, @awpsoras, @billsanto, @bonushenricus, @botan, @burgerga, @CareCT, @cderv, @cole-brokamp, @covid19ec, @datapumpernickel, @denskh, @deschen1, @DyfanJones, @erydit, @exetico, @fh-mthomson, @frzambra, @gergness, @GreenGrassBlueOcean, @guslipkin, @hadley, @i2z1, @isachng93, @IshuaWang, @JamesHWade, @jameslairdsmith, @JBGruber, @jcheng5, @jeroen, @jimbrig, @jjesusfilho, @jl5000, @jmuhlenkamp, @jonthegeek, @JosiahParry, @jwimberl, @krjaworski, @m-muecke, @maarten-vermeyen, @MarekGierlinski, @maxsutton, @mgirlich, @MichaelChirico, @mkoohafkan, @MSHelm, @mstei4176, @mthomas-ketchbrook, @NateNohling, @nick-youngblut, @pbulsink, @PietrH, @pkautio, @plietar, @pmlefeuvre-met, @rkrug, @romainfrancois, @salim-b, @shikokuchuo, @simplyalexander, @sluga, @stefanedwards, @steveputman, @tebancr, @thohan88, @tony2015116, @toobiwankenobi, @verhovsky, @walinchus, @werkstattcodes, and @zacdav-db.

Contents