To batch or not to batch

Eric: "Also because the cache gets confused if they are handled separately. After this commit then all of the account requests are happening server side for the initial request, no refetches from the client."

Alex: "And what about subsequent requests after client-side hydration? Wouldn't they become slower with N+1 requests compared to batched?"

Eric: "If batched is faster than a bunch of parallel requests, then maybe we need to use a different transport mechanism. But maybe I'm not sure the exact purpose of the batching. What bottleneck are we trying to address? Also I wonder what circumstances are we going to be re-fetching the accounts. Probably we shouldn't do that."

Horacio: "I don't understand why you think doing 40 small requests is similar to do 1 request. I would love to measure that."

Eric: "In which environment are we doing that? Usually between JS and the daemon, running on the same device. right? If we are doing it on the web, it will definitely be slower. I think that it might be a LOT fewer requests in total, if we can utilize the local cache correctly so thats why. 40 parallel requests will hopefully be only marginally slower than 1 big batch. And if we are clever, we can make that number much lower than 40 any time outside the initial request and I think we need to be clever."

...

All agreed to measure.

I want to clarify what problem batching is actually solving, because I think we’re talking past each other.

First, I agree that on the initial page load, where Remix is SSRing and making calls from the Remix server to the Go server on the same machine, batching vs individual requests is mostly irrelevant. Latency is tiny, connections are local, and the difference is unlikely to be user-visible. That part is not controversial.

Where batching matters is after hydration. After hydration, requests are no longer between two processes on the same machine, they involve network requests.

The cost model completely changes here. Even with perfect transport, keepalive, and parallelism, 40 requests are not equivalent to 1 request. They never are, even with sophisticated transport. Even with CDN caches, I doubt they are going to be faster than 1 batch.

I'm glad we agreed to measure this, because I'd be very curious to know. But just the physics of it doesn't add up for me. Even with a multiplexed connection there's speed of light, request and response headers, parsing, the cost of handling the request on the server, etc-etc.

Maybe with perfect parallelism it'll pass the bar. But browsers cap effective parallelism to same hostname (to 5 requests I think?), so we won't get perfect parallelism at all. We've seen this in our own app before where requests would stall because some other requests are slow.

This is also where I think the discussion about “refetching the same data” is slightly missing the point.

IMO, this is not about refetching the same data client-side. This is about navigation and pagination. Take a paginated list or an infinite scroll:

We fetch a page of results → a list of account IDs.

For each visible row, we need account details.

The the classic N+1 problem.

Now consider the options.

No batching, no caching:

40 IDs → 40 requests.

This is obviously bad. I think we all agree.

No batching, with application-level client-side caching:

Some accounts are already in memory.

Great — we skip those requests.

But the remaining uncached accounts still generate individual requests.

As the user scrolls or navigates pages, this keeps happening.

You still pay per-request overhead repeatedly.

Batching with application-level caching:

Client maintains a cache keyed by account ID.

When a page needs accounts: collect missing IDs, send one batch request for the missing ones, fill the cache per account.

Next page: same logic, usually fewer misses.

This amortizes request overhead while still caching at the correct granularity.

This is the key point: Batching and application-level caching are complementary, not competing ideas. Batching does not mean “we don’t cache per account”. It just means we don’t force the transport layer to pay the fixed cost 40 times.

Speaking of CDNs and browser caches. I think they are the wrong level of abstraction for caching API results. They no nothing about our application, and they are meant for mostly static resources. Application-level caching is the way to go, IMO.

I also want our system to work well for everybody, even for self-hosters who may not have set up a CDN. Assuming the CDN is always there is tying ourselves to a third-party, which I don't think is the right approach.