// HACKER NEWS — CYBERSECURITY
Principles for Fast Tokio Applications
I'm on my way back from RustConf. At the Unconf, we had a productive discussion about debugging and benchmarking async applications. Many interesting insights were shared. I'm attempting to enumerate some of them here, along with some of my own experiences. This is the first draft of what I hope can become a living document of best practices. Feel free to file an issue or open a PR. I'm hoping to also add a sample app in the coming days demonstrating these issues along with what the dial9 trace looks like.
There are few hard-and-fast rules for writing code that performs well on Tokio runtimes; the answer to so many questions is "it depends." The performance of a workload depends on what else is running on the runtime at that moment. This is why so many problems only show up in production! Writing async applications that perform well is a balance between fairness and batching, contention and isolation.
This post lays out some general principles and covers exceptions where I can. It assumes basic familiarity with Tokio's work-stealing runtime; a high-level summary is included in the appendix.
If you start looking for red flags in a Tokio application, you will find them. Almost every real application I have seen has polls (the time between .await points when the code yields back to the runtime) much longer than the 10-100 microseconds Alice Ryhl recommends in her excellent post What is Blocking?. These problems may or may not affect the application metrics or behavior you actually care about (see: long polls can be fine sometimes). It is important to work backward from a real metric you are trying to improve. For example, an application can have long polls that are completely benign; "fixing" them will not measurably impact user-facing metrics.
In the overwhelming majority of problems I have come across, the issue was in the application code itself, often in the interaction between multiple components of a distributed system (and not actually in Tokio). dial9 has given a lot of visibility into Tokio; at least as often as it finds a Tokio problem, it actually clearly demonstrates the lack of one (which gives folks the confidence to search elsewhere). Of course, sometimes it is a Tokio problem.
In terms of Tokio metrics, the most useful is the recently added schedule latency histogram. Schedule latency is the amount of time between your task being ready to run (e.g., because the socket has data) and Tokio actually polling the future. Although this won't tell you what the cause is, scheduling latency is the most common symptom of poor interactions between Tokio and your code.
Low latency across many requests requires fairness between connections.
Consider Redis (or any application that supports request pipelining). A naive implementation will read data directly off the connection while more data is available. When requests are pipelined, the entire pipelined request (or most of it) will end up in an in memory buffer. When you read frames off of it, each will be Poll::Ready (without going back to the network). This creates both long polls and unfairness between clients.
The impact on throughput is usually smaller: the same number of requests are processed. Latency, however, changes dramatically because one entire pipeline can wait behind another. Explicitly yielding after each request can reduce latency by roughly 10× in this example. You can do even better by yielding only after several consecutive immediately-ready reads.
Yielding after four consecutive immediately-ready reads makes pipelined requests much fairer without giving up batching entirely.