When Rust Lies: Debugging Memory, CPU and Async Failures in Prod

Memory safety guarantees get you to production. They don’t keep you there. Rust’s ownership model eliminates entire categories of bugs — use-after-free, data races, null dereferences — but Rust production debugging is still a real discipline, because logic bugs, async starvation, and Arc cycles don’t care about the borrow checker. This playbook covers the failure patterns that actually hurt in 2026: memory growth in K8s, Tokio executor saturation, P99 spikes in Axum/Actix, and the profiling stack to diagnose all of it.

TL;DR: Quick Takeaways

• Arc reference cycles bypass the borrow checker and leak memory silently — heaptrack catches them, the compiler doesn’t.
• Blocking a Tokio worker thread starves all tasks on that thread — spawn_blocking is not optional, it’s correctness.
• fmt::Debug in a hot loop can account for 15–30% of CPU time on high-throughput services.
• P99 latency spikes in Rust APIs are almost always connection pool exhaustion or serialization overhead — not the Rust code itself.

Rust Production Debugging Overview

Rust ships with a compiler that refuses to let you write memory-unsafe code. What it cannot do is refuse to let you write logically incorrect code. The borrow checker operates at compile time against ownership rules — it has zero visibility into runtime resource lifetimes, async scheduler state, or allocation pressure under load. In 2026, as more teams run Rust microservices at scale, the incident patterns are consistent: services that compiled cleanly, passed all tests, and then fell over under production traffic in ways that took hours to diagnose.

Why Rust services fail in production despite memory safety

Memory safety and correctness are different contracts. Rust guarantees the first. The second is your problem. A service can hold an Arc to a cache that never gets evicted — that’s a logical leak. A task can hold a Mutex guard across an await point — that’s a deadlock waiting for the right timing window. An unbounded channel can grow without limit because the consumer can’t keep up — that’s OOM. None of these violate Rust’s safety model. All of them will page you at 3am.

The deeper issue is that ownership lifetime ends at compile time. A value can be technically “owned” somewhere in your graph of Arcs and channels while being functionally leaked because nothing will ever drop it. The compiler proved you won’t use-after-free. It said nothing about when free actually happens.

Debugging Rust Memory Leaks in Production

Rust doesn’t have a garbage collector, which means memory management is explicit and deterministic. That’s the good news. The bad news is that “deterministic” doesn’t mean “obvious” — especially when Arcs, channels, and async tasks form reference graphs that the compiler can’t evaluate for you. Memory growth in a Rust service is almost always one of three things: Arc cycles, logical retention (you’re holding data you don’t need), or async tasks that never complete and never drop their captured state.

use std::sync::{Arc, Weak};
use std::cell::RefCell;

struct Node {
 value: u32,
 // Weak breaks the cycle — Arc here would leak forever
 parent: Option<Weak<RefCell<Node>>>,
 children: Vec<Arc<RefCell<Node>>>,
}

fn build_cycle() -> Arc<RefCell<Node>> {
 let parent = Arc::new(RefCell::new(Node {
 value: 1,
 parent: None,
 children: vec![],
 }));

 let child = Arc::new(RefCell::new(Node {
 value: 2,
 parent: Some(Arc::downgrade(&parent)), // Weak ref — correct
 children: vec![],
 }));

 parent.borrow_mut().children.push(Arc::clone(&child));
 parent
} // parent drops here, child drops, no cycle

The Weak::downgrade call is what makes this safe. Replace it with Arc::clone and you have a cycle — parent holds child, child holds parent, neither ever drops. In a server that builds these node graphs per request, you’re leaking every request’s allocation. heaptrack will show you growing allocation stacks that never free; the borrow checker showed you nothing.

Detecting memory growth in live systems

Two tools are worth knowing: heaptrack for post-mortem flamegraph-style allocation analysis, and bytehound for live tracing with a web UI. In Kubernetes, the first signal is usually container memory approaching its limit — set up a Prometheus metric on process_resident_memory_bytes and alert before OOMKill, not after.

# Attach heaptrack to a running process (Linux)
heaptrack --pid $(pidof your-rust-service)

# Let it run under load for 5–10 minutes, then detach
# Analyze the output
heaptrack_gui heaptrack.your-rust-service.*.gz

# Or for bytehound — instrument at build time
RUSTFLAGS="-C force-frame-pointers=yes" cargo build --release
# Run with bytehound server
bytehound record ./target/release/your-rust-service

heaptrack gives you a flamegraph of allocation call stacks — you’re looking for stacks that grow monotonically over time without a corresponding free. bytehound is heavier but gives you timeline views that make it easy to correlate memory growth with specific request types. In K8s, run heaptrack in a debug sidecar or use an ephemeral container — don’t instrument production builds by default.

Async memory retention problems in Tokio

The “Forever Future” problem: a Tokio task is spawned, captures state in its async closure, and then waits indefinitely on a channel that never sends, or a timer that never fires, or a condition that the rest of the system forgot to signal. The task is alive. Its memory is alive. Nothing will clean it up. This is especially nasty with tokio::spawn because the returned JoinHandle is often dropped immediately, giving you no way to cancel or observe the task.

use tokio::sync::oneshot;

// This task will live forever if tx is dropped without sending
async fn leaked_task(mut rx: oneshot::Receiver<()>) {
 // Captures potentially large state in this closure
 let _big_cache: Vec<u8> = vec![0u8; 1024 * 1024]; // 1MB

 // If the sender is dropped, this returns Err — handle it
 match rx.await {
 Ok(_) => println!("got signal"),
 Err(_) => println!("sender dropped — task can now exit"),
 }
}

// Better: use a CancellationToken for structured lifetime control
use tokio_util::sync::CancellationToken;

async fn cancellable_task(token: CancellationToken) {
 let _big_cache: Vec<u8> = vec![0u8; 1024 * 1024];

 tokio::select! {
 _ = token.cancelled() => { /* cleanup */ }
 _ = do_actual_work() => {}
 }
}

async fn do_actual_work() { /* ... */ }

CancellationToken from tokio-util is the structured concurrency answer here. Every spawned task gets a token derived from a parent. When the parent shuts down, all children get cancelled. Without this, your task count and memory footprint grow monotonically over the lifetime of the process.

Arc Reference Cycle — Memory That Never Frees

Rust High CPU Usage Debugging

CPU spikes in Rust services are counterintuitive. You wrote Rust specifically to avoid this. The reality is that a Rust service can burn CPU on things that are invisible in code review: allocation pressure in a hot loop, fmt::Debug calls on large structs, synchronization overhead, or a regex that compiles on every request. The profiling toolchain for Rust is mature enough in 2026 that there’s no excuse for guessing — flamegraphs will tell you exactly where the time goes.

Sudden spikes and gradual degradation have different root causes. A sudden spike that correlates with traffic is usually a hot path issue — something expensive happening per-request. Gradual CPU creep over hours or days is almost always lock contention or a data structure that degrades as it grows (think HashMap rehashing or a Vec that’s being scanned linearly). Start with your metrics: CPU per core (not aggregate), context switch rate, and lock wait time if you have it.

Flamegraph analysis for Rust performance bottlenecks

cargo-flamegraph wraps perf and produces a flamegraph in one command. Run it under realistic load — a flamegraph of idle service tells you nothing. The “hot loop” in a flamegraph is the widest stack frame near the top of the flame. That’s where your CPU is. If you see allocator functions (jemalloc::malloc, or the system allocator) high in the stack, you have allocation pressure.

# Install
cargo install flamegraph

# Run your binary under perf sampling (requires Linux perf or dtrace on macOS)
cargo flamegraph --bin your-service -- --config prod.toml

# Or attach to running process
sudo perf record -F 999 -g -p $(pidof your-service) -- sleep 30
sudo perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg

# With jemalloc for better allocator visibility
# In Cargo.toml: jemallocator = "0.5"
# In main.rs:
# #[global_allocator]
# static ALLOC: jemallocator::Jemalloc = jemallocator::Jemalloc;

jemalloc is worth enabling in production Rust services beyond just profiling — it consistently outperforms the system allocator under multithreaded allocation pressure, with benchmarks showing 10–40% throughput improvement on allocation-heavy workloads. The flamegraph will show you jemalloc frames separately from your application frames, making it easy to spot allocation-heavy paths.

Hidden CPU killers: allocations and logging overhead

fmt::Debug is implemented by derive macro and often formats entire structs — including nested collections — into a String. In a hot loop, every tracing::debug! or log::debug! call that formats a large struct allocates, even if the log level means the output never gets written. The format arguments are evaluated before the level check in older logging setups.

// BAD: allocates String even if debug logging is disabled
log::debug!("Processing request: {:?}", large_request_struct);

// GOOD: lazy evaluation — only formats if debug is enabled
log::debug!("Processing request: {}", large_request_struct.id);

// BEST in hot paths: use tracing with lazy fields
tracing::debug!(request_id = %req.id, "processing");
// tracing evaluates field values lazily when the span is active

// Also expensive: String::from in hot paths
// BAD
fn get_key(prefix: &str, id: u64) -> String {
 format!("{prefix}:{id}") // allocates every call
}

// BETTER: use a stack buffer or SmolStr for short strings
use smol_str::SmolStr;
fn get_key_cheap(prefix: &str, id: u64) -> SmolStr {
 SmolStr::from(format!("{prefix}:{id}"))
}

In a service processing 50k req/s, eliminating one format!() allocation per request can reduce allocator pressure enough to visibly lower CPU usage. The tracing crate’s lazy field evaluation is one reason to prefer it over log — fields are only evaluated when the span is actually active and at the right level.

Debugging Async and Tokio Runtime Issues

Tokio is a cooperative multitasking runtime. “Cooperative” means tasks voluntarily yield to the scheduler. When a task doesn’t yield — because it’s doing CPU-bound work or blocking I/O on the async executor thread — it starves every other task that the runtime scheduled on the same thread. This is the most common source of latency spikes in Rust async services, and it’s entirely invisible until you instrument or profile for it.

Tokio task starvation and blocking calls

The rule is absolute: never block a Tokio worker thread. A blocking call inside an async context — std::thread::sleep, synchronous file I/O, a long CPU computation — will stall the entire thread. If your runtime has 8 worker threads and 8 requests all hit the blocking path simultaneously, your service is effectively single-threaded until those calls complete. Use spawn_blocking for anything that blocks, and block_in_place when you can’t move the blocking work out of the current async context.

use tokio::task;

// BAD: blocks the worker thread, starves all other tasks on it
async fn process_file_bad(path: &str) -> String {
 std::fs::read_to_string(path).unwrap() // BLOCKING — never do this
}

// GOOD: spawn_blocking offloads to a dedicated blocking thread pool
async fn process_file_good(path: String) -> String {
 task::spawn_blocking(move || {
 std::fs::read_to_string(&path).unwrap()
 })
 .await
 .unwrap()
}

// block_in_place: use when you're inside a must-not-move context
async fn cpu_heavy_in_place(data: Vec<u8>) -> u64 {
 task::block_in_place(|| {
 // CPU-bound work — at least this thread is dedicated
 data.iter().map(|&b| b as u64).sum()
 })
}

// Configure the blocking thread pool size explicitly
let rt = tokio::runtime::Builder::new_multi_thread()
 .worker_threads(num_cpus::get())
 .max_blocking_threads(512) // default is 512, tune for your workload
 .enable_all()
 .build()
 .unwrap();

spawn_blocking offloads work to Tokio’s dedicated blocking thread pool (separate from worker threads), so it doesn’t stall async execution. The pool grows dynamically up to max_blocking_threads. If you’re seeing latency spikes that correlate with specific operations, enable TOKIO_CONSOLE and look for tasks that are polling for unusually long durations — that’s your blocking culprit.

Deadlocks in async Rust systems

Classical deadlocks (two threads waiting on locks held by each other) exist in Rust too, but async deadlocks are more subtle. The most common: holding a std::sync::Mutex guard across an await point. The guard doesn’t get dropped until the future completes. If any other task on the same thread tries to acquire that mutex, you have a deadlock — and it only manifests under specific scheduling conditions.

use std::sync::Mutex;
use tokio::sync::Mutex as AsyncMutex;

// BAD: std Mutex held across await — potential deadlock
async fn bad_lock_usage(state: &Mutex<Vec<u32>>) {
 let guard = state.lock().unwrap();
 some_async_operation().await; // guard still held here — DEADLOCK RISK
 drop(guard);
}

// GOOD: use tokio::sync::Mutex for async contexts
async fn good_lock_usage(state: &AsyncMutex<Vec<u32>>) {
 let guard = state.lock().await;
 some_async_operation().await; // tokio Mutex is await-aware
 drop(guard); // or just let it drop at end of scope
}

// ALSO GOOD: minimize lock scope, no await while holding
async fn minimal_lock_scope(state: &Mutex<Vec<u32>>) {
 let snapshot = {
 state.lock().unwrap().clone() // drop guard before await
 };
 process_data(snapshot).await;
}

async fn some_async_operation() {}
async fn process_data(_: Vec<u32>) {}

The rule of thumb: if you must hold a lock across an await, use tokio::sync::Mutex. If you don’t need to hold it across an await, use std::sync::Mutex (it’s faster). Never use std::sync::Mutex across an await boundary — it compiles fine and breaks at runtime under load.

Task leaks and uncontrolled spawning

tokio::spawn returns a JoinHandle. When you drop that handle immediately, you detach the task — it runs to completion with no supervision, no cancellation, and no backpressure. In a service that spawns a task per request, this is fine if tasks are short-lived. It’s a bomb if tasks can block, retry indefinitely, or hold resources. A service that’s been running for 24 hours with dropped JoinHandles can have thousands of tasks in flight with no way to inspect or cancel them.

use tokio::task::JoinSet;

// BAD: fire-and-forget with no supervision
async fn spawn_uncontrolled() {
 for req in requests() {
 tokio::spawn(handle_request(req)); // JoinHandle dropped immediately
 }
}

// GOOD: use JoinSet to bound concurrency and collect results
async fn spawn_controlled(requests: Vec<Request>) {
 let mut set = JoinSet::new();
 let concurrency_limit = 100;

 for req in requests {
 // Wait if we're at the limit
 while set.len() >= concurrency_limit {
 set.join_next().await;
 }
 set.spawn(handle_request(req));
 }

 // Drain remaining tasks
 while set.join_next().await.is_some() {}
}

struct Request;
async fn handle_request(_: Request) {}
fn requests() -> Vec<Request> { vec![] }

JoinSet is the right primitive for bounded concurrent work. It gives you a handle to all spawned tasks, lets you collect results, and cancels remaining tasks when dropped. For long-running background tasks, track them in a Vec<JoinHandle> and provide a shutdown signal — untracked tasks are the async equivalent of a thread that was never joined.

Tokio Worker Thread Starvation

Worker Thread #1

Task A — async I/O ✓

Task B — BLOCKING ✗

Task C — waiting…

Task D — waiting…

Task E — waiting…

std::thread::sleep
Thread blocked
C, D, E starved
P99 latency explodes

spawn_blocking
Offloads to blocking pool
Worker thread free
C, D, E run normally

Blocking Thread Pool
max_blocking_threads
= 512 (default)
separate from workers

Network and serialization overhead

serde_json is the default choice for REST APIs and it’s fine until it isn’t. On high-throughput internal services, JSON serialization can account for 20–35% of request CPU time. If you control both ends of the connection, bincode or prost (Protocol Buffers) are 3–8× faster for serialization and produce smaller payloads.

For public-facing APIs, stay with JSON — the ergonomics and tooling outweigh the performance delta. For internal service-to-service calls in the same cluster, prost gives you speed, smaller payloads, and schema validation. The migration path from serde_json to prost is non-trivial but the throughput gains on high-volume internal endpoints justify it.

Production Observability and Debugging Toolkit

You cannot debug what you cannot observe. The Rust observability ecosystem in 2026 is mature: the tracing crate for structured instrumentation, pprof-rs for continuous profiling, and OpenTelemetry for distributed traces. The gap between teams that find bugs in 10 minutes and teams that spend 6 hours is almost always the quality of their instrumentation, not the complexity of the bug.

Logging vs tracing in Rust production systems

The log crate gives you leveled log lines. The tracing crate gives you leveled log lines plus spans — structured, hierarchical timing contexts that flow across async boundaries and service boundaries. For any service that handles concurrent requests, tracing is not optional. Without spans, your logs are a flat stream of lines from N concurrent requests, interleaved with no way to reconstruct what happened to a specific request.

use tracing::{instrument, info, warn, Span};
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

// Setup: structured JSON output for log aggregators
fn init_tracing() {
 tracing_subscriber::registry()
 .with(tracing_subscriber::EnvFilter::from_default_env())
 .with(tracing_subscriber::fmt::layer().json())
 .init();
}

// instrument creates a span for each call — async-aware
#[instrument(skip(db), fields(user_id = %req.user_id))]
async fn handle_request(req: Request, db: &DbPool) -> Result<Response, Error> {
 info!("handling request"); // automatically scoped to this span

 let user = fetch_user(req.user_id, db).await?;

 if user.is_suspended {
 warn!(reason = "suspended", "request rejected");
 }

 Ok(build_response(user))
}

struct Request { user_id: u64 }
struct Response;
struct Error;
struct DbPool;
struct User { is_suspended: bool }

async fn fetch_user(_: u64, _: &DbPool) -> Result<User, Error> {
 Ok(User { is_suspended: false })
}
fn build_response(_: User) -> Response { Response }

The #[instrument] macro on an async function automatically creates a span that captures entry, exit, and duration. The skip directive prevents large arguments from being formatted into the span — critical for performance. Fields like user_id = %req.user_id are recorded in the span metadata and flow through to your log aggregator and trace backend.

Using profiling tools in live Rust systems

pprof-rs provides Go-style continuous profiling for Rust — CPU profiles on demand over HTTP without restarting the service. Integrate it as a handler in your Axum or Actix service and expose it on an internal port. Under load, hit the endpoint and get a flamegraph of what your service is actually doing.

use pprof::ProfilerGuard;
use axum::{routing::get, Router};

async fn pprof_handler() -> Vec<u8> {
 let guard = pprof::ProfilerGuardBuilder::default()
 .frequency(1000) // samples per second
 .blocklist(&["libc", "libgcc", "pthread", "vdso"])
 .build()
 .unwrap();

 tokio::time::sleep(std::time::Duration::from_secs(30)).await;

 let report = guard.report().build().unwrap();
 let mut buf = Vec::new();
 report.flamegraph(&mut buf).unwrap();
 buf
}

// Mount on internal port — not the public API
let internal_router = Router::new()
 .route("/debug/pprof/profile", get(pprof_handler));

Run the profiler for 30–60 seconds under production load to get a representative sample. The blocklist filters out low-level system frames that add noise. The resulting flamegraph SVG can be opened directly in a browser and shows you exactly which functions consumed CPU during the sample window — no guessing.

The three-layer stack that works in production: Prometheus for metrics (counters, histograms, gauges), tracing + OpenTelemetry for distributed traces, and Jaeger or Tempo for trace storage and querying. The tracing-opentelemetry crate bridges tracing spans to OTLP, so your #[instrument] annotations automatically generate traces without additional code.

use opentelemetry::sdk::export::trace::stdout;
use tracing_opentelemetry::OpenTelemetryLayer;
use tracing_subscriber::layer::SubscriberExt;

fn init_telemetry() {
 // In prod: replace stdout exporter with OTLP to Jaeger/Tempo
 let tracer = stdout::new_pipeline().install_simple();

 tracing_subscriber::registry()
 .with(tracing_subscriber::EnvFilter::from_default_env())
 .with(OpenTelemetryLayer::new(tracer)) // spans -> traces
 .with(tracing_subscriber::fmt::layer()) // spans -> logs
 .init();
}

One setup, three outputs: logs to stdout (for your log aggregator), traces to Jaeger (for distributed request tracing), and metrics to Prometheus (for dashboards and alerting). This is the minimum viable observability stack for a production Rust microservice. Teams that add this upfront spend significantly less time on incident diagnosis than teams that instrument retroactively.

Production Incident Debugging Workflow

When something breaks in production, the instinct is to look at code. The correct instinct is to look at data first — metrics, logs, traces — and only look at code once you know what you’re looking for. A structured triage workflow cuts mean time to resolution from hours to minutes, not because it’s a checklist, but because it prevents you from spending 45 minutes debugging the wrong component.

Step-by-step incident triage in Rust systems

The sequence matters: metrics first (what’s broken and when did it start), logs second (what was happening at that time), traces third (which specific requests were affected and where they slowed down), profiling fourth (what the code was actually doing). Most incidents are solved at step 1 or 2. Profiling is for the ones that aren’t.

• Step 1 — Metrics: CPU, memory, request rate, P99 latency, error rate. When did the anomaly start? Did anything change (deploy, traffic spike, dependency outage)?
• Step 2 — Logs: Filter to the time window from step 1. Look for error patterns, unusual request types, or log lines that appear near the anomaly timestamp.
• Step 3 — Traces: Pull traces from the affected time window. Find slow traces. Look at span timings — which span was slow, database, serialization, external call?
• Step 4 — Profiling: If the issue is CPU or memory, hit the pprof endpoint under similar load. Let it sample for 30–60 seconds. Read the flamegraph.
• Step 5 — Reproduce locally: Use the information from steps 1–4 to write a targeted reproduction. Don’t guess — reproduce the specific scenario that showed up in traces.

Reproducing production issues locally

Miri is the Rust interpreter for detecting undefined behavior — useful for catching memory issues in safe code that are hard to trigger in production. Loom is a concurrency testing tool that exhaustively explores thread interleaving — the right tool for async deadlocks and race conditions that only manifest under specific scheduling.

# Run tests under Miri (detects UB, memory issues)
cargo +nightly miri test

# Loom example — exhaustive concurrency testing
#[cfg(test)]
mod tests {
 use loom::sync::{Arc, Mutex};
 use loom::thread;

 #[test]
 fn test_concurrent_access() {
 loom::model(|| {
 let state = Arc::new(Mutex::new(0u32));

 let s1 = Arc::clone(&state);
 let t1 = thread::spawn(move || {
 *s1.lock().unwrap() += 1;
 });

 let s2 = Arc::clone(&state);
 let t2 = thread::spawn(move || {
 *s2.lock().unwrap() += 1;
 });

 t1.join().unwrap();
 t2.join().unwrap();

 // Loom will test ALL possible thread interleavings
 assert_eq!(*state.lock().unwrap(), 2);
 });
 }
}

Loom replaces std::sync primitives with its own versions that record scheduling decisions. The loom::model closure is run hundreds of times with different interleavings. If any interleaving produces a panic or assertion failure, Loom reports the exact sequence that triggered it. This is how you reliably reproduce the “happens once a week under load” race condition.

Root cause analysis workflow for Rust services

An RCA without a specific root cause is a timeline, not an analysis. For Rust backend failures, the structure that works: observed symptom (with metrics), contributing factors (what conditions were required), root cause (the specific code or configuration that failed), and fix (the change made and why it addresses the root cause — not just “we increased the pool size” but “we increased the pool size because connection hold time under concurrent load exceeded our release rate”).

For recurring incidents, add a prevention step: what instrumentation or alerting would have caught this earlier? The goal is that the same class of failure is caught by an alert next time, not a page at 3am.

FAQ

How do I detect memory leaks in a Rust production service that I can’t restart?

Attach heaptrack to the running process with heaptrack –pid. It instruments memory allocations without stopping the process, though it adds overhead — run it for 5–10 minutes under load, then detach. The resulting profile will show allocation call stacks that grow without a corresponding free. For live continuous monitoring without heaptrack overhead, expose process_resident_memory_bytes as a Prometheus metric and alert on sustained growth over a sliding window. That gives you early warning before OOMKill, at which point you can attach heaptrack during the next growth phase.

What causes Tokio task starvation and how do I confirm it’s happening?

Task starvation happens when a task on a worker thread blocks — using synchronous I/O, std::thread::sleep, or a CPU-heavy computation without yielding. Every other task scheduled on that thread stops making progress until the blocking call returns. To confirm: enable TOKIO_CONSOLE=1 and connect with the tokio-console CLI — it shows per-task poll duration in real time. Any task polling for more than ~100μs without an await point is a starvation candidate. The fix is spawn_blocking for blocking I/O or CPU work, which offloads to Tokio’s dedicated blocking thread pool.

Is serde_json a performance bottleneck in high-throughput Rust services?

At moderate traffic, no. At 50k+ req/s on internal services, yes — benchmarks consistently show JSON serialization at 20–35% of request CPU time for complex payloads. The alternative for internal service communication is prost (Protocol Buffers), which serializes 3–5× faster and produces smaller payloads. The migration cost is schema definition and code generation setup. For public APIs, keep serde_json — the developer experience and debugging ergonomics outweigh the performance delta. For high-volume internal gRPC or binary endpoints, prost pays for itself quickly.

How do I debug an Arc reference cycle in a long-running Rust service?

Arc cycles don’t trigger Rust’s drop machinery, so they’re invisible to normal memory analysis. heaptrack will show you allocation stacks that grow without freeing — look for your struct types in the flamegraph with an increasing allocation count and zero frees. Once you’ve identified the leaking type, search for all places where Arc::clone is called on it and trace the reference graph. The fix is Weak references on the “back” edge of any cycle. Going forward, prefer Weak for parent references in tree or graph structures — if ownership flows down, references flowing up should be Weak.

What’s the right observability stack for a production Rust microservice in 2026?

The minimal viable stack: tracing crate for instrumentation (replace log entirely), tracing-opentelemetry to bridge spans to OTLP, Prometheus via the metrics crate for time-series metrics, and Jaeger or Grafana Tempo for trace storage. The tracing crate’s #[instrument] macro does the heavy lifting — instrument async handler functions and any functions that do I/O, and you’ll have span-level timing for every request. Add pprof-rs on an internal debug endpoint for on-demand CPU profiling without service restart. This stack adds minimal overhead and gives you enough signal to diagnose 95% of production issues without touching code.

How does Loom help reproduce async concurrency bugs that only happen in production?

Loom replaces Rust’s standard synchronization primitives with instrumented versions that record every scheduling decision. The loom::model block runs your test code hundreds of times, systematically exploring every possible thread interleaving — not just the interleavings your development machine’s scheduler happened to produce. A bug that requires thread A to acquire a lock between thread B’s two operations, which occurs once a week in production, will be found by Loom in seconds. Loom is specifically designed for concurrency correctness testing, not performance — use it to validate lock-free algorithms and critical section logic during development, before the bug reaches production.