The firmware rollout to 120,000 field devices started Friday evening. By Tuesday morning, the dashboard says 96% complete. The remaining 4% are stuck in "updating," and some have been there since Friday night.
You pull the record for one stuck device. It did finish the update. It sent an "update complete" acknowledgment and, at almost the same moment, a routine telemetry message. Two consumers picked up those messages in parallel. Both read the device record, both modified it, both wrote it back. The telemetry handler wrote last, carrying the old "updating" status it had read milliseconds earlier. The acknowledgment was silently overwritten.
Nothing crashed. No alert fired. The device is healthy, but your platform believes it is mid-update, so the command scheduler refuses to send it anything new. Multiply that by a few thousand devices and you have a week of cleanup scripts.
Why device state is harder than it looks
Every device carries a small but busy bundle of state: last telemetry, configuration, firmware version, connectivity status, and pending commands. None of it is complicated alone. The difficulty comes from three properties that appear together at fleet scale.
Concurrent writers to the same record. One device can emit telemetry, acknowledgments, and alarms within milliseconds. A horizontally scaled consumer group spreads those messages across workers, each doing a read-modify-write against a shared database. The classic lost update follows.
Uneven load. A gateway aggregating hundreds of sensors, a unit stuck in a reboot loop, or a district of streetlights reconnecting after a power cut can concentrate traffic on a few keys. Those keys become hot partitions, and latency degrades for everyone.
Per-device time. Most fleets need per-device clocks: expect a heartbeat every five minutes, mark the device offline after three misses, retry a command if no acknowledgment arrives within a minute. That is potentially hundreds of thousands of independent timers that must survive restarts.
Each is solvable in isolation. The trouble is that they interact, and the code handling them spreads across ingestion workers, cron jobs, caches, and database triggers.
Why the usual workarounds fall short
Database locks. Pessimistic locking prevents lost updates but serializes work at the database, so bursty load trades data loss for contention and timeouts. Optimistic concurrency is better, but every handler now needs retry logic, and a hot device causes repeated conflicts.
Partitioning the stream by device ID. Keying the broker by device ID gives per-device ordering, which helps. But concurrency is tied to partition count, rebalancing pauses processing, and you still need somewhere to keep state between messages. Teams often add an in-memory cache per consumer, which then needs invalidation when partitions move.
Custom sharding layers. Eventually someone builds a "device service" that owns a slice of the fleet in memory, with a hashing ring for routing and snapshots for persistence. It can work. It is also a bespoke distributed system with failure detection and rebalancing logic your team maintains forever.
Cron-driven timers. Offline detection often starts as a job scanning for stale heartbeats every minute. At a few thousand devices that is fine. At hundreds of thousands, the scan gets expensive, results are always stale, and retries handled the same way arrive in bursts.
Each workaround pushes the team further into building infrastructure instead of device features.
The general approach: one virtual actor per device
The virtual actor pattern, popularized by the Orleans project and implemented in several runtimes, maps cleanly onto device fleets. Model each device as an independent actor with its own identity, state, and behavior, and let a runtime decide where it lives.
Turn-based concurrency. An actor processes one message at a time. If the acknowledgment and the telemetry for device 4711 arrive together, they are handled sequentially. The lost update from the opening scene cannot happen, because each device has exactly one writer, and your code needs no locks.
Virtual activation. You address an actor by ID; the runtime activates it on demand, loads its state, and deactivates it when idle. A device that reports once a day costs almost nothing in between.
Persisted state. Actor state lives in a durable store, so a node failure does not lose configuration or pending commands. The actor reactivates elsewhere with its state intact.
Per-actor timers and reminders. Instead of a global scan, each actor tracks its own heartbeat deadline: store the last-heartbeat time in actor state and use a periodic reminder to check it, rather than re-registering on every heartbeat. If the deadline passes, the actor marks itself offline and emits an event. Command retries work the same way.
Automatic distribution. A placement mechanism spreads actors across instances and rebalances as you scale. Work is partitioned by actor ID, not a fixed partition count.
The pattern has limits. A very chatty device still processes messages one at a time, so keep actor methods short and avoid blocking calls. But it turns "concurrency and placement for 120,000 device records" into "write the logic for one device," which is a much better problem.
Implementing it with Dapr Actors and pub/sub
Dapr, the CNCF graduated Distributed Application Runtime, includes a virtual actor implementation. It runs as a sidecar next to your application on Kubernetes, VMs, or self-hosted environments, so your service calls a local API instead of embedding a framework.
Dapr Actors activate on demand, use turn-based concurrency per actor, persist state in a pluggable store such as PostgreSQL, Redis, Cosmos DB, DynamoDB, or MongoDB, and support timers and reminders. The placement service distributes actors across instances. A typical device platform has three layers.
Telemetry ingestion via pub/sub. Your protocol gateway publishes into a broker. Dapr pub/sub works with Kafka, RabbitMQ, Redis, and cloud services, delivering CloudEvents with at-least-once semantics. A thin ingestion service subscribes and forwards each message to the actor for that device ID. Because delivery is at-least-once, make actor methods idempotent, for example by tracking the last processed sequence number in actor state. Configure a dead-letter topic for messages that keep failing.
Device actors as the source of truth. Each actor holds last telemetry, configuration, firmware version, and pending commands. The firmware rollout becomes an actor-level state machine ("pending," "updating," "complete," "failed") with every transition in one place.
Reminders for heartbeats and retries. Reminders are persisted and survive restarts; timers suit short-lived, non-critical work. Dapr resiliency policies add timeouts, retries, and circuit breakers to actor calls and components.
Teams already run this way at scale. A smart-lighting company uses Dapr Actors to control roughly 800,000 streetlights across 70+ countries, ingesting 20–30 million telemetry messages a day. A European IoT company tracks billions of device messages with Dapr on Kubernetes while reducing costs. Treat these as evidence the model scales, not as a performance promise, and load test with your own traffic shape.
For visibility, every sidecar and control-plane component emits Prometheus metrics, and OpenTelemetry tracing propagates W3C trace context across service invocation and pub/sub. You can follow one telemetry message from the broker to the actor that handled it, which matters when debugging one device out of hundreds of thousands.
Running it in production
The runtime removes custom sharding code, but you still operate a distributed system. The control plane includes the operator, sidecar injector, Sentry (the certificate authority behind mTLS workload identities), placement, and scheduler. Run it in high-availability mode with three replicas, and plan for upgrades, certificate rotation, and state store capacity.
That operational load can stay with your platform team or move partly to an outside provider. Free tooling exists for the Kubernetes side: the Dapr Ops Dashboard from Diagrid automates upgrades and certificate rotation, runs best-practice checks, and ships prebuilt metrics dashboards, and the same company sells commercial Dapr support. Before adopting it, confirm it fits your clusters and compliance rules.
A practical roadmap
- Pick one device type. Start with a single product line, not the whole fleet.
- Define actor state and methods. List per-device fields and the operations that change them.
- Route telemetry through pub/sub to actors. Run in shadow mode beside the current pipeline and compare results.
- Move timers into reminders. Replace the cron-based offline scan for that device type.
- Move command handling. Let the actor own pending commands, acknowledgments, and retries.
- Load test realistic bursts. Simulate mass reconnects and firmware rollouts, not just steady traffic.
- Expand gradually. Add device types once metrics and traces look healthy.
Before going live, confirm:
- ☐ Actor methods are idempotent under at-least-once delivery
- ☐ A dead-letter topic catches failed telemetry
- ☐ Reminders, not in-memory timers, drive offline detection and retries
- ☐ The actor state store is sized and backed up
- ☐ The control plane runs in HA mode
- ☐ Resiliency policies cover actor calls and components
- ☐ Tracing follows one device's messages end to end
Conclusion
The stuck rollout was never a firmware problem. It was a concurrency problem hiding in a data model that let many writers touch one device record. Locks, sharding layers, and cron jobs can patch around it, but each patch is infrastructure your team owns.
Modeling each device as a virtual actor moves concurrency, placement, and per-device timers into the runtime. When staged firmware campaigns grow into long multi-step processes, the same thinking extends to Dapr Workflow or a durable workflow platform built on it. Whichever runtime you choose, the principle holds: give every device a single owner, and a whole category of fleet-scale bugs disappears.