Can we prevent deadlocks and timeouts on ReliableQueue's in Service Fabric? -
we have stateful service in service fabric both runasync method , couple of service calls.
one service calls allows enqueue in reliablequeue
using(itransaction tx = statemanager.createtransaction()) { await queue.enqueueasync(tx, message); queuelength = await queue.getcountasync(tx); await tx.commitasync(); }
the runasync on other hand tries dequeue things:
using(itransaction tx = statemanager.createtransaction()) { await queue.trydequeueasync(tx); queuelength = await queue.getcountasync(tx); await tx.commitasync(); }
the getcountasync seems cause deadlocks, because 2 transactions block each other. if switch order: first counting , dequeue/enqueue?
this due fact reliablequeue today strict fifo , allows 1 reader or writer @ time. you're not seeing deadlocks, you're seeing timeouts (please correct me if not case). there's no real way prevent timeouts other to:
- ensure transactions not long lived - longer need , you're blocking other work on queue.
- increase default transaction timeout (the default 4 seconds, can pass in different value)
reordering things shouldn't cause change.