For an asynchronous data collection job, session TTL is easy to set and hard to reason about.
If the TTL is too short, a multi-step workflow may change identity halfway through. If it is too long, a failing route can remain attached to work that should have moved on. The correct value depends on the target, request sequence, and failure budget.
I am considering a policy based on workflow shape:
- independent pages: rotate per request or small batch
- related page sequence: keep one session until the item is complete
- long-running item: renew with an explicit handoff rather than silently extending TTL
- repeated failures: end the session and move the item to a bounded retry queue
The job should record session start, session end, reason for rotation, attempts, response bytes, and final validation result. This makes it possible to compare session behavior without treating every failure as an IP problem.
How do you choose a TTL for long multi-step jobs? Do you base it on elapsed time, request count, target behavior, or a combination?
If the TTL is too short, a multi-step workflow may change identity halfway through. If it is too long, a failing route can remain attached to work that should have moved on. The correct value depends on the target, request sequence, and failure budget.
I am considering a policy based on workflow shape:
- independent pages: rotate per request or small batch
- related page sequence: keep one session until the item is complete
- long-running item: renew with an explicit handoff rather than silently extending TTL
- repeated failures: end the session and move the item to a bounded retry queue
The job should record session start, session end, reason for rotation, attempts, response bytes, and final validation result. This makes it possible to compare session behavior without treating every failure as an IP problem.
How do you choose a TTL for long multi-step jobs? Do you base it on elapsed time, request count, target behavior, or a combination?