TJM (Tractor Job Management)

Despite reducing the problem of denoiser failure by isolating it on Tractor, we still needed to ensure that all images were available the next day. Hence, we developed a script called TJM (Tractor Job Management), which ensures to retry each task that failed by performing retries on each task in "error" status. TJM uses the Tractor API in Python 2.7, retrieving information from various JIDs (Job IDs), i.e., the different jobs present on the farm, and examining each TID (Task ID) to determine whether it needs to be restarted using the status of the TID.

The only issue lies in our Tractor procedure; each of the subtasks has multiple command IDs, the first handling denoising, followed by merging. If one of them fails, we are forced to retry the subtask from the beginning, even if the work has already been done. However, we can generate one image per blade, thus mitigating the problem. Additionally, we have the option to limit the number of PCs taken by the job to avoid flooding the entire render farm and prevent the aforementioned issue. Overall, this had minimal impact on our operation as errors are infrequent.