That sounds nice for the more involved big runs. I'll try to take a look at it sometime. I have some training runs that loop in just under 7 seconds at the shortest for the actual training portion of the run (maybe 9-10 including notebook spinup). After doing this for so many years, I've found that keeping it as short as humanly possible is usually the way to go. :D
That said, I can definitely take a look at it for scale once I need to go over something like 1-2 hours maybe! :D
Yep, definitely best to do somewhere else if you can and just need to iterate, but it’s a good option when you have to do a big one and eat some real GPU time.
That said, I can definitely take a look at it for scale once I need to go over something like 1-2 hours maybe! :D