It's a fairly trivial inconvenience. You can just add something to the effect of the below code, and you'll not get stuck and realistically not skip over crawling anything of value.
The odds of a payload that's smaller than the average <head> element taking 20 seconds to load, while containing something worth crawling is fairly low.
