Is it just me, or do (Perl) https downloads hang from time to time?
I can reproduce the problem, but can't get any more information than this on it. I'm crawling a whole load of web pages, which is working quite sucessfully. However, every once in a while, the "fetcher" (which is a Perl LWP::UserAgent script) hangs indefinitely. "strace" shows it's blocking in a read(), which lsof reveals is a socket that's connected to port 443 on the remote machine.
There seem to be two problems here - the remote machine seems happy to leave the connection open for days, but more importantly, the client doesn't seem to have a timeout on the read().
I've had a look in the Perl module code, but can't see a point where this could happen. I'm about to resort to semi-unnecessary select() calls before any sysread() calls to try and track down the problem.
Stuff in use: Fedora 1 (up2date to about June 2004) and libwww-5.79.