And now to answer my own question...
Turns out the 'f' command will help. The 'h' docs says that it let you view the code in another file, but in fact it switches your whole context to the other files (not the execution point, of course). You can then easily say 'b 1024' or whatever to set a breakpoint at a particular line, then switch back using 'f' again.
So, in my example above, I might run the debugger on try_it.pl, then immediately type f lib/Broken/Completely, then b 1234 to set a breakpoint at line 1234, then switch back to try_it.pl with f try_it.pl.