Linux connection error

[VamatoHD]

Description:
When using the tool in Linux, most of the modules result in Error (ConnectError: [errno -2] name or service not known). Since the module performs many requests at once, this may be DNS hitting the rate limit.

I'm not able to reproduce this in Windows, only in Linux (only tested in Linux Mint).

Possible solutions:
Perhaps lower MAX_CONCURRENT_REQUESTS on Linux.

[kaifcodec]

@VamatoHD I mean, try testing it with a lower MAX_CONCURRENT_REQUESTS value using the -C flag and see if the same issue still occurs.

I've run into similar issues before, such as intermittent DNS errors. In my case, they were usually caused by the OS or network stack just waking up from an idle state. After a few consecutive requests, everything started working normally again.

So it would be worth checking whether lowering the concurrency makes the issue disappear.

First test it before we update the default value.

[VamatoHD]

@kaifcodec reducing it to 10 (-C 10) did, in fact, remove the connection error and, in my opinion, make the search faster.

I believe that reducing it allowed the DNS to work properly, and every request was made on time, whereas, with 60 threads, most would wait for the DNS or even fail.

I've only tested this on Linux Mint (I can't test on Windows currently).

[kaifcodec]

Yeah, I think we may need to introduce platform-specific default configurations.

In my testing, it worked well on Windows 11, Ubuntu, and Termux overall. The only occasional issues I noticed were on Ubuntu and Termux, similar to what I mentioned earlier, intermittent DNS/network-related errors that usually resolved themselves after a few consecutive requests.

Anyways, do you have any ideas for addressing this? For example, are you thinking about introducing platform-specific default configurations (just for Linux Mint) or something else?

By the way did you test it that whether it get fixed after consecutive runs? Just like my case i.e. First run of the tool hit the error and the second or maybe 3rd run goes perfect?

[VamatoHD]

In my testing, it is also intermittent, but since there are so many workers, it ends up erroring many modules. And it's basically the same (just different modules) on each run.

Implementing platform-specific wouldn't work, as it can be hardware-specific.

We could keep a list of modules that failed due to a connection error (the DNS error as mentioned early) and try them again, maybe just by using a queue.

[kaifcodec]

@VamatoHD The main issue with retrying is that sites with unresolved DNS will genuinely stay down, so retrying them would just slow down the entire scan unnecessarily.

Also, I don't think this error is happening super often for most users. I tested it across Ubuntu, Termux, and Windows, and other collaborators tried reproducing it as well, and none of us hit the same error.

Instead of retrying every failed request, what if we just print a warning at the end of the run if more than 5 sites hit that specific error? Something advising the user to reduce concurrency using -C with a lower value.