hissho

or anyone here would be interested in putting what Lombi said into code and sell it to me?

thanks Lombi, at least it's possible now... are you selling it by any chance? I'm new to ubot and haven't got the time and expertise to figure this out myself. I'd be happy to pay for it. thanks again!

bump...any further help please? or it's just impossible for ubot to do it?

thanks for your help! well the thing is that the target sites don't necessarily always have a sitemap...also, not all pages are indexed in google...I want to be able to scrape unindexed URLs as well. e.g. if a site has got a simple opt-in form with "name" and "email address" fields, once clicked, visitors will be taken to another URL(thank-you page or whatever), which may or may not be indexed in google or listed in the sitemap. any further help would be much appreciated!

Hi all, Just wondering if anyone has done this before and possibly share a few tips on how to do this... Let's say my site is xyz.com and I want to use ubot to scrape ALL the URLs of that site. Nothing else, I just want the URLs (no meta description, no text, no images, no videos etc etc) and save it to a text file. Any help would be much appreciated!