Jump to content
UBot Underground

[Sell] Website Crawler - Crawl Websites And Extract That Succulent Data


Recommended Posts

  • Replies 69
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Plugins Required: HTTP Post (Paid) / Advanced File (Free) / File Management (Free) / Threads Counter (Free) / Large Data (Free) / Local Dictionary (Free)   Made and tested in Ubot Studio 4   This Is S

Just wanted to pop in here and give Nick some love with this software.   First off Nick creates really good useful tools. Second, his support is fantastic!   This is a tool with a LOT of different use

Update:   - Fixed: major bug that would close the program to crash when a PDF link (without the .pdf extension) was called - Default script to scrape external domains has been modified now to have gen

How is V2 coming along? Will the new version be able to deal with a website with 2-3 million pages?

 

It's nearly complete, I do want it to be able to crawl millions of pages that is the goal behind remaking it. Because of that I've introduced some new things like URL normalization so that it crawls in a smarter way and is more efficient. It also chucks data so that it is not pulling as large of lists so that the memory stays down. It uses files to do this (much like how Scrapebox saves the harvester sessions in a file so that it doesn't store the memory). The current version was already starting to do that but I wanted to do an overhaul and make it the best possible platform to build off of.

 

I am keeping the scraped data define (although in a new tab now, easier on everyone that way) so that you can easily make the transition, even if a lot of the rest of the code changes the transition should be seamless.

 

The current version is actually already capable of crawling a ton of pages I'll setup a test and see if I can kill it but it already manages things pretty well but eventually you will run into an issue because of the way it handles everything. But I am pretty sure there are not many (if any) other Ubot bots that can visit so many pages without crashing.

  • Like 1
Link to post
Share on other sites

Hi 

 

I am ubot dev 5 user. will it work in dev 5

 

Thanks

 

Yes this will work in Ubot 5, I just double checked for you. It was made in Ubot 4 and I still recommend Ubot 4 although Ubot 5 is now getting to the point where it's usable without many/any errors.

Link to post
Share on other sites

Hello Nick,

i got some Problems with latest Version. When he comes to above 300000 URL`s the CPU Goes up to nearly 100% and i must stop the Crawler due Taskmanager. Will this be repaired in new Version ?

Using latest Version build with UBot 4 on an VPS referred by you!

Greetings

Link to post
Share on other sites
  • 4 weeks later...

Hello,

any News on this ?

V2 was announced to middle of March, now its middle of April. Having Problems with Big Lists, Bot stops working ever!

Thanks

 

Sent you a PM about this and your other question!

Link to post
Share on other sites
  • 3 weeks later...

If you want to beta test a newer version of this shoot me a PM I'll give out compiled copies. It may not be totally bug free yet all I've done was make major changes in relation to the memory issues since that is the primary focus. With that in mind that is what you would be testing for if you want to try it out please let me know what you think of how it handles the memory.

 

Edit: for current customers only of course.

Link to post
Share on other sites
  • 1 month later...

Any sign of V2 Nick?

If you would like to beta test a newer version I have shoot me a PM. I'll probably end up releasing it since I haven't heard much feedback on it and it works well for me.

 

This offer is open to anybody who owns it actually so if you want to try the newer version PM me.

 

-Nick

Link to post
Share on other sites

Sorry for not responding sooner Nick... I did run the beta version and didn't have any issues at all. Ran for around 5 hours while monitoring memory etc... And it never ballooned once. Seemed to keep everything under control.

Link to post
Share on other sites
  • 3 months later...
  • 3 years later...

Notice: When V3 comes out on Elite Botters if you own this version you will have to do the following to upgrade:

 

1. Send me your IM Autobots username or email

2. Send me your Elite Botters usernname or email

 

If you don't have an account at Elite Botters you can grab Ubot Launcher 2 (for free if you want) and that will make an account.

  • Like 1
Link to post
Share on other sites

Saw your update email with the screenshots - super impressive! Can't wait to try it out when v3 drops.

 

Question: in the screenshot it shows most of the ram (which isn't even high at all) being used by what looks to be the internal ubot browser. If we were to use the feature in the Elite Ubot plugin to kill the ubot browser (I think that was a newer feature you'd mentioned, unless I'm mistaken), and were using the Xaml UI plugin (since it doesn't rely on the internal ubot browser), would that cut the ram usage by 50%?

 

Or am I getting confused somewhere? Doesn't really matter either way, just curious! Keep up the great work!

 

Sent from my SAMSUNG-SM-N920A using Tapatalk

  • Like 1
Link to post
Share on other sites

Saw your update email with the screenshots - super impressive! Can't wait to try it out when v3 drops.

 

Question: in the screenshot it shows most of the ram (which isn't even high at all) being used by what looks to be the internal ubot browser. If we were to use the feature in the Elite Ubot plugin to kill the ubot browser (I think that was a newer feature you'd mentioned, unless I'm mistaken), and were using the Xaml UI plugin (since it doesn't rely on the internal ubot browser), would that cut the ram usage by 50%?

 

Or am I getting confused somewhere? Doesn't really matter either way, just curious! Keep up the great work!

 

Sent from my SAMSUNG-SM-N920A using Tapatalk

 

Add them all up then kill off the browsers and then subtract the browser ram. Or just look at the Website Crawler 3 ram itself and that is what you should get as far as I know I will try to test this at some point as well to be sure.

  • Like 1
Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...