KevinMick 1 Posted March 12, 2015 Report Share Posted March 12, 2015 How is V2 coming along? Will the new version be able to deal with a website with 2-3 million pages? Quote Link to post Share on other sites
HelloInsomnia 1103 Posted March 12, 2015 Author Report Share Posted March 12, 2015 How is V2 coming along? Will the new version be able to deal with a website with 2-3 million pages? It's nearly complete, I do want it to be able to crawl millions of pages that is the goal behind remaking it. Because of that I've introduced some new things like URL normalization so that it crawls in a smarter way and is more efficient. It also chucks data so that it is not pulling as large of lists so that the memory stays down. It uses files to do this (much like how Scrapebox saves the harvester sessions in a file so that it doesn't store the memory). The current version was already starting to do that but I wanted to do an overhaul and make it the best possible platform to build off of. I am keeping the scraped data define (although in a new tab now, easier on everyone that way) so that you can easily make the transition, even if a lot of the rest of the code changes the transition should be seamless. The current version is actually already capable of crawling a ton of pages I'll setup a test and see if I can kill it but it already manages things pretty well but eventually you will run into an issue because of the way it handles everything. But I am pretty sure there are not many (if any) other Ubot bots that can visit so many pages without crashing. 1 Quote Link to post Share on other sites
mdegab 0 Posted March 18, 2015 Report Share Posted March 18, 2015 Hi I am ubot dev 5 user. will it work in dev 5 Thanks Quote Link to post Share on other sites
HelloInsomnia 1103 Posted March 18, 2015 Author Report Share Posted March 18, 2015 Hi I am ubot dev 5 user. will it work in dev 5 Thanks Yes this will work in Ubot 5, I just double checked for you. It was made in Ubot 4 and I still recommend Ubot 4 although Ubot 5 is now getting to the point where it's usable without many/any errors. Quote Link to post Share on other sites
pbsolution 4 Posted March 22, 2015 Report Share Posted March 22, 2015 Hello Nick,i got some Problems with latest Version. When he comes to above 300000 URL`s the CPU Goes up to nearly 100% and i must stop the Crawler due Taskmanager. Will this be repaired in new Version ?Using latest Version build with UBot 4 on an VPS referred by you!Greetings Quote Link to post Share on other sites
pbsolution 4 Posted April 15, 2015 Report Share Posted April 15, 2015 Hello,any News on this ?V2 was announced to middle of March, now its middle of April. Having Problems with Big Lists, Bot stops working ever!Thanks Quote Link to post Share on other sites
HelloInsomnia 1103 Posted April 15, 2015 Author Report Share Posted April 15, 2015 Hello,any News on this ?V2 was announced to middle of March, now its middle of April. Having Problems with Big Lists, Bot stops working ever!Thanks Sent you a PM about this and your other question! Quote Link to post Share on other sites
HelloInsomnia 1103 Posted April 15, 2015 Author Report Share Posted April 15, 2015 Update: Updated to work with new HTTP Post version Quote Link to post Share on other sites
HelloInsomnia 1103 Posted May 7, 2015 Author Report Share Posted May 7, 2015 If you want to beta test a newer version of this shoot me a PM I'll give out compiled copies. It may not be totally bug free yet all I've done was make major changes in relation to the memory issues since that is the primary focus. With that in mind that is what you would be testing for if you want to try it out please let me know what you think of how it handles the memory. Edit: for current customers only of course. Quote Link to post Share on other sites
HelloInsomnia 1103 Posted May 11, 2015 Author Report Share Posted May 11, 2015 All beta test requests sent out, still looking for a few more. Quote Link to post Share on other sites
pbsolution 4 Posted May 13, 2015 Report Share Posted May 13, 2015 Hello,please send me an Copy!Thanks Quote Link to post Share on other sites
HelloInsomnia 1103 Posted May 13, 2015 Author Report Share Posted May 13, 2015 Hello,please send me an Copy!Thanks Sent! Quote Link to post Share on other sites
Pete_UK 9 Posted June 23, 2015 Report Share Posted June 23, 2015 Any sign of V2 Nick? Quote Link to post Share on other sites
HelloInsomnia 1103 Posted June 23, 2015 Author Report Share Posted June 23, 2015 Any sign of V2 Nick?If you would like to beta test a newer version I have shoot me a PM. I'll probably end up releasing it since I haven't heard much feedback on it and it works well for me. This offer is open to anybody who owns it actually so if you want to try the newer version PM me. -Nick Quote Link to post Share on other sites
Pete_UK 9 Posted June 24, 2015 Report Share Posted June 24, 2015 Sorry for not responding sooner Nick... I did run the beta version and didn't have any issues at all. Ran for around 5 hours while monitoring memory etc... And it never ballooned once. Seemed to keep everything under control. Quote Link to post Share on other sites
HelloInsomnia 1103 Posted September 29, 2015 Author Report Share Posted September 29, 2015 New coupon code: 20off Take 20% off anything and everything at IM Autobots until Oct 3rd. Quote Link to post Share on other sites
HelloInsomnia 1103 Posted November 29, 2018 Author Report Share Posted November 29, 2018 Notice: When V3 comes out on Elite Botters if you own this version you will have to do the following to upgrade: 1. Send me your IM Autobots username or email2. Send me your Elite Botters usernname or email If you don't have an account at Elite Botters you can grab Ubot Launcher 2 (for free if you want) and that will make an account. 1 Quote Link to post Share on other sites
drewness 26 Posted December 1, 2018 Report Share Posted December 1, 2018 Saw your update email with the screenshots - super impressive! Can't wait to try it out when v3 drops. Question: in the screenshot it shows most of the ram (which isn't even high at all) being used by what looks to be the internal ubot browser. If we were to use the feature in the Elite Ubot plugin to kill the ubot browser (I think that was a newer feature you'd mentioned, unless I'm mistaken), and were using the Xaml UI plugin (since it doesn't rely on the internal ubot browser), would that cut the ram usage by 50%? Or am I getting confused somewhere? Doesn't really matter either way, just curious! Keep up the great work! Sent from my SAMSUNG-SM-N920A using Tapatalk 1 Quote Link to post Share on other sites
HelloInsomnia 1103 Posted December 1, 2018 Author Report Share Posted December 1, 2018 Saw your update email with the screenshots - super impressive! Can't wait to try it out when v3 drops. Question: in the screenshot it shows most of the ram (which isn't even high at all) being used by what looks to be the internal ubot browser. If we were to use the feature in the Elite Ubot plugin to kill the ubot browser (I think that was a newer feature you'd mentioned, unless I'm mistaken), and were using the Xaml UI plugin (since it doesn't rely on the internal ubot browser), would that cut the ram usage by 50%? Or am I getting confused somewhere? Doesn't really matter either way, just curious! Keep up the great work! Sent from my SAMSUNG-SM-N920A using Tapatalk Add them all up then kill off the browsers and then subtract the browser ram. Or just look at the Website Crawler 3 ram itself and that is what you should get as far as I know I will try to test this at some point as well to be sure. 1 Quote Link to post Share on other sites
HelloInsomnia 1103 Posted December 4, 2018 Author Report Share Posted December 4, 2018 New version is now live, I'll make a separate thread for it tomorrow: https://elitebotters.com/product/website-crawler-3/ Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.