BigEfromDaBX 25 Posted June 21, 2017 Report Share Posted June 21, 2017 So I created a bot that I need to scrape 100's of pages with. So im using the spawn thread. I set total number of spawns at 10 and max concurrent to 10 also. It will work fine for a bit but then the "in new browser" windows will stop popping up and it will just hang. Any suggestions. Quote Link to post Share on other sites
HelloInsomnia 1103 Posted June 21, 2017 Report Share Posted June 21, 2017 Maybe its a memory problem, its hard to say though for sure. If you can post your code maybe somebody can check it out. 1 Quote Link to post Share on other sites
BigEfromDaBX 25 Posted June 21, 2017 Author Report Share Posted June 21, 2017 Maybe its a memory problem, its hard to say though for sure. If you can post your code maybe somebody can check it out. Memory as in my Ram? Quote Link to post Share on other sites
HelloInsomnia 1103 Posted June 21, 2017 Report Share Posted June 21, 2017 Memory as in my Ram? Yes. Quote Link to post Share on other sites
BigEfromDaBX 25 Posted June 21, 2017 Author Report Share Posted June 21, 2017 I got my Virtual Box running windows 7 with 8GB's of ram and 4 Processors. Quote Link to post Share on other sites
deliter 203 Posted June 21, 2017 Report Share Posted June 21, 2017 You try this method http://network.ubotstudio.com/forum/index.php/topic/20500-asynchronous-http-gets-when-logged-into-a-site-from-ubot-browser/ 1 Quote Link to post Share on other sites
BigEfromDaBX 25 Posted June 21, 2017 Author Report Share Posted June 21, 2017 You try this method http://network.ubotstudio.com/forum/index.php/topic/20500-asynchronous-http-gets-when-logged-into-a-site-from-ubot-browser/Do I need an extra plugin to run that? Quote Link to post Share on other sites
deliter 203 Posted June 21, 2017 Report Share Posted June 21, 2017 also this if you want to use ubot selector tools on the http data http://network.ubotstudio.com/forum/index.php/topic/20577-use-ubots-scraping-tools-with-socketayman-http-gets/ 1 Quote Link to post Share on other sites
deliter 203 Posted June 21, 2017 Report Share Posted June 21, 2017 Do I need an extra plugin to run that?No just the URL decodeI think, which is in the ubot socket commands, or Ayman has a http text encode function if you do not have the socket commands 1 Quote Link to post Share on other sites
BigEfromDaBX 25 Posted June 21, 2017 Author Report Share Posted June 21, 2017 Am I going to have to man up and buy that http get or post plugin that everybody talks about? Will that help me scrape alot of pages fast? Quote Link to post Share on other sites
HelloInsomnia 1103 Posted June 21, 2017 Report Share Posted June 21, 2017 I got my Virtual Box running windows 7 with 8GB's of ram and 4 Processors. That doesn't matter so much, Ubot is 32 bit it can only use 2GB of ram anyways. So check to be sure its not running near that number or you could have issues. Am I going to have to man up and buy that http get or post plugin that everybody talks about? Will that help me scrape alot of pages fast? It will speed things up and reduce memory issues significantly. Just be sure to use HTTP Clear Headers and HTTP Clear Objects at the end of your thread if you decide to get the plugin (this will help with memory issues). 1 Quote Link to post Share on other sites
BigEfromDaBX 25 Posted June 21, 2017 Author Report Share Posted June 21, 2017 That doesn't matter so much, Ubot is 32 bit it can only use 2GB of ram anyways. So check to be sure its not running near that number or you could have issues. It will speed things up and reduce memory issues significantly. Just be sure to use HTTP Clear Headers and HTTP Clear Objects at the end of your thread if you decide to get the plugin (this will help with memory issues). You Rock. Stay Dry (FL Inside joke LOL) 1 Quote Link to post Share on other sites
BigEfromDaBX 25 Posted June 21, 2017 Author Report Share Posted June 21, 2017 i need to scrape one mail anywhere from 2000 to 10000 pages. That that http plugin the best way to do this? Or is the spawn thread just as good? Quote Link to post Share on other sites
HelloInsomnia 1103 Posted June 21, 2017 Report Share Posted June 21, 2017 i need to scrape one mail anywhere from 2000 to 10000 pages. That that http plugin the best way to do this? Or is the spawn thread just as good? Yes http plugin is probably the best way. It depends on the page but for most pages it will be the best way. I wouldn't worry about it but in some cases if there's like something to do with Javascript it wouldn't work since it's not a browser but for scraping this is almost never an issue. Quote Link to post Share on other sites
BigEfromDaBX 25 Posted June 22, 2017 Author Report Share Posted June 22, 2017 Yes http plugin is probably the best way. It depends on the page but for most pages it will be the best way. I wouldn't worry about it but in some cases if there's like something to do with Javascript it wouldn't work since it's not a browser but for scraping this is almost never an issue. Its a simple scrape of an email from each page. But there are alot of pages that i have to visit. Quote Link to post Share on other sites
BigEfromDaBX 25 Posted June 23, 2017 Author Report Share Posted June 23, 2017 Yes http plugin is probably the best way. It depends on the page but for most pages it will be the best way. I wouldn't worry about it but in some cases if there's like something to do with Javascript it wouldn't work since it's not a browser but for scraping this is almost never an issue.@HelloInsomnia I finally bought the http plugin after remembering that I bought a tutorial from you on scraping. In that tutorial you showed how to scrape and use the http post plugin. Watched your video a few mins. Bought the plugin and now my scraping is soooooo fast and smooth now. Thank you sir 1 Quote Link to post Share on other sites
HelloInsomnia 1103 Posted June 23, 2017 Report Share Posted June 23, 2017 @HelloInsomnia I finally bought the http plugin after remembering that I bought a tutorial from you on scraping. In that tutorial you showed how to scrape and use the http post plugin. Watched your video a few mins. Bought the plugin and now my scraping is soooooo fast and smooth now. Thank you sir Glad to hear it 1 Quote Link to post Share on other sites
BigEfromDaBX 25 Posted June 23, 2017 Author Report Share Posted June 23, 2017 Glad to hear it @HelloInsomnia I still need to practice on the encoder part as i was still getting the A's with the funny character on the top. But it didnt affect my scrape. Thanks Buddy. You rock 1 Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.