Bealad00 3 Posted December 12, 2014 Report Share Posted December 12, 2014 Hello, One of the bot i created today, is giving me this error, was wondering if anyone had this error before and know how to fix it? http://i.imgur.com/2anUz27.png Thanks in advance Quote Link to post Share on other sites
arunner26 51 Posted December 12, 2014 Report Share Posted December 12, 2014 Bealad00,Yes, I did see that error yesterday actually. I was changing the value of a column within a loop that was the subject of comparison in the loop while command.I changed that column used in loop while to another variable. Once done with changing the original variable inside the loop I set the new variable used in the loop while comparison.Sorry if the answer is confusing.I think there was a timing issue.That seemed to solve that issue above but eventually I moved the whole project back to UBOT version 4 from 5.5 because of other issues. Quote Link to post Share on other sites
itexspert 47 Posted December 13, 2014 Report Share Posted December 13, 2014 HAha i got the same thing on my error and nobody could help me with this error even Ubot support had no ideawhat to say to this! i was scraping Information from a site and i set table cells and after 18th loop this started happening,all i did is set user agent to internet explorerCleared cookies and i didn't allowed any flash or css on the website and this seemed to help! i am sharing my experience because it was frustrating that nobody had no idea how to help me i hope my answer will be helpful! Quote Link to post Share on other sites
cujo56 2 Posted February 26, 2015 Report Share Posted February 26, 2015 Ok, I am going to jump into this thread. I am having the same problem. I have tried increasing the wait time between page reloads in my scrapper (just make sure everything was fully loaded and account for sluggish internet connections). I get about 100 pages into a scrape and I get the same messages. I am working on a theory. Could Google Adsense on a site be a problem? After all, scripts like these could throw off their impression metrics (something they pride themselves on). Just not sure how to test the idea? How can we do ad blocking? Quote Link to post Share on other sites
deliter 203 Posted February 27, 2015 Report Share Posted February 27, 2015 post script,and show which node this happened in,I had the same problem too,but it stopped Quote Link to post Share on other sites
cujo56 2 Posted February 27, 2015 Report Share Posted February 27, 2015 Here is the Define it's happening in. I had this chuck of code nested before. I thought breaking it out might help me see the problem better. define Get Identity { set(#Identity,$scrape attribute(<class="info">,"innerhtml"),"Global") set(#Identity,$replace regular expression(#Identity,"\\t|\\r|\\n| ",$nothing),"Global") set(#Identity,$replace regular expression(#Identity,"(?<=>)\\s+(?=<)",$nothing),"Global") set(#Identity,$replace regular expression(#Identity,"(?<=>)\\s+(?=[a-zA-Z0-9\\:])",$nothing),"Global") set(#Identity,$replace regular expression(#Identity,"(?<=[a-zA-Z0-9\\:\\?])\\s+(?=<)",$nothing),"Global")} It's a basic scrape and cleanup of the HTML. This is the only scrape I am doing on each page as it comes up. From there I just pick out data and add it to lists... nothing special. It basically crashes on the first Set command after about 100 pages or so. Not sure if a Ubot gets any more basic than this. Quote Link to post Share on other sites
cujo56 2 Posted February 27, 2015 Report Share Posted February 27, 2015 Here is an additional error message I am getting after the JSON. Error converting value True to type 'System.Collections.Generic.List`1[system.String]'.Path ", line 1, position 4.Source: pagescraperbot -> -> set -> $scrape attribute (<class="info">, "innerhtml")-> $scrape attribute(<class="info">,innerhtml")->$scrape attribute Quote Link to post Share on other sites
cujo56 2 Posted February 27, 2015 Report Share Posted February 27, 2015 Ugh... OK, I moved just the working/running parts of my bot to a whole new file. (I had some other scripts/tabs in the original file that were still in development). Ran a test to loop 300 pages and it worked without error. I will test on a higher number next. I will post back here if there is still a problem. Otherwise, no news is good news. Thanks. Quote Link to post Share on other sites
cujo56 2 Posted March 1, 2015 Report Share Posted March 1, 2015 Looks like the problem still exists. I am back to square one on this. Any ideas as to what is triggering these errors? Quote Link to post Share on other sites
deliter 203 Posted March 1, 2015 Report Share Posted March 1, 2015 sounds like the scrape attribute is being fired before the page loads,the replace nodes,wouldn't cause an error like that,they would just return nothing,so the problem I reckon is with the scrape attribute loop while($comparison($exists(<class="info">),"=","false")) { wait(0.2)}set(#Identity,"","Global")set(#Identity,$scrape attribute(<class="info">,"innerhtml"),"Global")set(#Identity,$replace regular expression(#Identity,"\\t|\\r|\\n| ",$nothing),"Global")set(#Identity,$replace regular expression(#Identity,"(?<=>)\\s+(?=<)",$nothing),"Global")set(#Identity,$replace regular expression(#Identity,"(?<=>)\\s+(?=[a-zA-Z0-9\\:])",$nothing),"Global")set(#Identity,$replace regular expression(#Identity,"(?<=[a-zA-Z0-9\\:\\?])\\s+(?=<)",$nothing),"Global") 1 Quote Link to post Share on other sites
cujo56 2 Posted March 1, 2015 Report Share Posted March 1, 2015 Thanks for the suggestion and great support. I thought the loop would work but once again I get an error... http://i.imgur.com/UEfhNtL.jpg?1 Then this JSON error after hitting "Continue". http://i.imgur.com/b7qGD0Q.jpg I will try your $exists loop on the <id=genbtn"> html also... Hope that makes a difference. Quote Link to post Share on other sites
cujo56 2 Posted March 2, 2015 Report Share Posted March 2, 2015 The additional loop didn't make a difference. Same errors. Quote Link to post Share on other sites
cujo56 2 Posted March 3, 2015 Report Share Posted March 3, 2015 I have tried several things to address this problem in addition to the above. First, removing plugins that weren't needed in the bot. Second uninstalling recently added programs on my PC. I also disabled firewalls and security software. Finally, I uninstalled and reinstalled UbotStudio. I am still getting the same errors. Anyone else having these issues when scraping over 1000 pages? (mine will error out somewhere around 300 page +/-) Quote Link to post Share on other sites
deliter 203 Posted March 3, 2015 Report Share Posted March 3, 2015 yes I started playing with the scrape attribute node and I started getting this error,why not try $document text and regex it,to match multiple lines use (?s) 1 Quote Link to post Share on other sites
cujo56 2 Posted March 4, 2015 Report Share Posted March 4, 2015 Thanks for the suggestion. I gave the $document text a try. Got similar errors. The first one is different and says error converting value "<html.... (the rest the result $document text html follows) So I thought I would make a loop that tested the validity of the $document text just to make sure it was grabbing the HTML. Basically I would wait 4 seconds to make sure the page loads, grab the document, wait another 2 seconds just to slow it down, test if it was holding html, if not try again. That failed too. These errors feel like a memory leak to me... the bugs are impacting my ROI. Quote Link to post Share on other sites
deliter 203 Posted March 5, 2015 Report Share Posted March 5, 2015 so what did the document text say when it failed? nothing? actually,do you have an $eval node anywhere in your script? if so remove it,I don't know much javascript and I started playing with that and I got some VERY unexpected results,that was one of the problems for me 1 Quote Link to post Share on other sites
cujo56 2 Posted March 5, 2015 Report Share Posted March 5, 2015 This was the error from the $document text. error converting value "<html.... (the rest the result $document text html follows) No $eval nodes. I am really just grabbing HTML and picking it apart with regular expressions, storing data in lists for writing out. If the logic were bad it would fail on the first few times (I would think) and not wait till a few hundred loops into the script. It's almost like the browser is crashing or something? Could the target site be blocking me? Quote Link to post Share on other sites
itexspert 47 Posted March 6, 2015 Report Share Posted March 6, 2015 Did you notice your browser is crashing? Try to Open every new link from your List into new browser(I assume whatever you are scraping you first Scrape URL-s to collect all URL-s from some page and then you probably navigate to every page using $list item or $next list item So in your Loop set it up so every time Script navigates to the URL from $list item so here is the idea behind it set #list position = 0loop { Open new browser Scrape data close new browser increment #list position} This way your Browser will always be Fresh and Your script wont crash i am using the same logic with my Software and it works Flawlessly for my purposes,i just thought if this was a Browser issue maybe you could try it. Best Regards! 2 Quote Link to post Share on other sites
cujo56 2 Posted March 6, 2015 Report Share Posted March 6, 2015 itexspert - Good idea. I will have to give it some thought. The data I am scraping is generated dynamically on the page via button click and on page parameters so there isn't a url list to generate. It's doable I will just have to reset the parameters for each new browser... I will probably just get duplicate data. I will let you know. Quote Link to post Share on other sites
itexspert 47 Posted March 8, 2015 Report Share Posted March 8, 2015 Which website is this if you don't mind maybe we can help you with scraping logic? Quote Link to post Share on other sites
cujo56 2 Posted March 13, 2015 Report Share Posted March 13, 2015 Having some success. I scraped 1000 pages last night. The bot was failing at about 300 loops. Still don't know why. But I added an inner loop to wait 3 minutes every 100 iterations. Slowing it down seem to help. I am testing it now on a larger count. Quote Link to post Share on other sites
cujo56 2 Posted March 23, 2015 Report Share Posted March 23, 2015 OK I am back - I tried compiling the bot and running it that way. That failed even worse. So, I added a number of error handling routines along different places in the bot to try to capture the error and correct for possible errors. While I am able now to get up to 2000 iterations/pages it still locks up. Am I asking too much from Ubot Studio? Is it meant for bigger data collections? Don't take this wrong, I love the product and the support but I often feel like I am a beta tester and not a user. Quote Link to post Share on other sites
jpeter 1 Posted July 1, 2015 Report Share Posted July 1, 2015 I have exactly the same errors when trying to open Facebook pages and scroll them down and down again to scrape information about the posts published on that Facebook page. When and where it happens is a kind of random: sometimes it scrolls down normally, but sometimes (when the page is growing bigger than cca. 11 MBs), then it throws errors like the ones which are mentioned above. What should I do to prevent this from happening? (currently using the latest version, that is 5.7.1) Quote Link to post Share on other sites
pacbotter 1 Posted September 2, 2015 Report Share Posted September 2, 2015 I get this error on many sites. I am experiencing it on flippa now, but also many others. I published several trouble tickets on this - ubot always says they cannot reproduce the problem. Quote Link to post Share on other sites
Bot-Factory 602 Posted September 2, 2015 Report Share Posted September 2, 2015 There are mainly 2 reasons for this issue: 1. The awesomnium browser engine runs into some memory limitations. Or some other error.2. The website you try to automate is able to detect bots. Maybe you scroll to fast. You don't move naturally. There are lots of techniques out there to detect human behavior and to figure out if a human is navigating and using the site, or a machine. And I'm not talking about captchas. I talk about same timeouts when scrolling, mouse movements and so on. Kindest regardsDan Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.