cujo56

OK I am back - I tried compiling the bot and running it that way. That failed even worse. So, I added a number of error handling routines along different places in the bot to try to capture the error and correct for possible errors. While I am able now to get up to 2000 iterations/pages it still locks up. Am I asking too much from Ubot Studio? Is it meant for bigger data collections? Don't take this wrong, I love the product and the support but I often feel like I am a beta tester and not a user.

Having some success. I scraped 1000 pages last night. The bot was failing at about 300 loops. Still don't know why. But I added an inner loop to wait 3 minutes every 100 iterations. Slowing it down seem to help. I am testing it now on a larger count.

itexspert - Good idea. I will have to give it some thought. The data I am scraping is generated dynamically on the page via button click and on page parameters so there isn't a url list to generate. It's doable I will just have to reset the parameters for each new browser... I will probably just get duplicate data. I will let you know.

This was the error from the $document text. error converting value "<html.... (the rest the result $document text html follows) No $eval nodes. I am really just grabbing HTML and picking it apart with regular expressions, storing data in lists for writing out. If the logic were bad it would fail on the first few times (I would think) and not wait till a few hundred loops into the script. It's almost like the browser is crashing or something? Could the target site be blocking me?

Thanks for the suggestion. I gave the $document text a try. Got similar errors. The first one is different and says error converting value "<html.... (the rest the result $document text html follows) So I thought I would make a loop that tested the validity of the $document text just to make sure it was grabbing the HTML. Basically I would wait 4 seconds to make sure the page loads, grab the document, wait another 2 seconds just to slow it down, test if it was holding html, if not try again. That failed too. These errors feel like a memory leak to me... the bugs are impacting my

I have tried several things to address this problem in addition to the above. First, removing plugins that weren't needed in the bot. Second uninstalling recently added programs on my PC. I also disabled firewalls and security software. Finally, I uninstalled and reinstalled UbotStudio. I am still getting the same errors. Anyone else having these issues when scraping over 1000 pages? (mine will error out somewhere around 300 page +/-)

The additional loop didn't make a difference. Same errors.

Thanks for the suggestion and great support. I thought the loop would work but once again I get an error... http://i.imgur.com/UEfhNtL.jpg?1 Then this JSON error after hitting "Continue". http://i.imgur.com/b7qGD0Q.jpg I will try your $exists loop on the <id=genbtn"> html also... Hope that makes a difference.

Looks like the problem still exists. I am back to square one on this. Any ideas as to what is triggering these errors?

Ugh... OK, I moved just the working/running parts of my bot to a whole new file. (I had some other scripts/tabs in the original file that were still in development). Ran a test to loop 300 pages and it worked without error. I will test on a higher number next. I will post back here if there is still a problem. Otherwise, no news is good news. Thanks.

Here is an additional error message I am getting after the JSON. Error converting value True to type 'System.Collections.Generic.List`1[system.String]'.Path ", line 1, position 4. Source: pagescraperbot -> -> set -> $scrape attribute (<class="info">, "innerhtml")-> $scrape attribute(<class="info">,innerhtml")->$scrape attribute

Here is the Define it's happening in. I had this chuck of code nested before. I thought breaking it out might help me see the problem better. define Get Identity { set(#Identity,$scrape attribute(<class="info">,"innerhtml"),"Global") set(#Identity,$replace regular expression(#Identity,"\\t|\\r|\\n| ",$nothing),"Global") set(#Identity,$replace regular expression(#Identity,"(?<=>)\\s+(?=<)",$nothing),"Global") set(#Identity,$replace regular expression(#Identity,"(?<=>)\\s+(?=[a-zA-Z0-9\\:])",$nothing),"Global") set(#Identity,$replace regular expressio

Ok, I am going to jump into this thread. I am having the same problem. I have tried increasing the wait time between page reloads in my scrapper (just make sure everything was fully loaded and account for sluggish internet connections). I get about 100 pages into a scrape and I get the same messages. I am working on a theory. Could Google Adsense on a site be a problem? After all, scripts like these could throw off their impression metrics (something they pride themselves on). Just not sure how to test the idea? How can we do ad blocking?

ok... While I had UbotStudio set to Allow in Norton 360 I think the update program was being blocked. Disable Norton, restarted UbotStudio. It ran through the update process now works as normal.

smartscreen is off, .NET 4.5.2. I did manage to get an error message during one attempt. "error checking files" or something like that. Still locked up.

Version 5.5 won't start on my Windows 8 system... it's stuck at the initializing process. Cool new startup image though! Does anyone have any ideas? Thanks, Joe

yeah, I was thinking of how True works in PHP... (so long as the variable has a value and isn't false it is then considered True)

Thanks Steve and Dan, Yes 1=1, and true = $true. I am trying to see if the variable as a value, any value. Is $true just for checkboxes? . Asserting the negative, !=, with $nothing works. Related, is there a built in way to test a variable type (numeric integer/decimal, text, date, etc.) or do I have test my variable with a regular expression?

Is it just my system (or me) but is there something wrong with the Ubot Comparison function? set(#testvar, 1, "Global") if($comparison(#testvar, "=", $true)) { then { alert("Has Value") } else { alert("Does Not Have Value") } } I would expect this simple statement to pop up the Alert with "Has Value" because the variable #testvar has a value of 1 and is true. Instead ubot is telling be this is false and giving me the else Alert. What am I missing here?

Thanks for the quick and detailed reply. I think this will work for my needs... I am scraping all internal URLs from a website so I need to remove pages/urls already scanned from a "to do" list and place them in the "final list".

I have Ubot Studio and I am looking for a way to remove one list of items from another list. I understand the Developer version has a command to do this ($subtract lists) however that isn't available to me. Is there a "manual" way to do this? Thanks for your help. Joe

Version 4 seems to be solving all my problems so far. I think they need to include that as an optional download when you purchase version 5 (which seems to work ok on small bots).

Quick update for those following this thread. Tech support has been great in supporting me, so has the community at large. Thanks to all. While I don't have a Ubot 5 solution yet, Ubot 4 runs my bot much faster without crashing (so far). I was getting a lot of lag time when switching tabs in 5 which is gone now in 4. You need to ask tech support for a link to version 4 if you think you need it. With that I can't find a list of difference between the two versions anywhere. So I am not 100% if I have lost any functionality in my script due to the rollback. I hope this helps someone else

Thanks jomark3, I have installed version 4 and my bot is running much faster now in the editor. I need say that tech support has been very responsive with my version 5 issues and are working on a solution. Some of them are documented in the bug tracker like large bots running slow and regex issues. I trust they will get it worked out.

Thanks for the reply. Nesting where needed? Not sure I understand. I use it a lot for setting a variable to avoid having 5 or 6 individual reassignments of the same variable. Like in my example, Set Content Chunk - > $trim -> $replace -> $replace regular expression -> $find regular expression (in the last case I might be using a previous assignment of Content Chunk which is a larger section of the scrape). I don't need to do it that way but it seems to be cleaner and easier when going back to edit it. Is that a bad idea?