[FREE] Intuit - scrapper

[Security 17]

Hey guys.. Let me first say i have no programming experience! Im sure there are a lot of better ways to do the things I have done but i did try my best.

 

I actually spend hours on Regular expressions trying to show off, only to find out in the end there was a "better" way.

 

Right now im kinda stuck. Im trying to remove "Send to Phone" and " Get Directions". I tried to set the list as a variable then do the replace then change it back. But for some reason its not working..

 

Please check this source out. Let me know if there are things i could have done better. And if it turns out anyone can actually learn from my bot.. more power to you.

 

Thanks in Advance.

 

 

ui text box("Search for:", #search)
ui text box("City, State or Zip", #geo)
ui stat monitor("Leads Count:", $list total(%leads))
clear cookies
clear list(%links)
clear list(%leads)
set user agent("Chrome")
set visibility("Visible")
navigate("http://business.intuit.com/directory/", "Wait")
wait for browser event("Page Loaded", "")
type text(<full name field>, #search, "Standard")
type text(<name="location">, #geo, "Standard")
click(<id="searchSubmit">, "Left Click", "No")
wait for browser event("Page Loaded", "")
if($exists(<innertext="next >">)) {
   then {
       loop while($exists(<innertext="next >">)) {
           add list to list(%links, $scrape attribute(<href=r"directory/\\w+?-[a-z0-9-/]\{21,\}[^reviews\"]1">, "fullhref"), "Delete", "Global")
           click(<innertext="next >">, "Left Click", "No")
           wait for browser event("Page Loaded", "")
       }
   }
   else {
       add list to list(%links, $scrape attribute(<href=r"directory/\\w+?-[a-z0-9-/]\{21,\}[^reviews\"]1">, "fullhref"), "Delete", "Global")
       clear list(%links2)
       set list position(%links, 0)
       loop($list total(%links)) {
           navigate($next list item(%links), "Wait")
           wait for browser event("Page Loaded", "")
           add list to list(%leads, $scrape attribute(<id="phone">, "innertext"), "Delete", "Global")
           add list to list(%leadss, $scrape attribute(<id="address">, "innertext"), "Delete", "Global")
           add list to table as column(&leads, 0, 0, %leads)
           add list to table as column(&leads, 0, 1, %leadss)
       }
   }
}
save to file("{$special folder("Desktop")}/intuit.leads.csv", &leads)
stop script

[Legend 181]

Have you tried using regex and scraping everything up to "Send to Phone" and/or "Get Directions"?

http://ubotstudio.com/forum/public/style_emoticons/default/huh.gif

[Security 17]

I actually started to do it like that.. but then i noticed the id's gave me everything only added those extra words.. i was thinking this was the best way to do it.. Figured there would be some easier way to remove those extra things..

[Legend 181]

You can do something like this...

 

add item to list(%leads, $replace($scrape attribute(<id="phone">, "innertext"), "Send to Phone", $nothing), "Delete", "Global")

add item to list(%leadss, $replace($scrape attribute(<id="address">, "innertext"), "Get Directions", $nothing), "Delete", "Global")

 

or the regex route...

 

add item to list(%leads, $replace regular expression($scrape attribute(<id="phone">, "innertext"), "Send to Phone", $nothing), "Delete", "Global")

add item to list(%leads, $replace regular expression($scrape attribute(<id="address">, "innertext"), "Get Directions", $nothing), "Delete", "Global")

 

 

http://ubotstudio.com/forum/public/style_emoticons/default/rolleyes.gif

[Security 17]

Thanks a lot man.. Im going to check this out i been trying to figure out all the ways to use "replace" and i thought you had to do it by its self.. didnt know you could use it along with scrape attribute .. thanks again im going to check this out..

[Security 17]

or the regex route...

 

add item to list(%leads, $replace regular expression($scrape attribute(<id="phone">, "innertext"), "Send to Phone", $nothing), "Delete", "Global")

add item to list(%leads, $replace regular expression($scrape attribute(<id="address">, "innertext"), "Get Directions", $nothing), "Delete", "Global")

 

 

http://ubotstudio.com/forum/public/style_emoticons/default/rolleyes.gif

 

 

Shouldn't there be some regex in there ?

[Security 17]

You can do something like this...

 

add item to list(%leads, $replace($scrape attribute(<id="phone">, "innertext"), "Send to Phone", $nothing), "Delete", "Global")

add item to list(%leadss, $replace($scrape attribute(<id="address">, "innertext"), "Get Directions", $nothing), "Delete", "Global")

 

or the regex route...

 

add item to list(%leads, $replace regular expression($scrape attribute(<id="phone">, "innertext"), "Send to Phone", $nothing), "Delete", "Global")

add item to list(%leads, $replace regular expression($scrape attribute(<id="address">, "innertext"), "Get Directions", $nothing), "Delete", "Global")

 

 

http://ubotstudio.com/forum/public/style_emoticons/default/rolleyes.gif

 

 

Thanks a lot man.. worked great.

[Legend 181]

Shouldn't there be some regex in there ?

Technically there is.... any text string (like "Send to Phone" or "Get Directions" can work inside regex.. it is just a regular expression in its simplest form. You can replace them with "Send\sto\sPhone" or "Get\sDirections" and it should work the same way. UBot has a pretty unique regex flavor though, so you never know until you try it... http://ubotstudio.com/forum/public/style_emoticons/default/rolleyes.gif

 

Thanks a lot man.. worked great.

Cool! http://ubotstudio.com/forum/public/style_emoticons/default/smile.gif

[Security 17]

Technically there is.... any text string (like "Send to Phone" or "Get Directions" can work inside regex.. it is just a regular expression in its simplest form. You can replace them with "Send\sto\sPhone" or "Get\sDirections" and it should work the same way. UBot has a pretty unique regex flavor though, so you never know until you try it... http://ubotstudio.com/forum/public/style_emoticons/default/rolleyes.gif

 

 

Cool! http://ubotstudio.com/forum/public/style_emoticons/default/smile.gif

 

 

Well i was actually going to add the "\s".. believe me by now i know exactly what a regex string can be.. lol That video tut i sent you is just one of thousands of thing i have overloaded my brain with..

 

Anyways i got some new code to show you.. I'm officially DONE..!

 

ui text box("Search for:", #search)
ui text box("City, State or Zip", #geo)
ui stat monitor("Lead Count:", $list total(%leads))
ui stat monitor("Links collected", $list total(%links))
clear cookies
clear list(%cycles)
clear list(%links)
clear list(%leads)
set user agent("Chrome")
set visibility("Visible")
navigate("http://business.intuit.com/directory/", "Wait")
wait for browser event("Page Loaded", "")
type text(<full name field>, #search, "Standard")
type text(<name="location">, #geo, "Standard")
click(<id="searchSubmit">, "Left Click", "No")
wait for browser event("Page Loaded", "")
set(#cycles, $scrape attribute(<innertext=w"Results *-* of *">, "innertext"), "Global")
set(#cycles, $replace(#cycles, "Results 1-20 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 1-19 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 1-1 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 1-2 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 1-3 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 1-3 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 1-4 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 1-5 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 1-6 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 1-7 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 1-8 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 1-9 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 1-10 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "11-31 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 21-36 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 31-40 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 41-50 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 61-70 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 71-80 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 81-90 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 91-100 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 101-110 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 111-120 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 121-130 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 131-140 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 141-150 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 151-160 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 161-170 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 171-180 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 181-190 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 191-200 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 201-210 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 211-220 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 221-230 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 231-240 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 241-250 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 251-260 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 261-270 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 271-280 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 281-290 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 291-300 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 301-310 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 311-320 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 321-330 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 331-340 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 341-350 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 351-360 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 361-370 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 371-380 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 381-390 of ", $nothing), "Global")
set(#cycles, $replace(#cycles, "Results 391-400 of ", $nothing), "Global")
set(#cycles, $eval("Math.ceil({#cycles}/8);"), "Global")
loop(#cycles) {
   add list to list(%links, $scrape attribute(<href=r"directory/\\w+?-[a-z0-9-/]\{21,\}[^reviews\"]1">, "fullhref"), "Delete", "Global")
   click(<innertext="next >">, "Left Click", "No")
   wait for browser event("Results ", "")
}
set list position(%links, 0)
loop($list total(%links)) {
   navigate($next list item(%links), "Wait")
   wait for browser event("Page Loaded", "")
   add item to list(%leads, $replace($scrape attribute(<id="phone">, "innertext"), "Send to Phone", $nothing), "Delete", "Global")
   add item to list(%leadss, $replace($scrape attribute(<id="address">, "innertext"), "Get Directions", $nothing), "Delete", "Global")
   add list to table as column(&leads, 0, 0, %leads)
   add list to table as column(&leads, 0, 1, %leadss)
}
save to file("{$special folder("Desktop")}/intuit.leads.csv", &leads)
stop script

 

 

 

Please let me know if there is a better way to have done this.. ANYONE!

[Legend 181]

Yup... great intro to Regex... working my way through Friedl's book now... http://ubotstudio.com/forum/public/style_emoticons/default/wink.gif

 

Your code didn't work for me on first run... marketing in Denver gave me Results 1-12 which isn't in your replace list...

 

I would replace all those inefficient replaces with something like:

 

set(#cycles, $replace regular expression($scrape attribute(<id="resultsBar">, "innertext"), "Results\\s\\d-\\d\\sof\\s", $nothing), "Global")

set(#cycles, $replace regular expression(#cycles, "[^of]+(?=of)", $nothing), "Global")

set(#cycles, $replace(#cycles, "of ", $nothing), "Global")

 

Probably some way to condense that even further, but I don't have time to play with it...

 

http://ubotstudio.com/forum/public/style_emoticons/default/rolleyes.gif

[Security 17]

wow.. cool.. thnx.

[Security 17]

I would replace all those inefficient replaces with something like:

 

set(#cycles, $replace regular expression($scrape attribute(<id="resultsBar">, "innertext"), "Results\\s\\d-\\d\\sof\\s", $nothing), "Global")

set(#cycles, $replace regular expression(#cycles, "[^of]+(?=of)", $nothing), "Global")

set(#cycles, $replace(#cycles, "of ", $nothing), "Global")

 

Probably some way to condense that even further, but I don't have time to play with it...

 

http://ubotstudio.com/forum/public/style_emoticons/default/rolleyes.gif

 

wow.. cool.. thnx. Seems like who ever made this site made it randomize the "1-*'s" so every few searches it picks i new one. I didnt know i could use regex to do that.. I was watching the "advance yellow book scrape" tutorial They did it like that there, so i just coipied.

 

I remember John B. saying on the tut (in text) "i could have made a math expression to to this" meaning the replace, rather then copying and pasting. Would you happen to know how to do that ?

[Legend 181]

I remember John B. saying on the tut (in text) "i could have made a math expression to to this" meaning the replace, rather then copying and pasting. Would you happen to know how to do that ?

No... perhaps he'll enlighten us... I'm not sure about math expressions...

[Security 17]

No... perhaps he'll enlighten us... I'm not sure about math expressions...

 

 

HA.. good luck on that one.. Good old John B. was "helping" me with something, said he would get right back to me.. That was the last time i spoke to him.. Oh wait, no he did msg me to tell me i could not upload a exe/or use file share sites.. I pretty much wrote off any help from him.. But thanks to you i have actually been learning.. thanks a lot man..

[Legend 181]

HA.. good luck on that one.. Good old John B. was "helping" me with something, said he would get right back to me.. That was the last time i spoke to him.. Oh wait, no he did msg me to tell me i could not upload a exe/or use file share sites.. I pretty much wrote off any help from him.. But thanks to you i have actually been learning.. thanks a lot man..

I know John's a very busy guy and even though he is, he has found the time to be most instrumental in my learning the ins and outs of UBot (and I know I'm not alone)... give him a nudge if something's slipped his mind or hunt him down on Skype... he's almost always around to help... http://ubotstudio.com/forum/public/style_emoticons/default/smile.gif

[Security 17]

Just realized i could have injected the Zip + search word right into the URL. Rather then having to type/click.. that would make the code a bit smaller.

[Security 17]

Do you know anyone i could pay ? i would like to get a tutor.. Someone who i could either help in some way and or just pay..

 

If this was a hobby i could wait around for help but i got this program to make my company run more efficient. In my eyes im making good progress. Considering i have no coding experience and didn't even know what "Regex" ment before buying this program. But i feel with some stable assistance i could be doing a whole lot more.

[Security 17]

Do you know anyone i could pay ? i would like to get a tutor.. Someone who i could either help in some way and or just pay..

 

If this was a hobby i could wait around for help but i got this program to make my company run more efficient. In my eyes im making good progress. Considering i have no coding experience and didn't even know what "Regex" ment before buying this program. But i feel with some stable assistance i could be doing a whole lot more.

 

 

let me be a little more clear.. i am in no way looking for someone to make myt bots.. or even help me make bots.. I'm looking for a teacher.

 

Someone who knows how to lead me in the right direction enabling me to learn more with out having to guess/google/post everytime i want to do something new.

 

Im not looking to pay someone to code for me just so everyone knows..

[Security 17]

Ok there it is.. same program.. only WAY less code. + now it collects company name now as well..

 

 

Please let me know if there is anything i could have done better..

 

Thanks..

 

ui text box("Search for:", #search)
ui text box("City, State or Zip", #geo)
ui stat monitor("Lead Count:", $list total(%leads))
ui stat monitor("Links collected", $list total(%links))
clear cookies
clear list(%cycles)
clear list(%links)
clear list(%leads)
clear list(%leadss)
clear list(%leadsss)
set user agent("Chrome")
set visibility("Visible")
navigate("http://business.intuit.com/directory/", "Wait")
wait for browser event("Page Loaded", "")
type text(<full name field>, #search, "Standard")
type text(<name="location">, #geo, "Standard")
click(<id="searchSubmit">, "Left Click", "No")
wait for browser event("Page Loaded", "")
set(#cycles, $scrape attribute(<innertext=w"Results *-* of *">, "innertext"), "Global")
set(#cycles, $replace regular expression(#cycles, "Results\\s\\d\{1,3\}-\\d\{1,3\}\\sof\\s", $nothing), "Global")
set(#cycles, $eval("Math.ceil({#cycles}/8);"), "Global")
loop(#cycles) {
   add list to list(%links, $scrape attribute(<href=r"directory/\\w+?-[a-z0-9-/]\{21,\}[^reviews\"]1">, "fullhref"), "Delete", "Global")
   click(<innertext="next >">, "Left Click", "No")
   wait for browser event("Results ", "")
}
set list position(%links, 0)
loop($list total(%links)) {
   navigate($next list item(%links), "Wait")
   wait for browser event("Page Loaded", "")
   add item to list(%leads, $replace($scrape attribute(<id="phone">, "innertext"), "Send to Phone", $nothing), "Delete", "Global")
   add item to list(%leadss, $replace($scrape attribute(<id="address">, "innertext"), "Get Directions", $nothing), "Delete", "Global")
   add item to list(%leadsss, $replace($scrape attribute(<outerhtml=w"<h1>*<span id=\"speech-bubbles\"></span><a id=\"linkText\" rel=\"nofollow\" href=\"javascript:void(0);\" onclick=\"pageTracker._trackEvent(\'GC51.5-DryTest\',\'Link\',\'Email Me\');OpenPopup(\'Email Me\');\">Email Me</a></h1>">, "innertext"), "Email", $nothing), "Delete", "Global")
   add list to table as column(&leads, 0, 0, %leads)
   add list to table as column(&leads, 0, 1, %leadss)
   add list to table as column(&leads, 0, 2, %leadsss)
   save to file("{$special folder("Desktop")}/intuit.leads.csv", &leads)
}
stop script

[Legend 181]

Do you know anyone i could pay ? i would like to get a tutor.. Someone who i could either help in some way and or just pay..

Try TJ or Buddy... http://ubotstudio.com/forum/public/style_emoticons/default/smile.gif

 

 

 

[Security 17]

Try TJ or Buddy... http://ubotstudio.com/forum/public/style_emoticons/default/smile.gif

 

 

I did thanx..

[Security 17]

OK.. Intuit Scrapper Updated:

Timer added

Run time faster

can now type in city/city +state (with a comma), or zip..

 

Only problem im having now is on Large scrapes like say "Memphis" / "Automotive" seems like lock up and not move from scrapping links to scrapping leads..

 

I think the Answer to this would be to create a variable ending the click "next" when the variable is = to the last series of number Results In this case "1-20 of 50." the click "next" would end when the second series of numbers is = to the last. So when the "20" is displayed as "50" then it will end and move on to scrapping the data off of the links it collected.

 

Im pretty sure i know how to do this.. So im going to try now..

 

Here is my updated source.. Im very proud of this so far.. lol. Enjoy...

 

 

 

set visibility("Invisible")
allow images("No")
allow javascript("No")
allow css("No")
allow flash("No")
ui text box("Search for:", #search)
ui text box("City, State or Zip", #geo)
ui stat monitor("Lead Count:", $list total(%leads))
ui stat monitor("Links collected", $list total(%links))
ui stat monitor("Run Time:", "{#timer} seconds")
clear cookies
clear list(%cycles)
clear list(%links)
clear list(%leads)
clear list(%leadss)
clear list(%leadsss)
set(#startstop, 0, "Global")
set(#timer, 0, "Global")
thread {
   loop while($comparison(#startstop, "=", 0)) {
       increment(#timer)
       wait(1)
   }
}
set user agent("Chrome")
navigate("http://business.intuit.com/directory/search?name={#search}&location={#geo}", "Wait")
wait for browser event("Page Loaded", "")
set(#cycles, $scrape attribute(<innertext=w"Results *-* of *">, "innertext"), "Global")
set(#cycles, $replace regular expression(#cycles, "Results\\s\\d\{1,3\}-\\d\{1,3\}\\sof\\s", $nothing), "Global")
set(#cycles, $eval("Math.ceil({#cycles}/8);"), "Global")
loop(#cycles) {
   add list to list(%links, $scrape attribute(<href=r"directory/\\w+?-[a-z0-9-/]\{21,\}[^reviews\"]1">, "fullhref"), "Delete", "Global")
   click(<innertext="next >">, "Left Click", "No")
   wait for browser event("Results ", "")
}
set list position(%links, 0)
loop($list total(%links)) {
   navigate($next list item(%links), "Wait")
   wait for browser event("Page Loaded", "")
   add item to list(%leads, $replace($scrape attribute(<id="phone">, "innertext"), "Send to Phone", $nothing), "Delete", "Global")
   add item to list(%leadss, $replace($scrape attribute(<id="address">, "innertext"), "Get Directions", $nothing), "Delete", "Global")
   add item to list(%leadsss, $replace($scrape attribute(<outerhtml=w"<h1>*<span id=\"speech-bubbles\"></span><a id=\"linkText\" rel=\"nofollow\" href=\"javascript:void(0);\" onclick=\"pageTracker._trackEvent(\'GC51.5-DryTest\',\'Link\',\'Email Me\');OpenPopup(\'Email Me\');\">Email Me</a></h1>">, "innertext"), "Email Me", $nothing), "Delete", "Global")
   add list to table as column(&leads, 0, 0, %leads)
   add list to table as column(&leads, 0, 1, %leadss)
   add list to table as column(&leads, 0, 2, %leadsss)
   save to file("{$special folder("Desktop")}/intuit.leads.csv", &leads)
}
increment(#startstop)
stop script

[thefidler 0]

I just purchased and would also like a coach to work with

[Security 17]

Well its been a long time since i made this post. Im pretty sure by now i would qualify as a "coach".. PM me..