dbwebhosting 0 Posted May 11, 2014 Report Share Posted May 11, 2014 What am i doing wrong here guys, ui stat monitor("Company Names", $list total(%companies)) ui stat monitor("Contact Number", $list total(%contactnumbers)) ui stat monitor("Address", $list total(%address)) clear table(&Contacts) clear list(%address) clear list(%companies) clear list(%contactnumbers) navigate("http://www.thomsonlocal.com/listing/search?Phrase=publishers&Location=London", "Wait") define Scrape Data { add list to list(%companies, $scrape attribute(<tagname="h2">, "outertext"), "Don\'t Delete", "Global") add list to table as column(&Contacts, 0, 0, %companies) add list to list(%contactnumbers, $scrape attribute(<class="tel">, "outertext"), "Don\'t Delete", "Global") add list to table as column(&Contacts, 0, 1, %contactnumbers) add list to list(%address, $scrape attribute(<class="address ">, "outertext"), "Delete", "Global") add list to table as column(&Contacts, 0, 2, %address) } loop(100) { if($exists(<id="nextPg">)) { then { Scrape Data() click(<id="nextPg">, "Left Click", "No") wait for browser event("DOM Ready", "") } else { } } } save to file("{$special folder("Desktop")}/results.csv", &Contacts) When i run it my stats run off differently and then mess up my results. This is my first script so please help Quote Link to post Share on other sites
UBotDev 276 Posted May 11, 2014 Report Share Posted May 11, 2014 First you should start with adding "wait for element" commands on every step where page is reloaded. Also check t that all items you scrape have "tel" and "address", that way your stats won't go off. If "tel" is missing for example your tel column would be off from that item on. If this is the case I think you'll want to scrape with "$element offset" function. Quote Link to post Share on other sites
dbwebhosting 0 Posted May 11, 2014 Author Report Share Posted May 11, 2014 It does it perfectly when i only scrape the first page (25 results) when i loop it to scrape multiple pages it goes wrong Quote Link to post Share on other sites
UBotDev 276 Posted May 11, 2014 Report Share Posted May 11, 2014 As I said, first add wait commands, since without them you are not waiting for the page to get loaded (meaning that the content that you are scraping is still being loaded at that moment and is not available for scraping).Also I don't think there is a need to open multiple topics for the same issue, when you posted the code here already. Quote Link to post Share on other sites
dbwebhosting 0 Posted May 12, 2014 Author Report Share Posted May 12, 2014 Hi guys, i stopped the above code and changed to using Google. I have a small error with my code. When i scrape the address to a list it puts it onto 2 lines instead of appending it to 1 line. Can you advise me where i am missing ui text box("Keyword", #Keyword) ui text box("Location", #Location) ui text box("Email", #email) ui text box("File Name", #filename) ui stat monitor("<br>", "") ui stat monitor("<br>", "") ui stat monitor("<br>", "") ui stat monitor("<br>", "") ui stat monitor("Company Names", $list total(%companies)) ui stat monitor("Contact Number", $list total(%contactnumbers)) ui stat monitor("Address", $list total(%address)) clear table(&Contacts) clear list(%address) clear list(%companies) clear list(%contactnumbers) navigate("https://www.google.co.uk/maps", "Wait") wait for browser event("Everything Loaded", "") type text(<name="q">, "{#Keyword} in {#Location}", "Standard") wait for browser event("DOM Ready", "") click(<class="gbqfi gb_Aa">, "Left Click", "No") wait for browser event("Everything Loaded", "") define Scrape Data { click(<alt="A">, "Left Click", "No") wait for browser event("Everything Loaded", "") add list to list(%companies, $scrape attribute(<id="place-title">, "innertext"), "Delete", "Global") add list to table as column(&Contacts, 0, 0, %companies) add list to list(%address, $scrape attribute(<jscontent="$addrline">, "label"), "Delete", "Global") add list to table as column(&Contacts, 0, 1, %address) add list to list(%contactnumbers, $scrape attribute(<class="phone">, "innertext"), "Delete", "Global") add list to table as column(&Contacts, 0, 2, %contactnumbers) } Scrape Data() save to file("{$special folder("Desktop")}\\{#filename}.csv", &Contacts) Quote Link to post Share on other sites
dbwebhosting 0 Posted May 12, 2014 Author Report Share Posted May 12, 2014 Edit: Fixed the issue i just posted. Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.