Newbie - Help Needed On My First Bot

[echo 1]

Hello:

 

Hoping someone can point me in the right direction here or share some best practices as I'm currently working on my first bot and running into some issues. 

 

I tried to build this bot back in April to assist with some mundane work my boss has me do daily.  I watched all of the videos and looked through the forums and was making progress.  Then we had our first child and somehow three months went by without having any real free time to work on the bot. 

 

Well now I'm setting aside the time to get this done but am running into the same issue as back in April.  I've watched the videos again to brush up but to no avail. 

 

I work for a management company and every day my boss has me go to five competitors websites to check  pricing and availability.  At first I was going to try and script a bot that goes to each properties website and scrapes the data into a table/excel file.  Given I'm new to this I decided that might be to much to start with.  So instead I noticed that four of the five seem to syndicate to the apartments.com website.  The fifth has their info on there but it's not perfect (I can go to their site for now until I learn the software better). 

 

So to start with my first bot I decided that I'd try and scrape the info from the apartments.com website.  Once I had that working I'd then start working on really digging into each company's website and trying to scrape them for fun and to build up the Ubot skills.

 

I've been trying two different approaches and here is the code for each:

 

METHOD #1

 

navigate("http://www.apartments.com/the-larimore-omaha-ne/db8jnjh/","Wait")

clear list(%Apartment List)

scrape table(<class="availabilityTable  ">,&table scrape)

 

 

METHOD #2

 

navigate("http://www.apartments.com/the-larimore-omaha-ne/db8jnjh/","Wait")

clear list(%Bedrooms)

add list to list(%Bedrooms,$scrape attribute($element offset(<class="beds">,1),"innertext"),"Don\'t Delete","Global")

clear list(%Bathrooms)

add list to list(%Bathrooms,$scrape attribute($element offset(<class="shortText">,5),"innertext"),"Delete","Global")

 

 

Scraping the table in the first method seems to work with the only exception being that it pulls the beds &  baths in three separate ways.  I've tried digging into the HTML code and found the different classes but can't seem to get the bot to scrape specific class to just list it in the chart as "2 Bedrooms / 2 Bathrooms, etc…

 

I'm also not sure if this method will allow me to export the rows and columns into an excel file.

 

For the second method I've taken the approach of using variables for each item in the chart (beds, baths, rent, available date, etc…).  From there I'm hoping to add them to a list that I can then export to an excel file.  My issue with this method is that I've been able to pull the data for 1 row but not sure how to pull it for the rest of them on the page.  There is never a set number of rows since the data can change daily. 

 

Between work and having a newborn still my time is stretched so thin.  So if anyone can give me some pointers I'd really appreciate it.  The last thing I want to go is waste time on a method that end the end just won't work no matter how many hours I spend staring at it. 

 

I'm obviously new to this and have already hit a very frustrating wall…

 

Thank you!

 

 

 

[Brutal 164]

Here is a good start for you...

navigate("http://www.apartments.com/the-larimore-omaha-ne/db8jnjh/","Wait")
wait for browser event("Everything Loaded","")
set(#get stuff,$scrape attribute($element offset(<class="availabilityTable  ">,0),"innertext"),"Global")
add list to list(%stuff i got,$list from text(#get stuff,$new line),"Delete","Global")
with each(%stuff i got,#stuff i want) {
    if($contains(#stuff i want,"Not")) {
        then {
        }
        else {
            add list to list(%stuff i want,$list from text(#stuff i want,$new line),"Delete","Global")
        }
    }
}

[echo 1]

Thanks Brutal!

 

I'll give that a shot and see what happens.

[echo 1]

 

Here is a good start for you...

navigate("http://www.apartments.com/the-larimore-omaha-ne/db8jnjh/","Wait")
wait for browser event("Everything Loaded","")
set(#get stuff,$scrape attribute($element offset(<class="availabilityTable  ">,0),"innertext"),"Global")
add list to list(%stuff i got,$list from text(#get stuff,$new line),"Delete","Global")
with each(%stuff i got,#stuff i want) {
    if($contains(#stuff i want,"Not")) {
        then {
        }
        else {
            add list to list(%stuff i want,$list from text(#stuff i want,$new line),"Delete","Global")
        }
    }
}

So I gave it a shot this evening.  That code does pull the data from the table...now I'm just working with it to figure out how to get the data in the lists converted into a table that I can export into Excel.  

 

Thank you for the good start and hopefully I can work through the rest over the weekend.

[deliter 203]

You can also scrape data attributes,you'l see in this class it has all the data you need

clear list(%Bedrooms)
clear list(%Bathrooms)
add list to list(%Bedrooms,$scrape attribute(<class="rentalGridRow   bold first">,"data-beds"),"Don\'t Delete","Global")
add list to list(%Bathrooms,$scrape attribute(<class="rentalGridRow   bold first">,"data-baths"),"Don\'t Delete","Global")
set table cell(&rooms,0,0,"Bedrooms")
set table cell(&rooms,0,1,"Bathrooms")
add list to table as column(&rooms,1,0,%Bedrooms)
add list to table as column(&rooms,1,1,%Bedrooms)

[echo 1]

Hi Deliter,

 

Thank you for the information and direction! I'm going to get a chance to give this one a shot tonight as well.

 

I'll post my results in the morning!

 

You can also scrape data attributes,you'l see in this class it has all the data you need

 

clear list(%Bedrooms)
clear list(%Bathrooms)
add list to list(%Bedrooms,$scrape attribute(<class="rentalGridRow   bold first">,"data-beds"),"Don\'t Delete","Global")
add list to list(%Bathrooms,$scrape attribute(<class="rentalGridRow   bold first">,"data-baths"),"Don\'t Delete","Global")
set table cell(&rooms,0,0,"Bedrooms")
set table cell(&rooms,0,1,"Bathrooms")
add list to table as column(&rooms,1,0,%Bedrooms)
add list to table as column(&rooms,1,1,%Bedrooms)