Jump to content
UBot Underground

How To Get Ubot To Scroll Down A Page - As Content Only Loads On Scroll


Recommended Posts

Hi,

I have a client site I am trying to scrape the lot number, title and the url for each lot.

The site only seems to load more lots as you scroll down the page.

https://www.forumauctions.co.uk/Important-Books-Western-Manuscripts-and-Works-on-Paper-Day-One/15-11-2016?gridtype=listview

 

What action in Ubot will enable this scroll effect so it load more data, at present it will only load and scrape the data of the lots that I have already loaded, so have to manually scroll down the page to the end and then activate the bot.

 

also the urls its scraping from say Full Details for lot 1 are:

https://www.forumauctions.co.uk/index.php?option=com_auction&keyword=&name=&excl_keyword=&auction_no=1005&low_estimate=&high_estimate=&sort_by=lot_no&gridtype=listview&ipp=10&catId=&page_no=1&view=lot_detail&lot_id=25682&last_name=&first_name=&title=&auction_date=

 

but the correct url when you land on the page is:

https://www.forumauctions.co.uk/25682/Staffordshire.-Charter-c.-1217?auction_date=&auction_no=1005&catId=&excl_keyword=&first_name=&gridtype=listview&high_estimate=&ipp=10&keyword=&last_name=&low_estimate=&name=&page_no=1&sort_by=lot_no&title=&view=lot_detail

 

how do I get the end destination url scraped as I am only able to scrape the top level and not the level under?

 

If I manage to get the click - full details button working (not at the moment) how do I grab the url of a page I am on? cannot work that out

 

Many thanks

Andrew

 

Link to post
Share on other sites

To scroll you can use this:

run javascript("window.scrollTo(0,document.body.scrollHeight);")

Put that in a loop along with a wait and your code to scrape what you need.

 

As for the URL, there are a few ways but the easiest is to just navigate to the URL and then use the $url function located under browser functions to get the current URL - which will redirect for you when you navigate to it.

  • Like 1
Link to post
Share on other sites

Many thanks, for some reason I didn't get a notification you had responded so sorry for late thank you.

I will do as explained, and great on the url as a lot of my client sites have an initial url and then it pushes to a seo one which is best to scrape.

 

Kind regards

Andrew

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...