a-harvey 0 Posted October 25, 2016 Report Share Posted October 25, 2016 Hi,I have a client site I am trying to scrape the lot number, title and the url for each lot.The site only seems to load more lots as you scroll down the page.https://www.forumauctions.co.uk/Important-Books-Western-Manuscripts-and-Works-on-Paper-Day-One/15-11-2016?gridtype=listview What action in Ubot will enable this scroll effect so it load more data, at present it will only load and scrape the data of the lots that I have already loaded, so have to manually scroll down the page to the end and then activate the bot. also the urls its scraping from say Full Details for lot 1 are:https://www.forumauctions.co.uk/index.php?option=com_auction&keyword=&name=&excl_keyword=&auction_no=1005&low_estimate=&high_estimate=&sort_by=lot_no&gridtype=listview&ipp=10&catId=&page_no=1&view=lot_detail&lot_id=25682&last_name=&first_name=&title=&auction_date= but the correct url when you land on the page is:https://www.forumauctions.co.uk/25682/Staffordshire.-Charter-c.-1217?auction_date=&auction_no=1005&catId=&excl_keyword=&first_name=&gridtype=listview&high_estimate=&ipp=10&keyword=&last_name=&low_estimate=&name=&page_no=1&sort_by=lot_no&title=&view=lot_detail how do I get the end destination url scraped as I am only able to scrape the top level and not the level under? If I manage to get the click - full details button working (not at the moment) how do I grab the url of a page I am on? cannot work that out Many thanksAndrew Quote Link to post Share on other sites
HelloInsomnia 1103 Posted October 26, 2016 Report Share Posted October 26, 2016 To scroll you can use this: run javascript("window.scrollTo(0,document.body.scrollHeight);") Put that in a loop along with a wait and your code to scrape what you need. As for the URL, there are a few ways but the easiest is to just navigate to the URL and then use the $url function located under browser functions to get the current URL - which will redirect for you when you navigate to it. 1 Quote Link to post Share on other sites
a-harvey 0 Posted October 28, 2016 Author Report Share Posted October 28, 2016 Many thanks, for some reason I didn't get a notification you had responded so sorry for late thank you.I will do as explained, and great on the url as a lot of my client sites have an initial url and then it pushes to a seo one which is best to scrape. Kind regardsAndrew Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.