Scraping Instagram Insights Via Business.facebook/creator Studio

[ugakebet 4]

Hi guys,

 

I'm not sure if anyone has been able to do this.. But I have been trying to scrape Insta insights using the Content Studio option provided by FB..

 

I've taken a screenshot of this (though you might have tolog in via your own account to see the HTML code and stuff)

 

 

The problem I'm having with this is as follows:

1. Lazy load - When first loaded, uBOT only picks up the posts that are visible on the screen due to lazy loading. But if I scroll down either manually (or via run 

javascript("window.scrollTo(0,document.body.scrollHeight);"))

, what happens is that the top half of posts now disappear and gets hidden out of view from the HTML so again uBOT's $scrape attribute doesnt pick it up - it only sees ~5- visible posts at a time (ideally I'd want to scrape all 30-50 posts that I have in one shot)...

 

2. Also,I'm using the $scrape attribute feature for the scrapping. Since everything appears visually like a table, in reality the HTML/css is only <div's> rather than an HTML table so uBot doesnt recognize it with $scrape_table. Hence I have to scrape separately the post headings in one list, dates in another list, then the likes + comments in 3rd list and then due to the lazy load from the earlier point I have to manually see if there are any duplicated entries and take them out...

 

Is there a fast way to scrape: Name of post, datetime,likes,comments as column headings & then the data for each of them?

[SourceUltra 10]

You may want to try messing with the css height/width settings for the container the data is being displayed in to see if you can forcefully make it larger, possibly bringing more data into view. If more data is not displayed then there is a coded display limit and you may have to continue scrolling to fetch your data.

 

Also, check to see if there is an iframe on the page, because that may actually be where all of the data is. iframes are ancient technology but still great at screwing up the scrape attribute function. 

 

It can be good practice to ctrl+f the pages source code with some of the data you want to scrape, to see if it is being pulled from a script that's loaded in the site's footer. Sometimes the data is actually sitting in a footer script helplessly waiting for you to parse it.

 

And when all else fails you can try the "is it just me" test by seeing if a small tool like listly can scrape it. If that chrome extension can't do it then you know it's a tough scrape.