Problem Scraping Linkedin Review

[cjacobs 2]

Hi Guys,

I am wondering if you could analyze my code to see why it's not scraping a review from LinkedIn.

If I step the bot through it, everything works.

When I run it, nothing is getting scraped.

TIA.

-------

plugin command("ExBrowser.dll", "ExBrowser Set License", "YOUR-EXBROWSER-LICENSE-KEY")
clear cookies
clear table(&LinkedIn Data)
clear list(%LI)
ui open file("Load LinkedIn CSV File",#data)
ui text box("Email",#loginEmail)
ui password("Password",#loginPassword)
create table from file(#data,&LinkedIn Data)
set(#row,1,"Global")
plugin command("ExBrowser.dll", "ExBrowser CleanUp", "No")
plugin command("ExBrowser.dll", "ExBrowser Launcher", "Chrome", "", "{$plugin function("ExBrowser.dll", "$ExBrowser Chrome Disable Notifications")}{$plugin function("ExBrowser.dll", "$ExBrowser Chrome Disable Info Bars")}{$plugin function("ExBrowser.dll", "$ExBrowser Chrome Disable Info Bars Experimental")}{$plugin function("ExBrowser.dll", "$ExBrowser Use Portable Browser", "Chrome")}{$plugin function("ExBrowser.dll", "$ExBrowser Chrome Disable Save Password")}", "NO")
plugin command("ExBrowser.dll", "ExBrowser Navigate", "https://www.linkedin.com/")
if($plugin function("ExBrowser.dll", "$ExBrowser Element Exist", "x://input[contains(@autocomplete,\"username\")]")) {
    then {
        plugin command("ExBrowser.dll", "ExBrowser Type Text", "x://input[contains(@autocomplete,\"username\")]", #loginEmail)
        wait($rand(1,3))
        plugin command("ExBrowser.dll", "ExBrowser Type Text", "x://input[contains(@name,\"session_password\")]", #loginPassword)
        wait($rand(1,3))
        plugin command("ExBrowser.dll", "ExBrowser Click", "//*[@id=\"main-content\"]/section[1]/div[2]/form/button")
    }
    else {
    }
}
loop($table total rows(&LinkedIn Data)) {
    clear list(%reviews)
    wait(1)
    plugin command("ExBrowser.dll", "ExBrowser Click Special", "x://span[contains(text(),\"Messaging\")]")
    wait(1)
    plugin command("ExBrowser.dll", "ExBrowser Navigate", $table cell(&LinkedIn Data,#row,8))
    plugin command("ExBrowser.dll", "ExBrowser Scroll To Element", "x://h2[contains(text(),\"Recommendations\")]")
    wait(1)
    plugin command("ExBrowser.dll", "ExBrowser Click", "//*[@id=\"line-clamp-show-more-button\"]")
    wait(1)
    plugin command("ExBrowser.dll", "ExBrowser Change Attribute", "x://p[contains(@class,\"pv-recommendation-entity__headline t-14 t-black t-normal pb1\")]", "innertext", "")
    wait(1)
    plugin command("ExBrowser.dll", "ExBrowser Change Attribute", "x://p[contains(@class,\"t-12 t-black--light t-normal\")]", "innertext", "")
    wait(1)
    set(#reviewer,$trim($plugin function("ExBrowser.dll", "$ExBrowser Scrape Element", "x://div[contains(@class,\"pv-recommendation-entity__detail\")]")),"Global")
    wait(1)
    set(#reviews,$plugin function("ExBrowser.dll", "$ExBrowser Scrape Element", "x://span[contains(@class,\"lt-line-clamp__raw-line\")]"),"Global")
    wait(1)
    add item to list(%reviews,$trim($append line to text(#reviewer,#reviews)),"Delete","Global")
    add list to table as column(&LinkedIn Data,#row,4,%reviews)
    increment(#row)
}

 

 

[PRO 69]

Hello @cjacobs, I removed your ExBrowser license key from within your code 😉

Without having tried yet, I assume you want to try putting a wait right after:

plugin command("ExBrowser.dll", "ExBrowser Navigate", "https://www.linkedin.com/")

I assume that the content you try to scrape loads asynchronous.

So try to make sure it‘s really loaded before continuing (e.g. putting a simple "wait" or a "wait for element").

[cjacobs 2]

3 minutes ago, PRO said:

Hello @cjacobs, I removed your ExBrowser license key from within your code 😉

Without having tried yet, I assume you want to try putting a wait right after:

plugin command("ExBrowser.dll", "ExBrowser Navigate", "https://www.linkedin.com/")

I assume that the content you try to scrape loads asynchronous. So try to make sure it‘s really loaded before continuing.

 

Shoot! I forgot about that key!

It's navigating fine but it's not scrolling to 'Recommendations' nor scraping when in Run mode. Stepping through works which is why I put the waits in but still not working.

[PRO 69]

You tried my suggestion already? 

[cjacobs 2]

8 minutes ago, PRO said:

You tried my suggestion already? 

I did yes. Nothing.

[PRO 69]

Ok, try putting "wait" or "wait for element" after this line:

plugin command("ExBrowser.dll", "ExBrowser Navigate", $table cell(&LinkedIn Data,#row,8))

It's similar scenario. Right after the navigate command.
 

[cjacobs 2]

I tried using both wait and wait for element but it's still not working.

loop($table total rows(&LinkedIn Data)) {
    clear list(%reviews)
    wait(1)
    plugin command("ExBrowser.dll", "ExBrowser Click Special", "x://span[contains(text(),\"Messaging\")]")
    wait(1)
    plugin command("ExBrowser.dll", "ExBrowser Navigate", $table cell(&LinkedIn Data,#row,8))
    plugin command("ExBrowser.dll", "ExBrowser Wait For Element", "x://h2[contains(text(),\"Recommendations\")]", "Appear", 30)
    plugin command("ExBrowser.dll", "ExBrowser Scroll To Element", "x://h2[contains(text(),\"Recommendations\")]")
    wait(1)
    plugin command("ExBrowser.dll", "ExBrowser Click", "//*[@id=\"line-clamp-show-more-button\"]")
    wait(1)

 

[PRO 69]

Really sorry that I can't test your code atm. But I can suggest to put a "stop script" after questionable steps, then hitting run and looking inside the debugger if everything looks good until the step before the "stop script". 

Then move the stop script one command down the line, run again and check the debugger again. 

Repeating this should enable you to identify the exact step where the issue starts to appear. 

[cjacobs 2]

Ok, thanks for your advice.

[LoWrIdErTJ - BotGuru 904]

Has been resolved..