Jump to content
UBot Underground

Problem Scraping Linkedin Review


Recommended Posts

Hi Guys,

I am wondering if you could analyze my code to see why it's not scraping a review from LinkedIn.

If I step the bot through it, everything works.

When I run it, nothing is getting scraped.

TIA.

-------

plugin command("ExBrowser.dll", "ExBrowser Set License", "YOUR-EXBROWSER-LICENSE-KEY")
clear cookies
clear table(&LinkedIn Data)
clear list(%LI)
ui open file("Load LinkedIn CSV File",#data)
ui text box("Email",#loginEmail)
ui password("Password",#loginPassword)
create table from file(#data,&LinkedIn Data)
set(#row,1,"Global")
plugin command("ExBrowser.dll", "ExBrowser CleanUp", "No")
plugin command("ExBrowser.dll", "ExBrowser Launcher", "Chrome", "", "{$plugin function("ExBrowser.dll", "$ExBrowser Chrome Disable Notifications")}{$plugin function("ExBrowser.dll", "$ExBrowser Chrome Disable Info Bars")}{$plugin function("ExBrowser.dll", "$ExBrowser Chrome Disable Info Bars Experimental")}{$plugin function("ExBrowser.dll", "$ExBrowser Use Portable Browser", "Chrome")}{$plugin function("ExBrowser.dll", "$ExBrowser Chrome Disable Save Password")}", "NO")
plugin command("ExBrowser.dll", "ExBrowser Navigate", "https://www.linkedin.com/")
if($plugin function("ExBrowser.dll", "$ExBrowser Element Exist", "x://input[contains(@autocomplete,\"username\")]")) {
    then {
        plugin command("ExBrowser.dll", "ExBrowser Type Text", "x://input[contains(@autocomplete,\"username\")]", #loginEmail)
        wait($rand(1,3))
        plugin command("ExBrowser.dll", "ExBrowser Type Text", "x://input[contains(@name,\"session_password\")]", #loginPassword)
        wait($rand(1,3))
        plugin command("ExBrowser.dll", "ExBrowser Click", "//*[@id=\"main-content\"]/section[1]/div[2]/form/button")
    }
    else {
    }
}
loop($table total rows(&LinkedIn Data)) {
    clear list(%reviews)
    wait(1)
    plugin command("ExBrowser.dll", "ExBrowser Click Special", "x://span[contains(text(),\"Messaging\")]")
    wait(1)
    plugin command("ExBrowser.dll", "ExBrowser Navigate", $table cell(&LinkedIn Data,#row,8))
    plugin command("ExBrowser.dll", "ExBrowser Scroll To Element", "x://h2[contains(text(),\"Recommendations\")]")
    wait(1)
    plugin command("ExBrowser.dll", "ExBrowser Click", "//*[@id=\"line-clamp-show-more-button\"]")
    wait(1)
    plugin command("ExBrowser.dll", "ExBrowser Change Attribute", "x://p[contains(@class,\"pv-recommendation-entity__headline t-14 t-black t-normal pb1\")]", "innertext", "")
    wait(1)
    plugin command("ExBrowser.dll", "ExBrowser Change Attribute", "x://p[contains(@class,\"t-12 t-black--light t-normal\")]", "innertext", "")
    wait(1)
    set(#reviewer,$trim($plugin function("ExBrowser.dll", "$ExBrowser Scrape Element", "x://div[contains(@class,\"pv-recommendation-entity__detail\")]")),"Global")
    wait(1)
    set(#reviews,$plugin function("ExBrowser.dll", "$ExBrowser Scrape Element", "x://span[contains(@class,\"lt-line-clamp__raw-line\")]"),"Global")
    wait(1)
    add item to list(%reviews,$trim($append line to text(#reviewer,#reviews)),"Delete","Global")
    add list to table as column(&LinkedIn Data,#row,4,%reviews)
    increment(#row)
}

 

 

Link to post
Share on other sites

Hello @cjacobs, I removed your ExBrowser license key from within your code 😉

Without having tried yet, I assume you want to try putting a wait right after:

plugin command("ExBrowser.dll", "ExBrowser Navigate", "https://www.linkedin.com/")

I assume that the content you try to scrape loads asynchronous.

So try to make sure it‘s really loaded before continuing (e.g. putting a simple "wait" or a "wait for element").

Link to post
Share on other sites
3 minutes ago, PRO said:

Hello @cjacobs, I removed your ExBrowser license key from within your code 😉

Without having tried yet, I assume you want to try putting a wait right after:

plugin command("ExBrowser.dll", "ExBrowser Navigate", "https://www.linkedin.com/")

I assume that the content you try to scrape loads asynchronous. So try to make sure it‘s really loaded before continuing.

 

Shoot! I forgot about that key!

It's navigating fine but it's not scrolling to 'Recommendations' nor scraping when in Run mode. Stepping through works which is why I put the waits in but still not working.

Link to post
Share on other sites

Ok, try putting "wait" or "wait for element" after this line:

plugin command("ExBrowser.dll", "ExBrowser Navigate", $table cell(&LinkedIn Data,#row,8))

It's similar scenario. Right after the navigate command.
 

Link to post
Share on other sites

I tried using both wait and wait for element but it's still not working.

loop($table total rows(&LinkedIn Data)) {
    clear list(%reviews)
    wait(1)
    plugin command("ExBrowser.dll", "ExBrowser Click Special", "x://span[contains(text(),\"Messaging\")]")
    wait(1)
    plugin command("ExBrowser.dll", "ExBrowser Navigate", $table cell(&LinkedIn Data,#row,8))
    plugin command("ExBrowser.dll", "ExBrowser Wait For Element", "x://h2[contains(text(),\"Recommendations\")]", "Appear", 30)
    plugin command("ExBrowser.dll", "ExBrowser Scroll To Element", "x://h2[contains(text(),\"Recommendations\")]")
    wait(1)
    plugin command("ExBrowser.dll", "ExBrowser Click", "//*[@id=\"line-clamp-show-more-button\"]")
    wait(1)

 

Link to post
Share on other sites

Really sorry that I can't test your code atm. But I can suggest to put a "stop script" after questionable steps, then hitting run and looking inside the debugger if everything looks good until the step before the "stop script". 

Then move the stop script one command down the line, run again and check the debugger again. 

Repeating this should enable you to identify the exact step where the issue starts to appear. 

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...