Scrape Website Url

[flexfanatic 0]

I'm currently working on a clickbank marketplace scraper and I have all of the information like hoplinks, description, etc stored in a database. I'm now trying to obtain the actually website url by navigating to the page and capturing the url but I'm having some issues with figuring out how to obtain this. 

 

Looking at the page source there are a few tags that tell me what the full url is but is there a way to capture this from the address bar and save it back to the database table?

[Code Docta (Nick C.) 638]

$url

function gets the current url in the browser

[flexfanatic 0]

$url

function gets the current url in the browser

 

I love when its that simple. Works like a charm.

 

Thanks Code Docta

[HelloInsomnia 1103]

Just a heads up, you can get the entire marketplace via their feed that updates every day!

 

Here is the page that talks a bit about it: https://support.clickbank.com/entries/22824126-Marketplace-Feed

 

Or the direct link to the feed: https://accounts.clickbank.com/feeds/marketplace_feed_v2.xml.zip

[flexfanatic 0]

Just a heads up, you can get the entire marketplace via their feed that updates every day!

 

Here is the page that talks a bit about it: https://support.clickbank.com/entries/22824126-Marketplace-Feed

 

Or the direct link to the feed: https://accounts.clickbank.com/feeds/marketplace_feed_v2.xml.zip

 

Correct, I downloaded the marketplace and then imported it into a MySQL database. What I'm trying to accomplish is getting the actual website URL from the hoplink url. 

 

I'm able to get the results I want but now I'm running into a issue with updating the column (last line where I'm performing an update and I don't get a error but no update either).

 

plugin command("DatabaseCommands.dll", "connect to database", "mysql:server=\'localhost\';uid=\'myusername\'; pwd=\'password\'; database=\'cboffers\'; port=\'3306\'; pooling=false") {

    plugin command("DatabaseCommands.dll", "query with results", "select hoplinkurl from hoplinks where url IS NULL limit 1", &hoplinksurl)

    clear cookies

    navigate("{$table cell(&hoplinksurl,0,0)}.","Wait")

    clear list(%results)

    add item to list(%results,$url,"Don\'t Delete","Global")

    plugin command("DatabaseCommands.dll", "query", "UPDATE hoplinks SET url=\'%results\' WHERE hoplinkurl=\'&hoplinkurl\'")

[flexfanatic 0]

plugin command("DatabaseCommands.dll", "connect to database", "mysql:server=\'localhost\';uid=\'myusername\'; pwd=\'password\'; database=\'cboffers\'; port=\'3306\'; pooling=false") {
    plugin command("DatabaseCommands.dll", "query with results", "select hoplinkurl from hoplinks where url IS NULL limit 1", &hoplinksurl)
    clear cookies
    navigate("{$table cell(&hoplinksurl,0,0)}.","Wait")
    clear list(%results)
    add item to list(%results,$url,"Don\'t Delete","Global")

    plugin command("DatabaseCommands.dll", "query", "UPDATE hoplinks SET url=\'{%results}\' WHERE hoplinkurl=\'{$table cell(&hoplinksurl,0,0)}\'")
}

[HelloInsomnia 1103]

It looks like you are using the list %restults and the table &hoplinkurl when I think you should be using the list item and table cell you want, that may be the issue. I'm just looking at your code here and so it's kind of hard to say but that may be it off the top of my head.

[flexfanatic 0]

It looks like you are using the list %restults and the table &hoplinkurl when I think you should be using the list item and table cell you want, that may be the issue. I'm just looking at your code here and so it's kind of hard to say but that may be it off the top of my head.

 

Correct, I wrote the last line from the code view and forgot and messed up on the syntax.