Scraping Reddit Subreddit Subjects

[jisaacs1207 0]

I'm just working to figure out ubot. I was curious if someone could explain how to scrape every subject from a subreddit in reddit, and then go to the next page and do the same thing. This is without knowing what the subject text says in advanced.

 

Sorry if this seems very easy, but like I said... new. 

 

Ideally, I would like every subject to be added to a list.

 

I am on a dev license btw.

Edited August 24, 2016 by jisaacs1207

[deliter 203]

Heres a simple enough example at the bottom,I had a problem clicking next,so I scraped the href and navigate to it

 

At the top Ive added for using http get which is much faster for scraping,as you can see from the code,it is very similar to the below code,the plugin is available on a thread I started

define scrapeSubReddits(#subReddit) {
    plugin command("SocketCommands.dll", "socket container") {
        plugin command("SocketCommands.dll", "socket navigate", "GET", "https://www.reddit.com/r/{#subReddit}/")
        add list to list(%titles,$plugin function("DeliterCSS.dll", "Deliter CSS Selector", $plugin function("SocketCommands.dll", "$socket page html"), ".title.may-blank ", "TextContent"),"Delete","Global")
        loop while($comparison($plugin function("DeliterCSS.dll", "Easy HTML Parser", $plugin function("SocketCommands.dll", "$socket page html"), "a", "rel", "= (Attribute equals example \'class = myclass\')", "nofollow next", "href"),">",$nothing)) {
            set(#scrapePageLink,$plugin function("DeliterCSS.dll", "Easy HTML Parser", $plugin function("SocketCommands.dll", "$socket page html"), "a", "rel", "= (Attribute equals example \'class = myclass\')", "nofollow next", "href"),"Local")
            plugin command("SocketCommands.dll", "socket navigate", "GET", #scrapePageLink)
         
            add list to list(%titles,$plugin function("DeliterCSS.dll", "Deliter CSS Selector", $plugin function("SocketCommands.dll", "$socket page html"), ".title.may-blank ", "TextContent"),"Delete","Global")
        }
    }
}
scrapeSubReddits("LifeProTips")

define scrapeSubReddits(#subReddit) {
    navigate("https://www.reddit.com/r/{#subReddit}/","Wait")
    wait for browser event("Everything Loaded","")
    add list to list(%titles,$scrape attribute(<class="title may-blank ">,"innertext"),"Delete","Global")
    loop while($exists(<rel="nofollow next">)) {
        set(#scrapePageLink,$scrape attribute(<(tagname="a" AND rel="nofollow next")>,"href"),"Local")
        navigate(#scrapePageLink,"Wait")
        wait for browser event("Everything Loaded","")
        add list to list(%titles,$scrape attribute(<class="title may-blank ">,"innertext"),"Delete","Global")
    }
}
scrapeSubReddits("LifeProTips")

[jisaacs1207 0]

Just realized you had replied with such a terrific source. Thanks for that, and sorry for the (VERY BIG) delay doing so.