Error With Youtube Channel Operations (Load More Button)

[steelersfan 38]

I am having a hard time resolving this error in Ubot that is caused by loading more videos in YouTube. I am trying to get all of the videos of any channel, and my script works on nearly every channel I have tried it on. It merely starts by going to a channel, and hitting "load more" until there is nothing more to load.

 

The problem is that on some channels with a HUGE amount of videos (for example Yale Courses), it returns the following error:

 

"Error converting value True to type 'System.Collections.Generic.List`1 [system.String]'.

 

My code to keep loading more as long as the button exists:

if($exists(<class="yt-uix-button yt-uix-button-size-default yt-uix-button-default load-more-button yt-uix-load-more browse-items-load-more-button">)) {
    then {
        loop while($exists(<class="yt-uix-button yt-uix-button-size-default yt-uix-button-default load-more-button yt-uix-load-more browse-items-load-more-button">)) {
            wait for element(<class="yt-uix-button yt-uix-button-size-default yt-uix-button-default load-more-button yt-uix-load-more browse-items-load-more-button">,"","Appear")
            wait(1)
            click(<class="yt-uix-button yt-uix-button-size-default yt-uix-button-default load-more-button yt-uix-load-more browse-items-load-more-button">,"Left Click","No")
            wait(1)
            increment(#loadmorecounted)
        }
    }
    else {
        wait(1)
    }
}

If I monkey with the code after it returns an error, it almost always crashes Ubot.

 

I have hit a wall that I can't seem to get past. What can I do to fix this?

[deliter 203]

that error above is when Ubot is actually crashed,

 

here is a snippet I use a lot on youtube, and sites that are too complicated to automate background tasks

run javascript("
(function() \{
httpUrls = []
    var proxied = window.XMLHttpRequest.prototype.open;
    window.XMLHttpRequest.prototype.open = function() \{
       httpUrls.push( arguments[1] );
        return proxied.apply(this, [].slice.call(arguments));
    \};
\})();")
comment("run the above,then click load more button,a wait until loaded then initialise the bottom variable")
set(#re,$eval("httpUrls.toString()"),"Global")

this will return a string of URL's the browser is visiting in the background,so make sure the page is loaded before you run the run javascript bit, and that youve clicked load more, and that has completed before you initialise the re bit, turn it into a list,and have the bot find action_continuation URL

 

for load more on youtube I think the url call is something like "https://www.youtube.com/browse_ajax?action_continuation=1

 

find the URL, then just do http get of that address over and over, it should return the entire HTML block that would have loaded onto the page, and it wont crash the browser and will be much faster too

[steelersfan 38]

You know, I was hoping for a "get" solution! Thanks so much Deliter!

[deliter 203]

not sure how well this works, need to download my CSS Selector plugin too, I got 550 titles from yale campus channel, uses a bit of Javascript works on Chrome 49 latest, havnt tested on others

 

it uses eval so a bit slow, and will not be able to thread, I might add decoding javascript strings to the plugin if I can find an already writen function for doing that in C# and update this sometime

 

https://1drv.ms/u/s!AgO9AudYbciJgP8ZAkIdZ6ICeNRCWQ

usersVideos("https://www.youtube.com/user/YaleCampus/videos")
define usersVideos(#videoPage) {
    set(#get,$plugin function("HTTP post.dll", "$http get", #videoPage, "", "", "", ""),"Local")
    add list to list(%titles,$plugin function("DeliterCSS.dll", "Deliter CSS Selector", #get, ".yt-lockup-title .yt-uix-sessionlink", "title"),"Delete","Global")
    if($plugin function("DeliterCSS.dll", "Deliter CSS Selector Exists", #get, ".yt-uix-load-more")) {
        then {
            moreUserVideos("https://www.youtube.com{$plugin function("DeliterCSS.dll", "Deliter CSS Selector", #get, ".yt-uix-load-more", "data-uix-load-more-href")}")
        }
        else {
        }
    }
}
define moreUserVideos(#url) {
    set(#get,$plugin function("HTTP post.dll", "$http get", #url, "", "", "", ""),"Local")
    set(#removeEncoding,$eval("var re = {#get}
re.content_html"),"Local")
   add list to list(%titles,$plugin function("DeliterCSS.dll", "Deliter CSS Selector", #removeEncoding, ".yt-lockup-title .yt-uix-sessionlink", "title"),"Delete","Global")
    set(#next,$replace regular expression($find regular expression(#get,"browse_ajax\\?.+?\""),"\\\\\"$",""),"Local")
    if($comparison(#next,">",$nothing)) {
        then {
            moreUserVideos("https://www.youtube.com/{$eval("var re = \"{#next}\"
re")}")
        }
        else {
        }
    }
}

[deliter 203]

After trying to write a plugin and tearing my hair out, have a nice solution, using Aymans Free JSON Parser, much better script

 

http://ubot-plugins.com/jsonpath-parser-plugin/

usersVideos("https://www.youtube.com/user/YaleCampus/videos")
define usersVideos(#videoPage) {
    set(#get,$plugin function("HTTP post.dll", "$http get", #videoPage, "", "", "", ""),"Local")
    add list to list(%titles,$plugin function("DeliterCSS.dll", "Deliter CSS Selector", #get, ".yt-lockup-title .yt-uix-sessionlink", "title"),"Don\'t Delete","Global")
    if($plugin function("DeliterCSS.dll", "Deliter CSS Selector Exists", #get, ".yt-uix-load-more")) {
        then {
            moreUserVideos("https://www.youtube.com{$plugin function("DeliterCSS.dll", "Deliter CSS Selector", #get, ".yt-uix-load-more", "data-uix-load-more-href")}")
        }
        else {
        }
    }
}
define moreUserVideos(#url) {
    set(#get,$plugin function("HTTP post.dll", "$http get", #url, "", "", "", ""),"Local")
    set(#removeEncoding,$plugin function("JSONpath.dll", "$JSONpath parser", #get, "content_html"),"Local")
    add list to list(%titles,$plugin function("DeliterCSS.dll", "Deliter CSS Selector", #removeEncoding, ".yt-lockup-title .yt-uix-sessionlink", "title"),"Don\'t Delete","Global")
    set(#next,$plugin function("DeliterCSS.dll", "Deliter CSS Selector", $plugin function("JSONpath.dll", "$JSONpath parser", #get, "load_more_widget_html"), "[data-uix-load-more-href^=\"/browse_ajax?\"]", "data-uix-load-more-href"),"Local")
    if($comparison(#next,">",$nothing)) {
        then {
            moreUserVideos("https://www.youtube.com{#next}")
        }
        else {
        }
    }
}

[steelersfan 38]

I am thoroughly confused and overwhelmed at this point, lol! Deliter you are great! But I am still quite far away from your skill and understanding. I need some time to try and understand this all, but I know it will help me in trying to finish this bot. Thank you!

[deliter 203]

I am thoroughly confused and overwhelmed at this point, lol! Deliter you are great! But I am still quite far away from your skill and understanding. I need some time to try and understand this all, but I know it will help me in trying to finish this bot. Thank you!

Just run the code above, and check the titles list in your debugger, no need to understand it really

 

when you loaded you'l see an input for whatever youtube channel you want to scrape, it has yale loaded in there now

[steelersfan 38]

404 on "Aymans Free JSON Parser" too, so I'm SoL there.

[deliter 203]

use the script I posted above that , it doesn't use his parser, not as good but its still fine, but change the code to not delete duplicates in the titles list, I forgot to do that

[steelersfan 38]

Wow! So, it worked, and quickly! For the purposes of what I am trying to do, it is a much better solution than my own initial channel scraping method (Which was bulky, convoluted, and did not work on large channels!). I have to edit a bit of this to get it to work within my script, but that is trivial and easy. This replaces my clunky solution with a far better one. I have added slightly edited code below that gets the href as opposed to the titles, for anyone who may need that also.

usersVideos("https://www.youtube.com/user/YaleCourses/videos")
define usersVideos(#videoPage) {
    set(#get,$plugin function("HTTP post.dll", "$http get", #videoPage, "", "", "", ""),"Local")
    add list to list(%titles,$plugin function("DeliterCSS.dll", "Deliter CSS Selector", #get, ".yt-lockup-title .yt-uix-sessionlink", "href"),"Don\'t Delete","Global")
    if($plugin function("DeliterCSS.dll", "Deliter CSS Selector Exists", #get, ".yt-uix-load-more")) {
        then {
            moreUserVideos("https://www.youtube.com{$plugin function("DeliterCSS.dll", "Deliter CSS Selector", #get, ".yt-uix-load-more", "data-uix-load-more-href")}")
        }
        else {
        }
    }
}
define moreUserVideos(#url) {
    set(#get,$plugin function("HTTP post.dll", "$http get", #url, "", "", "", ""),"Local")
    set(#removeEncoding,$eval("var re = {#get}
re.content_html"),"Local")
    add list to list(%titles,$plugin function("DeliterCSS.dll", "Deliter CSS Selector", #removeEncoding, ".yt-lockup-title .yt-uix-sessionlink", "href"),"Delete","Global")
    set(#next,$replace regular expression($find regular expression(#get,"browse_ajax\\?.+?\""),"\\\\\"$",""),"Local")
    if($comparison(#next,">",$nothing)) {
        then {
            moreUserVideos("https://www.youtube.com/{$eval("var re = \"{#next}\"
re")}")
        }
        else {
        }
    }
}

So, for posterity, I am going to attempt to break down what is happening here.

 

1. usersVideos "videoPage" is the parameter variable of the define "usersVideos", allowing me to define the page. (can probably add a variable to this for UI input of a page)
2. define usersVideos first sets the "http get" function to the videoPage variable, then it adds the css selector attribute chosen to the list. (I altered my list name and selected href)
3. Next, the if statement checks if the gotten page info has a "load more" css selector, and if it does, it goes to the YouTube URL along with the css selected attribute for the actual "load more".
4.  Finally the "userVideos" define then calls to the next define "moreUserVideos" IF the load more button exists (selected by the css selector).
5. Now the "moreUserVideos" define/command sets the variable "#get" to its own parameter (#url) with "http get" calling the "#url" variable. (This is where I get lost! Will update again once I understand why this is done and how it works.)

 

So far, I somewhat get what is going on and I know it works, so big thanks to you Deliter! I am just trying to wrap my mind around the rest, so as to better understand what is being done, to expand my own knowledge and skill. While the problem is indeed resolved by this, I also want to understand how that was achieved, so that I can work around similar problems in the future, without needing help. Hopefully this will also help others who are trying to create similar bots!

[deliter 203]

Wow! So, it worked, and quickly! For the purposes of what I am trying to do, it is a much better solution than my own initial channel scraping method (Which was bulky, convoluted, and did not work on large channels!). I have to edit a bit of this to get it to work within my script, but that is trivial and easy. This replaces my clunky solution with a far better one. I have added slightly edited code below that gets the href as opposed to the titles, for anyone who may need that also.

usersVideos("https://www.youtube.com/user/YaleCourses/videos")
define usersVideos(#videoPage) {
    set(#get,$plugin function("HTTP post.dll", "$http get", #videoPage, "", "", "", ""),"Local")
    add list to list(%titles,$plugin function("DeliterCSS.dll", "Deliter CSS Selector", #get, ".yt-lockup-title .yt-uix-sessionlink", "href"),"Don\'t Delete","Global")
    if($plugin function("DeliterCSS.dll", "Deliter CSS Selector Exists", #get, ".yt-uix-load-more")) {
        then {
            moreUserVideos("https://www.youtube.com{$plugin function("DeliterCSS.dll", "Deliter CSS Selector", #get, ".yt-uix-load-more", "data-uix-load-more-href")}")
        }
        else {
        }
    }
}
define moreUserVideos(#url) {
    set(#get,$plugin function("HTTP post.dll", "$http get", #url, "", "", "", ""),"Local")
    set(#removeEncoding,$eval("var re = {#get}
re.content_html"),"Local")
    add list to list(%titles,$plugin function("DeliterCSS.dll", "Deliter CSS Selector", #removeEncoding, ".yt-lockup-title .yt-uix-sessionlink", "href"),"Delete","Global")
    set(#next,$replace regular expression($find regular expression(#get,"browse_ajax\\?.+?\""),"\\\\\"$",""),"Local")
    if($comparison(#next,">",$nothing)) {
        then {
            moreUserVideos("https://www.youtube.com/{$eval("var re = \"{#next}\"
re")}")
        }
        else {
        }
    }
}

So, for posterity, I am going to attempt to break down what is happening here.

 

1. usersVideos "videoPage" is the parameter variable of the define "usersVideos", allowing me to define the page. (can probably add a variable to this for UI input of a page)
2. define usersVideos first sets the get function to the videoPage variable, then it adds the css selector attribute chosen to the list. (I altered my list name and selected href)
3. Next, the if statement checks if the gotten page info has a "load more" css selector, and if it does, it goes to the YouTube URL along with the css selected attribute for the actual "load more".
4. The define; "moreUserVideos" is used to (This is where I get lost! lol! I'm not sure what happens here or how it is related to the first define.)

 

So far, I somewhat get what is going on and I know it works, so big thanks to you Deliter! I am just trying to wrap my mind around the rest, so as to better understand what is being done, to expand my own knowledge and skill. While the problem is indeed resolved by this, I also want to understand how that was achieved, so that I can work around similar problems in the future, without needing help. Hopefully this will also help others who are trying to create similar bots!

 

 

Thanks man, very flattered you really appreciate it and are trying to learn from it, I like helping people, kind of sucks when someone just copy pastas your work, doesn't "like" the post, nor tries to learn from it

 

the second part "moreUserVideos" is basically like if you were doing it manually, it hits the load more button, scrapes the titles, then the last thing is it checks if the load more button is present, if so, it keeps calling itself "moreUserVideos" to do that process again until the button is there no longer,if not present do nothing

 

if you want to start learning CSS Selectors, which I would really recommend, I have a plugin for Chrome to help you write selectors automatically

 

http://network.ubotstudio.com/forum/index.php/topic/20582-chrome-extension-for-auto-generating-scraping-code-for-my-css-plugin-exbrowser-too/

[steelersfan 38]

Ahh! Now I understand that second part better! It calls itself, brilliant! That is something that I never even thought of or conceived could be done with a define. A nice jump in my skill can be gained from such an example! I will now approach it better and continue my breakdown in the post above, both for my own analytical understanding, and to help others who may think as I do. Thank you for explaining it! Once I can understand enough myself to reiterate it in my own words, I will be quite happy!

 

Indeed, you are and have been a great help to me in the past. I don't like to just be a "taker" of knowledge, hence why I hope to buy some products from you in the future! Thanks again Deliter, for the great help!