Asynchronous Http Gets When Logged Into A Site From Ubot Browser

[deliter 203]

Hey

 

This example code does 50 HTTP Gets very very quickly, no need to work out how to log into sites neither without the browser

 

For this method to work, the browser needs to be on the website of the pages you want to scrape, not the actual page you want to scrape, just the same site, as it uses the browser to do the http gets, and the browser will only allow gets to the domain it is on

 

You can of course log into a site with sockets which takes time, but you cannot then use asynchronous HTTP Gets, you could start threading but that is a bit wasteful

 

This is much easier, just log into the site in your browser, once the browser page is on the site, it should allow HTTP Gets through the browser, this will asynchronously scrape, so your bot will not be held up waiting for the data of each request, so it is much much faster

 

What this script does is simply call a ubot function once it gets the response text from a http get, you just make a ubot command to handle that, I got this idea from a trick HInsomnia posted of URL encoding to pass arguments to functions outside of Ubot, which I didn't know it was possible, this is fantastic!!

 

simply add the AsyncGet Command to your script, and in the parameters, you put in the site, and the ubot command to call when it gets the response HTML , check it out

 

You will need the socket commands for the "URL Decode" function, can probably get it for free somewhere, if some free plugin has URL decode, then replace that function and should work

navigate("http://www.ubotstudio.com/","Wait")
wait for browser event("Everything Loaded","")
loop(50) {
    AsyncGet("https://www.ubotstudio.com/resources", "AsyncResponce")
}
define AsyncGet(#site, #UbotCommandName) {
    run javascript("function httpGetAsync(theUrl)
\{
    var xmlHttp = new XMLHttpRequest();
    xmlHttp.onreadystatechange = function() \{ 
        if (xmlHttp.readyState == 4 && xmlHttp.status == 200)
           
            ubot.runScript(\'{#UbotCommandName}(\\\"\'+encodeURIComponent(xmlHttp.responseText)+\'\\\",\\\"{#site}\\\")\')

    \}
    xmlHttp.open(\"GET\", theUrl, true); // true for asynchronous 
    xmlHttp.send(null);
\}

httpGetAsync(\'{#site}\')")
}
define AsyncResponce(#response, #site) {
    add item to list(%sites,#site,"Don\'t Delete","Global")
    add item to list(%responses,$plugin function("SocketCommands.dll", "$url decode", #response),"Don\'t Delete","Global")
}

[Code Docta (Nick C.) 638]

Very interesting, great thinking!!

 

 

Thanks, will have to check this out.

 

Regards,

Nick

[deliter 203]

Very interesting, great thinking!!

 

 

Thanks, will have to check this out.

 

Regards,

Nick

 

thanks Nick

 

Good talking to you again, the example is nothing fancy, just 50 HTTP Gets but I'm sure you'l see the possibilities

[luis carlos 94]

This seems awesome, but since I'am not a pro coder, I dont understand anything    Can you recommend a tutorial about this?  If you create your own video series, I will buy it for sure .

 

 

Thank you so much.

[deliter 203]

This seems awesome, but since I'am not a pro coder, I dont understand anything    Can you recommend a tutorial about this?  If you create your own video series, I will buy it for sure .

 

 

Thank you so much.

 

"I'am not a pro coder"

 

You mean you dont have Ubot professional? or just you yourself are not a pro coder

 

I guess you mean you do have professional, I think I was working with you before, No need to understand it really, if you want to learn though learn CSS selectors, I have a free plugin with a lot of functions in the free plugins thread, will make scraping a whole lot easier, all that happens is its a HTTP Get using Javascript, with Javascript unlike ubot it doesnt wait for a response, so it can send those 50 requests immediately, and process them as it receives the responses

 

Each time it gets a response it sends the url, and the document text to ubot(ubot.runScript)

 

I agree though, I credited HInsomnia for his tip which has gotten me able to build this, but its nothing short of a game changer so anyone reading just run the above script and enjoy!! oh and PM me for scripts while your at it !!

 

Then ubot process's it, so you can do whatever you want, just point it at whatever command you want to run on it

 

  AsyncGet("mySite", "MyUbotCommand")

[luis carlos 94]

Thanks!  I will start reading about "HTTP Get using Javascript".  I work with ubot pretty well while using the browser, but http is a new world for me

[deliter 203]

Anyone using this method, my customers are loving it, give it a try