Walter1 8 Posted December 5, 2013 Report Share Posted December 5, 2013 Although i checked all these pages a couple of times earlier for different solutions, i am not going trough all these pages again.Why wont this forum have a thread search option.... Anyways, i am using the httppost function to get the return of google.com in a local variable. But when i query i get a 'not allowed' comment. (405) I know my proxies are good so it aint those... Current code: set(#GOOGLERESULT, $plugin function("HTTP post.dll", "$http post", http://www.google.com/, "/search?output=search&sclient=psy-ab&q=site:{#CURRENT URL}&oq=site:{#CURRENT URL}", $random list item(%USERAGENTS), http://www.google.com/, #CURRENT PROXY, 5), "Global") Any solution for this problem guys? Thanks in advance! Quote Link to post Share on other sites
Aymen 385 Posted December 5, 2013 Author Report Share Posted December 5, 2013 i Know for fact that google search is a Get request not a post request , although that may work on some sites but some other sites can give you an error !try it as a GET request then if you still have the error , i'll take a look into it ! Quote Link to post Share on other sites
Walter1 8 Posted December 5, 2013 Report Share Posted December 5, 2013 Hi Aymen, Thanks for the swift reply, really appreciated! K, when i use this code it will only return the homepage as it seems and not the page with results.Thats why i was trying to use post option to make it happen... set(#GOOGLERESULT, $plugin function("HTTP post.dll", "$http get", "https://www.google.com/#q=site:www.test.com", $random list item(%USERAGENTS), "http://www.google.com", #CURRENT PROXY, 5), "Global") load html(#GOOGLERESULT) BTW: I am using v4+... Walter Quote Link to post Share on other sites
Aymen 385 Posted December 5, 2013 Author Report Share Posted December 5, 2013 thats not the correct url for searching , use something like this : https://www.google.com/search?q=ubot+studio the search part in red should be url encoded Quote Link to post Share on other sites
Walter1 8 Posted December 5, 2013 Report Share Posted December 5, 2013 Hey Aymen, Yeah that works perfectly. Ofcourse... *Having a dumb moment here* Thanks man, awesome support, as always! Quote Link to post Share on other sites
Aymen 385 Posted December 5, 2013 Author Report Share Posted December 5, 2013 Hey Aymen, Yeah that works perfectly. Ofcourse... *Having a dumb moment here* Thanks man, awesome support, as always! No problem!Glad i can help! 1 Quote Link to post Share on other sites
Code Docta (Nick C.) 638 Posted December 5, 2013 Report Share Posted December 5, 2013 I was curious one day and played on Google and it was'nt easy to getresults, so I stopped playing. Anyone get any further as in scraping the results? Quote Link to post Share on other sites
HelloInsomnia 1103 Posted December 5, 2013 Report Share Posted December 5, 2013 I was curious one day and played on Google and it was'nt easy to getresults, so I stopped playing. Anyone get any further as in scraping the results? Here is something that checks results I haven't looked at the code in a minute but I believe it still works, it might point you in the right direction. comment("check for domain and print out results with positions") ui drop down("Check Pages:", "1,2,3,4,5,6,7,8,9,10", #loops) clear table(&rankings) ui text box("Keyword:", #input_keyword) ui text box("Domain:", #domain) plugin command("HTTP post.dll", "Page Scraper", "https://www.google.com/setprefs?sig=0_99_9Ba26GvHykubSeuKp228oSs0%3D&hl=en&safeui=images&suggon=1&num=100&newwindow=0&q=&prev=https%3A%2F%2Fwww.google.com%2F&submit2=Save+Preferences", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)", #output) { } set(#page_start, 000, "Global") set(#rankings_row_num, 0, "Global") set(#last_page, "false", "Global") loop(#loops) { if($comparison(#last_page, "=", "false")) { then { plugin command("HTTP post.dll", "Page Scraper", "https://www.google.com/search?q={$replace(#input_keyword, " ", "+")}&num=100&&start={#page_start}", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)", #output) { } set(#look_for_next_button, $nothing, "Global") set(#look_for_next_button, $find regular expression(#output, "(?<=53px\\\"\\>)Next(?=\\<\\/span)"), "Global") if($comparison(#look_for_next_button, "=", $nothing)) { then { set(#last_page, "true", "Global") } else { } } if($contains($plugin function("HTTP post.dll", "$xpath parser", #output, "//div[@id=\'search\']", "InnerText"), #domain)) { then { clear list(%results) add list to list(%results, $list from text($replace($plugin function("HTTP post.dll", "$xpath parser", #output, "//h3[@class=\'r\']/a", "href"), "/url?q=", $nothing), $new line), "Delete", "Global") set(#cur_serp_postition, 1, "Global") loop($list total(%results)) { set(#cur_result_url, $next list item(%results), "Global") if($contains(#cur_result_url, #domain)) { then { set table cell(&rankings, #rankings_row_num, 0, $find regular expression(#cur_result_url, "http.*?(?=\\&)")) set table cell(&rankings, #rankings_row_num, 1, $add(#page_start, #cur_serp_postition)) increment(#rankings_row_num) } else { } } increment(#cur_serp_postition) } } else { } } wait(2) set(#page_start, $add(#page_start, 100), "Global") } else { } } } if($comparison(#rankings_row_num, "=", 0)) { then { alert("The site is not ranking for that keyword.") } else { save to file("{$special folder("Desktop")}\\rankings.csv", &rankings) alert("Results saved to your desktop as rankings.csv") } } 1 Quote Link to post Share on other sites
iDollarsteam 13 Posted December 6, 2013 Report Share Posted December 6, 2013 Hey Aymen, Yeah that works perfectly. Ofcourse... *Having a dumb moment here* Thanks man, awesome support, as always! Hi Walter, if you want to "play" with HTTP requests install Fiddler 2 on your computer and watch some tutorials on YT on how to read the requests. Then you will be able to see the required url in the request (like the google search req you needed earlier ), the type of request (GET, POST or other type), the headers, cookies and much more. After you learn a little about that you will have an entire "new world" at your feet using Aymens plugin. 1 Quote Link to post Share on other sites
Code Docta (Nick C.) 638 Posted December 9, 2013 Report Share Posted December 9, 2013 Here is something that checks results I haven't looked at the code in a minute but I believe it still works, it might point you in the right direction. comment("check for domain and print out results with positions") ui drop down("Check Pages:", "1,2,3,4,5,6,7,8,9,10", #loops) clear table(&rankings) ui text box("Keyword:", #input_keyword) ui text box("Domain:", #domain) plugin command("HTTP post.dll", "Page Scraper", "https://www.google.com/setprefs?sig=0_99_9Ba26GvHykubSeuKp228oSs0%3D&hl=en&safeui=images&suggon=1&num=100&newwindow=0&q=&prev=https%3A%2F%2Fwww.google.com%2F&submit2=Save+Preferences", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)", #output) { } set(#page_start, 000, "Global") set(#rankings_row_num, 0, "Global") set(#last_page, "false", "Global") loop(#loops) { if($comparison(#last_page, "=", "false")) { then { plugin command("HTTP post.dll", "Page Scraper", "https://www.google.com/search?q={$replace(#input_keyword, " ", "+")}&num=100&&start={#page_start}", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)", #output) { } set(#look_for_next_button, $nothing, "Global") set(#look_for_next_button, $find regular expression(#output, "(?<=53px\\\"\\>)Next(?=\\<\\/span)"), "Global") if($comparison(#look_for_next_button, "=", $nothing)) { then { set(#last_page, "true", "Global") } else { } } if($contains($plugin function("HTTP post.dll", "$xpath parser", #output, "//div[@id=\'search\']", "InnerText"), #domain)) { then { clear list(%results) add list to list(%results, $list from text($replace($plugin function("HTTP post.dll", "$xpath parser", #output, "//h3[@class=\'r\']/a", "href"), "/url?q=", $nothing), $new line), "Delete", "Global") set(#cur_serp_postition, 1, "Global") loop($list total(%results)) { set(#cur_result_url, $next list item(%results), "Global") if($contains(#cur_result_url, #domain)) { then { set table cell(&rankings, #rankings_row_num, 0, $find regular expression(#cur_result_url, "http.*?(?=\\&)")) set table cell(&rankings, #rankings_row_num, 1, $add(#page_start, #cur_serp_postition)) increment(#rankings_row_num) } else { } } increment(#cur_serp_postition) } } else { } } wait(2) set(#page_start, $add(#page_start, 100), "Global") } else { } } } if($comparison(#rankings_row_num, "=", 0)) { then { alert("The site is not ranking for that keyword.") } else { save to file("{$special folder("Desktop")}\\rankings.csv", &rankings) alert("Results saved to your desktop as rankings.csv") } } Thanks man, I actually used $HTML parser to get the results. I was playing with another bot and went hmmm? I wonder if this would work for SERPs? You know that working on bot always seems to get an answer for another. Noob's, if you havrn't you will. Thanks for the code tho ,it will speed up the rest of it. haha I just have a couple nodes in that one. TC Quote Link to post Share on other sites
Walter1 8 Posted December 11, 2013 Report Share Posted December 11, 2013 Hi Aymen, I have a question about the $pagerankchecker. In one of your source codes i noticed no proxy option was provided. Does this mean that we can bulk check the whole day long? I suppose not, just asking to be sure Quote Link to post Share on other sites
Sanjeev 46 Posted December 11, 2013 Report Share Posted December 11, 2013 Aymen? What resource does the $pagerankchecker use for getting the page rank info? Thanks. Quote Link to post Share on other sites
HelloInsomnia 1103 Posted December 11, 2013 Report Share Posted December 11, 2013 Aymen? What resource does the $pagerankchecker use for getting the page rank info? Thanks. It uses the Google toolbar. You should be able to see the exact url it calls with something like Fiddler. Quote Link to post Share on other sites
Aymen 385 Posted December 11, 2013 Author Report Share Posted December 11, 2013 Hi Aymen, I have a question about the $pagerankchecker. In one of your source codes i noticed no proxy option was provided. Does this mean that we can bulk check the whole day long? I suppose not, just asking to be sure yea that was just for testing , if you are going to bulk check with the same IP google may block your IP , so use proxies ! Quote Link to post Share on other sites
Sanjeev 46 Posted December 12, 2013 Report Share Posted December 12, 2013 It uses the Google toolbar. You should be able to see the exact url it calls with something like Fiddler.Thanks! Quote Link to post Share on other sites
Walter1 8 Posted December 12, 2013 Report Share Posted December 12, 2013 yea that was just for testing , if you are going to bulk check with the same IP google may block your IP , so use proxies ! Hi Aymen, thanks for the reply man. Yeah thats what i thought Quote Link to post Share on other sites
gandensang 11 Posted December 15, 2013 Report Share Posted December 15, 2013 i have small problem with http captcha dialog (in thread mode) problem is in thread only, please help to solve its thanks sample code: thread { set(#solve captcha, $plugin function("HTTP post.dll", "$http captcha dialog", "C:\\Documents and Settings\\Administrator\\Desktop\\testcaptcha.jpg"), "Global") } Quote Link to post Share on other sites
cicooboom 0 Posted December 15, 2013 Report Share Posted December 15, 2013 Follow module on twitter sockets exemple not working! Can you please verify? Quote Link to post Share on other sites
Aymen 385 Posted December 15, 2013 Author Report Share Posted December 15, 2013 i have small problem with http captcha dialog (in thread mode)problem is in thread only, please help to solve its thanks sample code: thread { set(#solve captcha, $plugin function("HTTP post.dll", "$http captcha dialog", "C:\\Documents and Settings\\Administrator\\Desktop\\testcaptcha.jpg"), "Global")} i'm aware of that bug , i'll fix it soon ! Follow module on twitter sockets exemple not working! Can you please verify?i'll check it out ! Quote Link to post Share on other sites
Marton 48 Posted December 15, 2013 Report Share Posted December 15, 2013 i'm aware of that bug , i'll fix it soon !+1 on the captcha dialog fix, I just created an application using it in thread mode too. Thanks Aymen p.s.: would it be possible to make it so it automatically gets focused (similar to the ubot manual captcha), and pressing enter would work too? (instead of having to click OK) Quote Link to post Share on other sites
a2mateit 395 Posted December 15, 2013 Report Share Posted December 15, 2013 +1 on the captcha dialog fix, I just created an application using it in thread mode too. Thanks Aymen p.s.: would it be possible to make it so it automatically gets focused (similar to the ubot manual captcha), and pressing enter would work too? (instead of having to click OK)+1 on using enter to make it work and getting it fixed. I use it when testing, and can't run it inside the thread. Quote Link to post Share on other sites
luis carlos 94 Posted December 15, 2013 Report Share Posted December 15, 2013 Could you add to the twitter script, how to upload the profile image or changing the header? Thanksi'm aware of that bug , i'll fix it soon ! i'll check it out ! Quote Link to post Share on other sites
gandensang 11 Posted December 16, 2013 Report Share Posted December 16, 2013 i'm aware of that bug , i'll fix it soon ! i'll check it out !thanks aymen, you and yor product is amazing Gandensang Quote Link to post Share on other sites
Marton 48 Posted December 17, 2013 Report Share Posted December 17, 2013 I'm trying to use Bypasscaptcha with the http plugin, but no matter what I do I can't get it working (I was able to solve captchas with 6 other captcha services using the http plugin).Do you guys have any idea what I might be doing wrong? I attached source code for you to see. If someone is interested, you could get a test key for free with 10 captchas:http://bypasscaptcha.com/contact.php?s=customized+softwareThen you can check the status of your key here: http://bypasscaptcha.com/tool.php Here's the API documentation: http://bypasscaptcha.com/api_doc/submit.php Thank you in advance!NOTE: If you get ERROR Invalid Key message, that doesn't mean you key is invalid, I've had this error with 2 other captcha services even though my key was correct (It just means there is an error in the code somewhere, and once I corrected the code everything went fine. I just don't know what to change in this case with bypasscaptcha)bypasscaptcha.ubot Quote Link to post Share on other sites
Aymen 385 Posted December 17, 2013 Author Report Share Posted December 17, 2013 +1 on the captcha dialog fix, I just created an application using it in thread mode too. Thanks Aymen p.s.: would it be possible to make it so it automatically gets focused (similar to the ubot manual captcha), and pressing enter would work too? (instead of having to click OK)i'll see what i can do ! I'm trying to use Bypasscaptcha with the http plugin, but no matter what I do I can't get it working (I was able to solve captchas with 6 other captcha services using the http plugin).Do you guys have any idea what I might be doing wrong? I attached source code for you to see. If someone is interested, you could get a test key for free with 10 captchas:http://bypasscaptcha.com/contact.php?s=customized+software Then you can check the status of your key here: http://bypasscaptcha.com/tool.php Here's the API documentation: http://bypasscaptcha.com/api_doc/submit.php Thank you in advance!NOTE: If you get ERROR Invalid Key message, that doesn't mean you key is invalid, I've had this error with 2 other captcha services even though my key was correct (It just means there is an error in the code somewhere, and once I corrected the code everything went fine. I just don't know what to change in this case with bypasscaptcha) the website is not loading here , i'll check it out later and see if i can make an integration Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.