Aymen 385 Posted April 22, 2013 Author Report Share Posted April 22, 2013 I'm still just starting to learn how to use http posts. Maybe I'm starting with things that are too complicated for a beginner, I don't know. Is there a sample code someone could post that will work for solving recaptchas? It looks like they are brought in with an iframe, so you can't simply scrape the captcha image url and sending that to your captcha solving service or software. How can I do this using HTTP post plugin? Below is what the code looks like. Thanks! <tr> <th align="left"><label for="spam">Captcha</label></th> <td colspan="2"> <script type="text/javascript" src="http://www.google.com/recaptcha/api/challenge?k=6LesgMUSAAAAACkxGFkFcBeCLcXKp3oUSas32fXS"></script> <noscript> <iframe src="http://www.google.com/recaptcha/api/noscript?k=6LesgMUSAAAAACkxGFkFcBeCLcXKp3oUSas32fXS" height="300" width="500" frameborder="0"></iframe><br/> <textarea name="recaptcha_challenge_field" rows="3" cols="40"></textarea> <input type="hidden" name="recaptcha_response_field" value="manual_challenge"/> </noscript> </td> </tr> I recorded 2 video tutorials a while back to show you how you can do this ! http://screencast.com/t/bVgQIrRBihttp://screencast.com/t/0SQdrJ6H0bbO Quote Link to post Share on other sites
SaHiN 14 Posted April 22, 2013 Report Share Posted April 22, 2013 hi aymen, http post - parameters problem; "username=jassmjasd325423&domain_id=0&pass=778877&re_pass=778877&name=jasdo323ksad&email=sjduj55haewrs77%40hotmail.com&PHPSESSID={#mywapblogphpsesid}&security={#mywapblogsolvecaptcha}®ister=Register" convert to: username=jassmjasd325423&domain_id=0&pass=778877&re_pass=778877&name=jasdo323ksad&email=sjduj55haewrs77%40hotmail.com&PHPSESSID=dcfe4e9749e4f3dea874e8567ed498ae&security=costally®ister=Register Problem : ®ister=Register conver to ®ister=Register confusion HTML Useful Character Entities ® - registered trademark - ® - ® Quote Link to post Share on other sites
Kev 69 Posted April 26, 2013 Report Share Posted April 26, 2013 Hi Aymen, Is it possible to use this plugin with Facebook's API? https://developers.facebook.com/docs/reference/api/search/ Thanks man, I love this plugin Quote Link to post Share on other sites
Aymen 385 Posted April 26, 2013 Author Report Share Posted April 26, 2013 Hi Aymen, Is it possible to use this plugin with Facebook's API? https://developers.facebook.com/docs/reference/api/search/ Thanks man, I love this plugin Yes possibleall you need is the API url as a URL and the token as post data ! Quote Link to post Share on other sites
iDollarsteam 13 Posted April 29, 2013 Report Share Posted April 29, 2013 Hi Aymen,I use your plugin to try to make a bot capable to login to youtube. I use the $http get function to scrape the login page and get all the parameters (visible and hidden ) and then I use regex to assign values to the dynamic hidden parameters, I put all of them in a $http post function and try to login... So far I get the error page with the message: There was an issue logging you into YouTube ... I see two main problems: When I scrape the page using the plugin I get one of the parameters - bgresponse: js_disabled but if I use fiddler to grab the http traffic I can see that bgresponse is a dynamic parameter and is generated by a javascript on the page so my question here is: does your plugin disable javascript when it is scraping? The second problem is: I don't think that the proxy command is working as it should or it is leaking my real IP, when I load the result from the POST I see that youtube show me the YT logo and my country code - I use private proxies that I can use in browser and google don't see my ip... I also use those proxies in one of my bots to make yt accounts ... this is the code that I came up with: clear list(%useragents) add list to list(%useragents, $list from file("{$special folder("Application")}\\UserAgents"), "Delete", "Global") set(#useremail, "YTLogin@gmail.com", "Global") set(#userpass, "YTPassword", "Global") set(#UserAgent, $list item(%useragents, $rand(0, $subtract($list total(%useragents), 1))), "Global") set(#Proxy, "proxy:port:proxyuser:proxypass", "Global") set(#URL, "https://accounts.google.com/ServiceLogin?service=youtube&continue=http%3A%2F%2Fwww.youtube.com%2Fsignin%3Faction_handle_signin%3Dtrue%26feature%3Dsign_in_button%26hl%3Den_US%26next%3D%252F%26nomobiletemp%3D1&uilel=3&hl=en_US&passive=true", "Global") plugin command("HTTP post.dll", "Clear Cookies", #URL) set(#GETloginpage, $plugin function("HTTP post.dll", "$http get", #URL, #UserAgent, "www.youtube.com", #Proxy, 15), "Global") set(#continue, "http://www.youtube.com/signin?action_handle_signin=true&feature=sign_in_button&hl=en_US&next=%2F&nomobiletemp=1", "Global") set(#service, "youtube", "Global") set(#dsh, $find regular expression(#GETloginpage, "(?<=id=\"dsh\"\\s* value=\")[\\w\\W]*?(?=\")"), "Global") set(#hl, "en_US", "Global") set(#GALX, $find regular expression(#GETloginpage, "(?<=name=\"GALX\"\\s*value=\")[\\w\\W]*?(?=\")"), "Global") set(#pstMsg, 1, "Global") set(#dnConn, $nothing, "Global") set(#checkConnection, "youtube:1000:1", "Global") set(#checkedDomains, "youtube", "Global") set(#timeStmp, $nothing, "Global") set(#secTok, $nothing, "Global") set(#_utf8, $find regular expression(#GETloginpage, "(?<=name=\"_utf8\"\\s* value=\")[\\w\\W]*?(?=\")"), "Global") set(#bgresponse, $find regular expression(#GETloginpage, "(?<=id=\"bgresponse\"\\s* value=\")[\\w\\W]*?(?=\")"), "Global") set(#Email, #useremail, "Global") set(#Passwd, #userpass, "Global") set(#signIn, "Sign in", "Global") set(#PersistentCookie, "yes", "Global") set(#rmShown, 1, "Global") set(#POSTloginpage, $plugin function("HTTP post.dll", "$http post", "https://accounts.google.com/ServiceLoginAuth", "continue={#continue}&service={#service}&dsh={#dsh}&hl={#hl}&GALX={#GALX}&pstMsg={#pstMsg}&dnConn={#dnConn}&checkConnection={#checkConnection}&checkedDomains={#checkedDomains}&timeStmp={#timeStmp}&secTok={#secTok}&_utf8={#_utf8}&bgresponse={#bgresponse}&Email={#Email}&Passwd={#Passwd}&signIn={#signIn}&PersistentCookie={#PersistentCookie}&rmShown={#rmShown}", #UserAgent, #URL, #Proxy, 15), "Global") load html(#POSTloginpage) Is there anyone here that can show me what I'm doing wrong and help me make it work? Thank You! Quote Link to post Share on other sites
Kev 69 Posted April 30, 2013 Report Share Posted April 30, 2013 Hi Aymen, I'm trying to understand how to use the http get with this plugin. For example, I want to use Facebook's API in order to grab the ID of (for example) this page: http://developers.facebook.com/tools/explorer?method=GET&path=kevin%3Ffields%3Did However, I dont want to have to load up that developers.facebook.com in order to grab this information, simply I want to use their API and grab the ID data. Any idea how I would use this plugin for this? Quote Link to post Share on other sites
Aymen 385 Posted April 30, 2013 Author Report Share Posted April 30, 2013 Hi Aymen, I'm trying to understand how to use the http get with this plugin. For example, I want to use Facebook's API in order to grab the ID of (for example) this page: http://developers.facebook.com/tools/explorer?method=GET&path=kevin%3Ffields%3Did However, I dont want to have to load up that developers.facebook.com in order to grab this information, simply I want to use their API and grab the ID data. Any idea how I would use this plugin for this? if you want to get the id , try something like this : set(#facebook, $plugin function("HTTP post.dll", "$http get", "https://graph.facebook.com/kevin?fields=id&method=GET&format=json", "Mozilla/5.0 (Windows; U; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)", "", "", ""), "Global") load html(#facebook) set(#id, $find regular expression(#facebook, "(?<=\\\"id\\\": \\\")[0-9]+(?=\\\")"), "Global") alert("id is : {#id}") Quote Link to post Share on other sites
Uncle Si 0 Posted May 1, 2013 Report Share Posted May 1, 2013 Bought this plugin last night, I must say this plugin is incredible. This takes building powerful bots to a whole new level.Thanks for the great plugin! Quote Link to post Share on other sites
orbital 15 Posted May 11, 2013 Report Share Posted May 11, 2013 is it possible to solve recaptcha when the site is using this library? https://www.google.com/recaptcha/api/js/recaptcha_ajax.js As far as I can tell since it is ajax it is created on the fly and thus no challenge code shows up when you scrape. Quote Link to post Share on other sites
Kev 69 Posted May 13, 2013 Report Share Posted May 13, 2013 Hey Aymen, Could you help me with a small request?I'm trying to use the page scraper node to search for the word "facebook.com/" (for example see this page: http://westinny.com/) How can I scrape this facebook url from the output file? I'm well versed in scrape attribute but in this case I'm not scraping the page - I need to search the #output for the Facebook text. Thanks in advance, Kev Quote Link to post Share on other sites
Aymen 385 Posted May 13, 2013 Author Report Share Posted May 13, 2013 Hey Aymen, Could you help me with a small request?I'm trying to use the page scraper node to search for the word "facebook.com/" (for example see this page: http://westinny.com/) How can I scrape this facebook url from the output file? I'm well versed in scrape attribute but in this case I'm not scraping the page - I need to search the #output for the Facebook text. Thanks in advance, Kev use find regular expression , and try this : (https://|http://)(www.|)facebook.com\/[a-zA-Z0-9./_-]+ Quote Link to post Share on other sites
orbital 15 Posted May 14, 2013 Report Share Posted May 14, 2013 When I try and replace your google scrape 100 results example code with the $http get function it doesn't work. It only works when you use the page scraper command. Here is the code. set(#output, $plugin function("HTTP post.dll", "$http get", "https://www.google.com/setprefs?sig=0_99_9Ba26GvHykubSeuKp228oSs0%3D&hl=en&safeui=images&suggon=1&num=100&newwindow=0&q=&prev=https%3A%2F%2Fwww.google.com%2F&submit2=Save+Preferences", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0", "", "", ""), "Global") set(#output, $plugin function("HTTP post.dll", "$http get", "https://www.google.com/search?q=payday+loans&num=100&&start=000", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0", "", "", ""), "Global") load html(#output) Here is your original example: plugin command("HTTP post.dll", "Page Scraper", "https://www.google.com/setprefs?sig=0_99_9Ba26GvHykubSeuKp228oSs0%3D&hl=en&safeui=images&suggon=1&num=100&newwindow=0&q=&prev=https%3A%2F%2Fwww.google.com%2F&submit2=Save+Preferences", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)", #output) { } plugin command("HTTP post.dll", "Page Scraper", "https://www.google.com/search?q=payday+loans&num=100&&start=000", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)", #output) { } load html(#output) I thought the $http get function and page scraper where the same thing? Quote Link to post Share on other sites
Aymen 385 Posted May 14, 2013 Author Report Share Posted May 14, 2013 When I try and replace your google scrape 100 results example code with the $http get function it doesn't work. It only works when you use the page scraper command. Here is the code. set(#output, $plugin function("HTTP post.dll", "$http get", "https://www.google.com/setprefs?sig=0_99_9Ba26GvHykubSeuKp228oSs0%3D&hl=en&safeui=images&suggon=1&num=100&newwindow=0&q=&prev=https%3A%2F%2Fwww.google.com%2F&submit2=Save+Preferences", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0", "", "", ""), "Global") set(#output, $plugin function("HTTP post.dll", "$http get", "https://www.google.com/search?q=payday+loans&num=100&&start=000", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0", "", "", ""), "Global") load html(#output) Here is your original example: plugin command("HTTP post.dll", "Page Scraper", "https://www.google.com/setprefs?sig=0_99_9Ba26GvHykubSeuKp228oSs0%3D&hl=en&safeui=images&suggon=1&num=100&newwindow=0&q=&prev=https%3A%2F%2Fwww.google.com%2F&submit2=Save+Preferences", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)", #output) { } plugin command("HTTP post.dll", "Page Scraper", "https://www.google.com/search?q=payday+loans&num=100&&start=000", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)", #output) { } load html(#output) I thought the $http get function and page scraper where the same thing? it worked for me but i get only 10 results per page , when i change another google extension i can get the whole 100 results per page !Page Scraper and http get are the same thing , you must of missed something Quote Link to post Share on other sites
Code Docta (Nick C.) 638 Posted May 24, 2013 Report Share Posted May 24, 2013 Here's something you guys might like to help you along. It works with Ubot too. http://fiddler2.com/home 2 Quote Link to post Share on other sites
Aymen 385 Posted May 24, 2013 Author Report Share Posted May 24, 2013 Limited time promotion 50% offcoupon code : MAYUBOT2013expires in 2013-05-31 Quote Link to post Share on other sites
earthlingj 13 Posted June 4, 2013 Report Share Posted June 4, 2013 Hey any of you guys that bought this able to figure out how to get twitter profile pics to upload? i could really use some help. also pinterest following i get how to do get / post ive gotten far.. just stuck on those 2 things since they work a bit differently. Quote Link to post Share on other sites
Code Docta (Nick C.) 638 Posted June 4, 2013 Report Share Posted June 4, 2013 Hey any of you guys that bought this able to figure out how to get twitter profile pics to upload? i could really use some help. also pinterest following i get how to do get / post ive gotten far.. just stuck on those 2 things since they work a bit differently. Million dollar question...I'd like to know as well please. 1 Quote Link to post Share on other sites
iDollarsteam 13 Posted June 6, 2013 Report Share Posted June 6, 2013 I don't know how twitter manage the upload of the pictures but if it is like youtube (the profile pic upload on YT) then it is done using flash so I don't think that you can use the plugin and http requests to do it Quote Link to post Share on other sites
seonid 4 Posted June 7, 2013 Report Share Posted June 7, 2013 Hi Aymen, Is this the right format to use private proxies? set(#Proxy, "proxy:port:proxyuser:proxypass", "Global") EDIT: yes it is Quote Link to post Share on other sites
Aymen 385 Posted June 7, 2013 Author Report Share Posted June 7, 2013 Hi Aymen, Is this the right format to use private proxies? set(#Proxy, "proxy:port:proxyuser:proxypass", "Global") EDIT: yes it is yes thats the right format Quote Link to post Share on other sites
seopolis 0 Posted June 14, 2013 Report Share Posted June 14, 2013 Hi Aymen, I bought your fantastic plugin yesterday, and has allready improved dramatically my scraping times! I need your help on this: I've set up a simple php page to post in my blog some simple text. I get the variables from browser like this: $title = $_GET['myTitle']; $content = $_GET['myContent']; $category = $_GET['myCat']; and then I process these and make a blog post. When I visit the page with my browser like this: http://mysite.com/post.php?myTitle=utf8-bla-bla&myContent=utf8-bla-bla&myCat=1 the post is published. Tryign to accomplish the same task through HTTP Post plugin, always fails... I tried to use html encode but nothing, I'm stack. Any suggestion? PS. Also tried with the navigate command and works fine too. But i preffer to not load the browser for this, and make the whole process quicker (for bulk posting). Quote Link to post Share on other sites
Aymen 385 Posted June 14, 2013 Author Report Share Posted June 14, 2013 Hi Aymen, I bought your fantastic plugin yesterday, and has allready improved dramatically my scraping times! I need your help on this: I've set up a simple php page to post in my blog some simple text. I get the variables from browser like this: $title = $_GET['myTitle']; $content = $_GET['myContent']; $category = $_GET['myCat']; and then I process these and make a blog post. When I visit the page with my browser like this: http://mysite.com/post.php?myTitle=utf8-bla-bla&myContent=utf8-bla-bla&myCat=1 the post is published. Tryign to accomplish the same task through HTTP Post plugin, always fails... I tried to use html encode but nothing, I'm stack. Any suggestion? PS. Also tried with the navigate command and works fine too. But i preffer to not load the browser for this, and make the whole process quicker (for bulk posting). if its a GET request , you can use the $http get function like this : set(#post, $plugin function("HTTP post.dll", "$http get", "http://mysite.com/post.php?myTitle=utf8-bla-bla&myContent=utf8-bla-bla&myCat=1 ", "", "", "", ""), "Global") Quote Link to post Share on other sites
seopolis 0 Posted June 14, 2013 Report Share Posted June 14, 2013 Thanks for the prompt answer. I send a POST request because (correct me if I'm wrong) GET has a limit of 1024 characters.. Since I'm using utf-8, I wanted to overcome this limit. But I will try your solution because my messages are about 100 characters, so I will take a stab and report back the results. Thanks. Quote Link to post Share on other sites
seopolis 0 Posted June 14, 2013 Report Share Posted June 14, 2013 I never used http get function before... In which command should I put it? PS. Ok, I know saw that you set the variable #post. Ignore my question. Quote Link to post Share on other sites
seopolis 0 Posted June 14, 2013 Report Share Posted June 14, 2013 Ok, this method works fine for small messages. Tested it several times and is blazing fast. The only downside is the character limit. When the message is large the GET truncates it to 1024 characters. In my case (utf-8 and lot of html markup) it truncates to around 40 words... I will get rid some of my html markup code, but the ideal solution would be to work like this with POST request. Thanks again Aymen for your help. UPD: I actually tried to declare the same variable with the "http post" function and it works just fine! Pretty happy right now, and I wouldn't find the solution without Aymen's help. +rep +1 Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.