zharfan 0 Posted May 22, 2014 Report Share Posted May 22, 2014 i check in the debugger when scrape google web page appear that massage Quote Link to post Share on other sites
Aymen 385 Posted May 22, 2014 Author Report Share Posted May 22, 2014 IndexOutOfRangeException It's not happening all the time. Mostly when I stop and start the bot multiple times. I was not able to fully isolate the issue yet. But I'm not doing anything fancy within the cookies container. Only using http get and xpath parser indexWithin a http container. No other commands are used from your plugin. Dan If you could reproduce the error or narrow it down that would be helpful Hey bro if there error massage when i use $http get "The remote server returned an error: (503) Server Unavailable." what the soluton?can you post the exact code here or pm it to me ? Quote Link to post Share on other sites
Bot-Factory 602 Posted May 22, 2014 Report Share Posted May 22, 2014 Aymen, what is http cookie response doing? Is that the cookie string that is also used for serializing? ThanksDan Quote Link to post Share on other sites
Aymen 385 Posted May 22, 2014 Author Report Share Posted May 22, 2014 Aymen, what is http cookie response doing? Is that the cookie string that is also used for serializing? ThanksDan that's the cookies string from the latest http response Regards Quote Link to post Share on other sites
Bot-Factory 602 Posted May 22, 2014 Report Share Posted May 22, 2014 that's the cookies string from the latest http response Regards So within a http container this should be exactly what the plugin should use for it's next http request right? Dan Quote Link to post Share on other sites
Aymen 385 Posted May 22, 2014 Author Report Share Posted May 22, 2014 yea , the plugin handles cookies automatically for you , in some cases some sites don't send all the cookies required because of excessive redirections , to bypass that , disable redirections and send cookies manually! Quote Link to post Share on other sites
Bot-Factory 602 Posted May 24, 2014 Report Share Posted May 24, 2014 Aymen, the error I get with my scraper when using http container is:"object reference could not set to an instance" Kev123 has a similar issue. But we are both not 100% sure where this comes from. If it's an issue in the http container, or some strange plugin stuff when we use it in combination with other plugins. Would be great if you could look into that when you work on the other issues. Thanks in advance for your helpDan Quote Link to post Share on other sites
kev123 132 Posted May 25, 2014 Report Share Posted May 25, 2014 Aymen, the error I get with my scraper when using http container is:"object reference could not set to an instance" Kev123 has a similar issue. But we are both not 100% sure where this comes from. If it's an issue in the http container, or some strange plugin stuff when we use it in combination with other plugins. Would be great if you could look into that when you work on the other issues. Thanks in advance for your helpDan aymen the issue is with http get it looks like it returns nothing hence the error on the set command i'll create some isolated code when I get chance. Quote Link to post Share on other sites
Aymen 385 Posted May 25, 2014 Author Report Share Posted May 25, 2014 aymen the issue is with http get it looks like it returns nothing hence the error on the set command i'll create some isolated code when I get chance. Can you poste a code snippet ? or steps to reproduce the error ? Quote Link to post Share on other sites
kev123 132 Posted May 25, 2014 Report Share Posted May 25, 2014 Can you poste a code snippet ? or steps to reproduce the error ?. Yeah I'll make something up tomorrow Quote Link to post Share on other sites
Bot-Factory 602 Posted May 27, 2014 Report Share Posted May 27, 2014 Not sure if it's possible at all, but one killer feature would be to share sessions / cookies between browser and http plugin.Maybe with session IDs, so that we could switch between different sessions. And load / save session and share them between plugin and browser. Just dreaming... :-) Dan 1 Quote Link to post Share on other sites
jamesfar 15 Posted May 27, 2014 Report Share Posted May 27, 2014 Not sure if it's possible at all, but one killer feature would be to share sessions / cookies between browser and http plugin.Maybe with session IDs, so that we could switch between different sessions. And load / save session and share them between plugin and browser. Just dreaming... :-) Dan ++ 1 for this it would be awesome if we can switch between http and browser Quote Link to post Share on other sites
Bot-Factory 602 Posted May 28, 2014 Report Share Posted May 28, 2014 Cookie Handling: The whole cookie handling still confuses me. Aymen, it would be great if you could explain that in detail. Are cookies sometimes shared with the ubot browser? Or are they always stand alone?Why are cookies downloaded to the cookies folder? When are they used?Are cookies within a http container unique? Why are files saved to the cookies folder when I use http container? How are they separated from each other? (seems to only happen with v5) I have done some additional tests. And there is definitely something wrong with UBot V5 and the plugin. Even in V4 I sometimes get an error like:the given key was not in the dictionary when I use: alert($plugin function("HTTP post.dll", "$http cookie response")) But in v4 the login always works. In v5 it doesn't work. Sometimes I'm logged in, sometimes not. Sometimes the cookie response is empty, sometimes I get that dictionary error and sometimes it contains a cookie string. I will send you a test script via PM so that you can reproduce the issue. Dan Quote Link to post Share on other sites
Bot-Factory 602 Posted May 28, 2014 Report Share Posted May 28, 2014 Also strange: in v5 it always writes files to c:\cookies in v4 it doesn't do that at all? Dan Quote Link to post Share on other sites
Marton 48 Posted May 28, 2014 Report Share Posted May 28, 2014 Why are cookies downloaded to the cookies folder? When are they used?Are cookies within a http container unique? Why are files saved to the cookies folder when I use http container? How are they separated from each other? I would like to learn about these too. Thanks. Quote Link to post Share on other sites
Marton 48 Posted May 28, 2014 Report Share Posted May 28, 2014 Hi guys, There seems to be demand for a working DeathByCaptcha source code (the one in the bonus package is not a full solution). DeathByCaptcha doesn't automatically return the answer to the CAPTCHA. As explained in the API, you need to "poll" for the uploaded CAPTCHA status manually (it's not as simple as with Decaptcher for example), so you need to make sure you give enough time for them to actually solve your CAPTCHA before the final GET request. You can make a loop and check every 5 seconds (wouldn't use a much lower value, because you could get banned for too many checks). I quickly updated the "DeathByCaptcha API.ubot" script to make it work (source code attached).Example image to test it with: http://content.screencast.com/users/urban.marton/folders/Camtasia%20Studio/media/bfd5ee9f-81a4-4b6f-acd6-2d15a9032764/example_captcha.jpg I added a 10+5 second wait before the first check, because in my experience it takes at least 15 seconds for DeathByCaptcha to give an answer. After that the script checks for an answer every 5 seconds, up to a maximum of 20 tries (I added this just so there is a timeout). I also added a "Check Balance" button for you guys. Hope this helps,Martonp.s.: Aymen feel free to include this in your bonus package if you like itDeathByCaptcha_byMarton.ubot 2 Quote Link to post Share on other sites
Aymen 385 Posted May 28, 2014 Author Report Share Posted May 28, 2014 Cookie Handling: The whole cookie handling still confuses me. Aymen, it would be great if you could explain that in detail. Are cookies sometimes shared with the ubot browser? Or are they always stand alone?Why are cookies downloaded to the cookies folder? When are they used?Are cookies within a http container unique? Why are files saved to the cookies folder when I use http container? How are they separated from each other? (seems to only happen with v5) I have done some additional tests. And there is definitely something wrong with UBot V5 and the plugin. Even in V4 I sometimes get an error like:the given key was not in the dictionary when I use: alert($plugin function("HTTP post.dll", "$http cookie response")) But in v4 the login always works. In v5 it doesn't work. Sometimes I'm logged in, sometimes not. Sometimes the cookie response is empty, sometimes I get that dictionary error and sometimes it contains a cookie string. I will send you a test script via PM so that you can reproduce the issue. Dan I just want to state that the http post plugin is built for the v4 not for the v5 , i've never tested it with the v5 nor used it with the v5 , my initial tests with the v5 showed that the threading system in both v4 and v5 are built differently as some of the http post functions (http last response , http cookies response etc...) requires the thread ID to keep data unique to each thread , looks like the plugin is not capturing the thread id with the V5! about cookies , cookies are unique per session , if you open multiple sessions in different threads , you will see different cookiesAnd yes cookies are supposed to be unique per http container and thread (if you are opening different threads and using different data of course) And finally , there is an option to save cookies files locally (exactly like CURL does) instead of in memory , if you specify a folder location in the save cookies to disk command then the cookies will be saved to disk instead of in memory! the folder shouldn't show up if you choose not to save to folder Keep in mind that what i just said is related to V4 not V5!the time when V5 becomes stable i'll make a patch for it! 2 Quote Link to post Share on other sites
Bot-Factory 602 Posted May 28, 2014 Report Share Posted May 28, 2014 I just want to state that the http post plugin is built for the v4 not for the v5 , i've never tested it with the v5 nor used it with the v5 , my initial tests with the v5 showed that the threading system in both v4 and v5 are built differently as some of the http post functions (http last response , http cookies response etc...) requires the thread ID to keep data unique to each thread , looks like the plugin is not capturing the thread id with the V5! about cookies , cookies are unique per session , if you open multiple sessions in different threads , you will see different cookiesAnd yes cookies are supposed to be unique per http container and thread (if you are opening different threads and using different data of course) And finally , there is an option to save cookies files locally (exactly like CURL does) instead of in memory , if you specify a folder location in the save cookies to disk command then the cookies will be saved to disk instead of in memory! the folder shouldn't show up if you choose not to save to folder Keep in mind that what i just said is related to V4 not V5!the time when V5 becomes stable i'll make a patch for it! Thanks a lot Aymen. Appreciate that. I think this was very important. Maybe some people try to build bots with v5 and the plugin. And are wondering about strange errors. So that's the reason why. Stay with v4. It's still much better in a lot of scenarios.Dan Quote Link to post Share on other sites
Aymen 385 Posted May 28, 2014 Author Report Share Posted May 28, 2014 Hi guys, There seems to be demand for a working DeathByCaptcha source code (the one in the bonus package is not a full solution). DeathByCaptcha doesn't automatically return the answer to the CAPTCHA. As explained in the API, you need to "poll" for the uploaded CAPTCHA status manually (it's not as simple as with Decaptcher for example), so you need to make sure you give enough time for them to actually solve your CAPTCHA before the final GET request. You can make a loop and check every 5 seconds (wouldn't use a much lower value, because you could get banned for too many checks). I quickly updated the "DeathByCaptcha API.ubot" script to make it work (source code attached).Example image to test it with: http://www.google.co...jquiAo_H3ywspkU I added a 10+5 second wait before the first check, because in my experience it takes at least 15 seconds for DeathByCaptcha to give an answer. After that the script checks for an answer every 5 seconds, up to a maximum of 20 tries (I added this just so there is a timeout). I also added a "Check Balance" button for you guys. Hope this helps,Marton p.s.: Aymen feel free to include this in your bonus package if you like it Thanks Marton , if you can open a new thread with this source i'm sure a lot of people will see it and will appreciate it !Thank you for the work Regards, Quote Link to post Share on other sites
Marton 48 Posted May 28, 2014 Report Share Posted May 28, 2014 Thanks Marton , if you can open a new thread with this source i'm sure a lot of people will see it and will appreciate it !Thank you for the work Regards, Great thanks, I'll do that right now! Quote Link to post Share on other sites
Cyrax 2 Posted May 28, 2014 Report Share Posted May 28, 2014 can someone please advise me what basic concepts i need to know before trying to use this plugin, as i am beginner in ubot and don't know why i need to make use of http post. and where http post and get fit in with ubot. Quote Link to post Share on other sites
Bot-Factory 602 Posted May 28, 2014 Report Share Posted May 28, 2014 can someone please advise me what basic concepts i need to know before trying to use this plugin, as i am beginner in ubot and don't know why i need to make use of http post. and where http post and get fit in with ubot.Ubot itself is using the built in browser. So you can navigate to sites and scrape / modify stuff on that page.The http plugin allows you to create http requests. A Http request is the stuff your browser will send to a webserver when you do something on a website (not for everything).So it's the action behind some of the things you do within a browser. When you click a button your browser will generate a http request (depends on the button) and send it to a webserver. If the button only executes local stuff (javascript), then there will be no http request. Advantages:Much faster than working with the browser Disadvantages:More complex. You need to understand http request.You have to know how to sniff and "deconstruct" http request with tools like fiddlerYou can't do everything with it (javascript limitations) One is not better than the other. It depends on what you want to do. My recommendation:Start with browser first. And then look at performance optimizations with http plugin later. Dan Quote Link to post Share on other sites
Marton 48 Posted May 28, 2014 Report Share Posted May 28, 2014 Disadvantages:You can't do everything with it (javascript limitations)Hi, Do you have some examples for stuff that is not possible with HTTP Requests? Maybe not with Aymen's plugin, but in general as far as I know we are supposed to be able to do everything with them if used correctly. Even AJAX stuff. Any insight on this? Thanks,Marton Quote Link to post Share on other sites
Bot-Factory 602 Posted May 28, 2014 Report Share Posted May 28, 2014 Hi, Do you have some examples for stuff that is not possible with HTTP Requests? Maybe not with Aymen's plugin, but in general as far as I know we are supposed to be able to do everything with them if used correctly. Even AJAX stuff. Any insight on this? Thanks,MartonPosted an example in another thread:http://www.ubotstudio.com/forum/index.php?/topic/16427-http-plugin-value-hidden-behind-javascript/ Not possible = I haven't found a way to do it :-) But if something is hidden within javascript, you need a way to execute that script to get the data. Dan Quote Link to post Share on other sites
Marton 48 Posted May 28, 2014 Report Share Posted May 28, 2014 Thanks Marton , if you can open a new thread with this source i'm sure a lot of people will see it and will appreciate it !Thank you for the work Regards, Actually Aymen, could you spare a moment to look at the code I posted (it's really short, just a few commands)? I just found something, I think this could be interesting. Try it in this order: 1. Run "Check Balance" (works as expected, I get: "Status OK! Balance: 2.83$")2. Run the script (works as expected, I get the answer to the CAPTCHA)3. Run "Check Balance" again (takes a long time, I get blank page in browser, then I get: "Status OK! Balance: $") According to my script, "Status OK!" means that it was actually able to get the balance, but then for some reason it doesn't display the right value (not a regex error though). Could this be an anomaly with the plugin? I actually had the same thing happen in one of my complex scripts, but I thought it was something in my code. But everything seems fine here. I even tried adding "http containers" to multiple places, doesn't seem to solve the problem. Anyone know what might be causing this? Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.