merkaba 36 Posted March 15, 2013 Report Share Posted March 15, 2013 I might be late to the party, but this plugin seriously reduced scrape times, on my home connection I'm scraping pages in fractions of a second. 1 Quote Link to post Share on other sites
Aymen 385 Posted March 15, 2013 Author Report Share Posted March 15, 2013 I have the plugin and it has been very useful, I was trying to ask Aymen if he had any conversations with Eddie about the api modification. I have a dream part was a joke (something that is possible but will never come). I hate the "it might come but we don't know when" hints on the part of the dev team. Or even worse, it's possible but it's not on the plate right now. yea i'm trying to talk with Eddy about the API and if it is possible to add few things to it ! Quote Link to post Share on other sites
Aymen 385 Posted March 17, 2013 Author Report Share Posted March 17, 2013 Version 1.5 updates : -$html encode : "converts characters that are not allowed in HTML into character-entity equivalents"-$html decode : "reverses the encoding"-$url encode : converts characters to their hexadecimal representation to be used in the http post requests-$url decode : reverses the encoding (You should use this method with care. Unescaping a string that has been previously unescaped can lead to ambiguities and errors.) Quote Link to post Share on other sites
dankass 52 Posted March 19, 2013 Report Share Posted March 19, 2013 This looks like an awesome plugin! I am waiting for some kind of confirmation about the multi-threading issue before I purchase any new plugins however. If speed is the issue (which I would think is one of the main reason for wanting this plugin) then multi-threading is like a must I would think. I understand this is a ubot issue and nothing you can do much about. I personally am not buying any more plugins from anyone until Eddie adds a way to implement "local" variables. I'm sure I'm not the only one, so it may be a good idea for all plugin devs to apply a little pressure on the ubot dev team to get this rolled out asap. Quote Link to post Share on other sites
Aymen 385 Posted March 19, 2013 Author Report Share Posted March 19, 2013 This looks like an awesome plugin! I am waiting for some kind of confirmation about the multi-threading issue before I purchase any new plugins however. If speed is the issue (which I would think is one of the main reason for wanting this plugin) then multi-threading is like a must I would think. I understand this is a ubot issue and nothing you can do much about. I personally am not buying any more plugins from anyone until Eddie adds a way to implement "local" variables. I'm sure I'm not the only one, so it may be a good idea for all plugin devs to apply a little pressure on the ubot dev team to get this rolled out asap.There is no problem with the speed , you get the default speed of the http requests , for multi threading i can convert the http post command to a function and that would fix the multi threading problem , however that would probebly cause few problems like everytime i update and add things to the function for example, it won't be possible to drag other created commands inside the function , you will need to update your code everytime to match the new function inputs ! Quote Link to post Share on other sites
Aymen 385 Posted March 20, 2013 Author Report Share Posted March 20, 2013 Hey Aymen, sent you a PM about a week ago and haven't hear from you. Can you please check it out and let me know. Also I see you added $html encode. how we should use it? I'm having the issue when scraping that if the site has special characters it will return weird characters. If that function doesn't work for scraping it would be nice to have the option to select the type of encoding we would like to have the text scraped as. Hey , first of all sorry about that , must of forget !the html encode function is use to encode the html special characters so you can send a http request with html content to the server without getting errors !http request can read only encoded strings so sending special characters in the http requests may lead to getting errors!the default Encoding for the http requests is UTF-8 , but i can make the option of choosing your own encoding ! Quote Link to post Share on other sites
iDollarsteam 13 Posted March 20, 2013 Report Share Posted March 20, 2013 I'm new to http requests but I have read a lot about http headers and requests today and I want to use this plugin for youtube ... I used fidler2 to see the http requests between the browser and youtube server and I want to ask the guys that know how this things works: have you tried this plugin to send http requests and answers for youtube?? (to confirm views, video ratings, subscribers on channels and so on)... it is working for the kind of http requests youtube and google use? Quote Link to post Share on other sites
Aymen 385 Posted March 20, 2013 Author Report Share Posted March 20, 2013 I'm new to http requests but I have read a lot about http headers and requests today and I want to use this plugin for youtube ... I used fidler2 to see the http requests between the browser and youtube server and I want to ask the guys that know how this things works: have you tried this plugin to send http requests and answers for youtube?? (to confirm views, video ratings, subscribers on channels and so on)... it is working for the kind of http requests youtube and google use?personally i didn't test it on youtube Quote Link to post Share on other sites
iDollarsteam 13 Posted March 20, 2013 Report Share Posted March 20, 2013 I just bought it waiting for the download link Quote Link to post Share on other sites
Aymen 385 Posted March 20, 2013 Author Report Share Posted March 20, 2013 I just bought it waiting for the download linkThank you Download link sent !Enjoy ! Quote Link to post Share on other sites
north_star 16 Posted March 21, 2013 Report Share Posted March 21, 2013 Hey Man, any idea how to GET the Boundary value on this kind a code ? Content-Type: multipart/form-data; boundary=---------------------------[value] Quote Link to post Share on other sites
Aymen 385 Posted March 21, 2013 Author Report Share Posted March 21, 2013 Hey Man, any idea how to GET the Boundary value on this kind a code ? multipart/form-data Content type is not supported for now it is used mostly with upload forms , i'm working on a command for that hopefully will be added next week 2 Quote Link to post Share on other sites
SaHiN 14 Posted March 21, 2013 Report Share Posted March 21, 2013 Hey Man, any idea how to GET the Boundary value on this kind a code ? QuoteContent-Type: multipart/form-data; boundary=---------------------------[value] Content-Type: multipart/form-data; boundary=---------------------------[value] name="user" john name="pass" 123456Convert ; user=john&pass=123456 nice working 1 Quote Link to post Share on other sites
north_star 16 Posted March 22, 2013 Report Share Posted March 22, 2013 (edited) user=john&pass=123456 Great trick man , thank you but I can't implement this on my node, please check your PM man .. Edited March 22, 2013 by north_star Quote Link to post Share on other sites
SaHiN 14 Posted March 22, 2013 Report Share Posted March 22, 2013 Great trick man , thank you but I can't implement this on my node, please check your PM man .. check your Pm 1 Quote Link to post Share on other sites
illmill 87 Posted March 22, 2013 Report Share Posted March 22, 2013 yeai found a problem clearning twitters cookies , i'm guessing it is because of the https , i'm doing tests on sites with the https , will keep you guys posted on this !Any update on this? Also, I think you mentioned on skype something about updating a source code for twitter, but I forget what it was for exactly. Is there any progress on that (and what were you going to share with the source code)? Quote Link to post Share on other sites
Aymen 385 Posted March 22, 2013 Author Report Share Posted March 22, 2013 Any update on this? Also, I think you mentioned on skype something about updating a source code for twitter, but I forget what it was for exactly. Is there any progress on that (and what were you going to share with the source code)?This is in progress , a fix for this will be released very soon ! 1 Quote Link to post Share on other sites
north_star 16 Posted March 24, 2013 Report Share Posted March 24, 2013 (edited) is there any trick to get ( download ) the Chaptcha image on our PC and then prompt it..? and then let user to fill the value..? Edited March 24, 2013 by north_star Quote Link to post Share on other sites
VentureOnline 49 Posted March 24, 2013 Report Share Posted March 24, 2013 Grab the image URL into a variable and then load the variable into the load html command. Quote Link to post Share on other sites
north_star 16 Posted March 24, 2013 Report Share Posted March 24, 2013 Grab the image URL into a variable and then load the variable into the load html command. I'm trying but no luck with this kind a chaptcha type http://url.org/imagebuilder.php Quote Link to post Share on other sites
Aymen 385 Posted March 24, 2013 Author Report Share Posted March 24, 2013 save the image file to a location , then load it to the load html using its local URL for example : file:///C:/Users/acer/Desktop/image.pnguse file:/// followed by the image path location Quote Link to post Share on other sites
SaHiN 14 Posted March 24, 2013 Report Share Posted March 24, 2013 save the image file to a location , then load it to the load html using its local URL for example : file:///C:/Users/acer/Desktop/image.pnguse file:/// followed by the image path location Hey Aymen, I have the same problem before and i think Load Html doesnt work on local images. You need to upload your images on your server. Quote Link to post Share on other sites
Aymen 385 Posted March 24, 2013 Author Report Share Posted March 24, 2013 Hey Aymen, I have the same problem before and i think Load Html doesnt work on local images. You need to upload your images on your server. oh i seenavigating to it using this code works too ! navigate("file:///C:/Users/acer/Desktop/image.png", "Wait") 1 Quote Link to post Share on other sites
SaHiN 14 Posted March 24, 2013 Report Share Posted March 24, 2013 oh i seenavigating to it using this code works too ! navigate("file:///C:/Users/acer/Desktop/image.png", "Wait") Very good Aymen Thank you. save the image file to a location , then load it to the load html using its local URL for example : file:///C:/Users/acer/Desktop/image.pnguse file:/// followed by the image path location .jpg, .png, ... => nice working captcha.php, captcha.asp => don't working. Quote Link to post Share on other sites
Code Docta (Nick C.) 638 Posted March 26, 2013 Report Share Posted March 26, 2013 Hey , any videos on the new updates? Particularly the API function. Maybe some resources on all this HTTP stuff to figure out what all this could really mean for us? Like practical uses. Thanks.TC 1 Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.