[Sell] - Http Plugin Tutorial Video Series

[Code Docta (Nick C.) 638]

Hey Dan!

 

Got this a lil bit ago and wanted to say great job!!!
 

I bought it for the G-Oauth tuts and Curl, mainly.

 

I have been sending people your way for a while and was curious too.

 

I mean, I like to see how others do things as well.

 

Now, I can safely say from first hand experience these are great tutorials and I hope to see more from you Sir!!

 

Can't tell how great these would've been when I started out.

 

If you guys want to speed things up this video series will cut the learning curve wayyyy down.

 

Thanks Dan,

 

TC

[Bot-Factory 602]

Hey Dan!

 

Got this a lil bit ago and wanted to say great job!!!

 

I bought it for the G-Oauth tuts and Curl, mainly.

 

I have been sending people your way for a while and was curious too.

 

I mean, I like to see how others do things as well.

 

Now, I can safely say from first hand experience these are great tutorials and I hope to see more from you Sir!!

 

Can't tell how great these would've been when I started out.

 

If you guys want to speed things up this video series will cut the learning curve wayyyy down.

 

Thanks Dan,

 

TC

 

Awesome! Thanks a lot for your feedback TC. 

If there is anything missing, just let me know guys.

 

Dan

[mamica 10]

All this HTTP thing is new to me, i have watch few videos and saw how to post. But i m interested if this plugin can click a button on a page? I know i asked same question few months ago about generating fake traffic and answer was that HTTP cant generate fake traffic because it dont load javascript counters. But what about clicking a button (like a vote button) can i increase votes or likes with this plugin?

[jamesfar 15]

All this HTTP thing is new to me, i have watch few videos and saw how to post. But i m interested if this plugin can click a button on a page? I know i asked same question few months ago about generating fake traffic and answer was that HTTP cant generate fake traffic because it dont load javascript counters. But what about clicking a button (like a vote button) can i increase votes or likes with this plugin?

 

Open your fiddler >> press F12 on your keyboard to snif >> do the vote manually on your browser >> check what fiddler gave you >> replicate fiddler result using http post plugin on ubot and checkt the result

 

Hope it helps

[Bot-Factory 602]

All this HTTP thing is new to me, i have watch few videos and saw how to post. But i m interested if this plugin can click a button on a page? I know i asked same question few months ago about generating fake traffic and answer was that HTTP cant generate fake traffic because it dont load javascript counters. But what about clicking a button (like a vote button) can i increase votes or likes with this plugin?

 

 

Hi Mamica,

 

it's very important to understand the concept of the http plugin. When you work with a normal browser, you have to distinguish between two things:

 

1. Stuff that is executed and done within your browser. Rendering of HTML code, executing of javascript, python and so on.

2. Communication between your browser and the webserver.

 

So the http plugin is able to simulate Number 2. We can simulate and executed http get and post commands. This is the same communication your browser is using when it communicates with the backend webserver.

 

So when we talk about "Clicking a button".

We are NOT simulating the button click on the website. 

We are simulating what the browser will send to the webserver when the button is clicked.

 

For example:

If a website has javascript. It could be that the button will execute some code locally. It could be used to change something on the site or to open / close menus. But it's mainly to control the behavior within the website of the browser. 

As soon as a javascript or a button needs additional data from the backend webserver, there will be another http call. 

 

So we can not execute jabascript with http plugin. That's true.. BUT we can simulate the HTTP calls the javascript might do. 

 

Example:

A website has a button with: More informations.

This is javascript code. So when you click the button the website will execute a small javascript snipped. But the necessary information is not available on the site. Then there will be another http call to receive the additional informations from the webserver. And you can "fake" that call with http plugin.

 

 

You have to think a little bit different when working with the http plugin.

 

So you can't say that javascript doesn't work with HTTP plugin in general. You have to investigate and trace what each website is doing. 

And then there are really a lot of possibilities with http plugin.

 

Hope that helps a little bit.

 

Cheers

Dan

[SaHiN 14]

Hi Dan,

 

new video idea = http upload code

 

sample = wordpress upload picture

[Bot-Factory 602]

Hi Dan,

 

new video idea = http upload code

 

sample = wordpress upload picture

Hello.

 

Do you run into some special issue with that task? 

 

Dan

[SaHiN 14]

Hello.

 

Do you run into some special issue with that task? 

 

Dan

 

Wp http post login, wp image upload  and the result image link

[Bot-Factory 602]

Wp http post login, wp image upload  and the result image link

 

Do you have some example code? A Fiddler trace when you execute that task natively via the browser. 

And a fiddler trace when you do it with your ubot script? Do you see a difference? Any special cookie that is not submitted? Some post data value that is different or missing when you do it with your script?

 

I always need a special challenge to accept something for a new tutorial video. A lot of websites are using complex http communication and it's more or less "just" assiduity work to figure out the right http procedure. But most of the time everything you need is covered in the tutorial videos already. 

 

But if there is something special necessary to make it work, I will definitely do another tutorial video. 

 

If you can send me some details, I will definitely take a look. 

 

Dan

[SaHiN 14]

Do you have some example code? A Fiddler trace when you execute that task natively via the browser. 

And a fiddler trace when you do it with your ubot script? Do you see a difference? Any special cookie that is not submitted? Some post data value that is different or missing when you do it with your script?

 

I always need a special challenge to accept something for a new tutorial video. A lot of websites are using complex http communication and it's more or less "just" assiduity work to figure out the right http procedure. But most of the time everything you need is covered in the tutorial videos already. 

 

But if there is something special necessary to make it work, I will definitely do another tutorial video. 

 

If you can send me some details, I will definitely take a look. 

 

Dan

 

special cookie  = yes login

 

important  = wp http upload picture code and result image link.

 

[Pete_UK 9]

Hi Mamica,

 

it's very important to understand the concept of the http...

...

...

Example:

A website has a button with: More informations.

This is javascript code. So when you click the button the website will execute a small javascript snipped. But the necessary information is not available on the site. Then there will be another http call to receive the additional informations from the webserver. And you can "fake" that call with http plugin.

Hi Dan,

 

I've been looking at using the http plugin with this is https://sellercentral.amazon.com/gp/fba/revenue-calculator/index.html/ref=xx_xx_cont_xx?ie=UTF8&lang=en_US

 

From the quote above it looks like this site falls into 'could be tricky' category.

 

It's easy to create a Post request with the 2 parameters (fiddler show METHOD & MODEL) required for the ASIN/PRODUCT search but as this search is controlled via javascript the results returned are not a webpage but son data instead. See image in the link below.

 

http://screencast.com/t/z4US7yXOwrl1

 

Do you have any pointers to help me figure out how to get the results rendered?

 

May be this would make a good video if you know of a solution. Or, maybe the page is too complex for http,

 

Thanks,

Pete

[Bot-Factory 602]

Hi Dan,

 

I've been looking at using the http plugin with this is https://sellercentral.amazon.com/gp/fba/revenue-calculator/index.html/ref=xx_xx_cont_xx?ie=UTF8&lang=en_US

 

From the quote above it looks like this site falls into 'could be tricky' category.

 

It's easy to create a Post request with the 2 parameters (fiddler show METHOD & MODEL) required for the ASIN/PRODUCT search but as this search is controlled via javascript the results returned are not a webpage but son data instead. See image in the link below.

 

http://screencast.com/t/z4US7yXOwrl1

 

Do you have any pointers to help me figure out how to get the results rendered?

 

May be this would make a good video if you know of a solution. Or, maybe the page is too complex for http,

 

Thanks,

Pete

 

Hello Pete,

 

I'm not 100% sure what your question is.

 

The search query is executed by a post request. That request has two parameters:

method

model

 

And the value for the model parameter looks like:

{"searchString":"B00CG5X4XC","lang":"en_US","marketPlace":"ATVPDKIKX0DER"}

 

So this should be very easy to create.

 

Now when you run that post request you will get the following return:

 

{"data":[{"link":"http://www.amazon.com/gp/product/B00CG5X4XC/ref=xx_xx_cont_xx","dimUnits":"inches","thumbnail":"https://images-na.ssl-images-amazon.com/images/I/41DxD4dyzqL._SL80_.jpg","subCategory":"2102020","dimensions":{"width":4.7,"length":8.2,"height":2.8},"gl":"gl_toy","image":"https://images-na.ssl-images-amazon.com/images/I/41DxD4dyzqL._SL120_.jpg","weightUnits":"pounds","productGroup":"21","weight":0.3,"asin":"B00CG5X4XC","whiteGlovesRequired":"N","title":"Beyblade Shogun Steel BeyWarriors BW-08 Pirate Orochi Battler [Toy]"}],"errorMessage":null} 

 

 

Now you can extract the product URL from that return.

http://www.amazon.com/gp/product/B00CG5X4XC/ref=xx_xx_cont_xx

 

 

Not sure if that answers your question? If not, I need more details about what you actually try to do.

 

Dan

[Pete_UK 9]

 

Hello Pete,

 

I'm not 100% sure what your question is....

 

The search query is executed by a post request. That request has two parameters:

method

model

 

And the value for the model parameter looks like:

{"searchString":"B00CG5X4XC","lang":"en_US","marketPlace":"ATVPDKIKX0DER"}

 

So this should be very easy to create.

 

Now when you run that post request you will get the following return:

 

{"data":[{"link":"http://www.amazon.com/gp/product/B00CG5X4XC/ref=xx_xx_cont_xx","dimUnits":"inches","thumbnail":"https://images-na.ssl-images-amazon.com/images/I/41DxD4dyzqL._SL80_.jpg","subCategory":"2102020","dimensions":{"width":4.7,"length":8.2,"height":2.8},"gl":"gl_toy","image":"https://images-na.ssl-images-amazon.com/images/I/41DxD4dyzqL._SL120_.jpg","weightUnits":"pounds","productGroup":"21","weight":0.3,"asin":"B00CG5X4XC","whiteGlovesRequired":"N","title":"Beyblade Shogun Steel BeyWarriors BW-08 Pirate Orochi Battler [Toy]"}],"errorMessage":null} 

 

 

Now you can extract the product URL from that return.

http://www.amazon.com/gp/product/B00CG5X4XC/ref=xx_xx_cont_xx

 

 

Not sure if that answers your question? If not, I need more details about what you actually try to do.

 

Dan

 

Hi Dan,

 

Sorry for not being clear on my question.

 

A Post request returns data fine but the main issue I have is... the data is just some 'data' that the existing webpage processes via javascript. Once the page updates it then makes the CALCULATE button available that generates another post request that brings back more data that is again used by javascript.

 

So my query is... how can I get the http requests trigger the page update using the json data that allows the Calculate button to become active.

 

Here is a video that I hope explains it a little better: http://screencast.com/t/LpMwZ8moP4W

 

Thanks,

Pete

 

PS. I will ad.... I think the answer will be something like .... there is no easy way around it. It's not an issue with the http request but more how the data is processed.    

Edited October 26, 2014 by Pete_UK

[Bot-Factory 602]

Hi Dan,

 

Sorry for not being clear on my question.

 

A Post request returns data fine but the main issue I have is... the data is just some 'data' that the existing webpage processes via javascript. Once the page updates it then makes the CALCULATE button available that generates another post request that brings back more data that is again used by javascript.

 

So my query is... how can I get the http requests trigger the page update using the json data that allows the Calculate button to become active.

 

Here is a video that I hope explains it a little better: http://screencast.com/t/LpMwZ8moP4W

 

Thanks,

Pete

 

PS. I will ad.... I think the answer will be something like .... there is no easy way around it. It's not an issue with the http request but more how the data is processed.  

 

Hi Pete,

 

thanks for the video. The main question is:

Do you want to render and see the webpage with all the details. Or do you only need the values and numbers who are returned?

 

The http plugin can not execute javascript. 

 

But when you press the Calculate button, that's another HTTP post request. So you can simulate that, and you will get all the data that is returned and shown on the page. So then you can use that data. 

But you can't render and show the page. 

 

I mean, using the http plugin has a different usecase, it's best for extracting lot's of data very quickly. Because we are not rendering and dealing with all the browser related stuff.

But if you want to see and show the website to your users, well.. then it's best to just work with the ubot browser. 

 

So the question really is, what information's do you need and what do you want to do with them? 

You can get all the numbers and values via HTTP plugin. Even the numbers that are shown after pressing the calculate button. 

 

But you can not render and show the exact same amazon html page afterwards. That's not possible. Well, at least not within a reasonable time. And I really don't see a reason why you want to do that.

 

So in my opinion there are two usecases:

1. You want to extract lot's of data.

Use http plugin. 

Show the value in a table or build your own html code to present the data

 

2. You want the exact same look and feel as on the amazon page.

Use the ubot browser.

 

If you can describe your overall goal with this, we can discuss this further. You can also PM me if you don't want to share it publicly. 

 

Cheers

Dan

[Pete_UK 9]

Hi Pete,

 

thanks for the video. The main question is:

Do you want to render and see the webpage with all the details. Or do you only need the values and numbers who are returned?

 

The http plugin can not execute javascript. 

 

But when you press the Calculate button, that's another HTTP post request. So you can simulate that, and you will get all the data that is returned and shown on the page. So then you can use that data. 

But you can't render and show the page. 

 

I mean, using the http plugin has a different usecase, it's best for extracting lot's of data very quickly. Because we are not rendering and dealing with all the browser related stuff.

But if you want to see and show the website to your users, well.. then it's best to just work with the ubot browser. 

 

So the question really is, what information's do you need and what do you want to do with them? 

You can get all the numbers and values via HTTP plugin. Even the numbers that are shown after pressing the calculate button. 

 

But you can not render and show the exact same amazon html page afterwards. That's not possible. Well, at least not within a reasonable time. And I really don't see a reason why you want to do that.

 

So in my opinion there are two usecases:

1. You want to extract lot's of data.

Use http plugin. 

Show the value in a table or build your own html code to present the data

 

2. You want the exact same look and feel as on the amazon page.

Use the ubot browser.

 

If you can describe your overall goal with this, we can discuss this further. You can also PM me if you don't want to share it publicly. 

 

Cheers

Dan

 

As a little history.... I started using the Ubot browser but for some reason with this page, it acts so inconsistently. I think this is something to do with the javascript rendering. I will not bother going into details on this but it was the inconsistent nature of the page in ubot that got me thinking of using the http plugin. I was hoping to get the data through post/get requests and then workout how to 'plug' the data back into the webpage to see the charts rendered.

 

It's the last step that really got me stuck.

 

Anyway, yes I totally agree that the http plugin is great for getting the data but it has nothing to do with working out what magic needs to be done to get the returned data back into the rendered webpage.

 

I've got the data coming through the plugin so I think my easiest route forward is to stick with the data and may be look at rendering it in a chart myself at a later date. Having the data is the main thing.

[sunny9495 42]

Just Purchased,Wish i could a solid foundation on HTTP Plugin following your tutorials

[Marton 48]

Great tutorials Dan, thank you, I had fun watching them. I definitely recommend this to anyone who wishes to cut the learning curve of the http post plugin (+ there are some other interesting stuff in there).

 

Some observation: *removed from thread by request, but I give permission to share my thoughts in the course. PMed OP. It's a simplification for the Xing script.*

 

All the best,

Marton

[Bot-Factory 602]

Thanks a lot for your feedback Marton.

 

I will share your optimization idea with the http tutorial members. 

Simplification is always a good thing after people have understood the basic concepts. 

 

Kindest regards
Dan

[Bot-Factory 602]

Black Friday SALE! 40% off
CODE: blackfridayx34

>>CLICK HERE<<

[HaHaItsJake 25]

 

Black Friday SALE! 40% off

 

How long is the coupon for? After Black Friday? 

 

Regards,

HaHaItsJake

[HaHaItsJake 25]

DAN IS THE MAN.

So much more understanding with HTTP Post. The Socket Generator is now useful to me too haha.

 

"oh this IP address requested 500 captchas from us today so lets show some tricky stuff"

 

Priceless. Haha

 

Thank you, the video you mentioned in the PM did work! perfectly I must say. I just followed that video along with my project and clicked play and it signed up 7 accounts without any problems. 

THANK YOU! Now need to research about the captcha solving for services. 

 

Anyone new to HTTP Post get this, you wont regret it. Watched one video and it blew my mind. Time to watch the rest tonight and see what else I can learn from them. 

Happiest Ubotter At The Moment, 
HaHaItsJake

PS.  Dan, check your PM.

[Code Docta (Nick C.) 638]

Hey Dan,

 

any plans to show how Ubots native sockets commands work?

 

Do you know if they have been debugged or work at all?

 

I know gogeta did twitter example but that was a while ago but think it broke in later updates, not sure tho.

 

Just wondering.

 

CD

[Bot-Factory 602]

Hey Dan,

 

any plans to show how Ubots native sockets commands work?

 

Do you know if they have been debugged or work at all?

 

I know gogeta did twitter example but that was a while ago but think it broke in later updates, not sure tho.

 

Just wondering.

 

CD

I've never tried the sockets commands.. Well... I took a quick look a while ago. But http plugin has a lot more options. 

And because everything is working fine for me, there was no need to look for something else. Ok, we could reduce the bot size. And people without the http plugin could save the plugin costs..

But for the things I do with http, sockets commands are to limited probably. But I might take a look at them in the future. To see if they are good enough. For some basic scrapers you can probably already use them.

 

But I would definitely miss the xpath parser from the http plugin. So that one alone is already a big win for me.

 

Dan

[Bot-Factory 602]

Black Friday SALE! 40% off
CODE: blackfridayx34

>>CLICK HERE<<
Valid till 11/29/2015

[Enigma 78]

Trying to PM you Dam but your inbox is full "The member Dan cannot receive any new messages"