Jump to content
UBot Underground

Recommended Posts

I am putting this case study up for myself and others to get a better understanding of ubot.

 

many before us has played with the program and got great results. Some of us ( including me let the program sit for a while out of frustration of not understanding ubot)

 

but after watching the tutorials and looking at the code. I am getting a little bit better at it now.

 

My hope with this case study is that advance members and newbies alike can dorp in a script here or their. and create a jing cast to explain a process slowly in bit size chunks so we can enhance one another.

Link to post
Share on other sites
  • Replies 94
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

PLEASE LET ME KNOW HOW I AM DOING SO FAR. BY clicking the plus or minus sign at the end of the posts

So lets recap   The Ubot System of operation known as USMEAC.   U: Utility - the Site or platforms over which the operation is to happen (may be considered to be part of the Situation heading)   S: Si

Alright here's the first beta of the finished product: http://www.filesend.net/download.php?f=c40382eff7f1c9abeebbcf02d3a5d424   Pretty much you type in City, State and it gets to it. .CSVs are output

to get a better understanding i went on a tutorial hunt.

First I would like to thank everyone fro creating the tutorials and the users for leaving tips and tricks. I have not looked at all of them. but the one that i have read i am truly grateful

 

ubotjunkie is the first place that i would stop at.

thanks frank for a great breakdown

 

He breaks down the process of a step by step format to what you want to do.

 

just click on the link above and watch the videos.

 

so since this first case study is on Arrons Nimocks yellow pages scraper his website is

ubottutorials

 

to get the script goto the other yellow paages post made by me and downlaod the script that Arron provided. And if you can leave him a donation for the contribution. anything a $5 or more will help and show the appreciation of the contributors work.

Link to post
Share on other sites

Yellow page scraper

Navigate to yellow pages

Enter a search term

Enter a search location

Click on button

Waite for page to finish

Scrape business name

Scrape business address

Scrape business city

Scrape business state

Scrape business zip

Scrape business phone

Ensure scrape info from the page

 

We also want to scrape the info from the sub pages under each business

 

Please leave feedback and best suggestions

Link to post
Share on other sites

the next thing what i did was look at Arron yellow pages scraper bot to see if i can break it down

 

and i downloaded the tutorial from elteejay and watched him explain the input strings

 

so what i am doing is taking the training from others and applying what they did that closely resemble what i am trying to create. Some of the explanation are moving to fast so i have to pause the vids so i can grasped what they are saying.

 

so the places that I have when to so far is ubot junkie,ubot tutorials, elteejay post in the forum, look them up under members and find the post. and the sales page for the ubot, they also have training in the training section

Link to post
Share on other sites

so now that i have some training tools and systems in play. followed along with some of the instructions, pausing the vids on the way so i can get a better understanding of the process and what they was doing to input the strings.

 

The UBOT system known as USMEAC.

 

U: Utility - the site or sites over which the operation is to happen (

(may be considered to be part of the Situation heading)

 

S: Situation - What do you want Ubot to do

 

M: Mission - a simple mission, normally delivered in a single sentence which is then repeated

 

E: Execution - Details of how the Utility is to be carried out, including:

 

* Commander's intent - what the bot is to do

* Scheme of maneouvre - how the bot is to do it

* Main Effort (optional) - where it is to be done

* Mission statements for any sub-units

* Timings

* Actions upon certain circumstances

* A summary of the execution plan

 

A: Administration and logistics

 

C: Command and signal

Link to post
Share on other sites

U= Yellowpages.com

S= Ubot to scrape the:


  •  

  • business name

  • address

  • city

  • state

  • zip

  • phone

  • email

  • website

M= create a bot to scrape the yellow pages for marketing list

E=

Commanders intent- goto yellow pages input field to select target to scraper

Scheme of maneuver- goto main site, input variables, have bot to wait to load page

 

Main Effort- done in cloud environment, upload results to forum for input…. This is a learning module give credit to all contribute link to pages.

 

Mission for sub-units- to scrape sub pages of all information above

Timings - record time of how fast you can create the bot after you learn the process

Actions- seek help from co members of the forum and the creators for simple instruction…. To anything there is a learning curve

Summary- give a break down of the creation of the bot once completed.

A= look over loops, see other samples of scripts, ask other s to look over the script that you created to spot any errors that you have made. And ask them if possible for a jing explanation

Create subs to code out progress.

 

C= the command for this case study is to give the complete update with videos to the bot contributor to be placed on their site if they choose to be compensated for the idea in the first place. In a nut shell they gave so give back

Link to post
Share on other sites

so now that we have a plan.

we need to talk out what we need to do.

 

we look at the code in the yellow pages scraper to see what strings was used to create the bot.

on my system it does not work so we need to make a new one based on the code of the old.

 

in the code

 

we see the codes

while -

search page

next

 

run sub

this one

next page

delay = 5

 

run sub

this one

scrape urls

run sub

this one

scrape listings

 

loop

 

 

 

the first is to open ubot and save as YP beta 1

navigate to the yellow pages site

Link to post
Share on other sites

Here we have the nav section to the bot after saving the bot we needed to goto the yellow pages

under the action command we choose the navigate we dragged it to script list and the www.yellowpages was automatically submitted in the code. then we save the file

 

having problems uploading the file

Link to post
Share on other sites

so what am doing is breaking it down into little steps so i can grasp the information better.

going back over the tutorials online especially Arron Nimocks and elte ejay. I am not saying that this is the best way to self teach yourself but this is what working for me.

 

the video that i am watching is the navigating a site and scraping product data by Arron, and the input interface by elteejay

 

please leave a feedback.

 

In the beginning when i purchased this program I wanted every thing now. the codes the scripts and tutorials.

well they are all their you just got to take the time to adjust to them, learn some thing new. and be humble to receive the information that others are kind enough to leave for you to help in your learning process.

 

and above all things is to be coach-able.

 

repetition is the key. going over the the videos repetitively and reading the other posts will help.

Link to post
Share on other sites

I tend to Bot in spurts... I don't touch it for a month or two then whip out a few bots (or a rather complex one) in short succession. One thing I have found that is really helping me is to be much more segmented in my bot building. Use subs A LOT and just make each sub do a small relatively simple task.

 

For example the first thing I usually create for almost any bot, is a login script. All it does is take a supplied username and password, navigate to whatever site I'm going to begin scraping (most of the ones I've been doing require logins) and log me in. Sometimes I give it a little bit of logic to detect for example if I'm already logged in and if so, not do it again. Then I can call this script from within another one if I want to, or run it manually.

 

Then in the main script, I break things up as much as I can. I create an initialize sub, which contains the UI elements, list clearings, variable defaults, etc. Then I take each core step of the bot and put it in its own sub.

 

Probably something else I should start doing is first to "sketch" out the bot by creating empty subs with the names of the MAJOR functions of the bot. The really BIG pieces. Then fill in each really big piece with smaller subs for the smaller functions. Then lastly begin actually creating the steps within those small functions. This would create a very solid structure for the bot.

 

Jonathan

  • Like 1
Link to post
Share on other sites

thank you webautomationlab for the kind words

 

thank you tool trainer for your process

 

can you provide a screen shot or a vid cast of the steps that you just explained.

 

if any advanced members that are out their that will be willing to leave an example of what tool trainer just stated

 

thank you

Link to post
Share on other sites

tooltrainer stated

  1. "sketch" out the bot
  2. creating empty subs
  3. with the names of the MAJOR functions of the bot. The really BIG pieces.
  4. Then fill in each really big piece with smaller subs for the smaller functions.
  5. Then lastly begin actually creating the steps within those small functions.
     
    This would create a very solid structure for the bot.
     
    can some one provide an example

Link to post
Share on other sites

I was asked a question of why am i doing this.

 

well first it is to improve my understanding of Ubot.

 

second why am I trying to get everyone involved.

 

by having advance member and nobs involved it is doing alot of things for the community

 

by the developers leaving an sample of what they know , customers and newbies alike will look at the rating and post that the contributor to see if he or she has an understanding of the program. your reputation will go up in the community and by leaving a vid explaining the process that they use will show an command over the utility.

 

nobs and customs will engage you more for project development of bots and want to outsource their projects to you. or a team of developers.

 

by teaching you get a better grasp of what you are doing and you begin to see things differently, develop differently.

 

not every one is interested in the same thing. and each of us have a different skill set. which can lead to join ventures, or a nob hiring an advanced member to build a bot while they are learning.

 

there is many project out their for data mining etc not just for internet marketers. ubot can become a hub to pool talents together like WPMU developer, or the Joomla help group, or drupal, to bid and complete a project.

 

so in turn the advance members are helping the nobs get a better understanding by leaving examples and vid samples.

 

donations and points are awarded to the advance members for their contribution. the ubot community will benefit as a whole by creating more developers other other platforms and other programs which in turn will bring in more clients and sales for the independent contractors and the developer of the Ubot

 

in turn it will bring in more money to hire ubot programmer to enhance the program by leaps and bounds. some great suggestions are being made by forum members like frank... and others that mention in having the url in the script. so by improving the overall quality of the community it will bring in extra coder and create developer to expand the functionality of ubot.

 

every one have a different level of learning. but the creativity in each of us will force us to another level of what has already been done

Link to post
Share on other sites

 

...For example the first thing I usually create for almost any bot, is a login script. All it does is take a supplied username and password, navigate to whatever site I'm going to begin scraping (most of the ones I've been doing require logins) and log me in. Sometimes I give it a little bit of logic to detect for example if I'm already logged in and if so, not do it again. Then I can call this script from within another one if I want to, or run it manually.

 

Then in the main script, I break things up as much as I can. I create an initialize sub, which contains the UI elements, list clearings, variable defaults, etc. Then I take each core step of the bot and put it in its own sub.

 

Probably something else I should start doing is first to "sketch" out the bot by creating empty subs with the names of the MAJOR functions of the bot. The really BIG pieces. Then fill in each really big piece with smaller subs for the smaller functions. Then lastly begin actually creating the steps within those small functions. This would create a very solid structure for the bot.

 

Jonathan

 

Hey Tool. That is almost exactly how I started doing things a little while back. One of the biggest advantages for me, is I always know exactly where I left off the last time I worked on it. I also try to "pre-plan" for as many subs as I can think of that I expect to call on several times, so I can get them setup early in the process.

 

This is great stuff Winston, thank you.

 

John

Link to post
Share on other sites

I'm all for sharing. It is a good idea. I benefit from smarter more talented users making creative bots I can't even imagine a need for today, but that I will enjoy using tomorrow.

 

I think that is the real strength of UBot.

 

I organize my bots in big subs usually.

 

Initialize

UI

Startup

Loop

Shutdown

 

Initialize clears cookies, clears list, sets constant variables.

 

UI defines all of the UI elements

 

Startup is my initial NAV, list processing, initial counter value setting etc that should be done before the main Loop.

 

Loop includes the bulk of my bot. If the bot does not require any looping, then this sub is redundant.

 

Shutdown takes list data and saves it, does completion NAV or pops up Javascript "finished" alert boxes and all other post loop processing before the bot stops.

 

Generally this framework works for 90% of what I use UBot for. Sometimes I need to add subs, but I almost always need these as a bare minimum.

 

http://i47.tinypic.com/oje8tc.gif

Link to post
Share on other sites

* "sketch" out the bot

* creating empty subs

* with the names of the MAJOR functions of the bot. The really BIG pieces.

* Then fill in each really big piece with smaller subs for the smaller functions.

* Then lastly begin actually creating the steps within those small functions.

 

biggest advantages

  • always know exactly where You left off the last time You worked on it.
  • Try to "pre-plan" for as many subs as You can think of that You expect to call on several times,
  • so You can get them setup early in the process.

This would create a very solid structure for the bot.

 

can some one provide an example

 

thanks johnB

 

can some one provide a written example or a video explaining this process

Link to post
Share on other sites

OK we now have part of the basics down. Now we are going to search around the forum to get some other great idea and tips.

 

Duke n Dave created a post in the scripting tutorials, tips and tricks

A good way to see what is going on in your script

Link to post
Share on other sites

BAMCIS process

  • Begin the Planning

sketch" out the bot

  • Arrange Recon
    visit and plan your bots function at the site
  • Make Recon
    See how other Ubot to scrape the Yellow pages
    • business name
    • address
    • city
    • state
    • zip
    • phone
    • email
    • website

    [*]Complete Planning

  • organize Your bots in big subs usually.
  • Initialize
  • UI
  • Startup
  • Loop
  • Shutdown
  • Initialize clears cookies, clears list, sets constant variables.
  • UI defines all of the UI elements
  • Startup is my initial NAV, list processing, initial counter value setting etc that should be done before the main Loop.
  • Loop includes the bulk of my bot. If the bot does not require any looping, then this sub is redundant.
  • Shutdown takes list data and saves it, does completion NAV or pops up Javascript "finished" alert boxes and all other post loop processing before the bot stops.

  • Issue Order
    run and test the bot

  • Supervise

Link to post
Share on other sites

So as we can tell learning how to develop a bot is 80% pre planning

 

so lets go back to our USMEAC and break that down

 

U

S

  • A. Site Functions and Processes
    1. Situation (SALUTE) Size, Activity, Location, Unit, Time (observed), Equipment
    2. Capabilities/Limitations (DRAW-D) Defend, Reinforce, Attack, Withdraw, Delay
    3. Most Probable Course Of Action and functions of the site (EMPCOA)

    [*]B. Ubot functions and operations

    1. Higher's Mission & Intent
    2. Adjacent Scripts
      • UI,Startup,Loop,Shutdown
      • Find open source scripts and bots

    [*]Supporting- find tutorials and bots that does what you want it to do

[*]C. loops/subs and sub loops

M

E

A

C

Link to post
Share on other sites

Yes this is based on the:

 

The five paragraph order is an element of United States Marine Corps, United States Army and United States Navy Seabees of small unit tactics that specifies instruction to a unit, based on an METT-TC Analysis (Mission, Enemy, Terrain & Weather, Troops & Fire Support, Time, and civilian) using the BAMCIS process (Begin the Planning, Arrange Recon, Make Recon, Complete Planning. Issue Order, Supervise) (i.e. fireteam, squad, platoon, company, etc.) prior to potential enemy engagement. It provides a structure for the unit to be able to understand and execute the mission of the unit leader. It is different from other instruction from higher authority in that it is given orally, instead of being issued as written orders. Officers and non-commissioned officers also use it informally to communicate relevant information prior to a non-combat movement (e.g. administrative travel/convoy, field exercise movements, weapon requalification, liberty, etc.).

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...