Clean the Clutter with Include

Speed of implementation is everything in this business. UBot Studio simplifies the process of web automation, reducing your build time from 2 weeks to 2 hours. But what happens when your bots start to get larger and larger? If you’re not careful, you’ll start getting consumed in your bot’s complexity, as each little task taking longer and longer to complete.

That’s why we’ve introduced the “include” function into UBot Studio 5.5.

Include lets your current script reference previously written scripts without rewriting that code again, which means you can create whole scripts that do a lot in just a few lines, keeping your bots small and complexity under control.

The UBot team recently explained the Include feature in a popular chat room to discuss this feature with the community. This Q&A session has been edited and reproduced below along with a demo video and a script. For more information on using Include, make sure to check out the wiki and the forum.

 

Community Chat: How to use the include command

 

Seth Turin:

I thought I’d take a minute to explain how the include command works, both to my team and to the community

 

Jason Kelley:

When did this feature come out?

 

Seth:

5.5.6 – It was actually in UBot 3.5, but got cut from UBot 4. It’s now back and ready for action.

 

Jason:

Awesome. My impression from previous conversations about include is that it’s a way to organize code and make it more reusable, which helps because people have some really long code in some of their bots. Am I close?

 

Seth:

Yes, you are. I’m going to explain it through the art of extended analogy. Imagine that you’re the CEO of a new car manufacturing startup. We’ll call it… mistubotsi. At first, the entire company is run by you and your trusty robot friend, Hal. You have to teach Hal how to do everything in the company, from manufacturing the cars, to sales and marketing, to customer service, etc. At first, this is fine, because there are only a few orders at a time, and Hal’s job is pretty simple, but soon, business picks up, and Hal finds himself doing more and more different kinds of jobs. Soon, Hal’s bot brain is overloaded and frazzled.

Hal gets angry. One day he locks you in a crash test car. You’re all like “Open the car door Hal!” and he’s like “I’m afraid I can’t do that, Dave”. The moral of the story is, you want to keep your bots happy by keeping them small, focused, and organized. Include is a means of doing this.

 
Jason:

So I have a bunch of bots that use a kind of complex series of change proxy commands. In that instance I’d build that as a separate bot basically and use the include?

 

Seth:

You’re pressing on another benefit of include, which I’ll return to a little later. But the short answer is yes.

So to keep your bots happy, keep them small and focus on a single job (or a single website). Happy bots have the benefit of being easy to maintain, less error prone, more stable, easy to change, and easy to understand.

 

Seth:

Unhappy bots are bots that are big and bloated, unorganized, and try to do too many things… they suffer from breaking often, being hard to debug, being hard to maintain, and possibly… becoming homicidal. When a bot starts to become too large, break it into separate, focused bots, and include those into one central, manager bot. By doing this, you greatly increase your capacity to make large, complex bot systems, that will be far more productive than one bloated unhappy bot.

 

Manny  Hernandez:

It is simple. Build your bot in modules. Like if it has account creation and posting. One bot for account creation, another for posting… Managed both in a central bot for the Ul.

 

Seth:

Exactly! Granted, if all those are small enough, you can just separate them into scripts. If account creation became a big operation, it makes sense to put it in a separate bot.

 

Jason Kelley:

So the idea is like, if you want to build a big bot, you can do that – just use include to break it into a bunch of separate pieces that you know all work. Then if you see a bug you know where it is and you can fix it (like normal programmers do i’d imagine).

 

Dave: 

But you are only including the defines from the bot right not the whole bot???

 

Seth:

Yes. Dave, the bot is technically minimally loaded in full, with all its plugins and its own included bots, but you can only access it through defined commands and functions.

 

Dave:

Is there a limit to how many you can include??? In the real world, I mean.

 

Seth:

It’s only limited by your machine’s hardware. In computer science, this idea is called “encapsulation”. It keeps the different sections of a bot clean and clutter free.

 

Manny:

How is it used? You call the include and automatically the defines are available?

 

Macster

Then include does pretty much the same as what the local bot bank does … but it actually loads the defines when requested from the UBot file!?… Isn’t the whole process the same in both features?

 

Seth:

Macster, to answer your question, the private bot bank is a little more limited than included bots.

In order of customizability, it goes 1) plugins, 2) included bots, 3) private bot bank, 4) custom commands

Jason, I want to address your point earlier… you said you had a complex proxy switch process that you use in a bunch of bots. You probably copy pasta from one into the next. The problem with copy pasta is that if, for instance, the server where you get your proxies from changes, you now have to go update every single bot separately

 

Jason:

Yes, or if I decide to change how it works in one bot and not in a bunch of others

 

Seth:

Instead, you could make a separate bot that just focuses on proxy switching, and include it into all the bots like you mentioned. If the server changes, you just change the one bot and the change is applied to all the bots that include it

 

Manny:

In a compiled bot that uses the include, do we need to have the .ubot in the file folder for commercial bots? Or it works like the plugins do?

 

Seth:

At the moment, you need to have it in the same folder, or specify that path to it

bgproinvest2:

How do we pass inputs (variables, parameters) to the included bots? How do we pass user input data to the included bot from the main UI?

 

Seth:

As a parameter, so the define in the included bot will take a parameter. Then in the master bot you just send in the variable as the parameter.

 

Manny:

How is it used? You call the include and automatically the defines are available?

Seth:

Once in the toolbox, you can use defined commands like any other command. Included bots can use plugins, and they can include their own bots, so you can potentially create highly expressive, large scale bots, while keeping everything small and manageable!

 

jay_ar233:

Looks good, does include reduce ram memory consumption?

 

Seth:

Jay, it will considerably reduce resources at dev time. UBot Studio does a lot of stuff to make coding visual and simple, and that also means that huge scripts will get laggy. By using include, you reduce the amount of code UBot Studio is processing at any given time

It does, however, slightly raise the resource usage in the compiled bots. This increase is not significant, and the tradeoff is well worth it.

 

Tompots:

The include command will definitely be worth its weight in gold, if I understand you correctly. For instance I use a very special bot recipe for all my bots, and it works on almost any site I make bots for. At current, I’m using the same recipe for all my bots functions, and just making new tabs for the sets of define commands for each of those functions. So with the include command I will be able to take the code out of those tabs and make them into their own .ubot file and call them from one central bot instead of having a bot with 20 tabs, I can have one bot with one tab for the UI and 20 .ubot files being called when the user select to run a certain function of the bot. Correct?

 

Seth:

Yes, you are exactly correct

 

Tompots:

That is going to make Ubot very very powerful. With limitless possibilities now.

 

A quick example of how to use include [code attached].

 

Seth:

So what we’re doing is making a bot that will search the big 3 search engines. This particular bot is small enough that it could all be done with one bot, but in this case we’re going use it to demonstrate how include works. So we’re going to separate the search functions into separate bots, and then consume all 3 bots in our manager bot.

Making one of those bots looks something like this:

Unable to display content. Adobe Flash is required.

 Once we have all three, we put them together like this:

Unable to display content. Adobe Flash is required.

Includes should always be the first thing in a script. You can see that by putting include into your script, it creates a new category under commands or parameters (in this case parameters) with all the defines from the other bot.

Bot 1: Google Scraper

define $search google(#search term) {
    navigate("http://google.com","Wait")
    type text(<name="q">,#search term,"Standard")
    click(<name="btnK">,"Left Click","No")
    wait(3)
    add list to list(%results,$scrape attribute(<tagname="a">,"href"),"Delete","Global")
    return(%results)
}

 

Bot 2: Bing Scraper:

define $bingbot(#bingbot) {
    navigate("http://www.bing.com","Wait")
    wait(3)
    type text(<name="q">,#bingbot,"Standard")
    click(<name="go">,"Left Click","No")
    add list to list(%bingresults,$scrape attribute(<tagname="a">,"fullhref"),"Delete","G    lobal")
    return(%bingresults)
}

 

Bot 3: Yahoo Scraper:

define $search yahoo(#search term) {
    navigate("www.yahoo.com","Wait")
    type text(<id="p_13838465-p">,#search term,"Standard")
    click(<id="search-submit">,"Left Click","No")
    add list to list(%results,$scrape attribute(<id=w"link-*">,"href"),"Delete","Global")
    return(%results)
}

 

Bot 4: Include in Action – substitute your directory for  [LocationOfScripts] 

set(#search term,"hal the happy bot","Global")
include("C:\Users\[LocationOfScripts]\bingbot.ubot")
include("C:\Users\[LocationOfScripts]\google scraper.ubot")
include("C:\Users\[LocationOfScripts]\search yahoo.ubot")
load html("<pre>{$run function("google scraper","$search google")}
{$run function("bingbot","$bingbot")}
{$run function("search yahoo","$search yahoo")}</pre>")

 

For more basic details of how to use include, see the wiki, and thanks to all who attended our chat!