mobileconsultant
-
Content Count
33 -
Joined
-
Last visited
-
Days Won
1
Posts posted by mobileconsultant
-
-
Hi Dan,
Disregard, I replied too soon and made newbie mistake, I didn't realize I could override the "attribute to get" field and type in "content". Duh!
Sorry for the bother... but thank you for more valuable training videos, I got even more nuggets of info and you've convinced me to purchase Cbrowser plugin.
Thanks,
-Rich
-
1
-
-
Hi Dan,
Thank you for the response and video links, I will watch those now. The issue for me may be with my HTTP Plugin, I don't have the option "content" under the "attribute to get" like in your screenshot, I only have available (Innerext,InnerHTML,OuterHTML,value,id,name,src,href,type,action,title). I'm currently running (Amen's) HTTP Post v2.5.1.0. I'll check with him and see if that's the most updated version.
Thank,
-Rich
-
Hi Dan,
Your previous suggestion is working great. The only major problem I'm have is that I have an Xpath expression that works perfectly in XBP, but it will NOT work in Ubot HTTP Plugin. Actually, anytime I try to extract data from //meta tags, XBP pulls the data but Ubot HTTP Plugin will not. I've tried everything I can think of and checked all over the web but to no avail. Here is what I have...
XBP returns the same data for all three versions of these, but nothing in Ubot HTTP Plugin (I'm using InnerText setting and tried the others). I even tried to use "description" meta tag in expression without success.
//meta[@name=keywords]/@content
//meta[contains(@name,"keywords")]/@content/head/meta[@name=keywords]/@content
-Rich
-
Hi Dan,
Thanks, yeah I guess pulling the data back and working with it via a List in Ubot is the easiest way to get a value. I was trying to do it on the fly with Xpath expression. I guess when I saw the XBP status bar easily reporting the number of elements for each expression it made me want to use Xpath to return a value in Ubot, but your suggestion works perfect.
Thanks,
-Rich
-
Hi Dan,
I've been out of the Ubot loop for the last 8 months due to accident injury, but I just purchased your Xpath Builder Pro and I wanted to say what an incredible job you did with the Xpath Builder videos! I also have your HTTP Plugin videos which were/are great, but you really did an excellent job of explaining the nuances between the different Ubot plugins (BigTable, Cbrowser, HTTP Post plug-ins) when using Xpath expressions and just using Xpath expressions in general (i.e. /text() versus InnerText, etc.) You've done a fantastic job of organizing your knowledge and the information when explaining the usage of Xpath Builder Pro/Xpath expressions with these over-the-shoulder type videos. All the live demonstrations are great training by example and are especially valuable where you dig down into some tricky elements/node identification stuff using XBP. Awesome job Dan!
I did have a quick question regarding the best way to return and store the value of the number of elements contained in an Xpath expression. I tried using the Xpath function count() in the HTTP $xpath parser function to store the number of elements in a variable but couldn't get it to work. For example, the expression below has Xpath Builder Pro identifying 16 elements...
//ul[@id=listing-tag-list]//li
I can access each of the 16 elements in Xpath Builder and Ubot using these expressions...
(//ul[@id=listing-tag-list]//li)[1]
...
(//ul[@id=listing-tag-list]//li)[16]
I'm struggling in Ubot to get the count() Xpath function to work to return and store the value in a variable to then write code to point to each element's number. Every web page has a different number of elements, so that's why I need to keep checking for the count from page to page. Here is what I've tried unsuccessfully...
set(#x3,$plugin function("HTTP post.dll", "$xpath parser", #get, "count(//ul[@id=\"listing-tag-list\"]//li)", "InnerText", "HTML"),"Global")
You are such a Ubot/HTTP Plugin/Xpath ninja, I figured you'd know the answer immediately, any help would be great!
Thanks,
-Rich
-
UPDATE: So far I've been using Proxy-N-vpn service for a couple weeks I have their PP-5 (private proxies service) and it's been working flawlessly!! I'm going to be upgrading to more of their private proxies, will probably jump on their Black Friday Sale!!
Other feedback:
Proxy-N-vpn is much better than the 100 shared proxies service I've purchase from proxybonanza (though PNV proxies are private), A lot of the proxybonanza shared proxies are not valid on their shared plan. I've also tried out HideMyAss (HMA) shared and private proxies for a year and I wasn't impressed (I heard they improved their service since then). Another service I can't say enough good things about is StrongVPN.com, I've been using their VPN services for 3 years now and they are rock solid and their customer support is awesome. I have all my overseas outsourcers running StrongVPN. Their was another proxy provider I tried out last year, I forgot their name right now, but they were bad, then they kept charging my credit card for months after cancelling.
Well there is some feedback for now... I'll keep posting updates.
Overall, I'm really happy with Proxy-N-vpn!!
-
1
-
-
I'm going to try out this proxy service... I just purchased the PP-5 (private proxies service), still waiting for the service to go live, it says "pending" in my account when I login to their service to get my proxies setup. I'll let all forum members know how this service rates.
Btw, I've been using various other vendors for proxy and VPN services for the last 7 years and there have been some good services and some really bad ones too (ones bordered on shady biz practices with my credit card). So, I'll give my feedback for Proxy-N-VPN soon.
-Rich
Ok, I got my proxies activated pretty quickly by Proxy-N-vpn now, that's a good sign! Going to test them out this week against Google's newest algo changes that crippled some of my scraping scripts this month due to big G's newest IP detection/timing/banning schema, will let everyone know my results.
-Rich
-
Hi Guys,
I thought I would give a bit of an update since I haven't commented lately on this post. I had purchased v5 a while back but immediately switched to v4 because everything and everyone was using v4 and v4 plugins plus all the trainings were using v4 (thanks Ubot Buddy). I learned Ubot Studio with v4 installed and Buddy's training, then I had a Microsoft Win Update that killed my Win7 OS (reboot loop issue), so I was forced to reinstall OS and everything else but this time I reinstalled Ubot v5.5.1 based on Dan's recent input and watching his training on Aymen's HTTP POST Plugin. So, the last month I've been writing and successfully running "new" small to mid-sized v5.5.1 scripts without major issues. I have noticed v5.5.1 loads faster and is running more stable. My scripts primarily use HTTP Post and SQLite Plugins and I do XPath Expressions to extract data (thanks to Dan's training and Aymen's plugins).
One bummer though, I had to upgrade my Ubot support without ever using v5 previously after I purchased it (due to v5 instability), just to re-download v5.5.1 support forced me to pay again. I sent them a ticket explaining that I thought it sucked that I bought v5, couldn't use it, had to use v4. Then when I finally heard v5.5.1 was working (from Dan) they forced me to pay upgrade again just to try it out and see if v5.5.1 truely did work now. Support basically indirectly told me too bad, you gotta pay up regardless if our product did NOT work when you purchased it, they felt it worked since last year. Yeah right! Well, I paid anyways just so I could download v5.5.1.
So far, I'm not having any major issues with v5.5.1, but my newer scripts are not huge either. I've started to dust off my old C programming skills and started to switch to learning C# .NET to port over my bigger Ubot v4 scripts for speed and stability purposes. I even thought about porting over my current mid-sized Ubot scripts to PHP's OOP code using HTTP Post commands. So, Ubot v5.5.1 for me now is a fast prototyping scripting language to get a concept working then I want to port the scripts to C# or PHP either myself or hire it out.
So, take a chance on v5.5.1 if you like!
-Rich
Btw, here's a few great vids on using simple PHP and Xpath to scrape website data, great if you want to run scraping scripts on a hosted dedicated or VPS and push data in MySQL DB easily:
Video 1: (videos 2 and 3 links are in the description of video 1)
-
I'm going to try out this proxy service... I just purchased the PP-5 (private proxies service), still waiting for the service to go live, it says "pending" in my account when I login to their service to get my proxies setup. I'll let all forum members know how this service rates.
Btw, I've been using various other vendors for proxy and VPN services for the last 7 years and there have been some good services and some really bad ones too (ones bordered on shady biz practices with my credit card). So, I'll give my feedback for Proxy-N-VPN soon.
-Rich
-
Hey The Fixer,
I believe most Ubotters run SQLite to write to the local disk on their PC and maybe on a hosted VPS or dedicated server. Shared web hosted accounts (usually Linux or Windows based) will most certainly not allow you to run SQLite directly on the server. Here is some SQLite info that may answer your question better, since I'm not sure what your end goal is with SQLite and your data. You may just be looking for a traditional RDBMS like MySQL to store and manipulate your data.
About SQLite
http://www.sqlite.org/about.html
Appropriate Uses For SQLite
http://www.sqlite.org/whentouse.html
"SQLite is not directly comparable to other SQL database engines such as Oracle, PostgreSQL, MySQL, or SQL Server since SQLite is trying to solve a very different problem."I hope that helps someone...
-Rich
-
Hey Aymen,
This looks like a really nice plugin! I just ordered.
Thanks,
Rich
-
Hey Buddy,
Glad to hear back from you, sorry you are having ear infection troubles, I wish you a speedy recovery. We were a bit worried for your welfare, you are an asset here in the Ubot community. We look forward to your other works and website whenever you get it going.
Thanks for all your hard work with the training videos, they really help and are a great reference when coding Ubot scripts.
-Rich
-
ok, I figured it out, it was my programming, lol
In the replace command, I needed to swap the variable in the Search Text field with the text to be replaced in the Original Text field, resulting in the code looking like this... which works perfectly:
set(#itemTransFullURLs, $replace(#TempURLs, "/transaction/", "https://www.domain.com/transaction/"), "Global")
-Rich
-
Hey Guys,
I'm scratching my head here also. I'm running UbotStudio v5.5.1 and I can't rule out that a simple replace command is not working due to a bug in the UbotStudio or my programming. I have two variables, one (#TempURLs) with a block of text relative URLs and a second variable (#itemTransFullURLs) with a replace command to replace a portion of text (/transaction/) in the relative URLs with text of full path (absolute) URLs (https://www.domain.com/transaction/). The replace command will not do the replace. I've tried quotes, escaping the characters, etc. but it won't perform the replace.
Any help on this very simple issue would be greatful.
Thanks,
Rich
Here is the code:
set(#TempURLs, "/transaction/237721973?
/transaction/237664169?
/transaction/237573269?
/transaction/237573268?
/transaction/237573267?
/transaction/237573266?
/transaction/237573265?
/transaction/237573264?
/transaction/237573263?
/transaction/237539559?
/transaction/237539558?
/transaction/237539557?
/transaction/237539556?
/transaction/237535818?
/transaction/237529132?
/transaction/237529131?
/transaction/237529130?
/transaction/237529129?
/transaction/237414323?
/transaction/237247724?
/transaction/237239665?
/transaction/237091749?
/transaction/237066567?
/transaction/237030483?", "Global")
set(#itemTransFullURLs, $replace("/transaction/", #TempURLs, "https://www.domain.com/transaction/"), "Global") -
I love and use windows automation scripts everyday that I developed with Macro Scheduler Pro scripting software. Frank here on the forum had posted a video about it a while back and I've been hooked on it ever since. I actually even use both UbotStudio and MacroScheduler to prompt each other for various tasks. Ubot rules in the browser world and I personally feel MS rules in Windows automation so they work hand in hand perfectly for me.
Macro Scheduler—Automation Software
“Advanced, Easy to Use, Windows Software Automation Tool”https://www.mjtnet.com/macro_scheduler_pro.htm
-Rich
-
I sent Buddy an email just now to check on him to see if he was doing okay and check with him and see if he may have missed his domain renewal. I know it happens to me sometimes, I manage over 400 domains across a ton of registrars for my company and some times an obscure registrar's renewal email slips through the cracks.
I hope that's the case because I love Buddy's videos and reference them all the time when I forget how to do something in Ubot Studio.
-Rich
-
1
-
-
@sktan7,
I'm running Ubot Studio v4.2.20 and it has database support.
-Rich
https://www.dropbox.com/s/rwpkbqt5hxxkxoj/Database-Support-Version4.jpg
-
Well, currently you have to run Ubot Studio v4.xxx to be able to take advantage of Aymen's HTTP POST plugin and several other things related to HTTP POST/Sockets. Also, when you purchase Ubot master Dan's HTTP POST tutorials he state's that you currently cannot do the things he is teaching in his videos with Ubot Studio v5 that you must be using Ubot v4.
So, until Ubot Studio v5 becomes more stable and these awesome advanced plugin features get ported and tested for use with v5 by the plugin developers, my v5 will sit and collect dust. I know it is unfortunate to say this, but I bought Ubot to implement it immediatley into production which I did with v4 after requesting it from Ubot support, I just don't have time to test, troubleshoot and report issues with v5.
Maybe, Ubot developers will give some type of incentive to the Ubot community and advanced users/developers to spend more time testing advanced stuff in Ubot v5 and that will resolve some of the v5 new release issues which in turn will cause the rest of the Ubot community to migrate over to v5. Microsoft does this constantly by giving away for free their .Net and developent language platforms like C#, SQLServer, VB, etc. to foster innovation and improvements.
I do want to say that Ubot Studio is an awesome innovation and this community built around Ubot is awesome. Ubot studio is really helping me at our company replace tedious manual tasks with automation scripts and in turn is improving efficiency and lowering costs. So, overall Ubot Studio is awesome and I truly believe version 5 will eventually overshadow it's predecessor and become a great version.
Oh, and did I mention the word "awesome", lol
-Rich
-
Hey entroqy,
You are welcome.
Btw, Dan has shown in his bonus training video how to use cURL to do HTTP POST along with Ubot Studio Advanced Shell Plugin to accomplish the same thing Aymen's HTTP POST Plugin does plus he mentions that you can leverage more HTTP commands with cURL than the HTTP plugin. But, Aymen has made the HTTP Plugin easy to use with simple dropdown selections where with cURL you need to know what you are doing with the cURL commands and paramenters that you input into the Ubot Studio "shell batch hidden" node.
Here is info on cURL: http://curl.haxx.se/
cURL is a command line tool for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, kerberos...), file transfer resume, proxy tunneling and a busload of other useful tricks
I would highly recommend if you want to do these more advanced techniques that you grab Dan's training, he will shorten your learning curve and troubleshooting headache times by weeks or months.
Also, you will want to get Aymen's "Socket Code Generator" to make your Ubotting life easier if you are going to be doing HTTP POST stuff.
-Rich
-
Hey entroqy,
I'm not Dan, but I'm just learning to use Aymen's HTTP POST Plugin by watching Dan's HTTP Plugin training videos and how he recommends to use it. The short version... the HTTP POST Plugin by Aymen allows you to add the plugin to your Ubot Studio v4 Pro/Dev and it allows you to directly access web servers and the web page data without having to display in the Ubot browser. You can do a direct HTTP request for a specific web page and your script gets all the html data stream directly and your script then can grab whatever you are scraping, but nothing is shown on your screen (no browser window, images, videos, flash, none of all the overhead associated with web browser), all data is access in the background in the Ubot code. So, basically, you could scrape 100's of pages of data in seconds versus 1,2,3 pages data with the web browser rendering everything and slowing down your PC and using up your memory, browser.exe crashes, etc. So, that's a newbie's take on what the plugin can do within your Ubot scripts, plus more.
Btw, Aymen is a very proficient programmer with Ubot Studio (and C#) and has developed many excellent Ubot Studio plugins and scripts that have been very helpful for the Ubot community. Many folks in this community have been requesting that there be some training on how to best utilized the HTTP Plugin developed by Aymen and Sockets with Ubot but, Aymen and others Ubotters have been too busy to really put together any comprehensive training until Dan decided to carry the torch and do it for everyone. Dan is a very strong Ubotter and has done a great job (over and above) creating walk-thru-videos on how to use the HTTP Plugin plus much much more.
-Rich
Just for your convenience here is what you should check out if you want to purchase and learn to use it:
HTTP POST Plugin developed by Aymen:
http://ubot-plugins.com/http-post-plugin/
HTTP Plugin Training by Dan:
-
1
-
-
Hey Dan,
I was holding out for the multithreading video and solution, just purchased now, looks good so far! Your checkout and delivery system worked flawlessly. Will give training feedback later after I review the videos.
Thanks,
-Rich
-
I just purchased! Looks awesome!
-Rich
I wanted to give a bit of a testimonial on UbotBuddy training. But first a quick background on me... for several years I was a corporate trainer teaching CompTIA certification preparatory classes (i.e. A+, Network+, Project+, etc). I had to teach hardware, software, networks, security technologies and project management plus sales presentation skills to different types of personalities at a large corporations. Imagine having a class of salespeople mixed with mid-level managers, IT guys and C-level corporate guys all in the same class room trying to teach them binary math, calculate computer optical/HD/RAID/NAS/SAN storage contraints, computer network protocols, PERT/Critical Path Method and who are stake holders in a project plus a ton of other things in 5 days to prepare them to take and pass a certification exam (I was paid handsomely by Ricoh corporation if you were wondering why someone would actually attempt this, lol).
So, back to UbotBuddy training, because have a background in programming and scripting languages and experience with putting together training material and teaching it to various personality types, I feel compelled to compliment Buddy on the way he approaches his UbotBuddy training. It's comprehensive in that it covers pretty much all the Ubot studio commands/functions discretely but he also teaches you how they interact with each other in a larger script. So many training books and manuals tend to focus just on the discrete explaination of a particular command or function, etc. and don't really give you examples of them in a foudation building aspect. Buddy also makes sure to cover particularly powerful elements of the Ubot scripting language rudundantly (this is important to drive it home, you'll thank him later).
UbotBuddy Training = 5 Stars (if you are new to Ubot, it's invaluable) Even if you have exprience in other scripting/programming languages it's great to ramp you up quick to the nuances of UbotStudio. Buddy's training will shave weeks off of your understanding on Ubot Studio. You can really cram and get his initial 30 day training videos done in a week or so but you need to watch and build scripts simultaneously to retain the training better than just watching and trying to remember. Also, he gives you all the Ubot scripts (over 300) to look at and use.
Excellent training Buddy!
-Rich
-
Awesome! Thanks!
This most recent update has been on my plate for quite a while since the research phase was sooo tedious.
My next step is to provide "How-To" videos from the beginning of a bot to the end for sites like Twitter, Facebook, Google, Craigslist, Fiverr, and others.
Buddy
Hey Buddy,
I love the training you put together, I left a psuedo-testimonial in the post above.
Btw, could you do some training on things like APIs (like LinkedIn API), Sockets, HTTP POST, Hidden browser scraping, browser.exe crash closing, restart Ubot commands/plugins, etc? That would be great in advanced training areas.
Thanks,
-Rich
-
Hey BotGuru,
Do you have any training vids showing this in use or how to use it efficiently? I noticed you had some good YT vids on your other scripts. I'm new at scraping emails from LinkedIn and messaging members. Let me know.
Thanks,
Rich
[SELL] SQLite Database Plugin
in Plugins
Posted · Edited by mobileconsultant
Hi Everyone,
I wanted to confirm the same issues with SQLite and NULL values and DOUBLE QUOTES in your literal strings. I wrestled with this for several hours till I had a breakthrough via trial&error and research on this forum and the web, so I wanted to make this post to save someone else a headache.
PROBLEM #1: SQLite databases and NULL values (you can't do it)
My Solution: In the Ubot code below, I had to do a $comparison when a variable contained nothing/NULL value, I chose to set the value to the string "None" in my case.
if($comparison(#xProdViews,"= Equals",$nothing)) {
then {
set(#xProdViews,"None","Global")
}
I also found out the hard way that when I tried to pass to SQLite literal strings that contained DOUBLE QUOTES via the Ubot plugin command "SQLite Table To DB". The results were the SQLite database would break or reject the transfer completely.
PROBLEM #2: SQLite databases and DOUBLE QUOTES (you can do it, but you have to use two double quotes)
My Solution: You have to replace DOUBLE QUOTES with a set of DOUBLE QUOTES like this ""
What I did in the node view was put a set of DOUBLE QUOTES "" and click the "" icon button in the SET() node.
In the Ubot code below, I'm pulling string data with $xpath parser expression and then $trim whitespaces off the string, then I $replace the html code """ with a set of DOUBLE QUOTES so that SQLite does not reject it and then Ubot is escaping the set of DOUBLE QUOTES with the node view "" icon clicked which are \ escaped characters in the Ubot code view below.
You'll also noticed that I $replace the html code "'" with single quote which oddly the Ubot plugin command "SQLite Table To DB" and SQLite database themselves don't have any problems with single quotes!? But, Ubot code view does escape character the single quote below.
set(#xProdTitle,$replace($replace($trim($plugin function("HTTP post.dll", "$xpath parser", #get, "//meta[@property=\"og:title\"]", "content", "HTML")),""","\"\""),"'","\'"),"Global")
SUMMARY: Watch out and replace those Nothing/NULL values and DOUBLE QUOTES in yuor strings when working with SQLite database and the Ubot plugin command "SQLite Table To DB"
Just wanted to give back to this awesome Ubot community and I truly hope this helps someone out and saves them coding time
Thanks,
-Rich