charliefinale
-
Content Count
45 -
Joined
-
Last visited
-
Days Won
4
Posts posted by charliefinale
-
-
I have tried javascript on on the external browser (in V6 not to be confused with the exbrowser plugin) and I do not think javascript executes there. Has anyone made that work?
-
This will solve half your problem. However if you use dropdowns, they will not work anymore. I have spent 5 days on that too, and concluded that only a Ubot update will fix it. The problem is not that they are VMs, it is that they have the latest dot net 4.8 updates, which breaks Ubot exe files.
-
Hey Secretus you have just made a thundering observation there. I have been battling with the dropdown issue, see other posts, which I am sure is connected to the dot net issues and C+. I created a tiny compiled Ubot dropdown and installed it on a brand new Windows Pro Virtual machine. It did not run at all, it hung. I then installed the C++ x86 disto, and it runs. It still has the bug mind you, but it runs.
Does this mean that despite the "downloading support files" part of a client installation of an exe file, it STILL WILL NOT WORK unless we ask the client to do this rather technical step too???!!!! What?!!
-
There is a Windows update for 4.8
https://support.microsoft.com/en-gb/help/4576947/kb4576947-cumulative-update-for-net-framework
My problems started after this update on all Versions from 6.0.1 up to 6.2.7. Do you have this Update installed? The other more worrying thing you might like to try, is whether your apps still run on a new Windows Home install (maybe from the cloud to test). None of mine that use dropdowns do anymore. In other words, if your clients do not have the right Windows setup, maybe an new one like this, your app will fail. It has nothing t do with whether they run on your machine, so it appears. Feel free to make a test on this machine of mine using Remote Desktop. It has had nothing installed on it, except a tiny complied Ubot dropdown, so it has the support files installed. It does not run. Does it work for any of your apps?
IP Address 72.18.198.141
User Administrator
pw Z?SU#H102tgX.C2TD
Until we know the answer, I cannot recommend anyone sends a compiled bot to a client.
-
As you know Bud, I have installed Ubot from scratch on a completely new clean Windows machine, sent you the video, and it replicates the bug. When you sent me your setup, I noted you had not updated your Windows since August. There is a dot net 3.5 and 4.8 Windows update on the 9th of September which co-insides with my problem. You might like to ensure that your system is up to date too. Or you could do as I suggested and test on a completed new totally up to date Windows Home or Pro and prove this matter one way or another. By doing this almost everything you have suggested to me about uninstalling previous versions etc is irrelevant. Also, if you want to be 100% sure that new customers with new versions of Windows won't be disappointed, testing on a virgin Windows setup is the only way.
-
Great, could you tell me what you think might be the reason that I still have the dropdown problem? What version of Windows do you have and is it completely up to date? My problem seems to have stated after a Windows update on the 9th of September where thy played around with dot net.
-
I have 6.2.7...I am battling with the dropdown issue on top of this, Can anyone confirm that the dropdown issue is unresolved, Ubot tech support tell me I am the ONLY person who has brought this to their attention. I was fine with v 6.0.1 . Has anyone got dropdowns working again????
-
It seems this bug is either back or never went away. I am in touch with support who have said that I appear to be the only one who has complained. So I am looking for others who can tell them this is not the case as it seems they do not seem to think it exists.
My story is this, I had 6.0.1 and was working on a client project 3 weeks ago when all of a sudden for absolutely no reason the dropdowns, checkboxes and buttons stopped working properly. I was told to pay for upgrades which I did, and I am now on 6.2.7. It is still there. My guess since they do not acknowledge the bug is that it effects machines that have the latest Windows 10 updates, that explaining why the bug manifested without any tampering with Ubot itself. I have tested Windows 10 home and Pro. Anyway, here is a screenshot taken from a brand-new machine install, a virtual machine that has never run anything else on it. The issue is not simply cosmetic, dropdowns separate from Ubot being left behind on top of any other program you may be running, checkboxes do not stay checked anymore, and buttons do not fire after one use. It manifests in compiled bots too.
Two things you guys out there can do to help.
1. Replicate the problem and report it here or to Ubot directly so we can let them know this is not isolated. If you have a Virtual Server account, spin up one as I did and try a virgin install.
2. Tell me if you have found a work around. My guess is it has something to do with dot net, but I have spent days on this to no avail.
You can test for the bug with this simple line of code
ui drop down("Label Again","test1,test2,test3",#test4)
If the dropdown appears next to the label, as opposed to below it, , you have the bug. Move the Ubot Window away from view, and the dropdown stays on top of Windows. See my other jpeg.
-
1
-
-
How did you get to add an extension in Ubot? I need to do the same thing.
-
The image referred to above
-
Very good reply, very helpful.
I am having a problem that may or may not be connected. I have tried to use the compile installer option available in the developer edition but have never got it to work. I was fairly amazed to see that that file is about 130 megs when the exe file is only 15 meg. So now after reading this I assume that it is downloading the 100 meg + of files that you mention here. What I did not appreciate is that if I send the non-installer exe file to a client, the first time he runs it, it will be nothing like the experience I have when I run it. Glad I know that, so I can tell him to expect it.
At least the compiler installer will respond more as expected....but I cannot get it to work. It takes about 20 mins to compile and it opens up a cmd window. But when it is run, it says it cannot find a gazillion files it needs.
Now knowing what I do, thanks to you, I really want to fix this. The first odd thing is that when I run the installer, it is files in the Ubot directory it cannot find. See image. Why on earth is it looking for these files there at all, it has after all just downloaded 100 meg of files!! The client will not have Ubot installed.
EDIT 10 mins later....hard to believe, but I needed to close Ubot studio in order to install a ubot bot complied with Ubot Studio as an installer. Now I really have no idea what the client without Ubot Studio installed is going to experience. Any ideas? Why does it not find the files it needs in its own 100 meg of files?
-
All OK. Downloaded another copy (same version I might say) from support site (for updates) and repaired the install and all is OK. Since I had the same issue on 2 PCs it makes me think that the version I downloaded was not the same one, even though it was also labelled 6.04. Anyway, for anyone else who thinks their US is playing up, this is what I did. I also wanted to get updates, but the payment gateway is down for PayPal and for CC payments which was a bit worrying.
-
This setup (see attached image) , with no code at all, just 2 tabs and two text boxes, takes 25 seconds to move form one tab to the other! I have just removed all plugins too, no different.
-
I have always assumed that once a bot was compiled the the time lag moving from one menu tab to another would go. But it doesn't. I have a very basic UI in each tab, one text field in one, a text box in another and moving from tab to tab takes over 15 seconds..and the another 15 seconds to go back. Is this normal? It is the first time I have used a UI in tabs for a client project, but I could not possibly deliver an app this slow. The app exe file also takes ages to load for the first time. It seems that rendering the UI is all too much. I have tried both the UI commands and the UI editor.
The delay is even worse switching from tab to tab and also from code view and back when I am in Ubot Studio. I had mainly given up using the UI for this reason, but now I have to. Does anyone find the same and is there a solution. Very worried the client is going to give me a bad review. I was sure I had seen apps with menu tabs on the market made with Ubot...was I mistaken?
I am V6.04 Developer version.
EDIT. Just watched this video. https://www.youtube.com/watch?v=BQPAl-Ppxug
One thing that strikes me is that when this tutor adds a text box to the UI, it appears almost immediately. When I do the same it takes 15 seconds to appear. It had not occurred to me until today that I may be experiencing something wildly wrong.
-
Yes, we will be finding work for Ubot Studio Developers. We will not take a cut over and above a processing fee if you bring a client to the Dataharvest platform and moreover you the developer can decide what the client is charged for your services.
If however we commission a bot to be developed for a client of ours, who came to us through our advertising for instance, we will take a cut, and that will depend from project to project.
There are also public bots….bots that our users can use based on their subscription, but were not made specifically for them.
Developers with active bots for public use on the site do not pay for a developer license, it is free. And they receive a commission when it is used. Those who do not, and simply perhaps want to use the system for its features, scheduling, IP rotation , Ubot Code generator, for instance, will get 2 months free, and then their account will be downgraded to free account which does not have the same developer features, or they can opt to pay for the developer account. Clearly it would make sense to developer to create a bot for commercial use and then get the account for free.
This is a slight departure from the original proposition, but a much better one for developers. In essence developer accounts are free and certain user accounts are also free. Those who signed up in the initial offering, and are working with me, have had their accounts extended to free permanently. We will also be publishing in the next few days very specific areas of interest from our clients but as food for thought they are
1. Journalism. Scraping news stories from all major news outlets.
2. Gambling (scraping odds, results, comparing bookies, and AI data for predictions)
3. Recruitment. Scraping job postings from Craigslist, LinkedIn etc.
Please note that currently the site is closed for new registrants, as we said it would be, as we document and publish the deal, continue its development, and I appreciate everyone's patience on this. Please watch this space in the next few days.
-
Licensed plugins must be supplied within a compiled Ubot exe file and therefore run on a dedicated server - so the developer must have a license to compile it. Currently we do not support any licensed plugins that are written in code only and uploaded to the servers. That way the licencor is protected.
We have 2 proxy suppliers and I would welcome anyone's opinion as to whether they are "good" . One is Luminati which can rotate IPs automatically each time a bot is run using their server app which we install on our server. This is probably our preferred method, that way our underlying IP is not blocked. The other s a more traditional way of doing it. Ezproxies. No username or password is required with either, the first one "just works" whether you set it up or not, the other is called in the change proxy command built into Ubot Studio. Since the Dataharvest load is spread over many servers, we can direct you traffic to a server that is reconfigured to your needs. We supply the list of IPs.
There is also no reason why you cannot bring your own proxies.
-
Thanks ATuringtest.
To answer you and Kreatus.
If I understand the question correctly , then yes this is SaaS hosting for bots. Dataharvest can run either Ubot exe files, or code that is fed into a Ubot exe, where the input that would normally be in the Ubot UI is provided by arguments. So in the background mybot.exe charliefinale@dataharvest.io mypassword123 https:/facebook.com could for instance login me into my facebook account. There are more secure ways of doing that, but that is the principle.
In Dataharvest we set up the input in a user interface, but that is how it is fed into the exe file.
Furthermore there are certain variables that the Ubot code uses in Dataharvest to know where to save the results. The main one is #filepath which means the personal directory of the logged in user. Now, since a user can also upload to this directory as well as download from it, we can upload a cookie here, or a list of FB accounts, or URLs and read them into the code using for instance.
set(#fb_list,$list from text($read file("{#filepath}/fblist.txt"),","),"Global")
Which would mean look in the personal directory of the logged in user and read in their CSV list from there. The code generator sets the #filepath as a local drive so the testing of such a piece of code can be tested locally. But if we were to write code that digresses from the code generator way of doing things, which I am hoping accomplished Ubot Coders will do, the trick is to substitute #filepath with any reference to your local drive path.
If someone was wanting to run only their own compiled exe files, feeding in arguments in the way I have described, then we can do that too, but it requires a dedicated server as we will never know what code is being executed, and therefore we need to isolate that for security reasons. We have that already in place, I can spin up a dedicated server in a few minutes and assign it to a developer account . It also 100% protects the developer code, which may be something that an FB guru such as yourself Kreatus, or any Ubot Developer whose IP goes beyond a simple scraper, might welcome.
-
Subscribe to the growing number of free tutorials for Ubot Studio from Dataharvest here.
https://www.youtube.com/channel/UC5iLmxpBMpqaw_HLe6rUKtA
Feel free to ask questions on these or any other topics. What does and does not work for you? We can probably create tutorials to cover those issues too.
-
Part 2 of scraping Craig's List.
In this video we look out how selecting an element in Ubot Studio does not always give the results we want; And how to limit the Xpath of a result by using the Xpath prefix to limit the scope of the Xpath. The result can be condensed to an Xpath and 2 rexeges which we then feed into the Dataharvest Code Generator to create a csv of the list we will then use in the UI of the webscaper.
The complex Xpath is this
//div[@id=\'center\']//*[contains(@href, \'search\')]
And the 2 Rexes are these
(?<=span class=\"txt\">)(.*?)(?=<)
(?<=href=\")(.*?)(?=\")
All of these can be used in Ubot Studio irrespective of whether you have signed up for Dataharvest or not.
Here is how it was done.
-
1
-
-
Mmm the attachment did not "take". So here is the code.
clear all data
set(#save_dir,"C:\\Users\\charl\\OneDrive\\bots\\craigs","Global")
set(#progress,"{#save_dir}/progress.htm","Global")
set(#filepath,#save_dir,"Global")
set(#botname,"filename_{$rand(100,999)}","Global")
set(#resultsname,"results.csv","Global")
add list to list(%form_arguments,$list from text("1 https://www.craigslist.org/about/sites"," "),"Don\'t Delete","Global")
define DH_save_to_file(#filename, #path, #content_to_save) {
if($comparison(#path,"!= Does not equal","")) {
then {
set(#slash,"/","Global")
create folder(#filepath,#path)
set(#pathslash,"/{#path}/","Local")
save to file("{#filepath}{#pathslash}{#filename}",#content_to_save)
}
else {
set(#slash,"/","Global")
save to file("{#filepath}{#slash}{#filename}",#content_to_save)
}
}
}
define DH_progress(#pages, #page_counter) {
set(#progress text,"\{\"pages\":{#pages},\"completed\":{#page_counter},\"timestamp\":\"{$now}\"\}","Global")
save to file(#progress,#progress text)
}
define DH_append_to_logs(#logfilename, #log_entry) {
set(#slash,"/","Global")
set(#logfilename,"","Global")
set(#log_entry,"","Global")
set(#logs,"logs","Global")
create folder("{#filepath}{#slash}",#logs)
append to file("{#filepath}{#slash}{#logs}{#slash}{#logfilename}","{#log_entry}{$new line}","End")
}
define DH_append_results_with_botnamepath(#content_to_append) {
set(#slash,"/","Global")
set(#content_to_append,"","Global")
create folder("{#filepath}{#slash}",#botname)
append to file("{#filepath}{#slash}{#botname}{#slash}{#resultsname}","{#content_to_append}{$new line}","End")
}
comment("---------------------------------Nothing required above this line when pasting to Dataharvest-----------------------------")
set(#pages,$list item(%form_arguments,0),"Global")
set(#Url,$list item(%form_arguments,1),"Global")
DH_append_to_logs("log.log", "{$now} - The Probot has started")
set(#page_counter,0,"Global")
navigate(#Url,"Wait")
wait for browser event("DOM Ready","")
loop(#pages) {
wait for browser event("Page Loaded","")
add list to list(%URL_name,$plugin function("XpathPlugin.dll", "$Generic Xpath Parser", $document text, "//li/a[1]", "innertext", "False"),"Don\'t Delete","Global")
add list to list(%URL_link,$plugin function("XpathPlugin.dll", "$Generic Xpath Parser", $document text, "//li/a[1]", "href", "False"),"Don\'t Delete","Global")
add list to table as column(&resultsTable,0,0,%URL_name)
add list to table as column(&resultsTable,0,1,%URL_link)
increment(#page_counter)
DH_progress(#pages, #page_counter)
DH_append_results_with_botnamepath(&resultsTable)
clear list(%URL_name)
clear list(%URL_link)
clear table(&resultsTable)
DH_append_to_logs("log.log", "{$now} - The bot has completed {#page_counter} pages")
if($comparison(#page_counter,"< Less than",#pages)) {
then {
click(<next>,"Left Click","No")
wait for browser event("DOM Ready","")
}
else {
DH_append_to_logs("log.log", "{$now} - The bot has done its job and is now shutting down")
close page
}
}
} -
Thanks. TJ. Appreciated.
In the belief that the more I help people on their way, and the more I assist people build profitable businesses, with or without Dataharvest, the more people will see this as a valuable trustworthy proposition.
So, here is a tutorial on how to scrape all 714 URLs of Craigs List in under 4 minutes (30 seconds without the tutoring) using the Dataharvest Code Generator. The resultant bot is attached.
I chose Craig's List because I know of recruitment agencies out there that want this. In part 2 I will feed this list into another bot. And in the third I will show how to create a UI for it that works in the cloud and then how to publish it. Hopefully they will all be under 5 minutes.
We will advertise these bots to the recruitment agencies for use. I believe there are about 20 or more slightly different commercial bots that can be created around this tutorial alone, so if any of you developers here want to be one of those who publish on Dataharvest, now is a good time to jump in. It’s 2 months free, with a guarantee that the early bird price never rises above 25% of the advertised developer price. It is pretty much fixed at $14.90 for life.
-
My name is Charlie and I have spent the last 15 years in sales and marketing, originally running a company that was a forerunner to WhatsApp. In my travels I have come across literally hundreds of companies that are looking for specific data from the net that can only be done by scraping. I believe that every company in the world needs some form of web-scraping data to assist them in their marketing strategies. What Dataharvest will do is find them for you, the developer. We will find those customers and offer them to you.
To allay one fear, the code belongs to you. It is not shared. I appreciate your concerns, but we are not looking for you to give up your intellectual property, but to become part of a community of developers to solve the data issues that confront our customers. So yes, you may publish on Dataharvest a scraper that let's say is focused on the house rental market, or Craigs list. Whilst this code will not necessarily be particularly different from any other, it is the marketing data that will attract users to us, and who will say, hey I want something similar to that, can you write it for us? And we will hand that back to you the developer to create at a price that you can determine. It is a two way street.
So what is in it for you as the developer? The opportunity to allow us to find customers for you. If it is your basic bot that an end user saw on Dataharvest and wants adapting to their specific needs, you get the business. As such even a simple scraper of Amazon will get your foot in the door. Or Airbnb. or Ebay. It will not be the $5000 code that will make this work. It will be the 20-30 lines that strikes a chord in the marketing director of ABC Inc.
TJ, I am a great fan of yours, and have been a member of your facebook page for some time. But you must agree surely that there are problems with the current business model for selling products created with Ubot Studio,. Selling our work as an executable file is old fashioned. It is not what our customers want.
Saas is a much better way of doing it. I cannot do it on my own. However similar each of the scrapers that may appear on Dataharvest are, I cannot write 1000 of them myself. Nor really as a developer do you have to create something so valuable that you would not trust it on our servers. Because even the simplest scrapers will attract customers to us, and then you the developer get paid for the needs of a customer who wants custom code.
Craigs list is a great example. There are 50 or more different websites out there, and someone looking for a job somewhere in America has to run a search on all 50 of them to get the complete picture. As a developer you could write that search bot and get paid $2 every time someone runs a job search. That could be a thousand times a month. I do not believe that can be sold as a downloadable exe file. That model does not work anymore.
So TJ, I have great respect for you. Your reputation and opinion will make a difference to how well Dataharvest is received. But do not be negative about it yet. please. This is a good idea for this community. I understand that trust is earned, so give me the opportunity to earn it. My terms and conditions are a work in progress and it is very easy for me to give you the developers as much protection as you deem necessary, but at the end of the day, I, like you, am a person, a Ubot developer whose reputation should be questioned, so that I am given the opportunity to earn your trust.
Dataharvest is everything that Seth Turin sells as the opportunity to create a business from his product. It is 100% loyal to his philosophy. The only difference here, is that everyone can include themselves in the Dataharvest vision, which is designed to create profitable entrepreneurs out of us all.
So please TJ, give it the thumbs up until such time as I have done anything that may cause an inkling of doubt. We can also chat on FB, I am member there and we have spoken several times before.
-
2
-
-
The platform is utterly amazing. If you are looking to get a return on your Ubot Studio investment, this is a good way to do it. There is a shorter video here. The tutorial videos will grow by the day so watch this space. It runs Ubot code without even compiling it.
-
You can find a reasonably comprehensive how to video here. We need your help to know what can bring you up to speed best.
Ui Dropdown - Not Working
in General Discussion
Posted
So basically you are saying that you may not be able to replicate this bug because you have not been subjected to the Windows updates that the rest of the world have? So, finally that makes sense. The problem is that many of these updates are irreversible, according to Microsoft, so what do we do?