Search the Community
Showing results for tags 'scraping'.
-
I haven't touched Ubot in a few years, so I'm really REALLY rusty. I'm sure there are some things I'm forgetting in scraping in general...but I've reviewed the videos and am having a hard time isolating the data I want. There is a Facebook page for a business that I work with where we're running a contest, and we need to pull all the "likers" from the recent history. I have the URL for this data, but there's no export. Hence the need to create my own scraping solution (as opposed to copying/pasting). I'll be pulling this type of data several times per year, so it's worth it to create a bot.
-
http://cbturbo.s3.amazonaws.com/image/cbturbo1.png Price: $25 - $47 (dimesale) http://cbturbo.s3.amazonaws.com/image/BuyNow.jpg http://cbturbo.s3.amazonaws.com/image/cbturbo3.png http://cbturbo.s3.amazonaws.com/image/BuyNow.jpg Price: $25 - $47 (dimesale) -
Hi, I need help from the Ubot community, I am new to Ubot but have played alot with the software and I must say this stuff is addictive However I'm on my first project and I'm pulling my hair out trying to figure out how to scrape an email address from a page, its quite complicated to explain so I produced a 4min video here it is: I hope someone can help me out here, I'm dying to continue with this project. Thanks -
So I decided to fire up my Ubot Studio and try to learn how to create a scraper as I need one from Cafepress. I'm able to scrape the results I wanted but I'm having some issues with the saved csv file. For some reason it's putting the Title and URL's on separate lines. I'm also wondering how to handle scraping multiple pages (over 100 just for this product type). ui text box("URL To Scrape:", #url) ui stat monitor("Product Titles:", $list total(%titles)) ui stat monitor("Product Thumbnails:", $list total(%thumbnails)) ui stat monitor("Product Images:", $list total(%images)) ui stat moni
-
Hi Guys, Firstly can I apologise for my poor coding terminology that is one of many things I hope to improve through this forum and this project that I am hoping you guys can help me with. So I have done masses of research on this subject and now it’s just time to ask for help!! So I am wanting to scrape Twitter, I have managed to create a .py code that does this for me when the shell is executed, however this is something currently done manually and the results are stored in a excel file and replaced every time the shell is run. What I would like is for the following; To create some c
-
Any ideas why i'm stuck to 50 urls scraped ? clear list(%urls) add list to list(%urls, $scrape attribute(<class="url-link">, "fullhref"), "Delete", "Global") I use this scrolling code and it works fine (it's a page like Pinterest) Last time i tried, i used 10 scrolls loop(#scrollnumber) { run javascript("window.scrollTo(0, document.body.scrollHeight);") wait(10) } It did scroll 10 times and i got lots of images but i just get the first 50 urls in the debugger ? No matter how many times i scroll down ? It refuses to scrape past the first 50 ? -
I have a bot that needs to scrape the seconds (#s), minutes (#m) or hours (#h) that is listed on the first 15 search results. It just won't save the times properly at all-I can't get them separated into separate rows or do a replace to add in commas or anything. I'm very stuck-what am I doing wrong? Here is my code: clear list(%search term) clear list(%times) navigate("https://twitter.com/search-home", "Wait") type text(<id="search-home-input">, "puppy", "Standard") click(<class="button btn primary-btn submit selected search-btn">, "Left Click", "No") wait for element(<titl
-
I'm trying to scrape the href urls from a page, however so far I've been unsuccessful... The page has different peoples names on it, with each of their names containing a link to their profile. This is the code that contains the hreflink (the person name contains the URL that i want to scrape <li class="clickable"> <input type="hidden" id="quick_follow_username_15768" value="MathewPeter"> <input type="hidden" id="quick_follow_name_15768" value="Mathew Peter"> <div class="image-container"> <a href="Mathew Peter"> <img src="http://img.com" alt="Mathew Peter
-
Hello, My name is Ricardo. I am trying to learn how to scrape URLS and are having a hard time trying to figure out how to get a list of URLs off of googles search page. I am looking here trying to find the "class" so I can select all the URLS on the page. I am looking in here: <a href=" " onmousedown="return rwt(this,'','','','8','AFQjCNET0Cgo85zpZkk1bf16E0ddjYO3Xw','','0CFkQtwIwBw','','',event)">Getting Started With <em>Ubot Studio</em> Drag And Drop - YouTube</a> Not working no matter what I try. Any help is greatly appreciated. Thanks in advance.
-
Hi guys, so I made this bot to scrape tumblr users by search keyword. Here's what I did: 1. Enter a keyword on the search box. 2. Scroll down the page 10 times 3. Scrape users Is there a faster way to do this?
-
Hi there, currently building an ebay scraper, but I'm having one problem - for search results, after the initial search results, it comes up with "more items related to" as well as "x items found from eBay international sellers". This can be avoided, it says how many results it found, so I can just scrape them all and then only keep the first *insert however many results it found* - how do I do this though? Is there a way to delete multiple items from a list, from a certain point onwards? Or is there a better way to get around this?
-
Hi guys, so I want to scrape the images in the media grid but scrolling down to the bottom doesn't seem to work. The media grid overlays the timeline. When I used the javascript to scroll to the bottom, it's the timeline that's being loaded instead of the media grid. When I use the mouse to scroll down, that's the only time that the media grid is shown. Any ideas? Here's my code: navigate("https://twitter.com/MariahCarey/media/grid", "Wait") wait for browser event("Everything Loaded", "") loop(5) { run javascript("window.scrollTo(0, document.body.scrollHeight);") wait(3) }
-
I am trying to get my bot to submit images to mobypicture (http://www.mobypicture.com). The bot logs me in. It gets me to the upload page. There I don't know how to interact with the image submission field (see image 01) When I click on the "Add Files" button it launches what seems to be an actual windows browser window (see Image 02) At this point I cannot add nodes to uBot the windows popup window won't allow any other interaction until you deal with it. If i note the window's tittle and add a window command with the window closed--like to tell it insert the path to the image
-
I am trying to build a bot that scrapes the page links and like count from the Facebook search page. For example: https://www.facebook.com/search/str/campus%20crusade/pages-named I have been successful in scraping the links but the likes count is just not working. The bot is missing a large number of the like counts and I just do not know why. The selector correctly tells me that there should be 12 scrapes per page, but when I run the bot I get significantly less. My code below: ui stat monitor("Page Links", %FB Page Links) ui stat monitor("Likes", %likes) clear list(%likes) clear list(%F
-
Today it just occurred to me that this forum lacks an external resources library or reference guide of sorts, for people looking at automation from (maybe) different standpoints. So here is this thread - please feel free to add what you guys read on other places, where you learn stuff, news, etc... This is NOT the place to write extended how to's or explain tips & tricks in detail, per se, but rather just what the links are about, in a few words, followed by the resource link itself. If you want to explain a new technique or have a tip or trick to share, open your own thread in the a -
So I came up with a small work around when scraping gigantic lists to reduce the memory used in my bots and keeps them from crashing. Before this fix the bots would get to about 40k scraped urls and then freeze up and the only way to even shut them down would be to end the process in the task manager. The work around is basically this... once the list reaches 10K it saves the list to a text file. The end result being a bunch of numbered text files like search-1, search-2, search-3 etc. Previously my stat monitor was tracking the total number of urls scraped by exactly that.... setting
-
I tried a few things but can't get it to work. What i want to do is loading a page, then using a regular expression that scrapes a particular strings (let's say phone number for example) and add that to a list. I need to scrape this way, because $scrape page or $scrape attribute is not possible on this particular site. So what i want is the webpage loads, it checks whether a particular string is present anywhere on the page, if yes then scrape that string and add it to a list.
