UBotBuddy 331 Posted April 20, 2010 Report Share Posted April 20, 2010 Has anyone had any luck at scraping Facebook profiles from the friends section? I can scrape the first page but I cannot get it to advance to the next page, scrape and then advance until it is done. I end up having to page through it manually and executing my bot without initializing my list so I don't wipe it out. Any ideas? thanks! Quote Link to post Share on other sites
Net66 54 Posted April 20, 2010 Report Share Posted April 20, 2010 Has anyone had any luck at scraping Facebook profiles from the friends section? I can scrape the first page but I cannot get it to advance to the next page, scrape and then advance until it is done. I end up having to page through it manually and executing my bot without initializing my list so I don't wipe it out. Any ideas? thanks! The attachment should help. You have to be on the first list of friends page. It scrapes the url of each friend and adds them to a friends list. Basically it is checking if the current url has changed or not after it has clicked the advance button. Because facebook keeps the button and its attributes present (it just greys it out) if it clicks it and the url does not change we can assume we were on the last page. You can then do a loop through each url in the friends list and scrape whatever data you require. For the sake of the demo this just runs through, builds the list and writes it out to a text file. Hope this helps Andyscrapefriends.ubot Quote Link to post Share on other sites
UBotBuddy 331 Posted April 21, 2010 Author Report Share Posted April 21, 2010 Andy, Thanks for the code. I'll check it out when I get back to work tomm. Buddy Quote Link to post Share on other sites
UBotBuddy 331 Posted April 21, 2010 Author Report Share Posted April 21, 2010 Andy, I navigated to the Friends section but when I executed the ubot it hit an error. I have attached it. Any ideas? Thanks! Buddy Quote Link to post Share on other sites
Net66 54 Posted April 21, 2010 Report Share Posted April 21, 2010 Andy, I navigated to the Friends section but when I executed the ubot it hit an error. I have attached it. Any ideas? Thanks! BuddyNo idea! What url are you starting from in facebook? Andy Quote Link to post Share on other sites
UBotBuddy 331 Posted April 21, 2010 Author Report Share Posted April 21, 2010 I clicked on my Profile tab and then in the left column under Friends I clicked on "See All" Then I ran the ubot. Quote Link to post Share on other sites
alcr 135 Posted April 21, 2010 Report Share Posted April 21, 2010 Looks like it doesn't got anything to click at? Quote Link to post Share on other sites
UBotBuddy 331 Posted April 22, 2010 Author Report Share Posted April 22, 2010 I see a next page triangle. Quote Link to post Share on other sites
Net66 54 Posted April 22, 2010 Report Share Posted April 22, 2010 I see a next page triangle. Sorry, should have been clearer! Login to Facebook. Account > Edit Friends. Left hand tab under "Lists" click "Friends" This loads the friends list on a main page rather than a pop up window. Run the bot sample. Andy Quote Link to post Share on other sites
UBotBuddy 331 Posted April 22, 2010 Author Report Share Posted April 22, 2010 Still no Workie! I didn't know that screen was there. Anyway, I tried the ubot and it still gives the same error as before. Does a button have to be selected before the bot is run? Quote Link to post Share on other sites
UBotBuddy 331 Posted April 23, 2010 Author Report Share Posted April 23, 2010 Andy, Is this working for you? I followed your instructions but no go. The same error that I reported earlier still pops up. Buddy Quote Link to post Share on other sites
Net66 54 Posted April 24, 2010 Report Share Posted April 24, 2010 Andy, Is this working for you? I followed your instructions but no go. The same error that I reported earlier still pops up. BuddyYes it works for me. I was going to try to find time to see if I could scrape from the pop up window on the view all link but been a bit busy. Facebook is a nightmare to scrape from! Quote Link to post Share on other sites
Net66 54 Posted April 25, 2010 Report Share Posted April 25, 2010 Been a bit busy this week but finally found a few moments to make the example better. Try this one. It should login to facebook, navigate to the correct places and scrape all friends into a file. Andyfacebookscrapefriends.ubot Quote Link to post Share on other sites
UBotBuddy 331 Posted April 26, 2010 Author Report Share Posted April 26, 2010 Andy I really do appreciate your efforts! It still is not working. Makes me wonder if it is my setup. I am running the newest ubot version. I attached the error I am getting. I am glad they fixed the wrapping of the error message. Buddy Quote Link to post Share on other sites
Net66 54 Posted April 26, 2010 Report Share Posted April 26, 2010 Andy I really do appreciate your efforts! It still is not working. Makes me wonder if it is my setup. I am running the newest ubot version. I attached the error I am getting. I am glad they fixed the wrapping of the error message. Buddy I am not using the latest beta because the lack of insert node and a few other things were slowing me up too much so I rolled back to the current stable release - I don't know if this makes any difference. Andy Quote Link to post Share on other sites
UBotBuddy 331 Posted April 26, 2010 Author Report Share Posted April 26, 2010 Thanks Andy. I dont know that I will roll back. I am thinking it might be my IE options that might be killing this. i will tweak some settings and retry. Buddy Quote Link to post Share on other sites
UBotBuddy 331 Posted April 26, 2010 Author Report Share Posted April 26, 2010 I figured out why it was not working for me. The login portion was not working right. So I removed that code and it appeared to work to some degree. While I was watching it, to me, it looked like it was skipping pages of profiles by one. So I added some delays before and after. When the "Click Chose" command executes it actually forces two advancements. So I only get half of the profiles. I am still tweaking it but to me I don't think "Click Chosen" should do that. If I could get it to backup right after the advancement then I could scrape all of the profiles but then it makes sense to correct the code to not have to do that. I will continue playing with it. This is the closet I have gotten so thanks for the code Andy. Buddy Quote Link to post Share on other sites
UBotBuddy 331 Posted May 2, 2010 Author Report Share Posted May 2, 2010 Andy, Sorry to trouble you with this again. The code you provided does work but when the actual scraping is done it is only getting half of the profiles. Upon further checking it looks like as soon as the "Next" page is selected that it is actually advancing two screens instead of one. Very odd. Are you seeing this as well? When you have time that would be great to get your thoughts. Thanks! Buddy Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.