Abdul 0 Posted April 20, 2012 Report Share Posted April 20, 2012 i was tryin to scrape usernames of all the followers of a guy everything was great and i was able to scrape the first 20 then i got stuck with the fact that in twitter you need to scroll down the page in order to see the rest of the followers list so is there a way around that i can scrape all the followers usernames? Quote Link to post Share on other sites
JohnB 255 Posted April 20, 2012 Report Share Posted April 20, 2012 Try this: navigate("http://twitter.com/", "Wait")run javascript("window.scroll(0, document.body.scrollHeight);") John Quote Link to post Share on other sites
Abdul 0 Posted April 20, 2012 Author Report Share Posted April 20, 2012 thanks... i'm trying it now Quote Link to post Share on other sites
Legend 181 Posted April 20, 2012 Report Share Posted April 20, 2012 Very nice piece of code John! http://ubotstudio.com/forum/public/style_emoticons/default/smile.gif Quote Link to post Share on other sites
magzmedia 6 Posted April 28, 2012 Report Share Posted April 28, 2012 Hey I like that bit of javascript...cheers John Another way of doing this would be to retrieve the feed from the Twitter API: here's how I do it: clear list(%userids) clear list(%usernames) set(#doctext, $read file("https://api.twitter.com/1/followers/ids.xml?cursor=-1&screen_name=tweetmagnate"), "Global") add list to list(%userids, $find regular expression(#doctext, "(?<=\\<id\\>).*(?=\\<)"), "Delete", "Global") loop(50) { navigate("http://twitter.com/account/redirect_by_id?id={$next list item(%userids)}", "Wait") add item to list(%usernames, $url, "Delete", "Global") load html($text from list(%usernames, "<BR>")) } In the above example I retrieve 5000 user ids of tweetmagnate's followersI then get the username by navigating to twitter. in the above example I'd get 50 usernames I've used similar routines for following and unfollowing. Hope this helps. Cheers Rob i was tryin to scrape usernames of all the followers of a guy everything was great and i was able to scrape the first 20 then i got stuck with the fact that in twitter you need to scroll down the page in order to see the rest of the followers list so is there a way around that i can scrape all the followers usernames? 1 Quote Link to post Share on other sites
Legend 181 Posted April 28, 2012 Report Share Posted April 28, 2012 That's very nice too, magzmedia... thanks for sharing! http://ubotstudio.com/forum/public/style_emoticons/default/rolleyes.gif Quote Link to post Share on other sites
bitrot 0 Posted January 15, 2014 Report Share Posted January 15, 2014 This is great magzmedia, but unfortunately it no longer works now that Twitter has updated their API. Do you have a solution for the new API? Thanks! Hey I like that bit of javascript...cheers JohnAnother way of doing this would be to retrieve the feed from the Twitter API: Quote Link to post Share on other sites
zenos 13 Posted January 24, 2015 Report Share Posted January 24, 2015 I would like to know if someone have another solution like magzmedia with the new API ? Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.