Scraping Usernames Using Keywords On Twitter

[polavotex 0]

Hello I am new to Ubot and I wanted to How do I scrape for usernames using keywords on twitter? Thank you very much

[BoosterBots 23]

Doing some testing on that site you can see the url structure is as follows:

https://twitter.com/search?f=users&vertical=default&q=baking&src=typd

replace the keyword baking with the keyword you want to search then perform your scraping.

[polavotex 0]

Thank you very it worked for me perfectly

[polavotex 0]

Thanks again you really helped me but the only problem is after I scroll down how do I get the new usernames that pop up

[BoosterBots 23]

Well you would want to use the run javascript to scroll down which would be:

window.scrollTo(0,document.body.scrollHeight);

Then I would recommend using regex to scrape out the usernames and add those to a list. Then just loop it for as many times you want to scroll down.

[polavotex 0]

Thanks for the info it really helped me out alot the only problem now is that I am able to scrape all the followers of the page but I cant separate the followers by keywords in there bio to get only the followers I want.

[BoosterBots 23]

Sounds like you are going to need to build a table and navigate to each profile then scrape the information you are looking for and put it into a table.

 

Username, Bio

 

Then you can analyze the data you are looking for that way. Or, you could build the filter right into the bot it's self. For example, navigate to the profile then scrape the bio and create an if statement which see's if that current users bio contains and of the words you choose. If the keyword exists then save those usernames to a list.

 

Play around with different features and you can do what you are asking fairly easy. The more you dig and play around the more you will learn.

[polavotex 0]

Thanks alot again but I am trying add the if statement but its still not searching for keywords no matter what parameter I use.I really appreciate all the info you gave it really helped me out