[Script] Scrape And Download All Images From A Tumblr Profile Using Curl

[whoami 26]

Hello everyone, I just did an enter in a blog explaining how you can download all images from a profile and doing pagination with a line of bash code.

This can be executed in any OS, also on Windows if you set up the curl module.

 

So what it is going to do, is using curl to mass scrape and paginate a tumblr profile, so you will get a list of URLs that will be processed with a while loop inside the curl, and then saving it on the folder you are running this command. But first…

You might need to install cURL in your server, dont worry is easy:

 

1

sudo

apt

-

get

install

curl

 

And if this doesnt works, try doing an apt-get update and then:

 

1

sudo

apt

-

get

install

libcurl3

php5

-

curl

 

Then just like that, type: curl -h  to see the help, if it display now you have it installed.

This is the main command you have to type inside the folder where you want to put all the downloaded images:

curl http://concept-art.tumblr.com/page/[1-7] | grep -o 'src="[^"]*.[png-jpg]"' | cut -d\" -f2 | while read l; do curl "$l" -o "${l##*/}"; done

Lets explain the code a little bit: the curl command is requesting the URL http://concept-art.tumblr.com/page/1starting with page 1, then when we put in [ ] and separate a bigger number at the right with a dash – , we will be able to request multiple pages doing a pagination. Then when you add  | means that it is separating the commands to run after, so grep is going to do a search for src attributes and only for those with termination on [png-jpg] or you can add more doing -gifinside the [ ]. Then the cut  command will collect the names, but what it is going to make download eveything will be the while loop, doing a curl on the found urls for the png or jpg images.

So make sure to add the while loop to the end for downloading, if you only want to see the images grabbed using the following command:

curl http://concept-art.tumblr.com/page/[1-7] | grep -o 'src="[^"]*.[png-jpg]"' | cut -d\" -f2

Here is a video that I recorded in order to show you how to do it:

 

If you want to read more like this post, visit the blog:

http://i.imgur.com/UUGHQQt.jpg

 

 

You can support by buying the courses and source code, I will continue posting tips.