Jump to content



Photo

[SELL]Scrape And Save Websites


  • Please log in to reply
34 replies to this topic

#1 illmill

illmill

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 458 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v3.5
  • License:Developer Edition

Posted 11 December 2013 - 08:26 PM

Introducing: Site Ninja

 

 

What it Does:

 

  • Scrapes All HTML Files
  • Downloads Images
  • Saves CSS and JS Files
  • Creates the File Paths on Your Local Machine
  • Saves the Entire Site on Your Computer

A few really cool uses for this:

 

  • Create killer UI's for your bots.  Like the UI of Site Ninja?  I used it to create it's own UI.
  • Improve your HTML5 and CSS3 skills by looking at what the professionals are doing.
  • Create awesome landing pages quickly and for FREE.

What it does not do:

  • It does not scrape and save wordpress themes or anything for any other CMS.
  • It does not save 100% flash websites.
  • It does not make you coffee or breakfast in the morning.

Frequently Asked Questions:

 

  • What about support?  Yes, I provide support on this.  If it's not working on a site you want to scrape, then I will take a look and see if it's possible to make a change to the software so it will work.
  • Are updates included?  Yes, updates are included.  What's nice about creating a bot that does this instead of the typical bot that does stuff on social networks is that the websites aren't updated all the time to make botting harder, so frequent updates shouldn't be necessary and it should always just work.
  • Money back policy?  Unfortunately because this scrapes and saves websites quick (only a few minutes per site) someone could easily just scrape all the sites they wanted and then ask for a refund.  Therefore, there's no money-back policy.  It works as advertised.

 

Only $27!

download.jpg



#2 the_way

the_way

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 557 posts
  • OS:Windows 8
  • Total Memory:3Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 11 December 2013 - 08:38 PM

does it work on wix sites?  thanks.


2VHxTj5.png


#3 illmill

illmill

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 458 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v3.5
  • License:Developer Edition

Posted 11 December 2013 - 08:46 PM

does it work on wix sites?  thanks.

As in http://www.wix.com/?  I don't know.  I haven't tried, but I will go and try now.

 

EDIT: I just tried and the short answer is: no, not really.  Wix stores all the images, css and js at wix.com/longFilePath (rather than just img/mypicture.jpg).  This wouldn't be much of a problem, but they are using javascript to pull all the text.  So this bot will save an index.html file that reflects the exact wix site and you could change the images, but not the text because it's being called with their javascript (or at least that's what I can tell from my quick glance).

 

Edit2: I might update it later where it will save all js and other files even if it's path is domain.com/longpath.... and then on your local machine it creates new paths and changes the html in your index so the new path is there and everything magically works, but when I do that the price will definitely go up.



#4 the_way

the_way

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 557 posts
  • OS:Windows 8
  • Total Memory:3Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 11 December 2013 - 10:32 PM

There is a large market for people wanting a tool to pull exact copies of wix sites, as their interface for designing is fast and easy to use, and very good for landing pages especially. Thats why they have their setup that way.

 

you would be better of creating a dedicated tool called wix scraper though.

 

Best of luck with sales anyway.


2VHxTj5.png


#5 illmill

illmill

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 458 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v3.5
  • License:Developer Edition

Posted 12 December 2013 - 12:29 AM

There is a large market for people wanting a tool to pull exact copies of wix sites, as their interface for designing is fast and easy to use, and very good for landing pages especially. Thats why they have their setup that way.

 

you would be better of creating a dedicated tool called wix scraper though.

 

Best of luck with sales anyway.

I'll keep this in mind.  Thanks!



#6 luis carlos

luis carlos

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 486 posts
  • OS:Windows 8
  • Total Memory:< 1Gb
  • Framework:v3.5
  • License:Developer Edition

Posted 12 December 2013 - 12:39 AM

Pretty nice bot.  When do you think the "wix update" would be ready? 



#7 illmill

illmill

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 458 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v3.5
  • License:Developer Edition

Posted 12 December 2013 - 01:50 AM

Pretty nice bot.  When do you think the "wix update" would be ready? 

I don't know.  I hadn't even considered this until quite_interesting's post.  I will take a look at wix when I have time (possibly over the weekend) and see how feasible it is to incorporate this.  

 

What exactly would you want a tool to do with wix?  Would you need to be able to edit the html in a text editor, or would it simply be so you can upload an exact copy of a wix page you already created to your own server?



#8 luis carlos

luis carlos

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 486 posts
  • OS:Windows 8
  • Total Memory:< 1Gb
  • Framework:v3.5
  • License:Developer Edition

Posted 12 December 2013 - 09:54 AM

Well, i have never done anything using wix, BUT my little doughter does, and she loves it (more than 20 sites about her favorite things,best friends, music and more). Guess she only wants to download, and upload it, since editing is easy in the wix site.

I don't know.  I hadn't even considered this until quite_interesting's post.  I will take a look at wix when I have time (possibly over the weekend) and see how feasible it is to incorporate this.  
 
What exactly would you want a tool to do with wix?  Would you need to be able to edit the html in a text editor, or would it simply be so you can upload an exact copy of a wix page you already created to your own server?



#9 Ptrick125

Ptrick125

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 285 posts
  • LocationNear Austin, Texas
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Professional Edition

Posted 12 December 2013 - 12:15 PM

I love it! It can help you lower server stress by turning dynamic pages into static pages! :)



#10 Sirfrank

Sirfrank

    Advanced Member

  • Members
  • PipPipPip
  • 109 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 12 December 2013 - 01:57 PM

hehe , 1000000x use for it, putting on my wishlist. (means i am buying as soon as i have a free day :))



#11 Sirfrank

Sirfrank

    Advanced Member

  • Members
  • PipPipPip
  • 109 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 12 December 2013 - 01:58 PM

saves entire site.... should i try scraping 10.000s page sites as well?



#12 illmill

illmill

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 458 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v3.5
  • License:Developer Edition

Posted 12 December 2013 - 08:03 PM

Well, i have never done anything using wix, BUT my little doughter does, and she loves it (more than 20 sites about her favorite things,best friends, music and more). Guess she only wants to download, and upload it, since editing is easy in the wix site.

If that's the case, then you can do that now with the software.  Every time you make changes on the wix site, just scrape it again and re-upload to your server..  It will save an identical copy of your wix site as one file - index.html.  You just can't edit it, you'll need to do that inside Wix.



#13 illmill

illmill

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 458 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v3.5
  • License:Developer Edition

Posted 12 December 2013 - 08:10 PM

saves entire site.... should i try scraping 10.000s page sites as well?

It probably won't work on a site with thousands of pages, but most of those are content management systems so it wouldn't work anyway.  It can definitely get a lot of pages, though, especially if there are links to them on the index page and the href tags are structured something like this: href="stuff/morestuff/mypage.html".  This is how it looks for pages because this is the common "best practices" for building static websites.  Otherwise, it would save all pages that it's linking to, even if it's on another domain, and it just wouldn't work very well.  This might change in the future, but that would be a much more complicated process and I would have to raise the price at that point.



#14 luis carlos

luis carlos

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 486 posts
  • OS:Windows 8
  • Total Memory:< 1Gb
  • Framework:v3.5
  • License:Developer Edition

Posted 12 December 2013 - 08:29 PM

If that's the case, then you can do that now with the software.  Every time you make changes on the wix site, just scrape it again and re-upload to your server..  It will save an identical copy of your wix site as one file - index.html.  You just can't edit it, you'll need to do that inside Wix.

Actually that is possible from the wix site, but if wix deletes the site (just imagine!),  It would be great if I can save the whole page in a new host, or even in my pc.



#15 XpownX

XpownX

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 196 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 13 December 2013 - 08:09 AM

Actually that is possible from the wix site, but if wix deletes the site (just imagine!),  It would be great if I can save the whole page in a new host, or even in my pc.

 

Does your bot scrape too .php files ?



#16 Sirfrank

Sirfrank

    Advanced Member

  • Members
  • PipPipPip
  • 109 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 13 December 2013 - 07:30 PM

Bought it :)

When scraping non English sites, the html-s are messed up. Characters shown for example “é” instead of “é” in the html file.

Any chance to correct that?



#17 illmill

illmill

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 458 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v3.5
  • License:Developer Edition

Posted 14 December 2013 - 10:24 PM

Does your bot scrape too .php files ?

No.  That's not really possible, as far as I know, especially if the php file redirects visitors.  Maybe it's possible to download php files?  I don't know, I haven't looked into this.



#18 illmill

illmill

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 458 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v3.5
  • License:Developer Edition

Posted 14 December 2013 - 10:26 PM

Bought it :)

When scraping non English sites, the html-s are messed up. Characters shown for example “é” instead of “é” in the html file.

Any chance to correct that?

Cool, thanks for your purchase!  Please send me a PM with the site you're scraping and I will take a look at it.  



#19 drb

drb

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 74 posts
  • OS:Windows 10
  • Total Memory:More Than 9Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 20 December 2013 - 03:42 AM

"Create killer UI's for your bots.  Like the UI of Site Ninja?  I used it to create it's own UI."

 

How do you do this?



#20 illmill

illmill

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 458 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v3.5
  • License:Developer Edition

Posted 20 December 2013 - 09:40 PM

"Create killer UI's for your bots.  Like the UI of Site Ninja?  I used it to create it's own UI."

 

How do you do this?

Well, while I was working on Site Ninja inside the UBS editor I browsed http://themeforest.n...g/landing-pages and found the theme you see in this UI.  I ran the URL through the bot and saved the theme to my local computer.  Then I did a bit of editing (really, I just deleted all the other stuff below the header and moved their search text box into the header area) and bam, I had a nice lookin UI.






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users