Jump to content
UBot Underground

[Sell] Website Crawler - Crawl Websites And Extract That Succulent Data


Recommended Posts

  • Replies 69
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Plugins Required: HTTP Post (Paid) / Advanced File (Free) / File Management (Free) / Threads Counter (Free) / Large Data (Free) / Local Dictionary (Free)   Made and tested in Ubot Studio 4   This Is S

Just wanted to pop in here and give Nick some love with this software.   First off Nick creates really good useful tools. Second, his support is fantastic!   This is a tool with a LOT of different use

Update:   - Fixed: major bug that would close the program to crash when a PDF link (without the .pdf extension) was called - Default script to scrape external domains has been modified now to have gen

I am trying to scrape php files from any website because I am studying php file source codes as examples thats it no for other purposes.

 

Well you can't see the source code of PHP files because they are served server side. Meaning, the end user can only see the output and not the source code. I suggest you take a look at places like Github where code is shared for free you can study PHP files there.

Link to post
Share on other sites

I compiled the code as is and when I try to run it with the exact settings in OP video, nothing happens.

 

Are there settings I need to initialize that I did not see?

 

Thanks

Link to post
Share on other sites

I compiled the code as is and when I try to run it with the exact settings in OP video, nothing happens.

 

Are there settings I need to initialize that I did not see?

 

Thanks

 

I'll send you a PM and we can try to figure it out!

Link to post
Share on other sites

Well it took forever and a day but I think the memory climbing issue is finally sorted: http://imgur.com/a/GCnkm

 

It was going between 90 and 120 mb during that test run, it just happened to increment for each photo but after each one it was down at 90 again at one point or more.

 

I am going to release 1.3 soon after one or two kinks are worked out. Because I know this update has taken forever but I've probably made 15 different versions of this and spent days of testing time.

 

After that version 2 will come out and it will make the code a lot cleaner and hopefully serve as a very strong foundation that can be built upon. Hopefully it will be able to crawl basically indefinitely.

  • Like 1
Link to post
Share on other sites

Great news Nick.

 

I do find it a little difficult at the moment to really pinpoint what code I should change for my own use.

Can you record a video or simple instructions to show an example of how someone would take the code and change it for their own use case. Just something simple like:

  • Here is the base code...
  • Now let change it for this scenario ...
    • Crawl site ABC looking for pages containing ZYX.
    • Save the URLS and scrape ZYX from the page.

 

Thanks,

Pete

  • Like 1
Link to post
Share on other sites

Update 1.3

 

To download the update login at: http://imautobots.com/wp-login.php

 

Then go back to the homepage and click on "Purchase History"

 

Also, there is a video showing how to make changes in 1.3 to customize what you scrape.

 

V 2.0 coming soon :D

  • Like 1
Link to post
Share on other sites

Update 1.4

 

- Lots of bug fixes

- 6 hour stress test ran with no errors and decent memory management

 

To download the update login at: http://imautobots.com/wp-login.php

 

Then go back to the homepage and click on "Purchase History"

  • Like 1
Link to post
Share on other sites

You found a bug in 1.4 Nick? Funny because it's running better than ever on my machine.

 

Yes when using multiple urls, since I had changed so much and was only testing on the big sites I didn't notice when it switches urls it won't work! I'll get 1.5 out today unless there is another major bug found - I just don't want to update too fast and have everybody downloading updates twice a day.

Link to post
Share on other sites

Updated to 1.5

 

- Fixed bug where you couldn't run mutiple urls

- Cleaned up a few unnecessary nodes

 

To download the update login at: http://imautobots.com/wp-login.php

 

Then go back to the homepage and click on "Purchase History"

Link to post
Share on other sites
  • 2 weeks later...

ETA for V2 is looking more like mid next week now, trying to get it done ASAP but also get it done the right way.

 

The price for this after V2 comes out is going to rise dramatically by the way. And of course the update to V2 for current users is free.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...