Check for Website Changes

[Bot-Factory 602]

Hi.

 

I'm working on a new bot to check for website changes.

The sites I want to check are 100% static HTML pages.

 

I though about scraping the page and compare the results against a txt file / list item.

 

Even better would be to get the HTML code of the site. Generate a MD5 checksum of the html code. And then compare the MD5 to a value in the csv. But I don't know if that's possible at all?

 

Would be great if one of the experienced guys could point me in the right direction?

 

Thanks in advance

Dan

[UBotBuddy 331]

Interesting idea. I must think on this.

[Seth Turin 223]

well, it depends on specifically what you want to do. are you just trying to find out whether or not a site is different than it was? that's simple enough. you can use download file to just download the html of the webpage. you can have one as a reference and then check it against new versions by using the evaluate qualifier. it's basically:

 

if evaluate old = new then it's the same.

[Bot-Factory 602]

Hi.

 

At the moment I load the URLs from a file. Navigate to the site.

And store a picture of the page in a folder.

 

Then you can look at all the pictures to see if the site still shows what you expect.

 

The goal is detect hacker attacks. Or if the website is offline / unavailable.

 

 

I will try to implement a more automatic way. With storing the HTML as baseline.

Then during the next run, will compare the HTML to the baseline.

 

And only if there's a difference, will download the picture.

 

Dan