First Bot

[Toxic Chili 2]

Finally started work on my first bot, so thought I would document some of my progress here :-) (hope that is okay).

 

Basically I have a few music related sites and the massive amount of content and information needed makes it very time consuming to add it all myself and expensive to pay to get someone else to do it.

 

So my initial challenge is.

 

1. to find a way to effectively scrape the information from sites on the internet (artist, album and track information).

 

2. to add the information to my site via the add artist/album forms.

 

So far I have managed to create something that can scrape the album information for a specified album, next step will be to work on getting all the album information for one artist and organizing it in a way that can easily be added to my site.

 

After that I will be looking at getting lists of artists for the specific music genre and adding the artist and all the album information to the site (if it doesn't already exist).

 

Once I have managed to figure all that out I hope to add some link building functionality to it also which can deep link to the newly added artists.

 

Probably going to take some time and any help/advice is appreciated :-)

Edited April 10, 2012 by Toxic Chili

[wakins4u 12]

kudos! the only advice I can give is, "Keep going, you're definately on the right track".

 

Don't stress if you ever hit a coding stumbling block (believe me, you'll hit one sooner or later), just come round here and ask the community... It'll all make you a better Ubotter i guess

 

keep it up!

[rusty 0]

Congrats! My golden rule is the more mistakes we make, the more knowledgeable we get.

[Toxic Chili 2]

Thanks :-)

 

My progress so far

 

My bot now can scrape the track listings from another site and input the tracks into my site.

 

When it posts the track listing onto my site, it counts the tracks in the file and then increments the list each loop adding the track number and track name.

 

Took my a few attempts to get the logic right, but working great so far :-)

 

Not completely automated yet, but getting there.

[Net66 54]

wtg... keep at it!

 

Andy

[Toxic Chili 2]

wtg... keep at it!

 

Andy

 

So far can do the scraping and posting, also can scrape every album of one artist and store the information in a file <album-name>-info.txt and <album-name>-tracklist.txt, then store a list of albums in a seperate file. Them it goes through the list of albums and loads the list based on the album name in that list.

 

My main stumbling block is that the information isn't always consistent. For example, one album might have the release date as 13 May, 1998. Where as another just has the year. The form for adding dates on my site is Year, then a drop down for the month. The year is the more important bit as it orders the albums by this, so I need to have a way of checking how the date is formatted and dealing with it according to that.

 

Adding the album tracks for each album is the most time consuming bit and got that bit sorted, but would be nice to get as much information as possible about the albums.