Automated Artist / Album Scraping (2 Viewers)

[jameson_uk]

Now I've found something very interesting and maybe a way to solve finding artists/albums. Google is your friend, site search with I'm feeling lucky to be precise:

Now this is looking good. I will try some stuff out tonight as this also does a little bit of fuzzy matching (eg. and = & etc)

 

[emphatic]

Attached installer worked a charm!

Emph

 

[jameson_uk]

irritatingly google does not seem to like scrapers you get http://www.google.com/sorry/?contin.../search?q=site:www.allmusic.com+Jeffrey&btnI=

Will see what I can do as the google stuff looks pretty good. unfortunately google seems to be pretty clever and want prevent scraping.

I have noticed something I can improve with albums and will do some digging on that

Edit: Should be working via MPEI installer now

 

[Pog]

irritatingly google does not seem to like scrapers

Bugger!!!

 

[jameson_uk]

irritatingly google does not seem to like scrapers

Bugger!!!

Indeed Seems the google API now allows 100 free searches per day but this is by application not user. Additional searches are $5 per 1,000 with a max of 100,000 per day (although I am not paying $500 a day for this ) Only alternative would be to have users sign up for their own google search API key but not really a user friendly solution.

As this is not time critical I may attempt to issue one query every 30 seconds or less and see if works but I think Google are being pretty clever here in stopping this kind of thing

 

[Pog]

Thought they might be more friendly toward non-commercial apps... but then policing for that would be a nightmare. Read this comment on some site: "somewhere there are two guys working away in a garage with the motto 'don't be google' "

I like your thinking, a few sneaky searches here and there. Don't sweat it, what's there now should help get skinners going to support this. As is, the info shown really opens up the music section specially if you have a lot of music. I'm finding myself listening to artists I'd forgotten about and kept bypassing and I like putting on some tunes and reading through the details.

The concern I have with the plugin as stands is constant scraping every time MP is restarted. Can it be set to just scrape for newly added stuff and maybe just do a complete scrape on a timer?

 

[Shangostar]

Seems to have worked fairly well for me though it has picked up a few incorrect ones, I'd like the option to have an interface of some sort where you can manually scrape rather than every time MP is started and be able to view and edit/delete current entries and maybe the option to choose when multiple entries are found.

 

[jameson_uk]

Seems to have worked fairly well for me though it has picked up a few incorrect ones

Can you let me have some examples please

I'd like the option to have an interface of some sort where you can manually scrape rather than every time MP is started

Remember that this will only scrape for missing details. I will also tie this into music database events so if you scan within MP this will prompt a new scan. Adding a UI is a bit of a pain and not sure it is worth the time ?? anyone else think this?

and be able to view and edit/delete current entries and maybe the option to choose when multiple entries are found.

Not sure I want to spent a large amount of time developing something that will end up having to be re-done when I implement some better scraping mechanism. The existing artist info page is pretty crap at showing info but I would end up having to re-implement that in order to do something like this and that is quite a lot of effort.

Multiple items again is a fairly massive thing that is actually not as useful as you might think. Based on my experiments most the stuff that could not be matched is because of difference between my tags and allmusic.com rather than duplicates. (this now silently copes with many duplicates in many circumstances).

I could do with a list of things that are not picked up to help do some analysis here.

I will post something up later that should produce a list of missing items so I can do some more digging

 

[jameson_uk]

I am on a very intermittent and slow internet connection on a train so I can not test this.

My hope is that it will create text files for missing artists and albums within the log directory.

If this does work could people please upload these as that would help improve missing stuff. Also please report and false matches.

Thanks

 

[ysmp]

Seems to have worked fairly well for me though it has picked up a few incorrect ones, I'd like the option to have an interface of some sort where you can manually scrape rather than every time MP is started and be able to view and edit/delete current entries and maybe the option to choose when multiple entries are found.

Hi, i think is a great idea ....
ther is a tool for moving pictures and is open sorce ...maybe this tool can be mod for Music DB , that way we will have option
to Edit/Clean/fix music DB......
link : iqump - Tools for movingpictures plugin of mediaportal - Google Project Hosting