Update allmusic.com scraper

Discussion in 'Area 51 - Testing Area' started by jameson_uk, August 4, 2013.

  1. jameson_uk
    • Premium Supporter

    jameson_uk Retired Team Member

    Joined:
    January 27, 2005
    Messages:
    7,258
    Likes Received:
    2,228
    Gender:
    Male
    Occupation:
    Data Warehousing Consultant
    Location:
    Birmingham
    Ratings:
    +2,405 / 0
    Home Country:
    United Kingdom United Kingdom
    Show System Specs
    Allmusic.com have updated their layout again so currently artist and album scraping inside of MP is broken.



    Attached is hopefully a fix which is https://github.com/MediaPortal/MediaPortal-1/tree/BUG_AMG_Scraper

    In C:\Program Files (x86)\Team MediaPortal\MediaPortal
    rename Databases.dll and place the following files from zip into that folder
    • Databases.dll
    • Databases.pdb
    • HTMLAgilityPack.dll
    In C:\Program Files (x86)\Team MediaPortal\MediaPortal\plugins\Windows
    rename WindowPlugins.dll and place the following files from zip into that folder
    • WindowPlugins.dll
    • WindowPlugins.pdb
    You will need to disable other scrapers (MusicInfoHandler) but scraping should then work inside of MP
     

    Attached Files:

    • AMG.zip
      File size:
      1.5 MB
      Uploaded:
      August 4, 2013
      Views:
      221
    • AMG_v3.zip
      File size:
      1.5 MB
      Uploaded:
      August 18, 2013
      Views:
      206
    • AMG_3a.zip
      File size:
      1.5 MB
      Uploaded:
      August 18, 2013
      Views:
      287
    Last edited: August 18, 2013
    • Thank You! Thank You! x 5
    • Like Like x 4
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. styrsven

    styrsven Portal Pro

    Joined:
    January 6, 2013
    Messages:
    115
    Likes Received:
    26
    Gender:
    Male
    Ratings:
    +51 / 1
    Home Country:
    Sweden Sweden
    Show System Specs
  4. jameson_uk
    • Premium Supporter

    jameson_uk Retired Team Member

    Joined:
    January 27, 2005
    Messages:
    7,258
    Likes Received:
    2,228
    Gender:
    Male
    Occupation:
    Data Warehousing Consultant
    Location:
    Birmingham
    Ratings:
    +2,405 / 0
    Home Country:
    United Kingdom United Kingdom
    Show System Specs
  5. jameson_uk
    • Premium Supporter

    jameson_uk Retired Team Member

    Joined:
    January 27, 2005
    Messages:
    7,258
    Likes Received:
    2,228
    Gender:
    Male
    Occupation:
    Data Warehousing Consultant
    Location:
    Birmingham
    Ratings:
    +2,405 / 0
    Home Country:
    United Kingdom United Kingdom
    Show System Specs
    Can anyone provide any feedback as to whether this is working for them?
     
  6. catavolt
    • Team MediaPortal
    • Administrator

    catavolt Design Group Manager

    Joined:
    August 13, 2007
    Messages:
    12,252
    Likes Received:
    4,988
    Gender:
    Male
    Occupation:
    Technical Documentation, Translation
    Location:
    Lanzarote
    Ratings:
    +7,408 / 38
    Home Country:
    Germany Germany
    Show System Specs
    Will test it later the day ;)
     
  7. styrsven

    styrsven Portal Pro

    Joined:
    January 6, 2013
    Messages:
    115
    Likes Received:
    26
    Gender:
    Male
    Ratings:
    +51 / 1
    Home Country:
    Sweden Sweden
    Show System Specs
    Yes, it looks like it is doing what it is supposed to do. I tried to update as much as I could, and it fetched the info correctly. A couple of times it seemed to hang when fetching data, and when I checked the logs I saw some error printouts. I'm attaching the actual logs (didn't run in debug mode).
     
  8. jameson_uk
    • Premium Supporter

    jameson_uk Retired Team Member

    Joined:
    January 27, 2005
    Messages:
    7,258
    Likes Received:
    2,228
    Gender:
    Male
    Occupation:
    Data Warehousing Consultant
    Location:
    Birmingham
    Ratings:
    +2,405 / 0
    Home Country:
    United Kingdom United Kingdom
    Show System Specs
    Any chance you can replicate this with debug level logs? I can stop the errors but these indicate cases where the code is not parsing the pages (might be because they are laid out differently or something) but I could really do with knowing which album / artist throw up these errors so I can check them
     
  9. styrsven

    styrsven Portal Pro

    Joined:
    January 6, 2013
    Messages:
    115
    Likes Received:
    26
    Gender:
    Male
    Ratings:
    +51 / 1
    Home Country:
    Sweden Sweden
    Show System Specs
    I'll have a go at it tonight.
     
  10. HomeY
    • Team MediaPortal

    HomeY Test Group

    Joined:
    February 23, 2008
    Messages:
    6,460
    Likes Received:
    2,627
    Gender:
    Male
    Occupation:
    Network Engineer
    Location:
    ::1
    Ratings:
    +4,737 / 16
    Home Country:
    Netherlands Netherlands
    Show System Specs
    Seems to work great Mike (y)
    20-26-32.png 20-26-52.png 20-27-38.png 20-28-00.png 20-28-28.png

    But... i have 1 artist that always seems to cause problems :)
    Frédéric Chopin
    [hide]
    Code (Text):
    1. [2013-08-08 20:27:04,467] [Log  ] [MPMain  ] [ERROR] - Exception: System.NullReferenceException: Object reference not set to an instance of an object.
    2.   at MediaPortal.Music.Database.MusicArtistInfo.Parse(String strUrl) in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\Databases\Music\InternetLookups\MusicArtistInfo.cs:line 194
    3.   at MediaPortal.GUI.Music.GUIMusicBaseWindow.ShowArtistInfo(String artistName, String albumName) in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\WindowPlugins\GUIMusic\GUIMusicBaseWindow.cs:line 1283
    4.   at MediaPortal.GUI.Music.GUIMusicBaseWindow.OnInfo(Int32 iItem) in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\WindowPlugins\GUIMusic\GUIMusicBaseWindow.cs:line 1464
    5.   at MediaPortal.GUI.Music.GUIMusicGenres.OnShowContextMenu() in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\WindowPlugins\GUIMusic\GUIMusicGenres.cs:line 723
    6.   at MediaPortal.GUI.Library.GUIWindow.OnAction(Action action)
    7.   at MediaPortal.GUI.Music.GUIMusicBaseWindow.OnAction(Action action) in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\WindowPlugins\GUIMusic\GUIMusicBaseWindow.cs:line 493
    8.   at MediaPortal.GUI.Music.GUIMusicGenres.OnAction(Action action) in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\WindowPlugins\GUIMusic\GUIMusicGenres.cs:line 336
    9.   at MediaPortal.GUI.Library.GUIWindowManager.OnAction(Action action)
    10.   at MediaPortalApp.OnAction(Action action)  Message: Object reference not set to an instance of an object.  Site  : Boolean Parse(System.String)  Source : Databases  Stack Trace:  at MediaPortal.Music.Database.MusicArtistInfo.Parse(String strUrl) in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\Databases\Music\InternetLookups\MusicArtistInfo.cs:line 194
    11.   at MediaPortal.GUI.Music.GUIMusicBaseWindow.ShowArtistInfo(String artistName, String albumName) in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\WindowPlugins\GUIMusic\GUIMusicBaseWindow.cs:line 1283
    12.   at MediaPortal.GUI.Music.GUIMusicBaseWindow.OnInfo(Int32 iItem) in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\WindowPlugins\GUIMusic\GUIMusicBaseWindow.cs:line 1464
    13.   at MediaPortal.GUI.Music.GUIMusicGenres.OnShowContextMenu() in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\WindowPlugins\GUIMusic\GUIMusicGenres.cs:line 723
    14.   at MediaPortal.GUI.Library.GUIWindow.OnAction(Action action)
    15.   at MediaPortal.GUI.Music.GUIMusicBaseWindow.OnAction(Action action) in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\WindowPlugins\GUIMusic\GUIMusicBaseWindow.cs:line 493
    16.   at MediaPortal.GUI.Music.GUIMusicGenres.OnAction(Action action) in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\WindowPlugins\GUIMusic\GUIMusicGenres.cs:line 336
    17.   at MediaPortal.GUI.Library.GUIWindowManager.OnAction(Action action)
    18.   at MediaPortalApp.OnAction(Action action)
    19. [2013-08-08 20:27:04,471] [Error  ] [MPMain  ] [ERROR] - Exception: Object reference not set to an instance of an object. Databases at MediaPortal.Music.Database.MusicArtistInfo.Parse(String strUrl) in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\Databases\Music\InternetLookups\MusicArtistInfo.cs:line 194
    20.   at MediaPortal.GUI.Music.GUIMusicBaseWindow.ShowArtistInfo(String artistName, String albumName) in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\WindowPlugins\GUIMusic\GUIMusicBaseWindow.cs:line 1283
    21.   at MediaPortal.GUI.Music.GUIMusicBaseWindow.OnInfo(Int32 iItem) in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\WindowPlugins\GUIMusic\GUIMusicBaseWindow.cs:line 1464
    22.   at MediaPortal.GUI.Music.GUIMusicGenres.OnShowContextMenu() in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\WindowPlugins\GUIMusic\GUIMusicGenres.cs:line 723
    23.   at MediaPortal.GUI.Library.GUIWindow.OnAction(Action action)
    24.   at MediaPortal.GUI.Music.GUIMusicBaseWindow.OnAction(Action action) in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\WindowPlugins\GUIMusic\GUIMusicBaseWindow.cs:line 493
    25.   at MediaPortal.GUI.Music.GUIMusicGenres.OnAction(Action action) in C:\Users\james_000\Documents\Code\MediaPortal-1\mediaportal\WindowPlugins\GUIMusic\GUIMusicGenres.cs:line 336
    26.   at MediaPortal.GUI.Library.GUIWindowManager.OnAction(Action action)
    27.   at MediaPortalApp.OnAction(Action action)
    [/hide]

    BTW: Lovin' the moods :D
     
    Last edited: August 8, 2013
    • Like Like x 1
  11. jameson_uk
    • Premium Supporter

    jameson_uk Retired Team Member

    Joined:
    January 27, 2005
    Messages:
    7,258
    Likes Received:
    2,228
    Gender:
    Male
    Occupation:
    Data Warehousing Consultant
    Location:
    Birmingham
    Ratings:
    +2,405 / 0
    Home Country:
    United Kingdom United Kingdom
    Show System Specs
    OK this was because he has no discography page. New version in first post which will now check that pages exist before loading them.
    Can you try and break this again please?
     
    • Like Like x 1
Loading...

Users Viewing Thread (Users: 0, Guests: 0)

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice
  • About The Project

    The vision of the MediaPortal project is to create a free open source media centre application, which supports all advanced media centre functions, and is accessible to all Windows users.

    In reaching this goal we are working every day to make sure our software is one of the best.

             

  • Support MediaPortal!

    The team works very hard to make sure the community is running the best HTPC-software. We give away MediaPortal for free but hosting and software is not for us.

    Care to support our work with a few bucks? We'd really appreciate it!