[Pending] - Utils: DownLoadImage failed when PageRequest need Header "User-Agent"

Discussion in 'Submit: code patches (MediaPortal/TV-Server/etc.)' started by morfius, August 22, 2014.

  1. morfius

    morfius Portal Pro

    Joined:
    November 10, 2011
    Messages:
    40
    Likes Received:
    24
    Gender:
    Male
    Occupation:
    Technician Engineer
    Location:
    Valladolid
    Ratings:
    +42 / 0
    Home Country:
    Spain Spain
    steps to reproduce the issue
    In configuration mode, select any Title from Database and go to CoverArt. Set the value of Image URL http://pics.filmaffinity.com/Carmina_o_revienta-570033301-large.jpg and Download Manually. You get the Error

    LogError
    Code (Text):
    1.  [2014-08-22 21:37:57,707] [Config ] [IMDBDetails] [INFO ] - Utils: DownLoadImage http://pics.filmaffinity.com/Carmina_o_revienta-570033301-large.jpg failed:The remote server returned an error: (403) Forbidden.
    2.  
    Now open your Internet Browser and download it. The image is OK.

    File Util.cs Routine DownLoadImage
    Code (Text):
    1.  
    2. public static void DownLoadImage(string strURL, string strFile, System.Drawing.Imaging.ImageFormat imageFormat)
    3. ...
    4. using (WebClient client = new WebClient())
    5. ...
    6. client.Proxy.Credentials = CredentialCache.DefaultCredentials;
    7. client.DownloadFile(strURL, strLogo);
    8.  
    Problem: some webs requires a header "User-Agent" (ex. FilmAffinity), but not all. IMDb don't work if you set the header "User-Agent".
    Solution:
    a) use a simple flag if (movieDetails.ThumbURL.Contains("filmaffinity")) client .Headers.Add("User-Agent", "Mozilla/"....
    b) use (HttpWebRequest)WebRequest instead of WebClient
    c) Team Media Portal have a lot of developpers with much more knowledge than me. In spanish Doctores tiene la Iglesia (translation from the Collins Dictionary there are plenty of people well able to pass an opinion (on that))

    Greetings from Spain



    P.S. This is my first report, i hope post it in the correct place
     
    Last edited: August 24, 2014
    • Like Like x 2
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. regeszter
    • Team MediaPortal

    regeszter Retired Team Member

    Joined:
    October 29, 2005
    Messages:
    5,335
    Likes Received:
    2,863
    Ratings:
    +5,151 / 18
    Home Country:
    Hungary Hungary
    Hi morfius,

    Can you create a patch file for Solution B? ;)
     
  4. morfius

    morfius Portal Pro

    Joined:
    November 10, 2011
    Messages:
    40
    Likes Received:
    24
    Gender:
    Male
    Occupation:
    Technician Engineer
    Location:
    Valladolid
    Ratings:
    +42 / 0
    Home Country:
    Spain Spain
    I'll try tomorrow, acording your code guidelines.
    Greetings!!
     
    • Thank You! Thank You! x 1
  5. morfius

    morfius Portal Pro

    Joined:
    November 10, 2011
    Messages:
    40
    Likes Received:
    24
    Gender:
    Male
    Occupation:
    Technician Engineer
    Location:
    Valladolid
    Ratings:
    +42 / 0
    Home Country:
    Spain Spain
    I regret very much the delay. Finally, i think found a elegant solution, without "if" and working at least for this two webs. (I guess that this solution work for all webs).

    Code (Text):
    1.     public static void DownLoadImage(string strURL, string strFile)
    2.       try
    3.       {
    4.         HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(strURL);
    5.         wr.Timeout = 20000;
    6. ...
    7.  

    PATCH

    Code (Text):
    1.     public static void DownLoadImage(string strURL, string strFile)
    2.       try
    3.       {
    4.         HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(strURL);
    5.         wr.Timeout = 20000;
    6.  
    7.         //Set UserAgent = "Any UserAgent"
    8.        wr.UserAgent = "Mozilla/8.0 (compatible; MSIE 9.0; Windows NT 6.1; .NET CLR 1.0.3705;)";
    9.  
    10.         //Accept All
    11.         wr.Accept = "*/*";
    12. ...
    13.  
    With this two lines, this code works fine for me No need for set any other header, like language, etc.


    Additional Observation

    This old procedure entry uses WebClient, but i think is not used anymore. (from 1.9)
    Code (Text):
    1.  
    2.     public static void DownLoadImage(string strURL, string strFile, System.Drawing.Imaging.ImageFormat imageFormat)
    3.  
    Greetings from Spain!!!
     
    Last edited: September 5, 2014
    • Thank You! Thank You! x 3
  6. morfius

    morfius Portal Pro

    Joined:
    November 10, 2011
    Messages:
    40
    Likes Received:
    24
    Gender:
    Male
    Occupation:
    Technician Engineer
    Location:
    Valladolid
    Ratings:
    +42 / 0
    Home Country:
    Spain Spain
    Finally, I decided to rewrite the entire procedure. You can test with this simple program.

    Code (Text):
    1.  
    2. #region using
    3. using System;
    4. using System.Diagnostics;
    5. using System.Net;
    6. using System.IO;
    7. using System.Threading;
    8. using MediaPortal.GUI.Library;//add ref Core & Utils
    9. #endregion using
    10.  
    11. /*
    12.  * Developper:      mOrfiUs
    13.  * e-mail:          apifilmaffinityimdb@gmail.com
    14.  * web:             https://sourceforge.net/projects/apifilmaffinityimdb/
    15.  * Forum post:      https://forum.team-mediaportal.com/posts/1098788
    16.  * Description:     New Core/Util/utils.cs DownLoadImage procedure
    17.  * FIX:             Set the necessary headers for proper WebRequest (ex. filmaffinity)
    18.  * Date             2014/09/06
    19.  */
    20.  
    21. namespace TestDLImage
    22. {
    23.     class TestDLImage
    24.     {
    25.         static void Main(string[] args)
    26.         {
    27.             string[] lDownLoadImages = new string[] {
    28.                 "http://fr.web.img2.acsta.net/r_640_600/b_1_d6d6d6/pictures/14/07/04/14/59/301675.jpg",
    29.                 "http://fr.web.img5.acsta.net/r_160_240/b_1_d6d6d6/pictures/14/07/04/14/59/301675.jpg",
    30.                 "http://st.kinopoisk.ru/images/film_original/450237.jpg",
    31.                 "http://image.tmdb.org/t/p/w396/1APNYPrWaZxZnBqnsHTou2MaHnx.jpg",
    32.                 "http://pic.filmaffinity.com/Monuments_Men-681632116-large.jpg",
    33.                 "http://pics.filmaffinity.com/Monuments_Men-681632116-large.jpg",
    34.                 "http://pics.filmaffinity.com/nts_Men-682116-large.jpg",
    35.                 "http://ia.media-imdb.com/images/M/MV5BMTQ5NTg5ODk4OV5BMl5BanBnXkFtZTgwODc4MTMzMDE@._V1__SX300.jpg",
    36.                 "http://ia.media-imdb.com/images/M/MV5BDk4OV5BMl5BanBnXkFtZTgwODc4MTMzMDE@",
    37.                 "http://ia.media-imdb.com/images/M/MV5BMjM4MjU3MDgzNV5BMl5BanBnXkFtZTgwODE3NzcwMTE@"
    38.             };
    39.             int iCount = 0;
    40.             foreach (string DLImage in lDownLoadImages)
    41.                 DownLoadImage(DLImage, @"c:\temp" + iCount++.ToString("000") + ".jpg");
    42.         }
    43.  
    44.         /// <summary>
    45.         /// Download a remote image and save it locally.
    46.         /// If it fails the first time, wait one second and try to download again.
    47.         /// UserAgent and Accept headers are set for correct request.
    48.         /// Unfortunately, synchronous downloads are often problematic, so it is necessary to correctly handle errors.
    49.         /// </summary>
    50.         /// <param name="strURL">remote image to download</param>
    51.         /// <param name="strFile">local image to save</param>
    52.         /// <param name="secondTry">bool used for the recursive call, after first fail</param>
    53.         public static void DownLoadImage(string strURL, string strFile, bool secondTry = false)
    54.         {
    55.             if (string.IsNullOrEmpty(strURL) || string.IsNullOrEmpty(strFile))
    56.                 return;
    57.             Uri uri = new Uri(strURL);
    58.             HttpWebRequest wReq = (HttpWebRequest)WebRequest.Create(uri);
    59.             wReq.Proxy.Credentials = CredentialCache.DefaultCredentials;
    60.             wReq.UserAgent = "Mozilla/8.0 (compatible; MSIE 9.0; Windows NT 6.1; .NET CLR 1.0.3705;)";
    61.             wReq.Accept = "*/*";
    62.             //should check the results before continue
    63.             //if wr.StatusCode == HttpStatusCode.OK
    64.             //but don't have reason if is a void procedure
    65.             try
    66.             {
    67.                 using (HttpWebResponse wr = (HttpWebResponse)wReq.GetResponse())
    68.                     using (BinaryReader br = new BinaryReader(wr.GetResponseStream()))
    69.                         using (FileStream fs = new FileStream(strFile, FileMode.OpenOrCreate, FileAccess.Write))
    70.                         {
    71.                             byte[] b = new byte[1024 * 128];//Should we increase the buffer size?. We live in 2014
    72.                             int bRead;
    73.                             do
    74.                             {
    75.                                 bRead = br.Read(b, 0, b.Length);
    76.                                 fs.Write(b, 0, bRead);
    77.                             } while (bRead != 0);
    78.                         }
    79.             }
    80.             catch (System.Net.WebException eWebException)
    81.             {
    82.                 Log.Info("Utils: DownLoadImage {1} failed: {0}", eWebException.Message + (secondTry ? " second try also fail" : string.Empty), strURL);
    83.                 //after second fail, return
    84.                 if (secondTry)
    85.                     return;
    86.                 //just wait a second, and then try again
    87.                 Thread.Sleep(1000);
    88.                 DownLoadImage(strURL, strFile, true);
    89.             }
    90.             return;
    91.         }
    92.  
    93.     }
    94. }
    95.  
    Greetings from Spain!!!
     
    Last edited: September 6, 2014
    • Thank You! Thank You! x 4
  7. regeszter
    • Team MediaPortal

    regeszter Retired Team Member

    Joined:
    October 29, 2005
    Messages:
    5,335
    Likes Received:
    2,863
    Ratings:
    +5,151 / 18
    Home Country:
    Hungary Hungary
    Can you create a git patch file or a fork from the current master on git?
     
    • Like Like x 2
  8. morfius

    morfius Portal Pro

    Joined:
    November 10, 2011
    Messages:
    40
    Likes Received:
    24
    Gender:
    Male
    Occupation:
    Technician Engineer
    Location:
    Valladolid
    Ratings:
    +42 / 0
    Home Country:
    Spain Spain
    firsr time... i will try ;)

    file Core/Util/Util.cs forked ;)
    Spanish translation: archivo forqueado correctamente.
     
    Last edited: September 6, 2014
    • Like Like x 1
  9. Sebastiii
    • Team MediaPortal

    Sebastiii Development Group

    Joined:
    November 12, 2007
    Messages:
    16,207
    Likes Received:
    6,173
    Gender:
    Male
    Location:
    France
    Ratings:
    +10,158 / 9
    Home Country:
    France France
    Show System Specs
    Thanks for your patch and defo need also feeling about @Deda :)
     
    • Like Like x 1
  10. morfius

    morfius Portal Pro

    Joined:
    November 10, 2011
    Messages:
    40
    Likes Received:
    24
    Gender:
    Male
    Occupation:
    Technician Engineer
    Location:
    Valladolid
    Ratings:
    +42 / 0
    Home Country:
    Spain Spain
    Sorry... i don't understand.... i am newbie ;)
    Please, the above code it's OK (i think), but i been sent several commits to git master. Last one it's the correct. Can you chek the last commit? Sorry for this inconvenience...
    You can try the above code. Just add remote image url to first list, even a fake image, for catching the error. Thanks in advance... and Greetings from Spain.
     
    • Like Like x 1
  11. Sebastiii
    • Team MediaPortal

    Sebastiii Development Group

    Joined:
    November 12, 2007
    Messages:
    16,207
    Likes Received:
    6,173
    Gender:
    Male
    Location:
    France
    Ratings:
    +10,158 / 9
    Home Country:
    France France
    Show System Specs
    Hi,

    Deda knows a lot about that part of code (if i'm not wrong) ;)

    So i tag him to see if he can look your change :)
     
Loading...

Users Viewing Thread (Users: 0, Guests: 0)

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice
  • About The Project

    The vision of the MediaPortal project is to create a free open source media centre application, which supports all advanced media centre functions, and is accessible to all Windows users.

    In reaching this goal we are working every day to make sure our software is one of the best.

             

  • Support MediaPortal!

    The team works very hard to make sure the community is running the best HTPC-software. We give away MediaPortal for free but hosting and software is not for us.

    Care to support our work with a few bucks? We'd really appreciate it!