[Pending] Utils: DownLoadImage failed when PageRequest need Header "User-Agent" (1 Viewer)

[morfius]

steps to reproduce the issue
In configuration mode, select any Title from Database and go to CoverArt. Set the value of Image URL http://pics.filmaffinity.com/Carmina_o_revienta-570033301-large.jpg and Download Manually. You get the Error

LogError

 [2014-08-22 21:37:57,707] [Config ] [IMDBDetails] [INFO ] - Utils: DownLoadImage http://pics.filmaffinity.com/Carmina_o_revienta-570033301-large.jpg failed:The remote server returned an error: (403) Forbidden.

Now open your Internet Browser and download it. The image is OK.

File Util.cs Routine DownLoadImage

public static void DownLoadImage(string strURL, string strFile, System.Drawing.Imaging.ImageFormat imageFormat)
...
using (WebClient client = new WebClient())
...
client.Proxy.Credentials = CredentialCache.DefaultCredentials;
client.DownloadFile(strURL, strLogo);

Problem: some webs requires a header "User-Agent" (ex. FilmAffinity), but not all. IMDb don't work if you set the header "User-Agent".
Solution:
a) use a simple flag if (movieDetails.ThumbURL.Contains("filmaffinity")) client .Headers.Add("User-Agent", "Mozilla/"....
b) use (HttpWebRequest)WebRequest instead of WebClient
c) Team Media Portal have a lot of developpers with much more knowledge than me. In spanish Doctores tiene la Iglesia (translation from the Collins Dictionary there are plenty of people well able to pass an opinion (on that))

Greetings from Spain

P.S. This is my first report, i hope post it in the correct place

 

[regeszter]

Hi morfius,

Can you create a patch file for Solution B?

 

[morfius]

Hi morfius,

Can you create a patch file for Solution B?

I'll try tomorrow, acording your code guidelines.
Greetings!!

 

[morfius]

I regret very much the delay. Finally, i think found a elegant solution, without "if" and working at least for this two webs. (I guess that this solution work for all webs).

    public static void DownLoadImage(string strURL, string strFile)
      try
      {
        HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(strURL);
        wr.Timeout = 20000;
...

PATCH

    public static void DownLoadImage(string strURL, string strFile)
      try
      {
        HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(strURL);
        wr.Timeout = 20000;

        //Set UserAgent = "Any UserAgent"
       wr.UserAgent = "Mozilla/8.0 (compatible; MSIE 9.0; Windows NT 6.1; .NET CLR 1.0.3705;)";

        //Accept All
        wr.Accept = "*/*";
...

With this two lines, this code works fine for me No need for set any other header, like language, etc.


Additional Observation

This old procedure entry uses WebClient, but i think is not used anymore. (from 1.9)

    public static void DownLoadImage(string strURL, string strFile, System.Drawing.Imaging.ImageFormat imageFormat)

Greetings from Spain!!!

 

[morfius]

Finally, I decided to rewrite the entire procedure. You can test with this simple program.

#region using
using System;
using System.Diagnostics;
using System.Net;
using System.IO;
using System.Threading;
using MediaPortal.GUI.Library;//add ref Core & Utils
#endregion using

/*
 * Developper:      mOrfiUs
 * e-mail:          apifilmaffinityimdb@gmail.com
 * web:             https://sourceforge.net/projects/apifilmaffinityimdb/
 * Forum post:      https://forum.team-mediaportal.com/posts/1098788
 * Description:     New Core/Util/utils.cs DownLoadImage procedure
 * FIX:             Set the necessary headers for proper WebRequest (ex. filmaffinity)
 * Date             2014/09/06
 */

namespace TestDLImage
{
    class TestDLImage
    {
        static void Main(string[] args)
        {
            string[] lDownLoadImages = new string[] {
                "http://fr.web.img2.acsta.net/r_640_600/b_1_d6d6d6/pictures/14/07/04/14/59/301675.jpg", 
                "http://fr.web.img5.acsta.net/r_160_240/b_1_d6d6d6/pictures/14/07/04/14/59/301675.jpg", 
                "http://st.kinopoisk.ru/images/film_original/450237.jpg", 
                "http://image.tmdb.org/t/p/w396/1APNYPrWaZxZnBqnsHTou2MaHnx.jpg", 
                "http://pic.filmaffinity.com/Monuments_Men-681632116-large.jpg", 
                "http://pics.filmaffinity.com/Monuments_Men-681632116-large.jpg", 
                "http://pics.filmaffinity.com/nts_Men-682116-large.jpg", 
                "http://ia.media-imdb.com/images/M/MV5BMTQ5NTg5ODk4OV5BMl5BanBnXkFtZTgwODc4MTMzMDE@._V1__SX300.jpg", 
                "http://ia.media-imdb.com/images/M/MV5BDk4OV5BMl5BanBnXkFtZTgwODc4MTMzMDE@", 
                "http://ia.media-imdb.com/images/M/MV5BMjM4MjU3MDgzNV5BMl5BanBnXkFtZTgwODE3NzcwMTE@"
            };
            int iCount = 0;
            foreach (string DLImage in lDownLoadImages)
                DownLoadImage(DLImage, @"c:\temp" + iCount++.ToString("000") + ".jpg");
        }

        /// <summary>
        /// Download a remote image and save it locally.
        /// If it fails the first time, wait one second and try to download again.
        /// UserAgent and Accept headers are set for correct request.
        /// Unfortunately, synchronous downloads are often problematic, so it is necessary to correctly handle errors.
        /// </summary>
        /// <param name="strURL">remote image to download</param>
        /// <param name="strFile">local image to save</param>
        /// <param name="secondTry">bool used for the recursive call, after first fail</param>
        public static void DownLoadImage(string strURL, string strFile, bool secondTry = false)
        {
            if (string.IsNullOrEmpty(strURL) || string.IsNullOrEmpty(strFile))
                return;
            Uri uri = new Uri(strURL);
            HttpWebRequest wReq = (HttpWebRequest)WebRequest.Create(uri);
            wReq.Proxy.Credentials = CredentialCache.DefaultCredentials;
            wReq.UserAgent = "Mozilla/8.0 (compatible; MSIE 9.0; Windows NT 6.1; .NET CLR 1.0.3705;)";
            wReq.Accept = "*/*";
            //should check the results before continue
            //if wr.StatusCode == HttpStatusCode.OK
            //but don't have reason if is a void procedure
            try
            {
                using (HttpWebResponse wr = (HttpWebResponse)wReq.GetResponse())
                    using (BinaryReader br = new BinaryReader(wr.GetResponseStream()))
                        using (FileStream fs = new FileStream(strFile, FileMode.OpenOrCreate, FileAccess.Write))
                        {
                            byte[] b = new byte[1024 * 128];//Should we increase the buffer size?. We live in 2014
                            int bRead;
                            do
                            {
                                bRead = br.Read(b, 0, b.Length);
                                fs.Write(b, 0, bRead);
                            } while (bRead != 0);
                        }
            }
            catch (System.Net.WebException eWebException)
            {
                Log.Info("Utils: DownLoadImage {1} failed: {0}", eWebException.Message + (secondTry ? " second try also fail" : string.Empty), strURL);
                //after second fail, return
                if (secondTry)
                    return;
                //just wait a second, and then try again
                Thread.Sleep(1000);
                DownLoadImage(strURL, strFile, true);
            }
            return;
        }

    }
}

Greetings from Spain!!!

 

[regeszter]

Can you create a git patch file or a fork from the current master on git?

 

[morfius]

firsr time... i will try

file Core/Util/Util.cs forked
Spanish translation: archivo forqueado correctamente.

 

[Sebastiii]

Thanks for your patch and defo need also feeling about @Deda

 

[morfius]

Thanks for your patch and defo need also feeling about @Deda

Sorry... i don't understand.... i am newbie
Please, the above code it's OK (i think), but i been sent several commits to git master. Last one it's the correct. Can you chek the last commit? Sorry for this inconvenience...
You can try the above code. Just add remote image url to first list, even a fake image, for catching the error. Thanks in advance... and Greetings from Spain.

 

[Sebastiii]

Hi,

Deda knows a lot about that part of code (if i'm not wrong)

So i tag him to see if he can look your change