Bug/Crash submission system

[krntea]

Originally Posted by infinityloop

could you please share the windows event log entries (system for TV-Server service and applications: for MediaPortal/setuptv.exe) after thie issue / crash occured?

Certainly, when I get back from the office. Perhaps if this information is useful to the MP developers it should be required as part of the forum bug submisison process and might be beneficial to mentioned so in the office forum bug submission guidelines?

Also, as you have also experienced the problems you could visit that time period in your own event logs to see how helpful they will be to this problem.

Originally Posted by krntea

and (most essential) provide us with enought ppl who can then analyze the automatically submitted reports in their sparetime ......
the idea is nice, but how do you expect that we should follow and analyze all these reports?

I suspect my first post linking the first blog discussing the topic in the other thread might have be misleading, my apologies. You do not need people to analyse reports. You need a debugger, and a developer. If the Developer has the desire to know what line of code the crash/hang is on, they load the dump information into the debugger with a click, navigate to the source code line in a few seconds, and then start debugging as normal. All variables and their contents all along the call stack are at the developers disposition. If the code where the crash occurred looks Ok then try-catch info can be placed around the offending point and the user asked to run it, OR the developer can look at machine state info to see if something might be affecting the offending source, but usually it is quicker not to bother.

Originally Posted by krntea

well maybe you want to code us such a feature + the required online facility then?

No. The benefits of auto bug submission are so popular and widespread that there is code to do it out there already. Take the time to try out the codeproject crashrpt as I suggested in the linked thread - it is easy to integrate and will show you quickly what I am talking about. And as mentioned, you will find all high volume open source projects with many users have to use such systems already (Mozilla suite, Azeurus etc) due to the problems with forums I have already mentioned.

Thanks,

Keith.

[krntea]

Please my post in this thread that expands on the codeproject crashrpt example a bit: http://forum.team-mediaportal.com/tv..._0-t25067.html

I think there is some misunderstanding here that is my fault, and I apologise: I posted above the first blog that I found on the theme to back up what i was saying. It contains this comment: "It will take some experience to read these dump files." which is in my experience incorrect. You need the crash report dump, a developer and a debugger. If the developer knows how to use the debugger (of course they do!) then that is all there is to it. No more "experience" necessary.

[infinityloop]

as i said before, if you can integrate a automatic error report system it into current MediaPortal version then this is highly appreciated.

we are looking forward to test it out then.

[darved]

I completely agree with krntea here - no matter who does it or when. The idea is solid and will have a huge impact on the stability of MediaPortal in the long term.

I am finding new bugs (some severe) every day on my system for the month I have been running it. To be sure the problems are with MediaPortal, I've run the FusionHDTV software for three days straight now without turning it off and it hasn't failed once. You will notice there are no bug reports from me as yet. Heres why:

Initially, I was new and figured most of my issues were likely to be user related (which many were). I also didn't want to fill the forums with bugs that were probably already reported.

Now that its part of my main entertainment setup things are different. We're watching tv or whatever and MediaPortal crashes or does something crazy (always at the worst possible moment ). To suggest that I tell everyone they can't watch TV until I gather some info on the issue is just a ludicrous expectation - of me and my family.

I was thinking of having a notepad on the coffee table and taking notes every time something happens, and then during a period when no-one is using the tv (rare) replicate all the issues and gather all the info and post them to the forums. Then I saw this post.

I think this is an awesome idea. I hope the developers see this thread. Maybe they will be more open to it.

[krntea]

Originally Posted by infinityloop

be aware, we simply do not have the required manpower to:

1. create these feature for MediaPortal and TV-Server
2. create the required online facilities to catch and store the reports
3. analyze all submitted reports

all developers are actively busy with working on bugfixes and/or new features. they do not have to waste much time with "trying to reproduce" something because of the lack of informations. thats something the test team takes care about.

I know the developers are busy (as I mentioned above). I also pointed out that they are so busy since the system of reporting bugs weighs them and the end users down... but no need to take my word for it go to any other high profile many user open source system. Let me get what your trying to explain about the bug fixing process with MP straight: Your saying that the developers "do not have to waste much time with "trying to reproduce" something because of the lack of informations.". You go on to say that some "test team" has the responsibility to find all the crash level bugs. If the developers don't try to reproduce submitted bugs because there is not enough information (there always is not enough information with forums posts), and nothing is done unless the test team can discover/reproduce the bug (which they usually cannot since they cannot possibly have the same same machine state and hardware as all the end users) So why bother asking end users to submit crash reports then? I don't follow the logic it just does not make sense in the framework of any project management I am aware of.

Originally Posted by infinityloop

many ppl might think now "i want that submission system. less work for me to report and developers fix it instandly".
the only thing which helps to get more bugs fixed are more developers.
the current "reporting-system/procedure" would even keep twice as much devs as we currently have busy.

what i miss is a inbuild feature which collects all logs and creates a zip file out of it. the user only has to attach it to the report then. a additional crash.dump would be nice then too.

This above quote is completely ignoring both what I have said above, the evidence present in other open source projects, and demonstrates clear lack of understanding of what an automatic bug submission system is.
Please, look into it before making rash comments about it and/or disregarding it.

Originally Posted by infinityloop

but i prefer manual bugreports in a forum.

I don't disagree with you. But as a project leader you should be less interested in what you like, and at least a little bit more open to what might improve the MP system. Most of this whole unfortunate thread could have been avoided with a simple "good idea, we the core developers might talk it over and consider" and I and others would be happy. I don't understand the level of negativity coming at me for such a simple idea... its not even new or radical! I have to wonder what happens with other more outlandish ideas with the project.

Originally Posted by infinityloop

~1year ago the team was thinking about it (as i allready said). the conclusion was that the efforts are to much that we could do it.

Going by the responses to the suggestion above, it really comes across that perhaps the MP developers are not aware of what a crash report system is at the basic level - so it would be hard to imagine serious consideration was done. It takes 10 minutes or less to do a really basic integration of the codeproject crashrpt project but I am certainly not using that as a complaint, merely an example. This is probably not the best out there for MP but an easy example with test code to point to.

Originally Posted by infinityloop

as i said before, if you can integrate a automatic error report system it into current MediaPortal version then this is highly appreciated. we are looking forward to test it out then.

An excellent suggestion and by all means I certainly may and put some code where my mouth is at some point in the future. However for the immediate time I will leave the topic open as a improvement suggestion and see if the developer community is even the least bit open to the idea. So far the reaction is quite negative (and scary).

[and-81]

Hey guys,

As a developer on the project I thought I'd jump in and offer my perspective on this...

Generally, I must agree with infinityloop.

Just so everyone know's where I'm coming from:

I'm a professional software engineer, university trained. I've worked on software projects in the local arm of a fortune 500 company (read: big) and I have worked with bug-submission systems before.

I've worked on MediaPortal for some time now, and if you look through the forums you'll see that I've dealt with many users and many bugs in my plugins and other various coding. And my experience has been that talking with the user is more valuable that reading fault reports.

I'd swap one thousand auto-generated fault reports for one well written (by a person) fault report.

This project has thousands of users and only a handful of active developers, of which even less actively bug fix. In my personal experience (as a professional software engineer) I have found that when you give the end user a magic "submit fault report" button the quality of the "additional information" that is provided drops through the floor. That is to say, it's bad.

You end up with lots of useless fault reports. You still do get good ones, but you have to sift through the rubbish to find them. In my opinion, on a project like MediaPortal with a very high user-to-developer ratio that just wont work. If we had people who's job it was to fix faults, then it might work, but MediaPortal is no-one's job. It's a "spare-time" project, not a full-time job.

I know this is in contrast with krntea's experience. And I say it with full respect. Because it doesn't mean one of us is right and the other is wrong, it just means that on the projects we have worked on there have been differing situations in place that have lead us to different conclusions.

The important lesson here is that it's not clear cut that an automatic error log submission system would benefit the project. And it is my opinion that it would not.

However, I do believe that any extra information we could be given would be a benefit. Anything that would give us more details about what's happened will be of benefit. Just don't put a "Submit" button next to it, please!

I think the best way to do this might be to have a dialog pop up explaining that an error has occurred (you could turn this off in configuration) and give the user the chance to compile a "log collection file". For example, it could zip up all the relevant logs, and maybe even generate some additional ones.

The user would then be given instructions on how to submit the fault report. A link to the forums and the choice to proceed.

We've had a fair few bugs where the user's written feedback was far more important than the logs. Things like "Go to Music, then go to TV, and back to music ..." In a fault log, that just looks like random behaviour, but when the user spells out that those are the steps to produce the bug you have a lot more to go on.

And what happens when a new Release hit's the users, and there is a commonly reproduced bug in it. We'll get one hundred thousand emails in the first 3 days all saying the same thing. And one of those emails will be for a different bug, but will get lost in the mix. I really worry about flooding the developers with emails.

I could go on, but I wont, so just to summarise (cos I do ramble on):

1. I think an automated system for bug reporting is dangerous for this project.

2. But I do think more information for the developers is a good thing, so maybe a semi-automatic system would be worthwhile.

3. I am open to having my mind changed on this, but it would only be through seeing it in action on this project. Maybe we could trial it? Either way, it's going to take someone stepping up and coding it, and honestly that's probably going to take a new developer to jump in (krntea ?).

Anyway, it's all healthy discussion, just don't get too carried away believing it's some sort of panacea.

Cheers,

[infinityloop]

Originally Posted by darved

I was thinking of having a notepad on the coffee table and taking notes every time something happens, and then during a period when no-one is using the tv (rare) replicate all the issues and gather all the info and post them to the forums.

that would be highly appreciate!

since this is exactly what brought us from 0.1.xx to the last very stable 0.2.2.0.
and it also is what the developers NEED from us

clear, detailed reports

dont fear to post a issue which is allready "know".
you can have a look at our ::bucktracker:: to check if that issue is allready there.

Testing and reporting is a timeconsuming and somtimes frustrating task.
But if you do it right, then it realy helps.

[flokel]

An excellent suggestion and by all means I certainly may and put some code where my mouth is at some point in the future. However for the immediate time I will leave the topic open as a improvement suggestion and see if the developer community is even the least bit open to the idea. So far the reaction is quite negative (and scary).

No matter how this ends,
for my part so far a big for adressing this topic.

[krntea]

G'Day Aaron, thanks for weighing in. Don't let the Spanish flag fool you I'm an Aussie in spain :-).

First up I think you might be mis-informed on one thing: I see in your post that you have repeated the idea that all this is is a way to auto submit the logs: "The important lesson here is that it's not clear cut that an automatic error log submission system would benefit the project. And it is my opinion that it would not.".

To repeat, I am not talking about an primitive auto log and/or comment submission system. Please see the referenced codeproject and take up the suggestion to run a test on it. If you have never used auto submission of debug info systems before this will amaze you, as it did me the first time I used it.

Originally Posted by and-81

I'd swap one thousand auto-generated fault reports for one well written (by a person) fault report.

Some hypothetical questions here I am not actually looking for you to answer them:

1) As a developer and given the choice between debugging a crash/hang problem of some software with your development environments debugger, or alternatively going without it and only allowed to use logging output/event log to track down the offending source code - which of the two would you choose? I think most programmers would take the debugger.

2) If the logging output does not contain the information needed to debug, what do you do.. do you turn to the debugger?

All auto submitted crash reporting does is give the developer the power to debug the crash that the user exprienced in their development environment, as, where and how it happened on the end users machine.. .looking at where it happened in the the source code along with all variables and stack trace availabel to click through and inspect. Of course you can do more, but that is its greatest core strength of what I am talking about.

Originally Posted by and-81

You end up with lots of useless fault reports. You still do get good ones, but you have to sift through the rubbish to find them.

See my earlier in this thread comments on scripts and submission. This information is also helpful satisitically and can help improve the system, track down the cough, less adept developers. Auso submisison does not take more developer time, it saves it. Again we are not simply talking about sending logs and user comments and some system state information. We are talking debugging info for your development environment.

Originally Posted by and-81

In my opinion, on a project like MediaPortal with a very high user-to-developer ratio that just wont work. If we had people who's job it was to fix faults, then it might work, but MediaPortal is no-one's job. It's a "spare-time" project, not a full-time job.

See my earlier in this thread comments on high profile open source projects that have turned to these submission systems for exactly the opposite reason. Even better chat to their diverse group of developers I am sure they would share their experiences and inform as to why they went to using them.

Cheers,
Keith

Just a quick follow-up: There is some real or percieved developer fear here that auto submission leads to high volume of data. Note You can only usually submit a report when your system crashes or hangs. All other reports, user interface requests etc etc cannot use the auto system (unless they cause crashes). Your not going to get massive high volume unless the software crashes that frequently. Might be the case in the first week the submission system goes live, and after that a new crash free release comes out...

Think the little dialog "send to microsoft box" that pops up when MP or other programs crash. Only the debgging info dumps needed by a development environment are sent to MP developers, not microsoft, where they can actually do some good.

[infinityloop]

Originally Posted by krntea

I don't disagree with you. But as a project leader you should be less interested in what you like, and at least a little bit more open to what might improve the MP system.

Project Leader?

I am only the Test Team Leader

So i do what the developers want.
And they want decent reports (clear, step-by-step description with logfiles and the communication with the reporter(s))

-----

So here we are again.
Someone new steps in, tells us what we are doing wrong, identified whats is slowing down the developers and provides us with a solution what he thinks is "best for the project".

Your suggestion is not the part that i find annoying, it is the way how you "present" it.
Claiming to know and understand whats currently slowing down the developers and the team is quite arrogant imo.

And those who know me know which kind of response follows to such posts.



Fact is, as i allready written 3 times, that we have seriously discussed the possibility of an automatic reporting system not that long ago.

we came to the conlusion that the development would not benefit from the "send button".
and-81 allready summed it up in his post.

This means that we will not spend time to implement such a feature.
But we are thinking about ways to make the reporting process easier for the enduser.


@krntea is there any reason why you demand that this feature must be automatic?
where is your problem with having the user starting a bugreport describe step-by-step what he did and attach the crash.dump file?