18:41:07 #startmeeting 18:41:07 Meeting started Wed Nov 19 18:41:07 2014 UTC. The chair is colonelqubit. Plugin info at http://wiki.debian.org/MeetBot. 18:41:07 Useful Commands: #action #agreed #help #info #idea #link #topic. 18:41:42 First off, marvelous work on UNCONFIRMED bugs, team ;-) 18:42:27 we broke 600, and after a brief bump up, we're headed back down below that point (612 currently) 18:42:52 Okay, who's here for the meeting? 18:42:52 yep I was quite vicious with them last week.. I've commented on 445 bugs in Nov 18:43:30 Hi, all. 18:44:05 hi 18:44:29 We had a screw-up with sorting in Calc. I wonder what QA can do to avoid the next such screw-up. 18:44:40 * colonelqubit sees a few ux-advise bugs in UNCONFIRMED that are probably from jphilipz_meeting. You going to join us? ;-) 18:45:49 TerrenceEnger: I think it's important for QA to do as much as possible to identify potential issues 18:46:20 colonelqubit: in the design meeting now, but there shouldnt be many ux-advise bugs as unconfirmed from me :D 18:46:46 i think i see 2 only 18:46:54 jphilipz_meeting: what's up with cross-scheduling a Design meeting for the same time slot? :P 18:47:03 Indeed. But that suggestion is short on specifics. 18:47:27 colonelqubit: well it as 1 hour earlier and then daylight saving times :D 18:47:42 * colonelqubit nods 18:47:50 TerrenceEnger: Sure. I think there were several factors that all aligned to magnify the problem 18:47:51 I came close to noticing the upcoming problem with Calc sorting. In the push during the smmer to reduce ... 18:48:12 I think one big piece is being able to identify problems early-on 18:48:46 TerrenceEnger: I guess in the more general level you are referring to problems caused by changing default behavior 18:48:47 the UNCO count, I tried to--pardon the pun--sort out the sorting bugs. I gave up, thinking that the confusion was entirely mine. 18:49:56 So that means a few things: 18:49:58 beluga_: Yes, but more generally, changing something "important". 18:49:58 1) More unit tests 18:50:00 2) Reducing our number of untriaged/non-de-duplicated bugs 18:50:02 3) Improving our workflow from bug reporting to bug fixing 18:50:34 TerrenceEnger: and what you are asking for QA is to recognize that something is important early on 18:51:07 Exactly. 18:51:35 that requires experience and I think it helps, if our more experienced members have plenty of free time.. free time is bought by less experienced members dealing with the regular stuff 18:51:48 I fear that that requires user involvement. And of course, any user who would care about the sorting is mostly interested in her own job. 18:52:09 So early on with this bug I believe one issue was that we needed test cases 18:52:21 one of my objectives in my crazy spree of unco checking was to kind of clear the playing field, so we can sit down in peace for awhile and see, where we can go from here 18:52:52 we had some reports, but there were multiple people claiming different things about spreadsheets and sorting 18:53:22 unfortunately for a segment of our users, there was no institutional clarity about how sorting should work with spreadsheets 18:53:34 * colonelqubit wishes that JBF was here to give more of a summary on that 18:53:51 * colonelqubit thanks beluga_ for all the help :-) 18:54:01 colonel qubit: I am sure that the test cases would have been forthcoming if we had realized the importance of what we were going to change. 18:54:30 TerrenceEnger: oh, I definitely agree -- it's hard to know what's the big critical change vs. what's much smaller 18:54:35 So, looking forward: How can we know when we are getting into deep water? 18:54:48 It's a really tough question 18:55:14 We certainly don't have complete coverage of Calc or any other LibreOffice component with tests, nor will we ever be perfect there 18:56:14 I have never heard of *user* meetings. Is ther such a thing? Perhaps a special interest group within an organization of accountants, or something? 18:56:16 TerrenceEnger: Do you think that some users would be willing to use a dev/beta version earlier in the release cycle? 18:57:23 That would certainly help us! What can we offer the user in return? 18:57:26 enoki: what do you think? 18:58:22 As I think back to my professional programming work, I am aware that a friendly user has often saved my bacon. Of course, I tried really hard to make sure that the relationship was a two-way street. 18:59:14 TerrenceEnger: triage their bugs first? 18:59:42 So put an emphasis on bugs reported against master, then alpha/betas, then going back in time 19:00:01 Bravo! I like that idea. To whom can we best propose it. 19:00:28 the one issue there is that we encourage people to file bugs against the earliest version we can repro 19:01:04 user feedback is very little( in japan). 19:01:37 enoki: okay, that's fair 19:01:55 With bibisect, is not a report against any version in the bibisect range about as useful as any other version? 19:02:37 #idea To find bugs that affect groups of users earlier, put an emphasis on bugs reported against master, then alpha/betas, then going back in time 19:03:16 TerrenceEnger: I believe so, but in that case we're finding when we introduced the bug 19:03:38 a bibisect result doesn't tell a dev if the bug is still in master 19:04:22 enoki: Do many people in Japan test the beta and rc builds? 19:04:44 Hmm. What we need is not more bug reports, but review by somebody with knowledge to say "Hold on a minute!" 19:05:19 TerrenceEnger: no one person is knowledgeable about all of LibreOffice 19:05:39 and has an understanding of our our users apply LibreOffice to their data 19:05:40 The daily dbgutil bibisect repository is usually within 24 hours of master. Close enough for horeshoes. 19:05:47 * colonelqubit agrees 19:07:09 Re "no one person ..." Yes indeed. And the person who most *uses* an application within LibreOffice is already busy using it. 19:07:09 I think that statistics are our friend here 19:07:11 colonelqubit: no. 19:07:29 in general, all QA team should have the knowledge to keep their eyes peeled for suspicious propositions - I mean these demands with potential to break things are enhancement requests basically? New team members should become conscious of the importance of this 19:07:32 i will try test events in study meeting. 19:07:52 enoki: yeah, I don't know a lot of people in the US who test with dev builds that much either 19:08:34 beluga_: In the calc sorting example, we had a bug report 19:08:56 that said "Behavior xyz is a bug" 19:09:37 I think for the most part, people agree on how the mechanics of these tools should work 19:09:41 beluga_: As I said, I had the opportunity to foresee the Calc problem, bug I wrote it off as *my* confusion. Sigh. 19:09:43 colonelqubit: well that includes knowing when something is an enhancement request/request to change a behavior.. so more caution with setting those as NEW 19:09:59 TerrenceEnger: JBF alerted about it in the summer 19:10:07 there was plenty of time to stop it 19:10:12 it seems to be tested after release at many user organizations. 19:10:22 it's just that there are some corner cases that aren't as visible/understandable as we'd like them to be 19:10:52 in my categorized list of "bugs that are difficult to the QA team" I have a category called Change a default behavior 19:11:06 and those are *not* marked as enhancements.. 19:11:19 7 of them in there 19:11:19 Okay, so I do want to get to some other stuff in the meeting. Do we have an action item here? 19:11:46 I don't think we can have an action when it's about diffusion of knowledge 19:12:16 beluga_: you did suggest better identification of when an enhancement request will change existing behavior 19:12:17 Well, "be careful" is hardly an action item. Big Sigh. 19:12:55 we can't have an action: "be very knowledgeable about all the things in LibO so we can prevent disasters" 19:13:00 I have actually spent quite some time thinking, and I have no concrete suggestions. 19:13:36 beluga_: Yes. And while you are at it, "be perfect" too. 19:13:42 ;-) 19:13:44 if we are smart, we can build a healthy environment where smart things and decisions occur naturally 19:14:04 I think my initial ideas stand 19:14:40 reducing our 'incoming' box so that we can identify dupes 19:14:53 get more people (real users) testing our versions earlier-on 19:15:46 I'm not sure that unit tests would have helped us in this situation, as the 'correct' behavior wasn't clear 19:16:07 yeah they would have been considered broken tests :) 19:16:22 larger numbers of test documents aren't a bad thing 19:16:59 Can we expect our approved service providers to give feedback based on their knowledge of there customers' workflows? 19:17:04 This is why I encourage bug reporters to attach an example file, even if the steps to create a document that exercises a bug are pretty simple 19:17:57 TerrenceEnger: I think it's in their best interests to do so, but that's definitely up to them 19:18:48 Or poll those providers: what organization do the heavy users belong to? Hope to find at least one organization per application. 19:18:48 Unfortunately people show up to test when stuff breaks 19:19:06 it's hard to gather this info remotely 19:19:11 Then approach the organization for help. 19:19:34 same problem with reproducing bugs that occur in some environment that we don't have access to 19:19:49 For example, empathy is failing to reconnect for me after I resume my laptop, so I'm much more likely to test it out now. We need to get ahead of that curve 19:20:47 Okay, I think we have some good ideas. Let's move on to some other topics for a bit, and we can chat more about this later :-) 19:21:02 yeah I have some less gloomy topics :) 19:21:11 ... Speaking of bugs that are important to address, does anyone have something for me to take to the ESC? 19:21:13 beluga: Yeah! 19:21:28 * colonelqubit wants to head-off future problems 19:21:45 any idea of some "pastebin for html" where I could share my categorized list of difficult bugs? 19:22:24 you can use etherpad 19:22:50 ok tdf's pad 19:22:55 The wiki does not use html, but otherwise seems like the natural place. 19:23:15 http://pad.documentfoundation.org/p/qa 19:23:23 beluga_: we have a pad for qa 19:23:24 Do you mean difficult to triage or difficult to fix? 19:23:37 to triage 19:23:51 beluga_: if it's a search term, you can just paste the url here in the chat so it goes into the minutes 19:24:17 but can I paste html in there? 19:24:35 beluga: No. 19:24:43 ok then I won't do it 19:25:07 * colonelqubit isn't sure where the html came from 19:25:15 hmm maybe I should just use a service like jsfiddle 19:25:16 did you write it by hand, or is this generated from some tool? 19:25:53 I copied & pasted the titles of the bugs by and when the list was done, I regexed the Bug xxx to be links 19:26:12 then ran through a text to html converter to add paragraphs and breaks.. just very simple stuff 19:26:18 to make it look decent 19:26:27 beluga_: okay. I think the wiki is a good place for stuff like that to live 19:27:10 If it's more of a personal list, feel free to put it on https://wiki.documentfoundation.org/User:Beluga/some/page 19:27:55 That way you can use wiki-syntax for the markup. Also make sure to check out the {{fdo|1234}} template, etc... 19:28:29 yeah that could be for later 19:28:34 * colonelqubit nods 19:28:49 The thing I like about the wiki is that infra keeps it running 19:29:23 That ensures that whatever data we store today is going to be around tomorrow and a while from now :-) 19:29:46 ok here we are http://jsfiddle.net/61o8vyay/ 19:30:14 that list represents an analysis of all our unconfirmed bugs 19:30:24 there are 272 bugs in that list 19:30:33 The category "Not sure how to proceed" contains a lot of bugs that could be closed after a review, but I'm not confident enough to be the closer. 19:30:38 huge list! 19:30:43 I have added some notes (in parentheses) to the end of some of the bug titles. For example, which distro, which language.. 19:30:54 (or if it has a 34 essay included) 19:31:00 34 page I mean 19:31:28 ;-) yeah, I think we've all run into that kind of bug report 19:32:38 if put in the wiki, the list could become a regular feature 19:32:40 News from asklibo: LibreOffice looks just plain ugly 19:34:23 beluga_: so how can we work with you on the list of UNCONFIRMED bugs? 19:34:57 I think that keeping track of which bugs require specific repro environments is an important sorting step 19:35:02 colonelqubit: I think we will figure it out in time :) You don't have to work with me, we can have ideas of how to tackle them based on the categories 19:35:17 so I hope it will be an inspiration basically 19:35:24 * colonelqubit nods 19:35:28 Sure, that sounds great 19:36:27 other stuff I have to bring up: Julien would like the QA team to communicate other achievements and goals than unconfirmeds all the time 19:36:43 beluga_: I don't know how much you nudge individuals in the team about helping to repro certain bugs, but that's a good way to use our resources 19:36:58 I nudge Jay about Arabic stuff all the time :) 19:37:22 I don't always remember who has what capabilities (system, skills), so I use https://wiki.documentfoundation.org/QA/Team as a reference 19:37:55 related to the communication: there's the annual report to fill and we should somehow put into song and verse all the stuff that happened this year 19:38:29 Definitely -- I encourage everyone here and everyone not on the call to list the things they've done this year :-) 19:38:42 It's great for us to have a record of the accomplishments 19:38:57 should we list them straight to the pad or somewhere else as a rough sketch to work on? 19:39:07 the annual report pad I mean 19:39:16 http://pad.documentfoundation.org/p/annualreport 19:39:40 beluga_: I think listing them in the pad is fine. If anyone has specifics they'd like to discuss about QA, feel free to post to the qa list 19:40:34 Our second-most important task is keeping the developers happy. But we cannot get feedback on that for an "accomplishment" without taking developers away from what they want to be doing. 19:40:44 here is Julien's post about communication: http://lists.freedesktop.org/archives/libreoffice/2014-November/064624.html 19:41:19 So I think there are a number of different things we can focus on 19:41:51 I think I have written how our upcoming own bugzilla could make such feedback easy. 19:41:51 I've encourage people to help with regressions, work on some bibisecting, etc.. 19:42:14 * colonelqubit nods 19:42:46 I'll reiterate my invitation for people to start thinking/planning some improvements for Bugzilla 19:43:06 it's good to push newbies to improve their triaging.. the feedback I've received in this channel has done it for me 19:43:33 improvements to the source code will (in general) not be merged-in until after the migration, but we can start brainstorming and even writing code in a separate branch now, if people want to get started 19:44:32 Rob (ertai_NL) has some work to pull out stats from Bugzilla. I think he'd be happy to chat with people on that topic 19:45:41 enoki: Any topics you'd like to bring up for the meeting? 19:46:31 Question for the ESC: how often do the developers feel QA could have done something to avoid wasting their time? 19:46:54 Are there patterns in the wastage that we can fix? 19:46:55 I have a question. 19:47:02 I will use Moztrap in BugHunting Session.(in real Events) 19:47:05 TerrenceEnger: that's perhaps a loaded question :-) 19:47:24 4.4.0 beta setup is not complete? 19:47:28 Yes. It would take some courage to ask it. 19:47:34 enoki: Let's see 19:47:35 in moztrap. 19:47:58 Unfortunately sophie isn't here in IRC right now 19:48:27 One action item we have is pinging >= two year old NEW bugs per tommy's suggestion. Is jmadero afk or present? 19:48:52 There are only 266 NEW bugs that are 2 year old and not enhancements, easyhacks or meta. 19:49:11 enoki: Have you chatted w/sophie about Moztrap? 19:50:41 hi guys 19:50:49 I do not 19:51:01 tommy27: hi we were just talking about the pinging. Here is the search query for those 2 year olds: https://bugs.freedesktop.org/buglist.cgi?bug_status=NEW&chfieldto=2012-11-19&f1=bug_severity&f2=status_whiteboard&f3=short_desc&list_id=493769&o1=notsubstring&o2=notsubstring&o3=notsubstring&product=LibreOffice&query_format=advanced&v1=enhancement&v2=easy&v3=[meta 19:51:01 I'm still in time 4 the QA call or it's alredy over? 19:51:17 or I was talking about the pinging and no one replied yet :) 19:51:19 beluga_: I thought we had some reservations about the bulk-ping idea (from a don't-harass-the-devs perspective and from a technical filter in/out perspective) 19:51:43 colonelqubit: jmadero is ok with the NEW ping with filters like in that query 19:51:43 enoki: okay, why don't you email sophie. Feel free to cc me :-) 19:51:47 yes let's take easy hacks out of it 19:51:58 jmadero should do the pinging in any case 19:52:17 Well we can use the 'QA Admins' account for that kind of mass-ping 19:52:29 at one point I was considering to give those 266 bugs a quick once over, but I guess if they get changed to needinfo anyways, I won't bother 19:52:32 * colonelqubit wonders what the collateral damage will be in emails 19:52:59 we should not turn them under NEEDINFO 19:53:32 yeah, unless we review them individually and ask (or confirm that we're still waiting) for new information, I'd avoid bulk-updating the status in that fashion 19:53:42 tommy27: ok then that complicates things a bit.. because then the old bugs get turned into "recently changed" ones and we just sit on our hands and wait, if the reporters respond 19:53:43 let's only ask to retest and tell if they become WORKSFORME during LibO developemtn 19:53:59 if we receive no feedback we leave thte status to NEW 19:54:06 colonelqubit: thanks, i will writing email to sophie. 19:54:28 we change status only to WFM if the users say the bug is gone 19:54:43 beluga_: if a bug is marked as NEW, it's possible that we can identify and fix it without any further interaction from the OP 19:54:59 tommy27: the option I considered was that I would test all the ones that are testable and comment/change status accordingly and then we could ping the ones that remain 19:55:03 * colonelqubit puts on his structured data hat 19:55:27 what would be slick would be if we could keep track of the versions of LO against which we've tested a bug 19:55:50 I could test those 266 bugs before December 19:55:52 ...which we'll work on implementing after migration :-) 19:56:19 ok but the aim of the ping is to ask users to do the retest to avoid QA having to amnually retest all them 19:56:46 tommy27: so what's the long-term plan? 19:57:03 let's say we ping 300 bugs... 19:57:06 Re "if the users say": I would think anybody who has reproduced the bug should be able to set RESOLVED WFM. Then it is up to the reporter to set VERIFIED WFM. 19:57:09 tommy27: yeah, but we risk losing the possiblity of closing more bugs as WFM by waiting for confirmation that never happens 19:57:42 based on my FILEOPEN bug tests, it is possible that 10-20% of those oldies can be closed as WFM 19:57:52 we ping them and if we receidve no feedback after a week we go on restesting manulaly one by one 19:58:03 TerrenceEnger: usually, I'd say anyone with a very similar setup (OS/version) can say something is WORKSFORME 19:58:15 especially if we have 2 people confirming the result 19:58:54 the aim of the ping is to get rid of as many hidden WFM oldies so we don't have to retest ourselves 19:59:00 tommy27: ok I guess it's pretty easy to keep track of them as the pinging happens at once :) so just save the date and get back to those NEWz 19:59:10 TerrenceEnger: so you asked earlier "how can QA help devs avoid wasting time?" 19:59:51 As long as we're cautious about how we do it, I agree with tommy27 that getting users to retest their own bugs is a really effective use of our time 20:00:03 Yes, wasting as in "I spent 5 minutes reading this report, only to decide that I am completely uninterested in it." 20:00:25 tommy27: I think it's important to have that testing strategy in place.. that we commit to testing by ourselves in a week (maybe two weeks is better :)) 20:00:39 Bibisecting more of the outstanding bugs is another big win for us 20:01:26 until a Windows bibisect tool will be available I can't help with that 20:01:30 I'm working on the bibisect server this week, so we can have more complete coverage through 4.4 20:01:44 tommy27: have you tried a virtual machine with Linux? 20:02:11 sorry i have too few spare time to start learing linux 20:02:17 I'm using an Ubuntu VM in my work Win 7 laptop and it's quite powerful for testing 20:02:28 colonelqubit: re "big win": I think I remember a dev saying that a bibisect was "excessively useful". I try to remember that when I get discouraged. 20:02:51 tommy27: ok, but a VM is a very safe way to learn it :) so you don't mess up your computer 20:02:51 tommy27: Perhaps I can whip together a new screencast demonstrating how to use bibisect in a VM 20:02:57 News from asklibo: Duplicate embedded images wasting space  || Different language 20:03:28 Re Windows bibisect: How many Windosws-only bugs do we have? 20:03:41 actually I do some kind of rudimental bibisect using the portbale versions from Winpenpack from here: http://sourceforge.net/projects/winpenpack/files/X-LibreOffice/releases/ 20:03:42 TerrenceEnger: yes, indeed. The close we can get to identifying a single commit that caused a regression, the sooner the devs can focus-in on the problem 20:04:08 TerrenceEnger: it's hard to track windows-only bugs as users set the OS field themselves 20:04:30 I leave the room for a few minutes. i'll be back 20:05:03 And they are quite right to do so. We cannot query the distinction "windows only" vs. "only observed on Win". 20:05:24 TerrenceEnger: Well, that's where I think stats + repro results come in handy 20:06:13 if we can repro on Win7 but not OSX 10.8 or Ubuntu 14.04, then chances are good it's a Windows-only bug 20:06:30 Okay, I see by the clock that we're pushing an hour and a half here. It's been a great meeting, but I don't want to keep y'all much longer 20:06:41 Yes. 20:06:45 One task suggested by jmadero was: Going through all NEEDINFO bugs to see, if the reporter forgot to change to UNCONFIRMED. I'd add that one should test, if the bug is actual. I guess this would be a good task for a newbie member. 20:06:47 Anyone have anything else that they want to bring up? 20:07:31 * colonelqubit nods -- Yes, users often do not change the status back from NEEDINFO 20:08:12 colonelqubit: jmadero also brought up in the mailing list that we should evaluate enhancement request.. I started thinking that maybe in the long term we should go over the NEW enhancements and see, if there are any silly requests (or those potentially disrupting ones we discussed earlier) 20:08:29 * colonelqubit hmms 20:08:59 I do think that evaluating enhancements is an eventual goal, but I think that (like enhancements themselves) it's generally a lower priority 20:09:15 in any case, my next bug snacks will be of this kind: going through old open bugs to see, if some of them can be closed as WFM 20:09:47 I'll start after the mass pinging :) 20:09:50 I still think that UNCONFIRMED bugs are high priority, because I really want us to be able to get to newly-reported bugs within 1-2 weeks, ideally 1 week 20:10:40 That will both help us identify potential breakage (e.g. spreadsheet sorting) as well as make sure that we connect with bug reporters before they lose interest in the bug and/or LibreOffice 20:10:44 IMO we need a couple more regulars 20:11:06 maybe increased communication would help attract more members 20:11:08 it's always nice to have more help 20:11:21 yet the team is quite healthy as it is 20:12:18 I trust the colonel to be our recruiting officer 20:12:39 :-) 20:12:43 We'll see what I can do 20:14:34 Have a great week, guys! We've got Thanksgiving next week in the US, so Thurs-Sat I might be busy with feasting and family, but will try to get back to emails as soon as possible :-) 20:15:02 Thanks. Enjoy the turkey. 20:15:16 Keep the good ideas coming, recruit your friends to join us, and beluga: Enjoy your bug snacks -- we love what you're doing :-) 20:15:24 * colonelqubit waves 20:15:28 bye 20:15:37 enoki: thanks for participating this week! 20:15:54 scientific definition of bug snax: http://en.wikipedia.org/wiki/Entomophagy 20:16:09 #endmeeting