18:00:52 #startmeeting 18:00:52 Meeting started Thu Jul 9 18:00:52 2020 UTC. The chair is paddatrapper. Information about MeetBot at https://wiki.debian.org/MeetBot. 18:00:52 Useful Commands: #action #agreed #help #info #idea #link #topic. 18:01:03 #topic Roll call 18:01:08 0/ 18:01:08 hi 18:01:15 * olasd does a barrel roll (call) 18:01:23 Agenda: http://deb.li/oNCD 18:02:36 #topic Actions from last meeting 18:03:24 #link http://meetbot.debian.net/debconf-video/2020/debconf-video.2020-07-02-17.58.html 18:03:37 o/ 18:03:39 tumbleweed: thanks, was struggling to find it 18:04:03 um, what happened to meetbot? 18:04:11 nattie and I haven't done the presenter docs yet 18:04:11 deaded for now 18:04:19 mmkay 18:04:43 I've started working on the voctoweb ansible role: https://salsa.debian.org/debconf-video-team/ansible/-/merge_requests/205 18:04:57 I should be able to have something I think is mergeable today 18:05:04 cool 18:05:08 #action nattie to edit and compile presenter advice to video-team docs 18:05:12 #action paddatrapper to edit and compile presenter advice to video-team docs 18:05:31 #info pollo working on ansible voctoweb role 18:05:35 paddatrapper: fyi, you can also say "#action foo and bar to xyz", and it shows up as an action for both ;-) 18:05:44 wouter: TIL thanks! 18:05:52 eww php 18:05:57 yup 18:06:02 and it's not pretty php either 18:06:03 #action paddatrapper to setup a Jitsi VM and add to ansible 18:06:17 considering the code it'd take all of 10 minutes to re-do it in a sensible language 18:06:37 yeah, definitely room for improvement there 18:06:47 but working is the first important step 18:06:52 :) 18:06:54 agreed 18:06:55 olasd: have you had a chance to work on reducing the stream delay? 18:06:55 well, it works, I don't think we need to reimplement everything first? ;) 18:07:10 paddatrapper: as I said last time I'll work on that during the sprint 18:07:14 so, no 18:07:19 ah right, yes 18:07:40 #action olasd to try minimize stream delay during the sprint 18:07:56 I poked at OBS, and got it working 18:08:04 I haven't worked on the geoip things this week 18:08:04 (well technically I guess the sprint started, I merged a pollo MR:P) 18:08:10 hehe 18:08:11 #info tumbleweed managed to get OBS working 18:08:38 #action tumbleweed to find a solution to the geoip issue 18:08:54 pollo: did you look at the audio sync, or was that also sprint? 18:09:00 nope 18:09:08 I also experimented a little more with android + OBS locally, and got some pretty low-latency results (under 1s) 18:09:15 #action olasd to add a list of available recordings to voctoweb 18:09:27 tumbleweed: android and OBS? 18:09:34 paddatrapper: android camera into OBS 18:09:41 #action pollo to test audio sync 18:09:42 tumbleweed: ah 18:10:26 lowest latency setup I've found is droidcam's mjpeg http stream. It puts a timestamp on the top of the video (I think to encourage you to buy the paid version), but one could overlay something else over that 18:10:53 and I didn't realise, but OBS can import the srt streams directly, making larix+OBS better than I'd thought before 18:11:01 anyway, enough about that :) 18:11:16 #info lowest latency setup tumbleweed found is droidcam's mjpeg http stream, but includes timestamp in the video 18:11:25 ooh that is cool 18:11:32 #action wouter to work on review for prerecorded talks 18:11:43 I actually did look at that 18:11:57 wouter: ah awesome 18:12:06 I think the best we can do is have Q&A and prerecorded things be separate events in SReview 18:12:19 then we can inject the prerecorded talks without review, and transcode them as is 18:12:37 and review the Q&A which we then release as a separate file 18:13:07 and potentially combine downstream of SReview? 18:13:13 anything else will require me to figure out a way to either stitch together two files that will not stitch together very well, or implement a full nonlinear editor 18:13:22 which I don't think is a good idea 18:13:27 eh, perhaps, we could do that too 18:13:31 but I don't think we need to 18:13:54 I think we should, IMO the Q&A is part of the talk 18:13:58 For YouTube and PeerTube it would be useful to have just one 18:14:02 file 18:14:07 and that 18:14:11 I don't think it's essential to have it as 1 file 18:14:16 okay, I'll see how to make that possible then 18:14:18 +1 on Q&A being part of the talk 18:14:21 context is important 18:14:22 is there more action items to review or should we move on to more specific topics? 18:14:50 although the easiest way to accomplish that is to just include the prerecorded as part of the review 18:15:02 #topic Blocked salsa issues 18:15:02 i.e., don't inject prerecorded stuff, just review as it was broadcast 18:15:09 olasd: nope :) 18:15:38 (well, I did have the VM usage action item, but I forgot that was even assigned to me, so didn't do much) 18:15:39 I think we covered the issues in the previous topic, anything in particular to raise here? 18:15:50 not from me 18:16:00 wouter: we did have the discussion with zigo after the meeting last week though 18:16:35 yes 18:16:43 #topic Video stack 18:16:47 what I'm saying is that I missed it was for me and I'm not sure we discussed everything 18:16:56 ah I see 18:17:15 \o/ 18:17:18 wb 18:17:22 somebody set up vocto, and I played with it a little, and was able to make it hang 18:17:32 hang, in what way? 18:17:38 it seems vocto is very peculiar about the exact format of the video coming in 18:17:38 and, how? 18:17:51 oh, that's not so good 18:17:52 the video I tried had different interleaving 18:17:53 are you talking about the interlaced example we spoke about here? 18:18:04 oh right, you did say that 18:18:07 that just needed to be deinterlaced before injection 18:18:09 and after that, that specific input didn't work anymore, until voctocore was restarted 18:18:17 tumbleweed: that's not the point 18:18:20 we could theoretically fix that by re-encoding everything 18:18:24 ivodd: no, it is the point 18:18:24 there might be other setting we don't think about 18:18:29 voctomix expects consistent inputs 18:18:37 if we build a system that gives it that, it's happy 18:18:39 I have a few scripts that use the SReview::Video API to ensure that things are encoded the same way, re-encoding if necessary 18:18:39 but as wouter said, I think we need to re-encode everyting beforehand 18:18:57 wouter: the video's I tried were all encoded with sreview in the past, so no consistency there either 18:19:01 I can set up something that will allow people to upload things and that re-encodes them ready for broadcast 18:19:02 not necessarily 18:19:16 ivodd: only because I have changed the output profiles a few times over the years 18:19:21 ivodd: that won't be the case for this system 18:19:26 what would be the impact on video quality of reenconding everything? 18:19:34 the problems you'd have to fix are identical for injecting or re-encoding? 18:19:40 s/?/./ 18:19:41 pollo: depends on the encoder settings 18:20:00 well, that's my main worry 18:20:03 pollo: if we set them high enough (which I intend to do) then there shouldn't be much loss 18:20:04 it would be good to have an automated way to test the re-encoded result as well, obviously 18:20:12 pollo: I'd expect negligable loss 18:20:18 We definitely ran into issues at SotM for not encoding 18:20:30 re-encoding I mean 18:20:34 I'll try to whip up something simple this weekend 18:20:35 paddatrapper: what did you use for SotM? vocto? 18:20:41 ivodd: OBS 18:20:47 why did OBS care? 18:20:50 if you rely on the file to be in a format, then you can't change at the last minute 18:20:52 where a speaker can upload a talk, and then gets an email when it's re-encoded to our settings 18:20:57 wouter: we actually also want review of the incoming talks, so maybe just use review :) 18:21:01 so they can test if everything looks fine 18:21:02 tumbleweed: multiple resoltions, formats, etc 18:21:04 or that :) 18:21:11 if fix the format on the fly, you get it right every time 18:21:14 ivodd: that's not a bad idea! 18:21:15 wouter: s/review/sreview 18:21:15 we definitely shouldn't support multiple resolutions 18:21:24 +1 18:21:32 CarlFK: no 18:21:35 tumbleweed: I agree the problems are the same between ingest and re-encode, but re-encoding everything to have a consistent input format to feed voctomix means we have consistent resource usage during the live event 18:21:42 CarlFK: you have to make sure you don't forget any setting that might be wrong 18:21:55 olasd: true. But at the same time, playback is so cheap, that doesn't worry me at all 18:22:01 tumbleweed: is it though? 18:22:14 in my experience, yes 18:22:20 my experience has been very different 18:22:22 anyway, as I was saying (before everyone started making suggestions): 18:22:33 ivodd: ingest.py asks the server for the current settings, so no human needs to figure it out 18:22:35 it depends on the file size and the codec, but H.264 is easy on today's hardware 18:22:43 (where "easy" means "cheap") 18:22:44 CarlFK: except ingest doesn't detect interlaced video 18:22:47 that's a bug, that we should fix 18:22:50 - vocto is difficult about input format, and we need to get it EXACTLY right, or we need to restart voctocore 18:23:01 - inputs coming and going don't seem to be a problem 18:23:14 ivodd: I can't imagine anything except interlacing causing that problem 18:23:15 CarlFK: also, H.264 has many many fiddly settings that can cause issues if you don't re-encode 18:23:19 * wouter found that out the hard way 18:23:21 CarlFK: I used ingest.py, and it doesn't work correctly in all cases 18:23:24 ivodd: it wouldn't even accept video in other resolutions 18:23:32 tumbleweed: I think I got it - want me to file an issue, or do you have details handy? 18:23:39 ivodd: so I hear ;) 18:23:39 ivodd: and there aren't that many other free parameters 18:23:54 CarlFK: interlaced video 18:24:01 tumbleweed: I have no idea, I'm just reporting what I found out 18:24:08 this is why I think re-encoding everything to *exactly* the same settings is the best issue 18:24:12 er, best solution I mean 18:24:20 then there won't be any problems 18:24:33 wouter: unless the settings change, then ka-boom 18:24:36 then there will be a single set of problems 18:24:41 well, don't do that then :) 18:24:49 wouter: that has the advantage that we can publish the result (without Q&A) immediately after the talk 18:24:52 wouter: assuming your re-encoding stack deinterlaces, in that case 18:24:53 olasd: true, yeah 18:25:02 tumbleweed: I can make it do that if necessary 18:25:03 a typical re-encoding would use the same interlacing/progressive on output, as input 18:25:10 we're likely to re-encode the videos we get from presenters anyway before releasing them right? 18:25:20 I don't think that's necessary 18:25:22 ffmpeg has options to force re-encoding, so that's not an issue 18:25:24 but other people here seem to 18:25:33 so, whatever :) 18:25:45 SotM feedback - pre-recorded talks and Jitsi Q&A worked well. We published a pad per session for questions, that worked better than filtering questions from IRC in my opinion 18:26:00 paddatrapper: that's a different topic 18:26:09 tumbleweed: if Q&A and prerecorded video is not encoded at the same settings, one of them will need to be re-encoded or you can't combine them 18:26:20 ivodd: we don't have an agenda point for yet 18:26:21 wouter: that's not what we're talking about, at all 18:26:25 s/yet/it/ 18:26:29 well, from all of this, I think it's more prudent to have presenter videos go by Sreviews, have them reviewed by a Human that can check for obvious mistakes, and re-encode them for release & playback 18:26:43 paddatrapper: i'm not saying it's a bad topic :) 18:27:53 tumbleweed: I thought it was, but meh, doesn't matter that much 18:27:57 that doesn't prevent us/CarlFK from hardening ingest against different interlacing settings too 18:27:57 pollo: sure, sounds reasonable. And I guess that makes the later combination wouter spoke of trivial 18:28:41 #action wouter to look at setting up SReview for pre-recorded talks 18:29:04 (although I'm not sure I can actually do that...) 18:29:19 #action wouter to look at setting up SReview for pre-recorded talks 18:29:32 no idea if you can, but makes sure :P 18:29:34 can't do the work? or can't tell the bot? 18:29:42 the latter :) 18:29:52 so, high level video stack 18:30:06 it sounds like no real arguments against voctomix at this point 18:30:26 have we tried feeding jibri output to voctomix yet? 18:30:28 OBS is still there as a fallback, but we're assuming vocto is the way to go 18:30:34 olasd: not yet 18:30:42 we need to get the jitsi up 18:30:50 I don't think anyone tested the audio sync either 18:30:54 I think we should test that ASAP, before deciding on whether or not to OBS? 18:30:55 Yeah I plan on getting that up this week 18:30:56 or maybe ivodd did? 18:31:16 we do have a sprint tomorrow+sat+sun :) 18:31:22 pollo: I was searching for a video where it would be easy to see the audio sync when I hit the lockup issue 18:31:31 so, no :( 18:31:38 good to know, thanks 18:31:46 pollo: BTW it was your talk that caused the lockup :) 18:31:51 (and olasd's) 18:31:52 IME audio sync was always a resource exhaustion issue on the voctocore; if we run into that without a voctogui, we should worry :P 18:32:12 pollo: I did test switching audio sources, and that just works as expected 18:32:21 yeah, me too 18:32:37 what will be different is switching audio sources. Not something we've ever needed before 18:32:50 yeah, audio source switching has always worked; we just disable the GUI to do that because we don't need it 18:32:52 but it works 18:33:17 also, the latency on the preview stream from here was acceptable 18:33:31 from here, too 18:33:33 (doesn't mean we shouldn't make sure it works within our setup, just that it shouldn't be an issue in general) 18:33:36 I didn't try the giu remotely 18:33:42 * tumbleweed did 18:33:50 but I only had 2 sources 18:34:09 if we have a few more (IRC, jitsi, gobby), it'll certainly add up 18:34:36 I think we should probably stress test with more at some point 18:35:04 that's this weekend :) 18:35:08 shall we move on? 18:35:13 yes, but we should also try to make sure we don't have too many sources 18:35:17 but we can move on 18:35:19 #topic Streaming setup 18:36:02 Don't think anything here is changing from last week? 18:36:12 so my plan for the weekend is to set up rtmp push and see where that gets us wrt. latency 18:36:27 (instead of pulling hls over https) 18:36:39 (instead or in addition of, actually) 18:38:51 anything to add to that? 18:39:16 and I'll finish the geoip rework 18:39:33 I guess not; we can move on 18:39:35 #topic Advice/training for directors 18:39:48 paddatrapper: I added your previous topic to the agenda before that 18:39:58 because we need to know how we do thing before we give advice to directors :) 18:40:22 I'll work on this after we have finalised the stack, so probably after the sprint 18:40:51 #topic How to run Q&A 18:41:24 SotM found direct access to pads for each session was quite nice 18:41:35 paddatrapper: can you explain the setup you had? 18:41:48 Easier than filtering from IRC and allowed answers to be recorded there too 18:42:01 the talkmeister/director and the presenter in jitsi and everyone else on the pad? 18:42:32 of was the pad managed by specific people 18:42:36 s/of/or 18:42:42 I think the main problem we have is how to capture the questions to get them into voctomix 18:42:47 ivodd: We had a etherpad for each talk which was linked from the talk detail page on the website (needs to be better published) 18:43:00 paddatrapper: and everyone could add questions to it? 18:43:03 the jank solution I worked on involves running IRC in the framebuffer and capturing that 18:43:05 They were then read out by the talk meister in the Jitsi session 18:43:08 ivodd: yes 18:43:14 ok 18:43:21 what happens at CCC (when there's lots of questions in irc): the signal angel (person responsible to relay questions) sends the URL of a pad to the audience; the audience and the signal angel work together on collating questions and ordering them by priority / interest; at Q&A time, the signal angel asks the questions in order 18:43:35 olasd: that sounds like the way to go 18:43:46 In the Jitsi session (1 per talk), there was the director, mixer (OBS output), presenter and talk meister 18:43:48 and the pad can be a source in vocto / jitsi too 18:44:15 the OBS output was there for allowing the talk meister and presenter to follow along with the pre-recorded talk 18:44:29 tumbleweed: what renders the pad text into video? 18:44:32 tumbleweed: by capture a browser? 18:44:40 yeah 18:44:57 paddatrapper: why did you need a director, a 'mixer' and a talk meister? 18:45:01 jibri seems to be doing that and is pretty heavy 18:45:13 ivodd: the mixer is the OBS output feed from the pre-recorded talk 18:45:34 oh, that's not a separate person, but a separate source in jitsi? 18:45:40 pollo: it'll be slightly better, without playing videos in the browser. But CPU usage is hardly a critical concern for us, here 18:45:45 the director is the one controlling Jibri and talk meister interacts with presenter 18:46:08 ivodd: yeah, but not intended for going out on air, rather as feedback for people in the Jitsi room 18:46:32 paddatrapper: ok, I was thinking about how many people we would need per talk, but you were talking about sth else :) 18:46:46 ivodd: ah, yeah we had 2 - director and talk meister 18:46:56 paddatrapper: and that worked ok? 18:47:13 wait 3 - technical assistant for helping the presenter before 18:47:39 ivodd: 3 worked very well, but then again we didn't have any presenter technical issues that I know of 18:47:48 ok, good to know 18:47:59 and did you have any live events/bofs? (apart from Q&A after a talk) 18:48:20 One talk that eventually was live because the pre-record just wouldn't play 18:48:37 welcome, closing and 1 quiz thing 18:48:47 they worked well with enough pre-testing 18:48:52 for bofs people usually use gobby, if we can't get a web client for that, we'll need to tell people to use $approved_webpad 18:49:21 err, I guess we can capture the gobby X11 window... 18:49:22 why would we need a web client? 18:50:09 we don't, I think pollo was confused :) 18:50:40 but setting those up will surely involve some clickedy-clicks via a VNC 18:50:58 yeah, but that's not a problem, is it? 18:51:07 it means director needs to have a VNC access 18:51:08 worst case, just have one of the presenters share gobby over jitsi/bri 18:51:27 pollo: seems reasonable to me 18:51:31 if the talkmeister could do that, I think it would be better 18:51:55 Given good enough internet connection, that would be good 18:52:24 you don't need a //good// internet connection to scroll down a static window full of text 18:52:36 olasd: but you need it to share it over Jitsi 18:52:46 (if we do it static over vnc) 18:52:48 if you're using a jitsi, and the talkmeister has a good connection, they can do this themselves 18:53:05 right 18:53:06 for live mixing in vocto, a capture from a local gobby / IRC client would probably be much better quality 18:53:20 and we don't need to have a whole VNC setup (and possibly a VNC web interface) for directors 18:53:26 would also ensure consistent look and feel 18:53:44 ^ a local capture in vocto 18:53:48 we should test things like that this weekend and see what works well 18:53:51 is anyone planning on trying the vnc stuff? 18:54:06 yeah 18:54:10 who? :P 18:54:18 (I'm up for playing/experimenting/testing over the weekend) 18:54:21 I'll poke at it 18:54:26 #agreed test sharing Gobby though Jitsi and capturing on voctomix during the sprint 18:54:37 if possible, using SPICE would be interesting 18:54:41 #action tumbleweed to poke and VNC on vocto machine 18:55:00 pollo: I've never used spice 18:55:20 whatever we use needs to have a display bitmap to grab and stream, server-side 18:55:23 it's much more efficient than vnc (closer to rdp) 18:55:24 tumbleweed: spice is more bandwidth-efficient 18:55:38 rigth, but RDP is not the right solution for this problem 18:55:43 not sure where spice fits on that spectrum 18:55:44 no, but spice can be :) 18:56:01 yeah but... *sigh* 18:56:04 it's used a lot in the qemu/kvm space 18:56:24 tumbleweed: there's the capture side (fed through to vocto), and the control side (which the talkmeister uses?) 18:56:36 spice/rdp/vnc are the control side 18:56:36 err 18:56:49 yes 18:56:59 if those are clearly separate then we're good 18:57:01 and there is https://packages.debian.org/buster/spice-html5 if we need to fallback to a webinterface 18:57:09 if the capture has to capture from the talkmeister's machine, we screwed up 18:57:19 tumbleweed: spice is like vnc but more clever in sending only bits that changed, and afaik if you do things like drag a window (or scroll in a webpage) it knows how to shift similar bits around better 18:57:21 tumbleweed: yeah, no, that wouldn't make sense 18:58:16 Anything else here? 18:58:25 spice also has features to send along alternate streams, which qemu uses to piggyback a USB channel -- not that that matters here ofc 18:58:36 (afaict we need 1/ a remote x11 server to run gobby / etherpad / whatnot; 2/ x11grab that through to vocto; 3/ have spice/rdp/vnc for the director to control that) 18:58:56 yeah, sounds about right 18:59:10 I think that can all live on the voctomix VM 18:59:26 maybe 18:59:39 it needs setting up, testing 18:59:43 given the massive VMs we have a tthe moment, it could 18:59:47 but I don't like that 18:59:58 I just mailed zigo to ask for more credits for smaller VMs 19:00:06 don't want to run out of credits on the weekend 19:00:06 +1 19:00:20 but I think he'll need to ask permission for that 19:00:31 hopefully he can tomorrow 19:00:33 that's why he wanted to stay under 4 19:00:39 I asked on IRC a few days ago, but didn't get a response 19:01:25 oh, I did. I missed that 19:01:26 worst case scenario we can spin up DO droplets? 19:01:40 yeah, or use personal accounts elsewhere 19:01:50 highvoltage seemed amenable to refunding reasonable expenses there 19:02:32 #action Advice/training for presenters 19:02:41 nattie and I will work on this tomorrow 19:02:47 (and even more amenable to setting it up long-term so that video team can spin up/down whenever needed and it gets billed to TO automatically) 19:02:48 #undo; #topic 19:02:58 #undo 19:02:58 Removing item from minutes: 19:03:03 urg thanks 19:03:09 :) 19:03:12 #topic Advice/training for presenters 19:03:16 I was confused what that was supposed to mean :) 19:03:25 too many actions this meeting 19:03:29 nattie and I will work on this tomorrow 19:03:34 yep 19:03:36 #topic Stats on talk submissions 19:03:56 the submission deadline has passed now 19:04:14 #link https://debconf20.debconf.org/talks/statistics/ 19:04:20 nice 19:04:34 will there be language tracks? 19:04:47 doesn't look like any happened 19:05:03 tumbleweed: do we know how many are bof-type events? 19:05:42 20 BoFs 19:05:54 that's quite a bit, I expected less of them 19:06:03 indeed 19:06:13 2 sprints (both pollo :P) 19:06:21 muahha 19:06:26 won't be videoed though 19:06:32 rejected! 19:06:55 it might be a good idea to have a way to switch jitsi from 720p to 480p for BoFs 19:07:11 especially for large ones 19:07:37 with that many bofs, we'll need to look into the details of the bof setup and do test runs 19:07:59 pollo: yeah 19:08:14 maybe we should ask people who want to join in a bof to let themselves known, so we can see whether much planning is necessary? 19:08:19 pollo: I seem to recall that requires a config change and a Jitsi restart... could be wrong though 19:08:25 but will have a look 19:08:28 paddatrapper: that's about it yes 19:08:30 during MDCO we had ~20 people in a room, which worked quite well 19:08:47 if we don't expect any bofs with more people (which seems likely) then there's not really an issue 19:08:52 paddatrapper: but it would be nice not to have to do it manually (and to forget it's at 480p when it shouldn't be) 19:08:57 #action paddatrapper to investigate easy switching of Jitsi from 720p to 480p for large bofs 19:09:03 pollo: +1 19:09:54 wouter: large jitsi rooms are hard on bandwidth and CPU for users 19:09:57 pollo: it's 480p now bit it wasn't for mdco 19:10:09 it's not really a problem on the server side 19:10:14 lazy answer could be 2 jitsis :P 19:10:17 pollo: hm, okay, that could make a difference 19:10:24 tumbleweed: and then have a link between them? ;) 19:10:36 no, send people to the right one for the event 19:10:38 (480p is actually ok for most of the webcam quality we end up getting, screen shares seem to be displayed at higher resolutions) 19:10:38 well each room requires a link anyway 19:10:52 #topic Any other business 19:10:55 #undo 19:10:55 Removing item from minutes: