Diversity at NDC London

Jakob BradfordJon Skeet, Chris O’Dell and I have exchanged a number of emails, Google Hangouts, and suggestions in the past few months that could be used to improve diversity at NDC conferences. I was a speaker at NDC Sydney, and it was one of the most heavily gender skewed events I had attended in years.

Ultimately, NDC London 2017 ended up at 23% female speakers. That’s a great result compared to the gender diversity numbers at the Sydney NDC conference. I am keen to see if the attendee ratio changes as a result. NDC have also promised a blog post on the history of their diversity numbers, and what their plans are for the future to continue to improve.

For other reasons, I’m not involved with NDC any more, but I wanted to say congrats to the London committee, and especially to Chris O’Dell, my ex 7digital colleague and friend, who helped to shape that agenda. She’ll be speaking on How to get your submission accepted at NDC London.

The Agenda is out – NDC London 2017 from NDC Conferences on Vimeo.

NDC Sydney

NDC Sydney wrapped up last month, and because it happened between a bunch of other things (preparing for our move to Seattle, recording the podcast with Scott Hanselman, a conference call with Jon Skeet & Jakob Bradford – post coming soon!), I hadn’t take the time to digest it all.

My talk

First, my talk – Video Transcoding at Scale for ABC iview. It was the last rendition of this talk I’d be giving, especially since I’ve already left the ABC. I’m glad I got the chance to present it one last time. I’ve written about my general experience with public speaking, and lessons learned. Here are the slides:

It was so nice to see feedback from my talk on Twitter 🙂

Other talks

A lot of the content at NDC was quite Microsoft heavy (not surprising), however I really enjoyed some alternative talks that weren’t about MS tech.

Lynn Langit’s session on Teaching Kids Programming was a highlight! Having done a lot of work writing tutorials before, and running sessions for people to learn, I understand how much work has been put in to produce the course materials. It was so inspiring hearing what the kids can do, and seeing some examples.

Lars Klint’s talk on Mastering Your Inner Developer was a really personal story.  Judging by twitter traffic, a lot of people really loved the session.

Tess Ferrandez’s a Developer’s Guide to the UX Galaxy was also really interesting. What are the tenets of design that we should be aware of, and what happens when they go wrong? She walked us through Microsoft’s 12 UX guidelines. Unfortunately I can’t find a link to the cards she was using, but it was a great checklist.

Niall Merrigan’s session on black hat tools was eye opening. I learned that all devices on the internet can be scanned in 2m30s if you have enough machines to do it. So, don’t rely on security through obscurity.

I met some pretty awesome people, and got selfies

Yes, that’s right, I totally did.

From left to right:

  • Scott Hanselman
  • Niall Connaughton – my other half who was also speaking at NDC
  • Tess Ferrandez
  • Troy Hunt
  • Jon Skeet – note the #HeForShe shirt! I love, love, loved that he publicly called out for better diversity.

Diversity

As I mentioned above, Jon Skeet publicly issued a challenge to improve diversity, both with a #HeForShe shirt, and a mention at the beginning of both his talks. It was my favourite moment of the conference. ❤

Subsequent discussions led to a conference call with Jon and Jakob Bradford (one of the NDC organisers) to discuss strategies to increase diversity at future events. More on that in a separate post shortly.

Organisation & Social events

The most challenging aspect is seven speaker tracks, which makes it difficult to pick which session you go to. (And as a speaker, knowing you’re up against 6 other people!)

Having said that, NDC is very well organised. Some really big name speakers, there’s plenty of space in the agenda to mingle, professional AV and video recording, and the social events – a boat cruise, drinks, and lightning talk PubConf – are well catered. Love the conference swag too – a black hoodie that I’m actually using!

Looking forward to next time, and keen to see next year’s speaker lineups!

My podcast with Scott Hanselman

Yes, that’s right – I recorded a podcast with Scott Hanselman 😀 There’s a story behind it too…

I bumped into Scott at NDC Sydney. It was the evening before the main conference started, and we were both helping out with the Kids Code Club. We started chatting about what I was presenting on Thursday – video transcoding systems, cloud services, various price points, why our new system was worth building.

Scott: “Hey, would you like to do a podcast? It’s just a really casual chat, just like this.”
Me: “Sure! Sounds good.”

Just to make sure I was prepared, I wrote some notes about video transcoding, AWS services, and reviewed bits we’d built in our system. A couple of weeks later, we caught up via Skype:

Scott: “Great, thanks for doing this! It’s usually best to talk about something you’re really passionate about. What would that be?”
Me: “Um…” thinking: video transcoding is pretty awesome, but it’s not my #1 super-fun-happy-thing. But now I have to come up with thirty minutes of content I haven’t prepared…

I’m not really good at impromptu. That’s why formats like my blog (where I can draft things), and public talks (where I can prepare), work okay for me. So doing a thirty minute discussion flying blind was a terrifying prospect.

However, Scott was really, really supportive. After some back and forth, we came up with a blend of topics that include continuous learning, becoming more social, finding your tribe, and making it work for you. You can listen to the podcast here.

I’ve got a healthy respect for Scott’s work now that I’ve seen how much time, effort and support he puts in to the things he does.

If you haven’t heard Hanselminutes before, give it a try. There’s some great slices of developer life in there. And now I’m one of them. 🙂

 

Lessons learned in Public Speaking

Every year, I try to learn something new. 2014 was javascript, 2015 was back to CS fundamentals, and 2016 is my Year Of Public Speaking.

I’ve been lucky to be able to present the same talk multiple times this year, which is an overview of the new video transcoding system we built at the ABC:

  1. Internal presentation at the ABC (March)
  2. Public presentation at ABC’s tech talk night (March)
  3. YOW! Night talk in Brisbane (April)
  4. YOW! Night talk in Melbourne (April)
  5. YOW! West Conference in Perth (May)
  6. YOW! Night talk in Sydney (May)
  7. NDC Sydney (August)

Each time I presented it, something changed before I gave the next version. The talk has evolved a lot during these months, and I wanted to share some of the lessons I learned doing multiple renditions, and public speaking in general.

Invest time in your talk abstract

Put the time in to write your abstract well. it lives everywhere, it sells the talk before you arrive, it entices people to attend. It’s your advertising. Don’t neglect it. I wish I’d done mine better in early versions. I was pretty happy with the last one at NDC Sydney.

Show it to someone and ask for feedback. Would they attend your session? Does it accurately represent your talk?

Sarah Mei has done a great breakdown of what your conference proposal is missing. It’s also good to trawl through conference agendas and see which proposals interest you, and why – take notes!

Say less than you think you should

If you’ve done some research, then heed the advice you read everywhere: leave space to breathe, and say less. Craft your message around a central point and chop out the excess, no matter how interesting.

Humans are creatures of relativity; we forget how much we know about something, given that you’ve probably spent weeks, months, or even years on it.

The audience doesn’t have the benefit of those months or years of background understanding the topic like you do. They won’t absorb everything. Nor will you be able to cram everything from one year into one hour.

What’s the most important thing you want to say? Or the top 3 things? Select content to include to support your message, rather than selecting things to exclude. You’ll have a more coherent message, and the audience will feel less overwhelmed. Don’t be afraid to repeat things often.

Craft your story

Read some advice on how to construct a narrative, and tell a story. Don’t just list the facts – make it something they can get involved in.

There are a few story “archetypes” detailed in the TED Talks Official Guide to Public Speaking that I found useful.  An example story archetype is introducing a challenge, and then unmasking step-by-step how you solved it. The audience gets to play along, and “solve” the challenge with you, which leads to more engagement.

Ultimately, as Zach Holman says, talks are entertainment.

Devote time to your slides

A well-designed slide set can support and reinforce your message in amazing ways. Great slides are really valuable. I believe they’re worth as much time as you can spare, and they can give you more confidence if you’re a newer (or nervous) public speaker. For example, you can use funny images to inject humour if you’re not normally great with joke delivery. The slides help you cheat. The better you are at public speaking, the less you need the slides as backup.

Here’s an example of using slides to reinforce your message. It’s much nicer to see these 3 slides in sequence, while you explain what’s happening – we take a watermark, make it 30% transparent, then resize it:

output_vddsYs

And here are the 3 separate slides:

Compare those visual ones to a text slide:

watermark-text

Sure, the second slide takes less work to create. But the first set of three look prettier. They easily convey your point, and nobody has to read text to digest it. If your audience understands what you’re saying – through word, or through visuals – then you’ve done your job well 🙂

Learn some public speaking techniques

Invest in some classes, read some books, or watch videos to learn about public speaking techniques.

This can include a training course, (I was lucky be part of YOW’s Women in Tech comp in 2015, and absolutely encourage you to enter if you are a woman in Australia), but I have also since done a course with Public Speaking For Life in Sydney, which focuses on more general delivery techniques. I found both very valuable, and PSL offer multiple courses throughout the year.

I highly recommend watching Damian Conway’s Instantly Better Presentations on how to deliver a good technical presentation. He specifically covers technical issues, like demonstrating code. His intensive public speaking courses are great.

The recent TED Talks book is an easy read, and Amy Cuddy’s TED talk on body language was really helpful.

Another great thing to do is to watch other people deliver talks, and see which ones you found compelling. What did they do to make you feel engaged?

Practice. Lots.

Although I ran through my hour long presentation multiple times, it’s way too much content to know off the bat. But the more I ran through it, the more comfortable I got with the content, and the less nervous I would feel on stage (this was the theory).

This Wait But Why article explains the science of different levels of memorisation (with graphs, and everything!) and is seriously entertaining to boot.

A few techniques I used to familiarise myself with the talk:

  • Recorded my voice rehearsing various sections, a) standing up and b) including intonations and pauses and then played it back to myself on the way to work, or at lunch.
  • Rehearsed in a similar position to the real talk – Standing up, not facing my computer, etc.
  • Bought a clicker and practiced with it.
  • Checked the speaker notes were visible, and easy to focus on.
  • Recorded myself to see if I should adjust anything with body language or gestures.
  • Got someone to watch me in practice, and welcomed their feedback. I did this multiple times.

My story

A few years ago, I’d never want to stand up in front of an audience and talk about something. I didn’t feel qualified enough, or interesting enough.

I’m really proud that I got to the stage of doing all those talks, and how they turned out. The talk feedback has been great, and it’s fun being a speaker. People are interested in what you have to say. You also get to be in situations you wouldn’t be otherwise, like chatting to Tess Ferrandez, whose debugging tutorials I used years ago.

However, the reality is that someone else always pushed me to try. Dave Thomas, who organises YOW, repeatedly encouraged the Women in Tech competition speakers to submit talks for proposals. Sam Newman told me I should submit to NDC Sydney.

My next goal? Submit a talk somewhere, and get it accepted, without someone pushing me to do it. And to get my fellow women in tech up and talking too. 😀

 

Presenting at YOW! Nights in April

Following on from last night’s presentation at the ABC on our new transcoding service, Metro (which went really well!) I’m excited to announce that I’ll be presenting the same content in Brisbane, Melbourne and Sydney, thanks to YOW!

Metro currently transcodes all the content for ABC iview and has successfully processed thousands of pieces of content since launch in December 2015. It’s built using Node.js, Go, FFmpeg and various AWS services such as queues, notifications, autoscaling groups, and a hosted database.  If you’re curious to learn more, come along!

I’ll be presenting on the following dates:

If you’d like a sneak peek, slides from last night’s presentation are underneath – but obviously it’s going to be much better in person 😉

Hope to see you there!

“Developer-esque” relations

It’s not technically my official job, but I do some “developer-esque” relations for the ABC’s Digital Network division. I started doing it because I like it, and it’s been very educational.

To clarify, the phrase  “developer relations” could be interpreted in a couple of ways:

  1. Your company has products or services it sells, and you’re trying to help people in the community to make better use of them, to encourage new users, and to get an idea about future features that people are seeking. The end goal is to get more people using your product, and be happier while they’re doing so.
  2. Your company is trying to be more open about the way it builds things, and to build some community. You’re sharing knowledge about your internal processes and decisions, mistakes you’ve made, and what’s coming up in the future. The end goal is to let people find out more about your environment, get people interested in the products you build, exchange ideas, and perhaps even entice some future employees to join you.

My “developer-esque” relations work at the ABC falls firmly in the second camp, and it includes work over the last year such as GovHack 2015, our hackathons, and a new tech talk series we’re starting up. You can sign up here! The first talk is on our internal transcoding system, Metro.

I’m also pretty excited about the latest endeavour, the launch of the ABC developer blog developers.digital.abc.net.au 🙂

The ABC has some really interesting products that millions of people use daily. However, when it comes to who, how, or why we build those products, there is zero external visibility. If you were trying to find information about the ABC’s development team, there really wasn’t anything to see. In an age where almost every company has engineering blogs, talks at conferences or events, and has a community presence, we were falling desperately behind.

The lack of information correlates to a difficulty in attracting good candidates. When you have choices, why would you choose the place that you know the least about?

So the theory is that by sharing more, we hope to get more people in the door. I’m pretty excited to be part of that. If you are too, then please join us.

Metro: the ABC’s new Media Transcoding Pipeline

In December last year, the ABC launched a new video encoding system called Metro (“Media Transcoder”), which converts various sources of media into a standardised format for iview.

It’s been a fantastic project for the ABC’s Digital Network division – we’ve built a cheap, scalable, cloud-based solution that we can customise to suit our specific needs.

Metro has been live for a month, successfully transcoding thousands of pieces of content. Here’s an overview of how it’s been designed and what it does.

Background

Our previous transcoding system had a fairly straightforward job: produce the same set of renditions for each piece of content it was given. Both input and output files were fairly standardised. The previous system was convenient, but there were some aspects we couldn’t customise, and we didn’t use its #1 proposition: on-demand transcoding. Most of the content the ABC publishes is available to us days in advance, so we just need to make sure that it’s transcoded before it’s scheduled for release online.

We calculated that we could replace the existing system for less than the previous system cost, and take advantage of AWS services and their scalability. Other systems like the BBC’s Video Factory have been successfully built using the same model. Writing our own system would allow us to start batching up jobs to process in bulk, or use different sized instances to help reduce costs in the long term.

Our first step was to replicate what the existing system did, but allow it to scale when needed, and shut down when there’s nothing to do.

Architecture

Metro is a workflow pipeline that takes advantage of queuesautoscaling compute groups, a managed database, and notifications. Logically, the pipeline follows this series of steps: File upload > Queue Job > Transcode > Transfer to CDN > Notify client

 

transcodearchitecture

The pipeline is coordinated by the “Orchestrator”, an API written in node.js that understands the sequence of steps, enqueues messages, talks to our database, and tracks where each job is in the system. It’s also responsible for scaling the number of transcoding boxes that are processing our content.

Each step in our pipeline is processed by a small, isolated program written in Golang (a “queue listener”), or a simple bash script that knows only about its piece of the pipeline.

We are able to deploy each piece independently, which allows us to make incremental changes to any of the queue listeners, or to the Orchestrator.

Interesting bits

Autoscaling the Transcoders

The transcoders are the most expensive part of our system. They’re the beefiest boxes in the architecture (higher CPU = faster transcode), and we run a variable number of them throughout the day, depending on how much content is queued.

Before a piece of content is uploaded, we check to see how many idle transcoders are available. If there are no spare transcoders, we decide how many new ones to start up based on the transcoding profile. Higher bitrate outputs get one transcoder each; lower bitrates and smaller files might share one transcoder over four renditions. Once we process everything in the queue, we shut down all the transcoders so that we’re not spending money keeping idle boxes running.

Here’s a snapshot of the runtime stats (in minutes) on boxes over a 4 hour window:

ec2_transcoders2

There’s definitely some optimisation we can do with our host runtime. In future, we’d like to optimise the running time of our transcoders so that they run for a full hour, to match Amazon’s billing cycle of one hour blocks. We’d also like to take advantage of Amazon’s spot instances – using cheaper computing time overnight to process jobs in bulk.

FFmpeg

FFmpeg is the transcoding software we use on our transcoders. It’s open source, well maintained, and has an impressive list of features. We’re using it to encode our content in various bitrates, resize content, and add watermarks. We create an AMI that includes a precompiled version of FFmpeg as well as our transcoder app, so that it’s ready to go when we spin up a new box.

There’s still a way to go before we’re using FFmpeg to its full extent. It’s capable of breaking a file into even chunks, which would make it perfect to farm out to multiple transcoders, and likely giving us even faster, consistent results every time. We can also get progress alerts and partial file download (e.g taking the audio track only, avoiding downloading a bunch of video information that you won’t use).

SQS Queues

We utilise SQS queues to keep our pipeline resilient. We’ve got different queues for various step in our system, and each queue has a small app monitoring it.

When a new message arrives, the app takes the message off the queue and starts working. If an error occurs, the app cancels its processing work and puts the message back at the head of the queue, so that another worker can pick it up.

If a message is retried a number of times without success, it ends up in a “Dead Letter Queue” for failed messages, and we get notified.

Things seem to be working well so far, but we’d like to change the queues so that consumers continually confirm they’re working on each message, rather than farming out the message and waiting until a timeout before another consumer can pick it up.

In Production

Metro has been transcoding for a month, and is doing well. Our orchestrator dashboard shows all of the jobs and renditions in progress:

orchestrator2-small

And some of the work done by transcoders in a 4 hour window:

transcode_chart

The Future

We have more features to add, such as extracting captions, using cheaper computing hardware in non-peak times, and building priority/non-priority pipelines so that content can be ready at appropriate times. Metro has been really interesting to build, much cheaper than our previous solution, and we can customise features to suit our needs. I’m really looking forward to where it goes next.