My Ode to Twitter

One thousand, nine hundred and seventy tweets. That’s my entire tweet history, fourteen years of my digital life (more like the first seven until I stopped tweeting, around 2016). That tiny number doesn’t adequately convey what Twitter meant to me, the relationships it created, and the opportunities I received. If I were to write all those words collected in a single story, they wouldn’t have the same impact. Their power came from the changes that elapsed between them in perspective, and in time. 

I joined in 2009, when I got my first smartphone. It was kind of exhilarating to be able to send your thoughts into the ether and see someone respond in real time, even random people you’d never met. Circles of friends joined and we’d all banter, or send a funny photo to elicit a response. I began to attend a lot of tech conferences, and I’d tweet my thoughts or reactions while I was there. As someone who benefits from time to think before responding, the asynchronous and flat nature of twitter was helpful to join the conversation. The limit of 140 characters forced me to be succinct and thoughtful at the same time. I loved that the bar was low; you didn’t have to write an essay.

Twitter was my community. I joked with my colleagues while sitting next to them. I connected with other female engineers as we supported and cheered each other on. As a minority group in tech, that support was something I didn’t realise I was missing until I’d found it, completely by accident.

Twitter was my tech library. A place I found tweets, curated and vetted by others, that linked to amazing resources, blogs to engineering culture, and particularly other phenomenal women in tech. Or tragic stories showing the ways women were being failed or ignored. I curated a collection of those articles as a reference to give to others, when the inevitable “why aren’t there more women in tech?” question came up. It was inspiring to watch people claim their voices and be part of it in real time. 

Twitter was a curated stream of consciousness; a place to share things too good to be kept to myself. Some stunning pictures of places I’d visited, or a particularly delicious cake. A one liner that I knew would make a specific person laugh when they read it. A place to explain my weird Australian perspective on British or American things.

Twitter was my expanded CV. My tech tweets connected me to authors, speakers, and prospective employers. I learned about new technologies and upcoming events. I linked to my projects, my talks, my podcasts; I could retweet others’ praise or feedback to validate my perspective. That track record was enough to open doors to me, later in my career – people offered me jobs, invited me to conferences, asked me to connect them to people, or wanted to discuss collaborations. (And then I went to work for Amazon, where it all came to a halt, but that’s another story).

I feel a surprising pang thinking about Twitter going away, even though I don’t actively post on it today. There is more than a decade of me catalogued – a lot of my professional growth, and serendipitous connections. Even now, I can’t get some of that back. People have deleted their own tweet histories – I acknowledge that going back 10+ years is a long time to preserve – but also, Twitter, after being horrendously gutted in the last few months, seems on a slow slide towards the garbage heap.

I hope that Twitter survives, but if it doesn’t, I want to acknowledge what it’s meant to me. Thanks for everything.

Six Months in Seattle

Edit: I found this in my drafts folder after six years (!) I did some minor edits, but it’s otherwise a snapshot of our lives in mid-March, 2017, and it made me laugh as well as brought up a bunch of nostalgia.

Yes, I can’t believe it’s been six months already!

There’s a lot to process when you move countries. Even simple things, like going to the supermarket (“grocery store”) and getting some takeaway food (“takeout”) can be a challenge, because you have no point of reference on what’s good or bad, or even the correct terms to communicate (“they use the word entree for the main meal?!”)

If you’re Australian, and you’ve thought about moving to Seattle for Amazon, you’ve probably run into Diary of a Pampered Housewife. If not, she’s written an awesome guide to Aussies relocating to Seattle and if you’re thinking about moving here, you should go read it.

Weather

There’s no getting around it: Seattle is pretty grey. It doesn’t rain all the time, but the clouds have been here since October. Also, they (whoever “they” are) told us it doesn’t rain very heavily, so you can get away with not needing an umbrella. They were lying.

People tend to walk around with waterproof shells – think North Face, Marmot, Columbia, etc – to keep the water out. This works to a point, but if you’re trying to protect a laptop on your back, you need an umbrella or rain cover for your bag, too.

Waterproof shoes are also a key to success. Somehow, you can tackle anything when your feet are dry. We became friends with REI.

I own an ultra light down coat from Uniqlo that squashes down to nothing. I love it, it’s vaguely water resistant, and it keeps me warm through anything from -5 to 15 degrees (that’s 22 to 59 in foreign-speak) using additional layers when it’s below 5.  I supplement it with a Columbia rain jacket or an umbrella when it’s really heavy.

When it’s raining solidly for a whole month, like what happened in October 2016, even an umbrella and wellingtons won’t save you.

Fashion

The outdoors and the rain make Seattle a very casual place, which suits me and my capsule-like wardrobe just fine. I have ditched all shoes that have heels on them. I have four “active” pairs of shoes, a far cry from the 50+ pairs I used to own ten years ago.

Trail shoes are pretty normal around the office. A lot of restaurants and bars are super casual. I don’t feel weird carrying my backpack around. I’ve refactored my handbag (“purse”) into pockets on my jacket, plus my phone with credit cards. I no longer have that annoying problem when you switch handbags to suit an outfit and forget to transfer everything over.

Money

US dollars aren’t very accessible, in the making-it-easy-to-use-for-everyone sense. Every note is the same size, and the same colour. I get confused carrying a wad of monotonous green notes around, so I solve the problem by not using any at all. Everything goes on my card.

On the one hand, I love it because I don’t have to carry any physical money around. On the other hand, America is so far backward that I can’t believe I’m even going to type this out. We pay for our rent by cheque. Yes, you read that correctly. Cheques, in 2017. But at least I don’t have to write it out every month, because our “internet banking” service automatically writes the physical cheque for you and physically mails it to the person you want to pay.

…I know, right?

Also, unlike most modern economies, you can’t transfer random amounts of money to other people’s bank accounts, so people start using third party services. I guess you win some, lose some.

Outdoors

If you like the outdoors, Seattle is a great place to be. It’s surrounded by mountains, which are nice to hike on in summer, and all that rain gets converted into a lovely dusting of snow for winter.

We can drive to several snow fields that are 1-2 hours away, and have done so multiple times since December. Since we’re so close to snow, it opens up a lot more possibilities for snow activities besides just downhill sports. We’ve been snowboarding, cross-country skiing, snowshoeing, and just plain hiking on snow. It’s all beautiful.

In Autumn, the trees start changing colour, the air is crisp, and it’s fresh. I like that I can get to this less than an hour from my house. In Sydney, after an hour of driving, I’m probably still somewhere on Parramatta Road.

The latitude of Seattle means that it gets more sunlight than London does, so standing in the sunlight in February actually feels warm. And it’s further west in its time zone, so in February it’s still light at 6pm. There’s a lot you can do with that.

We’ve been told the summer here is amazing – reliable dry, clear weather. Can’t wait 🙂

Transport

When we moved here, I had very firm visions of continuing to catch public transport. I’d embraced the habit in London, and we lived near a train line in Sydney. I hate driving, and I hate traffic, so public transport is a win-win. Seattle is also a liberal city as US cities go, so I figured the transport would be reasonable.

However, Seattle doesn’t have a lot of different types of public transport. Most of the city relies on buses. There is one rail line, but it doesn’t go near to work. Unfortunately, buses get affected by road traffic, so timetables are a bit fictional. The buses are also crowded. It’s not uncommon to watch a bus or two pass your stop in the evening. Anytime between 4:45pm and 6:30pm is a royal pain to get home.

After one month of constant rain (the wettest October on record), followed by the coldest December on record, we capitulated. We started driving our car to work.

Food

The pacific northwest has a reputation for some good local produce. It’s around, but compared to the quality of food you can get in Sydney, it’s just not the same. You will, in fact, be disappointed if you randomly stop in somewhere. We’ve found food to be very salty and very “blunt”, for lack of a better word. There’s usually one overwhelming flavour profile – e.g. sweet or smoky – and a lack of any other dimension.

I miss being able to stop into a random cafe in Sydney and trust I’d get a decent meal. We’ve spent a lot more time cooking food ourselves, and researching any restaurants before we drop in.

Culture

It’s a small-big city, which means you get all the amenities of a city, but the perks of less crowds. Parking is easy. Getting tickets or going to restaurants is generally easy. There’s space to breathe.

If you’re a nerdy introvert, Seattle is pretty awesome. There’s a big board game culture, including game cafes which let you play games for free. Since I’ve passed the stage of getting trashed on a Saturday night, the games are fun. And cheaper than alcohol, and open until 11pm on a Saturday night. I love it!

The city has a reputation for the Seattle Freeze. I am unsure if it’s an American thing in comparison (i.e. Americans are overwhelmingly open and chatty people by default, so the relative introvert-ness of Seattle is confronting) but we haven’t found it particularly freeze-like, but we hang around with a lot of foreigners, since Amazon is full of them. We do, however, notice the distinct lack of swearing at work. Also, people don’t really do drinks or much after-work socialising. Drinks are over in an hour (only an hour?!). We’re also not sure if this is an American thing, or an Amazon thing. More research to come.

Miscellaneous

The amount of junk mail here is ridiculous. There’s some kind of weird scenario where the US postal service is an enabler of junk mail, because it’s one of their main sources of income. It’s incredibly screwed up.

Americans are also big subscribers of things. People are always asking if you’ve joined their club, so they can send you a 0.1% discount and a message on your birthday every year, in exchange for mining your purchasing history and selling your data to broker. Thanks, but I’m good.

Donating money to an organisation means you’re suddenly on their email and physical junk mail lists, with no way to opt out (yes, I am looking at you Planned Parenthood – way to say thanks). It’s made me think twice about who to give my money/data to in future.

Diversity at NDC London

Jakob BradfordJon Skeet, Chris O’Dell and I have exchanged a number of emails, Google Hangouts, and suggestions in the past few months that could be used to improve diversity at NDC conferences. I was a speaker at NDC Sydney, and it was one of the most heavily gender skewed events I had attended in years.

Ultimately, NDC London 2017 ended up at 23% female speakers. That’s a great result compared to the gender diversity numbers at the Sydney NDC conference. I am keen to see if the attendee ratio changes as a result. NDC have also promised a blog post on the history of their diversity numbers, and what their plans are for the future to continue to improve.

For other reasons, I’m not involved with NDC any more, but I wanted to say congrats to the London committee, and especially to Chris O’Dell, my ex 7digital colleague and friend, who helped to shape that agenda. She’ll be speaking on How to get your submission accepted at NDC London.

The Agenda is out – NDC London 2017 from NDC Conferences on Vimeo.

NDC Sydney

NDC Sydney wrapped up last month, and because it happened between a bunch of other things (preparing for our move to Seattle, recording the podcast with Scott Hanselman, a conference call with Jon Skeet & Jakob Bradford – post coming soon!), I hadn’t take the time to digest it all.

My talk

First, my talk – Video Transcoding at Scale for ABC iview. It was the last rendition of this talk I’d be giving, especially since I’ve already left the ABC. I’m glad I got the chance to present it one last time. I’ve written about my general experience with public speaking, and lessons learned. Here are the slides:

It was so nice to see feedback from my talk on Twitter 🙂

Other talks

A lot of the content at NDC was quite Microsoft heavy (not surprising), however I really enjoyed some alternative talks that weren’t about MS tech.

Lynn Langit’s session on Teaching Kids Programming was a highlight! Having done a lot of work writing tutorials before, and running sessions for people to learn, I understand how much work has been put in to produce the course materials. It was so inspiring hearing what the kids can do, and seeing some examples.

Lars Klint’s talk on Mastering Your Inner Developer was a really personal story.  Judging by twitter traffic, a lot of people really loved the session.

Tess Ferrandez’s a Developer’s Guide to the UX Galaxy was also really interesting. What are the tenets of design that we should be aware of, and what happens when they go wrong? She walked us through Microsoft’s 12 UX guidelines. Unfortunately I can’t find a link to the cards she was using, but it was a great checklist.

Niall Merrigan’s session on black hat tools was eye opening. I learned that all devices on the internet can be scanned in 2m30s if you have enough machines to do it. So, don’t rely on security through obscurity.

I met some pretty awesome people, and got selfies

Yes, that’s right, I totally did.

From left to right:

  • Scott Hanselman
  • Niall Connaughton – my other half who was also speaking at NDC
  • Tess Ferrandez
  • Troy Hunt
  • Jon Skeet – note the #HeForShe shirt! I love, love, loved that he publicly called out for better diversity.

Diversity

As I mentioned above, Jon Skeet publicly issued a challenge to improve diversity, both with a #HeForShe shirt, and a mention at the beginning of both his talks. It was my favourite moment of the conference. ❤

Subsequent discussions led to a conference call with Jon and Jakob Bradford (one of the NDC organisers) to discuss strategies to increase diversity at future events. More on that in a separate post shortly.

Organisation & Social events

The most challenging aspect is seven speaker tracks, which makes it difficult to pick which session you go to. (And as a speaker, knowing you’re up against 6 other people!)

Having said that, NDC is very well organised. Some really big name speakers, there’s plenty of space in the agenda to mingle, professional AV and video recording, and the social events – a boat cruise, drinks, and lightning talk PubConf – are well catered. Love the conference swag too – a black hoodie that I’m actually using!

Looking forward to next time, and keen to see next year’s speaker lineups!

My podcast with Scott Hanselman

Yes, that’s right – I recorded a podcast with Scott Hanselman 😀 There’s a story behind it too…

I bumped into Scott at NDC Sydney. It was the evening before the main conference started, and we were both helping out with the Kids Code Club. We started chatting about what I was presenting on Thursday – video transcoding systems, cloud services, various price points, why our new system was worth building.

Scott: “Hey, would you like to do a podcast? It’s just a really casual chat, just like this.”
Me: “Sure! Sounds good.”

Just to make sure I was prepared, I wrote some notes about video transcoding, AWS services, and reviewed bits we’d built in our system. A couple of weeks later, we caught up via Skype:

Scott: “Great, thanks for doing this! It’s usually best to talk about something you’re really passionate about. What would that be?”
Me: “Um…” thinking: video transcoding is pretty awesome, but it’s not my #1 super-fun-happy-thing. But now I have to come up with thirty minutes of content I haven’t prepared…

I’m not really good at impromptu. That’s why formats like my blog (where I can draft things), and public talks (where I can prepare), work okay for me. So doing a thirty minute discussion flying blind was a terrifying prospect.

However, Scott was really, really supportive. After some back and forth, we came up with a blend of topics that include continuous learning, becoming more social, finding your tribe, and making it work for you. You can listen to the podcast here.

I’ve got a healthy respect for Scott’s work now that I’ve seen how much time, effort and support he puts in to the things he does.

If you haven’t heard Hanselminutes before, give it a try. There’s some great slices of developer life in there. And now I’m one of them. 🙂

 

Lessons learned in Public Speaking

Every year, I try to learn something new. 2014 was javascript, 2015 was back to CS fundamentals, and 2016 is my Year Of Public Speaking.

I’ve been lucky to be able to present the same talk multiple times this year, which is an overview of the new video transcoding system we built at the ABC:

  1. Internal presentation at the ABC (March)
  2. Public presentation at ABC’s tech talk night (March)
  3. YOW! Night talk in Brisbane (April)
  4. YOW! Night talk in Melbourne (April)
  5. YOW! West Conference in Perth (May)
  6. YOW! Night talk in Sydney (May)
  7. NDC Sydney (August)

Each time I presented it, something changed before I gave the next version. The talk has evolved a lot during these months, and I wanted to share some of the lessons I learned doing multiple renditions, and public speaking in general.

Invest time in your talk abstract

Put the time in to write your abstract well. it lives everywhere, it sells the talk before you arrive, it entices people to attend. It’s your advertising. Don’t neglect it. I wish I’d done mine better in early versions. I was pretty happy with the last one at NDC Sydney.

Show it to someone and ask for feedback. Would they attend your session? Does it accurately represent your talk?

Sarah Mei has done a great breakdown of what your conference proposal is missing. It’s also good to trawl through conference agendas and see which proposals interest you, and why – take notes!

Say less than you think you should

If you’ve done some research, then heed the advice you read everywhere: leave space to breathe, and say less. Craft your message around a central point and chop out the excess, no matter how interesting.

Humans are creatures of relativity; we forget how much we know about something, given that you’ve probably spent weeks, months, or even years on it.

The audience doesn’t have the benefit of those months or years of background understanding the topic like you do. They won’t absorb everything. Nor will you be able to cram everything from one year into one hour.

What’s the most important thing you want to say? Or the top 3 things? Select content to include to support your message, rather than selecting things to exclude. You’ll have a more coherent message, and the audience will feel less overwhelmed. Don’t be afraid to repeat things often.

Craft your story

Read some advice on how to construct a narrative, and tell a story. Don’t just list the facts – make it something they can get involved in.

There are a few story “archetypes” detailed in the TED Talks Official Guide to Public Speaking that I found useful.  An example story archetype is introducing a challenge, and then unmasking step-by-step how you solved it. The audience gets to play along, and “solve” the challenge with you, which leads to more engagement.

Ultimately, as Zach Holman says, talks are entertainment.

Devote time to your slides

A well-designed slide set can support and reinforce your message in amazing ways. Great slides are really valuable. I believe they’re worth as much time as you can spare, and they can give you more confidence if you’re a newer (or nervous) public speaker. For example, you can use funny images to inject humour if you’re not normally great with joke delivery. The slides help you cheat. The better you are at public speaking, the less you need the slides as backup.

Here’s an example of using slides to reinforce your message. It’s much nicer to see these 3 slides in sequence, while you explain what’s happening – we take a watermark, make it 30% transparent, then resize it:

output_vddsYs

And here are the 3 separate slides:

Compare those visual ones to a text slide:

watermark-text

Sure, the second slide takes less work to create. But the first set of three look prettier. They easily convey your point, and nobody has to read text to digest it. If your audience understands what you’re saying – through word, or through visuals – then you’ve done your job well 🙂

Learn some public speaking techniques

Invest in some classes, read some books, or watch videos to learn about public speaking techniques.

This can include a training course, (I was lucky be part of YOW’s Women in Tech comp in 2015, and absolutely encourage you to enter if you are a woman in Australia), but I have also since done a course with Public Speaking For Life in Sydney, which focuses on more general delivery techniques. I found both very valuable, and PSL offer multiple courses throughout the year.

I highly recommend watching Damian Conway’s Instantly Better Presentations on how to deliver a good technical presentation. He specifically covers technical issues, like demonstrating code. His intensive public speaking courses are great.

The recent TED Talks book is an easy read, and Amy Cuddy’s TED talk on body language was really helpful.

Another great thing to do is to watch other people deliver talks, and see which ones you found compelling. What did they do to make you feel engaged?

Practice. Lots.

Although I ran through my hour long presentation multiple times, it’s way too much content to know off the bat. But the more I ran through it, the more comfortable I got with the content, and the less nervous I would feel on stage (this was the theory).

This Wait But Why article explains the science of different levels of memorisation (with graphs, and everything!) and is seriously entertaining to boot.

A few techniques I used to familiarise myself with the talk:

  • Recorded my voice rehearsing various sections, a) standing up and b) including intonations and pauses and then played it back to myself on the way to work, or at lunch.
  • Rehearsed in a similar position to the real talk – Standing up, not facing my computer, etc.
  • Bought a clicker and practiced with it.
  • Checked the speaker notes were visible, and easy to focus on.
  • Recorded myself to see if I should adjust anything with body language or gestures.
  • Got someone to watch me in practice, and welcomed their feedback. I did this multiple times.

My story

A few years ago, I’d never want to stand up in front of an audience and talk about something. I didn’t feel qualified enough, or interesting enough.

I’m really proud that I got to the stage of doing all those talks, and how they turned out. The talk feedback has been great, and it’s fun being a speaker. People are interested in what you have to say. You also get to be in situations you wouldn’t be otherwise, like chatting to Tess Ferrandez, whose debugging tutorials I used years ago.

However, the reality is that someone else always pushed me to try. Dave Thomas, who organises YOW, repeatedly encouraged the Women in Tech competition speakers to submit talks for proposals. Sam Newman told me I should submit to NDC Sydney.

My next goal? Submit a talk somewhere, and get it accepted, without someone pushing me to do it. And to get my fellow women in tech up and talking too. 😀

 

Presenting at YOW! Nights in April

Following on from last night’s presentation at the ABC on our new transcoding service, Metro (which went really well!) I’m excited to announce that I’ll be presenting the same content in Brisbane, Melbourne and Sydney, thanks to YOW!

Metro currently transcodes all the content for ABC iview and has successfully processed thousands of pieces of content since launch in December 2015. It’s built using Node.js, Go, FFmpeg and various AWS services such as queues, notifications, autoscaling groups, and a hosted database.  If you’re curious to learn more, come along!

I’ll be presenting on the following dates:

If you’d like a sneak peek, slides from last night’s presentation are underneath – but obviously it’s going to be much better in person 😉

Hope to see you there!

“Developer-esque” relations

It’s not technically my official job, but I do some “developer-esque” relations for the ABC’s Digital Network division. I started doing it because I like it, and it’s been very educational.

To clarify, the phrase  “developer relations” could be interpreted in a couple of ways:

  1. Your company has products or services it sells, and you’re trying to help people in the community to make better use of them, to encourage new users, and to get an idea about future features that people are seeking. The end goal is to get more people using your product, and be happier while they’re doing so.
  2. Your company is trying to be more open about the way it builds things, and to build some community. You’re sharing knowledge about your internal processes and decisions, mistakes you’ve made, and what’s coming up in the future. The end goal is to let people find out more about your environment, get people interested in the products you build, exchange ideas, and perhaps even entice some future employees to join you.

My “developer-esque” relations work at the ABC falls firmly in the second camp, and it includes work over the last year such as GovHack 2015, our hackathons, and a new tech talk series we’re starting up. You can sign up here! The first talk is on our internal transcoding system, Metro.

I’m also pretty excited about the latest endeavour, the launch of the ABC developer blog developers.digital.abc.net.au 🙂

The ABC has some really interesting products that millions of people use daily. However, when it comes to who, how, or why we build those products, there is zero external visibility. If you were trying to find information about the ABC’s development team, there really wasn’t anything to see. In an age where almost every company has engineering blogs, talks at conferences or events, and has a community presence, we were falling desperately behind.

The lack of information correlates to a difficulty in attracting good candidates. When you have choices, why would you choose the place that you know the least about?

So the theory is that by sharing more, we hope to get more people in the door. I’m pretty excited to be part of that. If you are too, then please join us.

Metro: the ABC’s new Media Transcoding Pipeline

In December last year, the ABC launched a new video encoding system called Metro (“Media Transcoder”), which converts various sources of media into a standardised format for iview.

It’s been a fantastic project for the ABC’s Digital Network division – we’ve built a cheap, scalable, cloud-based solution that we can customise to suit our specific needs.

Metro has been live for a month, successfully transcoding thousands of pieces of content. Here’s an overview of how it’s been designed and what it does.

Background

Our previous transcoding system had a fairly straightforward job: produce the same set of renditions for each piece of content it was given. Both input and output files were fairly standardised. The previous system was convenient, but there were some aspects we couldn’t customise, and we didn’t use its #1 proposition: on-demand transcoding. Most of the content the ABC publishes is available to us days in advance, so we just need to make sure that it’s transcoded before it’s scheduled for release online.

We calculated that we could replace the existing system for less than the previous system cost, and take advantage of AWS services and their scalability. Other systems like the BBC’s Video Factory have been successfully built using the same model. Writing our own system would allow us to start batching up jobs to process in bulk, or use different sized instances to help reduce costs in the long term.

Our first step was to replicate what the existing system did, but allow it to scale when needed, and shut down when there’s nothing to do.

Architecture

Metro is a workflow pipeline that takes advantage of queuesautoscaling compute groups, a managed database, and notifications. Logically, the pipeline follows this series of steps: File upload > Queue Job > Transcode > Transfer to CDN > Notify client

 

transcodearchitecture

The pipeline is coordinated by the “Orchestrator”, an API written in node.js that understands the sequence of steps, enqueues messages, talks to our database, and tracks where each job is in the system. It’s also responsible for scaling the number of transcoding boxes that are processing our content.

Each step in our pipeline is processed by a small, isolated program written in Golang (a “queue listener”), or a simple bash script that knows only about its piece of the pipeline.

We are able to deploy each piece independently, which allows us to make incremental changes to any of the queue listeners, or to the Orchestrator.

Interesting bits

Autoscaling the Transcoders

The transcoders are the most expensive part of our system. They’re the beefiest boxes in the architecture (higher CPU = faster transcode), and we run a variable number of them throughout the day, depending on how much content is queued.

Before a piece of content is uploaded, we check to see how many idle transcoders are available. If there are no spare transcoders, we decide how many new ones to start up based on the transcoding profile. Higher bitrate outputs get one transcoder each; lower bitrates and smaller files might share one transcoder over four renditions. Once we process everything in the queue, we shut down all the transcoders so that we’re not spending money keeping idle boxes running.

Here’s a snapshot of the runtime stats (in minutes) on boxes over a 4 hour window:

ec2_transcoders2

There’s definitely some optimisation we can do with our host runtime. In future, we’d like to optimise the running time of our transcoders so that they run for a full hour, to match Amazon’s billing cycle of one hour blocks. We’d also like to take advantage of Amazon’s spot instances – using cheaper computing time overnight to process jobs in bulk.

FFmpeg

FFmpeg is the transcoding software we use on our transcoders. It’s open source, well maintained, and has an impressive list of features. We’re using it to encode our content in various bitrates, resize content, and add watermarks. We create an AMI that includes a precompiled version of FFmpeg as well as our transcoder app, so that it’s ready to go when we spin up a new box.

There’s still a way to go before we’re using FFmpeg to its full extent. It’s capable of breaking a file into even chunks, which would make it perfect to farm out to multiple transcoders, and likely giving us even faster, consistent results every time. We can also get progress alerts and partial file download (e.g taking the audio track only, avoiding downloading a bunch of video information that you won’t use).

SQS Queues

We utilise SQS queues to keep our pipeline resilient. We’ve got different queues for various step in our system, and each queue has a small app monitoring it.

When a new message arrives, the app takes the message off the queue and starts working. If an error occurs, the app cancels its processing work and puts the message back at the head of the queue, so that another worker can pick it up.

If a message is retried a number of times without success, it ends up in a “Dead Letter Queue” for failed messages, and we get notified.

Things seem to be working well so far, but we’d like to change the queues so that consumers continually confirm they’re working on each message, rather than farming out the message and waiting until a timeout before another consumer can pick it up.

In Production

Metro has been transcoding for a month, and is doing well. Our orchestrator dashboard shows all of the jobs and renditions in progress:

orchestrator2-small

And some of the work done by transcoders in a 4 hour window:

transcode_chart

The Future

We have more features to add, such as extracting captions, using cheaper computing hardware in non-peak times, and building priority/non-priority pipelines so that content can be ready at appropriate times. Metro has been really interesting to build, much cheaper than our previous solution, and we can customise features to suit our needs. I’m really looking forward to where it goes next.