Databases, Kindness, and Building a Learning Culture in Tech.

[Note: Abridged version lives here.]

Let me tell you about Gigi.

Now Gigi is a friend of mine who’s a systems engineer. She had the problem of being bitingly unhappy: and she didn’t know if it was the tempo of work that she had to do, or the fact that the demands of her job stopped her from making friends. She didn’t know if it was the isolation, or the exhausting politics, or the fact that while she was getting things done, she didn’t really get the time to celebrate her successes. And no one to celebrate them with.

She didn’t know if, maybe, she was just wired wrong.

Unhappiness can be cumulative — small things here and there that pile up until you reach a certain tipping point. It felt like the slow pile-up of dishes in her sink she didn’t have the time or energy to put away — until she reached a volume where she just couldn’t not do anything about it anymore. She had reached hers. Lost and rudderless, she went to her manager to see if he had any advice — halfway expecting that he’ll say, “Hey, you should take a vacation”.

Instead, the manager said, “You should try doing something for other people.”

Lacking any sort of life context outside of work, she decided to try and train other people in the organization. Now, she hated it at first: she hated the jitters of standing in front of people, she hated the questioning looks, she hated the nagging doubt that she wasn’t in a position to be teaching anyone anything.

Then, she slowly grew to like it. Hell, she even loved it at times.

Then, she forgot she was unhappy.

Teaching as Data Transfer.

We’ll get back to that story shortly. For now, I’m going to be talking about the various experiments I’ve done with regards to the propagation of a teaching culture within my current company. Before we go any further, I’d like to outline the frame by which I try to attack this problem: “What if teaching is just data transfer?”

You have Idea X which exists within the head of Person A, which needs to be transferred to Person B. In a way that preserves as much of the content as possible.

Now, if we’re thinking of teaching as data transfer, we have to think about the two ways that data is transferred: synchronously and asynchronously. As various definitions exist, I’m going to use the one that best fits with my current agenda (ha!) — synchronous operations block both processes until the operation finishes, asynchronous ones do not. To put it in human terms, the distinction between the two is the WAIT — blocking the teacher until the teachee has understood the material.

“Do you understand it yet? If not, we’re going to go through it again.”

Broadcasts, and Fan-out Training.

At some point in a company’s lifespan, they will start to think of teaching as a critical component of the system of work. Enable developers, make cheaper mistakes faster, all that fancy jazz. And when a company thinks of how to teach other engineers, they will frequently lean into the model that people are most familiar with: broadcasts.

Think a pub-sub model, or perhaps Amazon SNS — you have a standardized message, being fanned out to several subscribers and endpoints. When you publish a message to the topic, the subscribers then receive the message in a way that is specific to the protocol — HTTP endpoints get POST-ed to, email addresses get it via SMTP.

In very much the same way, a company will create a syllabus of content that is easily copied and reproduced, and that content is then broadcasted in batches. In my current company, this is done via an internal AWS training program — we take 25–30 developers every quarter, then sit with them for three days, going over various AWS products. Every now and then we’ll need to uplift the content, but this is a relatively cheap process we can deploy and scale quick.

Sounds good so far? Well, we have a couple of problems.

SPOF. Easiest to identify would be the fact that the program suffers if I get hit by a bus. Having inherited the program from (awesomer, and better) other people, it personally took me a long time before I got a handle of the tempo of work. Add the fact that teaching requires extensive knowledge of the material I had to deliver, which means I had to have the comprehensive lexical knowledge of three days worth of teaching material.

Inconsistency. Let’s say we had a hypothetical quiz to test knowledge retention at the end of the program. And because we’re all fantastic, smart people, let’s say everyone got 90% across the board (awesome!). My main point of interest is in the 10% that did not get retained — specifically, the fact that this will be different across everyone. Now this is where the analog of SNS falls apart: SNS tailors message delivery according to each individual recipient protocol, where individual lecturers cannot do that.

Anti-synchronicity. In-place lectures are frequently referred to as synchronous training — but for this specific case, I’d like to present a counterpoint: there is an immense social pressure that comes with not wanting to be the person who drags the class down. Synchronous processes block until the operation is finished. A trainee who is mindful of the fact that A.) The training program has a time limit, and B.) Other attendees may want to get to the next set of material will not raise their hands to say, “Hey, I don’t get this yet.”

WAIT calls don’t get issued. The class continues on.

Better Broadcasts.

Let’s focus on the first problem as the last two are endemic to the broadcast format. At least I can fix that.

The first problem was addressed by opening up the floor to people who held a particular interest in the subject matter. Tech geeks are often passionate about very specific things, and I’m a big fan of letting people work on things they care about. This meant that security geeks could raise their hands to talk about VPCs and KMS, and automation nerds could teach Cloudformation.

Now this worked fantastically, because people who care about things passionately are A.) More likely to own it, B.) More likely to improve the quality of their space, and C.) Way more likely to put effort into making other people understand why they care about the specific subject matter.

The last part is particularly important, because it had the beautiful side effect of generalizing the training modules. People are usually more passionate about knowledge domains and practices than they are about specific products. And if you want to make someone understand your passion, you do it by explaining why you care about the topic within a bigger context.

In practical terms, this meant that VPC modules turned into a broader networking piece. Instead of talking about KMS, we talked about asymmetric/symmetric cryptography. Instead of talking about Cloudformation, we started talking about the principles of automation and why infrastructure as code is important.

Take people who care, then empower them.

Gossip Protocols and Guilds.

The modern tech organization often has an irregular shape; change happens quick, and change is often distributed unevenly. How do you transfer data across a weird topology?

Enter gossip protocols: The easiest way to think of this is to think of a crowd of people sharing gossip. You send a message to person A (“Casey farts in the elevator!”), who then gossips it to the people they know (“Did you know that Casey farts in the elevator?”), who will then propagate it to their peers.

Think of a search query which you post to Node A. Node A will have its list of neighbor nodes, to whom it will then forward the search query. Other nodes will have their own, different list of peers that they forward to, and so on and so forth. In logarithmic time, every node will have the query.

(And everybody will know that in fact, Casey farts in the elevator. Whether this is the actual case is unconfirmed.)

In very much the same way, some companies propagate knowledge through guilds — a slice of people across tribes and organizations who care about a specific topic, who meet a couple of times a month to trade knowledge. This knowledge then gets transferred to the attendees’ tribes.

Practically, this looks like the DevOps Guild I run for my current company — which has internal meetups that hover from 50–100 people strong. We take an hour twice a month and talk shit. It’s nice. We get multiple senders, arm domain experts with a platform, and get a nice coverage across the company.

There’s a couple of problems though:

Peer variance. The people who attend guild meetings will have different levels of topic comprehension. This puts us in a position where we constantly have to balance between A.) providing a content that helps everyone reach the same topic comprehension, and B.) providing content that’s compelling to people who know the space well. You want to be inclusive, and you don’t want to bore subject experts.

Still a broadcast. The format of a guild meeting is of someone delivering a presentation on something they care about. This still suffers from the same lack of WAIT as in broadcast training — although to a lesser extent, as guild attendees tend to be more familiar with each other, and more open to questions.

At this point, the shoe is starting to drop that hey, we need more synchronicity.

Better Guilds?

And so we split the guild into working groups — essentially, smaller groups of people who care about a specific subject domain. It won’t matter if they’re experts at it or not; even people who were interested in learning about the subject matter were welcome.

These groups took the shape of several domains: Monitoring, VPCs, Searching, and so on. This paid dividends from a learning perspective, as it’s easier to be synchronous when you’re a much smaller group. Working groups had the luxury of stopping and waiting until everyone was on the same baseline.

This also allowed people to collaborate on problem spaces. Having an agreed channel on how to communicate issues and interests means that it’s less likely for someone to implement a solution in isolation; having a license to spend time on shared problems means that solutions aren’t being made on the weekend, or out of band. Then, they can post the results to the larger guild.

Take people who care, then link them with peers.

Database Volumes and Mentorship.

Synchronous replication is good for cases where you want a guarantee that data is the same across both nodes, and you have a reasonably fast mechanism to transfer deltas across. When you write to a synchronous node, the transaction is not finished until both nodes have the data committed.

What does this look like practically?

Imagine a two-way relationship that allowed for the space for people to stop until information has been fully understood. Imagine that since it’s impossible to do it for a large amount of people, it took place instead between one teacher and one student. Now imagine if the teacher developed a relationship where they understood the student’s particular learning patterns and preferred teaching methods.

Sounds awesome, right? Sounds like mentorship.

Now a lot of companies will have programs in place to support mentorship, but very few will have it widespread. The approach after all, has a couple of issues:

Scaling. To be quite honest, the tech space still needs more mentors. Mentorship is time-expensive, and doesn’t allow for the same efficiency as a broadcast. Finding people who are willing to commit the time for regular catchups who only serve to better the life of one person is a bit of a task.

Initialization. A fresh mentor who is a first-timer will take a lot of time to learn the ropes. There’s a messload of soft-skills for which material does not exist and is hard to find. Add the fact that there’s no one canonical approach to mentorship, and you end up doing it via trial and error.

But when it works, it works. So let’s make it better.

More Mentorships.

Cheap and lo-fi is my game, so I started by creating ticket lists for people who want to learn specific things — and appending names of the people who are interested in teaching. Think of it as like the telephone signs you see on the street. “Interested in a flatmate? TAKE ONE.”

This allowed for people to experience how great of an experience teaching is. This also meant that we had a fleet of people who can stop and wait until the teachee has successfully absorbed the information.

As always, there are good surprises. The people we taught started signing up to be teachers — and they actually fared better, as they still had the experience of being lost in the topic fresh in their minds. They were more likely to understand the best way to relay and transfer information without loss.

They were more synchronous. They were more empathetic. We took people who cared, which created more people who cared.

Why the fuss?

Why do I even care this much? As of now, the word count is at 2,100+, and I would have spent a good beautiful Sunday afternoon writing this up.

Well, it’s my opinion that there’s a huge bulk of our work that we don’t actually talk about. When people write or do conference speeches, it’s usually for the 10% where we’re winning or losing or learning. But what about this huge grind?

Consider this scenario: you come into work, and it’s Monday and somehow you’re already exhausted. Your first message of the day is a calendar invite from your manager for a meeting where you can explain the thing you explained three times last week. You accept the meeting, because it’s easier than having the meta-conversation about having explained everything before.

Then another email comes in. You get an email from Trello noting that the priority tag for your issue has been lowered, all of a sudden. It’s going to have to wait until the next sprint, and you just want to shake your DL’s shoulders and say, “Being an asshole isn’t in the fucking manifesto”. But you don’t do that.

Frustrated, you decide to take it out on someone else — the engineer who hasn’t replied to your ticket about creating an AD OU. You walk to his desk, ready to complain that it’s been sitting there for two weeks, and it takes 10 minutes, and it’s blocking your work and goddammit are you the only one who cares about the work that you do?

You come to his desk. He tells you he’s done it already. Two weeks ago.

This happens day after day after day.

It’s my personal opinion that shit like this has a cost. Maybe it’s tiny, maybe it’s 0.001% of your total enthusiasm or passion or well-being. But this accumulates and accrues as you pay that tax on your soul. Day after day after day.

In this post, we’ve talked about synchronicity, and how it enabled people to teach others better. But if you think about it, synchronicity is really just the space and capacity to interrogate and understand how the other person feels, what the other person needs right now.

…synchronicity is really just the space and capacity to interrogate and understand how the other person feels, what the other person needs right now.

A synonym of the word ‘kindness’ is considerate — the root word being ‘consider’, and I submit that that’s not an accident. Kindness is just the ability to empathize and think about what other people feel, and to understand what you can do for them. Nietzsche describes kindness as the most curative herb for the human soul.

Synchronicity is empathy. Empathy generates kindness. And kindness is restorative.

Alternatives.

Now consider the previous scenario with a lens of synchronicity — not as an imperative, but as an alternate choice. Maybe it’s sitting down with your manager and getting to know him to figure out the best way to transfer your ideas across. Maybe it’s normalizing the idea with your DL to make sure that prioritization is communicated well. Maybe it’s just having regular lunches with the other engineers and talk to them more.

Of course, this could all be bullshit. This requires active, deliberate action. To be honest, I forget to do this on most days. Some days I just don’t want to.

This could be bullshit, because Gigi, the sad engineer? She doesn’t exist. She’s me. And really, this whole thing has just been a journey of understanding how important being synchronous with other people really is. How it’s rejuvenating and restorative. How the fact that while I may not do it 99% of the time, the fact that the option exists makes all the difference.

There’s a lot of sad people in tech. There’s also a lot of people who are just starting out in this terrifying, messy space. I hereby submit the following proposal:

Let’s be deliberate in creating the Broadcasts, Gossip, and Synchronous Pairs that new people in organizations need.

Which really, is another way of saying that we should have the Teachers, Communities, and Friends that we needed when we were starting out.

Which, really? Boils down quite easily:

Be the people we needed when we were younger.