As leaders, we are always striving to bring greater ownership/accountability to our organizations, and visibility to our internal customers. When it matters, the question of “Who did what and when?” matters more than ever. However, especially in larger microservices-based CI/CD environments, with anywhere from dozens to hundreds of app releases happening across many dev teams per day, making your team’s changes visible is often easier said than done. Because when everything becomes visible, nothing really is anymore, right? This isn’t even a signal to noise ratio thing. It’s pretty much all signal now.
Let’s take infrastructure code changes as an example. Because of their continued ability to impact up to and including everything else, these continue to deserve special awareness in realtime. Especially if anything goes sideways. But it doesn’t always work out that way, does it? The main problem of making your infrastructure code changes visible is no longer a technology problem. There are friendly UI dashboards for everything these days. The problem isn’t really a process problem either. Everything you are doing is already code-driven, test-driven and management-approved by now. No, the main problem is (still) a people problem. Namely, making sure the right people are actually seeing your change notifications at the right time.
Consider the default visibility of a basic Ansible playbook run. If you have at least thousands of instances and at least tens of app teams working in a proper self-service model, and Ansible is a core part of your infrastructure automation approach, playbooks are being run pretty much all the time. Which makes the default visibility of any one of those individual runs basically nil. If you are running Ansible Tower (or AWX), yes, you have a dashboard– but one that probably no one outside of the TOC is looking at continuously. There are lots of “simpler” ways for one of your DevOps engineers to inform her colleagues in realtime when hers do. But deciding how to tell other people she is making a change can feel like a Choose Your Own Adventure book where almost all the endings are bad. Does she send an email? Who even reads email anymore? Does she stand up in the middle of the room and shout? Everyone else ended up working from home today. What about Slack? Hmm, sounds promising. But we thought the whole point of automation in the cloud is that you are supposed to let the platform do the hard stuff. And sometimes the hard stuff includes your DevOps engineer actually remembering to notify the rest of the team when she’s doing something late on a Friday afternoon. Without meaning to wax philosophical too much here, you are trying to keep this simple for everyone. Because your team has complex problems surrounding you constantly, and you are merely trying to reduce the number of times per week someone is coming up and asking you, “Did you guys change something?” And as Alan Kay said, “Simple things should be simple, complex things should be possible.” Plus in general, to increase speed and accuracy, you are trying to eliminate decision points and have that platform automatically enforce the process you want to have, all without your engineers or internal customers having to remember to do anything except the task itself. You have Ansible and you have Slack. People never seem to miss the Slack messages that matter to them. How about bringing these technologies together for the greater good?
Slack uses app-based webhook integrations to let external tools including Ansible send messages to channels. It’s absolutely trivial to first create a Slack app, second enable incoming webhooks, and third create an incoming webhook. These incoming webhooks are created per channel. Which means that Slack gives you a token per channel for your app.
You can then include Slack module code in each of your playbooks, which might look something like this:
---
- hosts: all
gather_facts: no
connection: local
tasks:
- name: Notify DevOps channel
slack:
token: T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
msg: 'test msg'
channel: '#devops'
delegate_to: localhost
That’s really all there is to it! Obviously, you can and should include other details in the msg string using variables for the “{{who}}” (the DevOps engineer) and the “{{what}}” (the Jira ticket, the playbook name, the host(s) it ran on). The “When” is automatically supplied by the Slack message timestamp. For larger numbers of hosts per playbook run, you’ll want to explore using threads. For emphasis, you’ll want to explore color coding and emoji icons. But with 10 or so lines of Ansible code, you have brought greater visibility to your organization, you have gone from confusion to clarity, and you have found a receptive audience where you know they will actually be paying attention to your message in realtime. Or at least they can read the scrollback on Monday morning.