Building customer-facing security features in partnership with dev teams helps you better serve your customers, unlocks additional revenue, and bidirectionally transfers knowledge between teams—a concept at the very core of DevSecOps.
The security industry generally agrees on the value of enabling developers in an agile environment—although we don't agree on what to call it… “Shifting Left,” “Creating a Paved Path,” “DevSecOps.” Regardless of the name, we tend to focus on teaching developers how to Sec, but there’s less focus on security engineers learning how to Dev.
This article will focus on how to create a meaningful partnership between security and software engineers.
Building customer-facing security features in partnership with dev teams helps you better serve your customers, unlocks additional revenue, and bidirectionally transfers knowledge between teams—a concept at the very core of DevSecOps.
If joining a development team seems daunting and you’re thinking to yourself, “I’m not a software developer, how do I get started with coding?” Unfortunately, this is not the blog to kickstart your coding career, but it will help you understand important business processes related to delivering software.
How to apply some of the lessons of DevOps to security engineering
How to make your first change in production code
How to speak the language of product and engineering
Ensure that security features are prioritized and built correctly
Steps to take after you’ve cooperatively delivered a feature
How to show appreciation and recognize those that helped you succeed
Some people cringe when they hear the phrase “DevSecOps,” they’re reminded of poorly executed and overpromised vendor marketing pitches. I believe the real goal of DevSecOps is for developers, ops people, and security folks to learn more about each other's roles so that we all build better systems together. The tech industry saw the success of this cross-pollination in DevOps, which gave us infrastructure as code and also resulted in developers releasing more stable software.
At any technology company, your devs are going to outnumber your ops and security people. Empowering devs to safely manage operations and security responsibilities as autonomously as possible will result in more stable and more secure software.
Now, a lot of people think of shifting security left as teaching developers about security concepts, fixing systemic vulnerability classes, and building tooling to prevent security mistakes from happening. While these things are certainly important, shifting left is also about people.
In recent years there’s been an industry focus on eradicating classes of bugs through easy-to-use tooling or libraries and on shifting security responsibilities to developers. Building a helpful library that also does things securely with a friendly developer experience is a win-win for software and security engineering.
There will always be gaps in your tooling, and when this is the case you’ll want to empower developers to make autonomous security decisions. Your security engineering team can be available to help, but they can’t possibly be experts on every technology being used across your company. Security-conscious developers will often be able to make better decisions more quickly. When implemented well, this approach can create meaningful improvements to the security posture of your org.
To be able to tackle these challenges, security engineering teams must partner across large swaths of your engineering organization. They’ll naturally become more like cross-cutting engineering teams, such as those that build engineering tooling.
To do this type of work it helps to have a breadth of experience that makes you relatable to various teams, while having a firm grasp on security concepts. Some call this “T-shaped,” meaning well-rounded with a given specialty.
Your security knowledge gives you important business context that will help teams build security features correctly. As a security person, you might inherently notice and analyze things like Single Sign-On (SSO) setup procedures and password strength requirements when you visit websites. This focus gives you a valuable perspective from which to advise other engineers. You’ve seen what works and what is too cumbersome. Devs will learn about security from your active presence on their team, reviewing Pull Requests (PRs), discussing designs, as well as through general osmosis.
Helping deliver features also results in a transfer of knowledge. You’ll feel more confident in future design reviews and your recommendations and questions will be more relevant and realistic. It will be easier for you to find the source of a bug reported through your bug bounty program which means you’ll write up better vulnerability tickets, and better tickets get fixed faster.
As mentioned in the intro, this post is not going to teach you how to write code. There are many people significantly more qualified to help you get started with writing software. Once you have a decent grasp on writing code, this document will provide tips about how to make the most of those skills at your company.
Historically, not many people (myself included) have worked as a software engineer prior to becoming a security engineer. Although I do have a Computer Science degree and had written some amount of tooling before joining Segment, I had never written any customer-facing code. If the tools I wrote for work broke, I could fix them on my own schedule.
A great way to get started is by fixing small bugs in customer-facing code. Pick something you think is worthy of fixing, but is a lower priority in the engineering or security backlog. A great place to find these types of issues is to look through your bug bounty program’s unresolved queue, you can also poke around in Jira backlogs, or chat directly with coworkers to see if they have some suggestions about where to get started. Lean on your existing relationships with software engineers and product managers, or use this as an opportunity to build new relationships, because they will be invaluable on your journey. Engineering managers will be grateful that you fixed a lower priority item on their backlog.
Once you’ve found some bugs to fix, really spend time trying to understand the full impact of the change. Where is this component used? How well-tested is the existing implementation? Do you understand what this part of the app is doing and why? Once you feel like you’ve done sufficient research, confirm your learnings with someone familiar with this part of the codebase. If you have a proposed solution, run this by them to make sure it won’t break anything. Ask for relevant documentation or guidance. Are there some good PRs or tests you should be looking at and emulating? If so, take the time to really understand what’s going on and avoid copying and pasting even if it might result in working code. Remember the goal here is for you to learn more about your company and its technologies, not arrive at a solution in the fastest way possible.
Submitting fixes with well-written PRs and strong testing is paramount to your success, even if it may seem cumbersome at first. Maybe you’ve spent hours or days researching your fix, a great way to demonstrate and document this knowledge is in comments, tests, and PR summaries in addition to the code you’ve written.
If there are a lot of comments on your PRs, don’t get discouraged. This is a great opportunity for you to learn more about writing code at your company. It can be helpful to schedule one-on-one time with reviewers, try to absorb information from experts at your company.
Once your PR is approved, your next task is to learn how code is deployed. Luckily at Segment, this is as easy as merging to the default branch in many cases, but you’ll need to consult with your colleagues for guidance in this area. If you find documentation gaps or things that are unclear, offer to take notes and try to improve (or create) relevant documentation. This will help solidify your understanding of the process, earn you points with engineering teams, and make it easier for the next person.
Make sure you have a plan to test your changes after they get deployed, and find out in advance how to revert your changes in case something goes wrong. If this happens, don’t be crestfallen! Everybody breaks something eventually, I once pushed a change that resulted in customers not being able to create new sessions in our app for about 20 minutes 😬
Once you’ve successfully fixed a few bugs and feel somewhat confident in a part of your company’s codebase, look for an opportunity to create a small but impactful feature.
For me, this was creating a password strength meter for our customers. I talked about this process in a previous blog so I'll skip the details here. With the help of my friend and coworker, Frances, my crude designs created with colored pencils were transformed into sleek React components using Segment’s design system, Evergreen.
During this process, you’ll gain an appreciation for the various steps required to deliver a piece of production code. The design process, writing good tests, documentation, A/B testing, metrics, etc. All of these things take time, and will make you think twice the next time you think “why hasn’t engineering fixed this yet? It seems like such a quick fix!”
An important benefit of learning more about software engineering is building empathy for your coworkers.
After delivering my first feature, I was excited to do more of this type of work and quickly started to plan another project. I had my sights set on building 2FA for our customers. This was seen as a fundamental security control, and luckily it was easy to find an engineering team willing to help deliver this feature. We had originally envisioned this as a security → engineering embed, and although the feature was delivered successfully, it felt more like a relay race.
I was mostly involved in the creation of the backend for 2FA, and although I learned a lot during this process in retrospect I could’ve been more proactive about being “part of the team.” Once the backend portion was done, I met with the person delivering the front end and handed off the project. I was in no way excluded from the team, it was just the first time we had done something like this and we learned how to improve for next time, which will be covered later in this post.
At Segment, I have found engineering teams to be interested in building security features, and they have welcomed additional help from security. Teams will have additional confidence that these features are being built correctly and in a security-approved way. In a B2B environment, security can be a big differentiator, especially with large enterprise customers.
Having a security person embedded in a software team can provide a lot of value. As a security person, when someone says “Let’s build 2FA,” you might already have examples in your head about companies that have built 2FA really well (or really not so well). You may think of some interesting and unique test cases to codify. You might be a great internal beta tester. That said, you will also need training and onboarding, which will take time away from the team’s tasks, so be aware of those very real costs to them.
Take the time to talk to some people early and figure out how planning within your engineering organization works, and try to conform as much as possible. Your counterparts in engineering, product, and design have gone through this process before, don’t hesitate to ask them about their experiences.
Once you understand the planning process, it’s time to get another team bought in and excited. Why should another team help deliver your idea? Even if it’s a great idea, engineering teams typically have a large backlog of great ideas.
Let’s make yours stand out.
An easy way to get your idea prioritized is to be aware of what other teams are working on, and look for efficient opportunities to work together.
For example, early last year I caught wind that a team wanted to combine a few smaller authentication-related services into a single service. They also needed to expand this service to handle authentication for the upcoming release of our new customer management API. There were some improvements I had been wanting to make to our JWTs after seeing Louis Nyffenegger’s presentation at AppSec California. This was a really convenient time to make changes since the authentication system was already getting an overhaul.
Barring an opportunistic partnership, try to approach things the same way your Product Managers prioritize things. Have conversations with PMs on teams you might work with and learn from the experts at your company. Learn how to use your company’s business intelligence tools to build support for your feature, “This enhances an existing feature used by 60 customers representing $25 million in annual recurring revenue (ARR).” Search your company’s repository of customer requests and see if this is something that has been asked for previously. You can also dig through customer security questionnaires if you’re in the B2B space and see if you can find references to the feature you want to build, e.g. “Prospects A, B, and C have asked if we offer SCIM.”
If you can find customer demand for your idea, that will greatly strengthen the chances your feature gets funded.
If you have the chance to embed within another team, make the most of it! You have an incredible opportunity to learn from talented folks at your company, and build a valuable skill set.
Once you’ve got another team bought in, start working with the leaders on that team. This is likely going to include the engineering manager, but you also need to identify other leaders such as senior engineers or PMs.
If this team has hosted people before, great! Figure out what worked well last time, and what should be modified. If not I’d start with onboarding applicable to a new engineer for that team. Schedule a recurring 1:1 with the manager, look at the Jira board and planning docs, ask about documentation you can review in advance.
If you can do these things in the weeks leading up to the project commencing you’ll enjoy a nice head start. This will also reduce the burden on the host team and help make you more self-sufficient.
Once the project starts, join their Slack channel, attend their planning meetings, participate in standup, and any other events they invite you to join. Try to get involved in areas that are both strengths and weaknesses for you.
Maybe you’re really good at reading RFCs and research. Maybe you attend a lot of conferences or read a lot of newsletters and have some influential work in mind that you can share with the team. Or maybe you’re trying to improve your writing skills and have the ability to partner with someone strong in this area.
Don’t limit yourself to just working on security tasks; try to get exposure to how engineering functions in your company. This is a great chance to learn more about logging, metrics, testing, reliability, and other aspects of building good software.
If you do complete a security task, such as a threat model, make sure you fill the role of the developer. Someone else from the security team should represent the security organization. Not only will this expose you to additional parts of the process, you’ll also gain empathy for the engineering side of this work and maybe you’ll realize there are parts that aren’t so great. Also, it avoids a conflict of interest. This new perspective will help you redesign security workflows with additional pragmatism.
Make sure you stay involved with the team until the feature is launched, if possible. Were there any issues caused by the launch? If so, this is a great opportunity to jump into the incident and help debug what went wrong, and learn how you can avoid a similar mistake next time. If the issue is serious enough to warrant a post-mortem, make sure you attend, even if you just listen.
Once things are in a stable state, how is success being tracked? Were the metrics established during previous phases of development meaningful? Were the expectations accurate? Revisit some of the data you collected to get buy-in. How many customers are using your new feature? Were customers that had requested this feature notified? Is the go-to-market team aware of what your team has just built? Not all of these questions will be relevant to every deliverable, and there are certainly many others to be asked, but these might be helpful for you to keep in mind.
After you’ve recovered from the sadness caused by leaving your temporary team, try to spend time reflecting on your experience. Even though you’ve returned to your original team, keep the learnings top of mind.
What went well, what could be improved for next time? Can you convert any of the things you’ve learned into documentation or a mini-training that you could deliver to your security teammates? A team just spent days or weeks working with you on a project, help them out by spreading the knowledge they gave you.
Now that you’re familiar with this system, try to be available when issues come up in the future, this will help build lasting goodwill with members of the engineering org. This could mean hanging out in various Slack channels or just being available to help if someone DMs you with a question. The important thing is that the team knows you still have their back.
Try to incorporate this new knowledge into systems or processes your team has in place, or will put in place in the future. Think back to your time as a software engineer and imagine how you would feel going through the various security requirements. Would this be an easy task to complete, or would it be too arduous and provide little value? Hopefully, this new perspective and additional empathy will lead to you creating more efficient and automated processes in the future. Try to pave a path where things are both easy and secure.
One of the main goals of learning more about software development is to help you think like your software engineering counterparts. With this mindset, you’ll be more easily able to create scalable security tools and processes that empower developers to make good security decisions with minimal input from your team.
Make sure the people that helped you feel appreciated. At Segment, we have a Security Leaderboard where we add points for people that help out our team. Check out this presentation by my director, Eric Ellett, if you want to learn more. I’d also highly recommend dropping a note to someone’s manager or offering to write a review for them during the next performance review cycle. This advice is applicable to many projects but certainly applies here as well.
Additionally, if you find yourself presenting your work at a company All Hands, make sure everyone is included on an intro or outro slide. This type of recognition doesn’t take too much effort and goes a long way to building good relationships along the way. It also helps security allies get promoted, and it’s nice to have friends in high places!
At the end of the day, your company doesn’t exist to be secure, it exists to serve your customers. Security plays an important role in building customer trust and is a component of good software, but it is far from the only piece. Security (especially in the B2B space) can also be a huge differentiator to winning business and maintaining customer confidence.
Different teams at your organization have different motivations, but at the end of the day they should have the common goal of delivering something great for your customers. Take the time to meet people throughout your company and look for opportunities to partner with them and exchange information. With more people knowledgeable about security, better choices will be made and with more autonomy. When a security team is more knowledgeable about other parts of the business, pragmatic tools and processes will be implemented. All of this is rooted in building strong relationships with your coworkers, so remember to be excellent to each other!
If this type of work sounds interesting to you, please check out our careers page!
To everyone that took the time to review my blog and provide feedback, especially Coleen (Segment’s CISO), Rachel (Product Management Superstar), and Clint (of tl;dr sec), I appreciate you! Thank you to everyone that has helped me grow at Segment, especially the Beam Team, our customer security story would not be where it is today without all of you!
This section is meant to provide you with concrete action items and things to think about at various phases of your quest. Think of this as a checklist for you to revisit along the way.
Learn about adjacent parts of your company through documentation and meeting people. If your company has a buddy system or uses an app like donut, these can be great ways to meet people if you’re new.
Engineering training, design reviews, threat models, really anything where you’re working with other teams should be viewed as an opportunity to meet new people and build relationships.
Try to stay up-to-date on your company’s roadmap. Listen in on quarterly planning sessions and product review sessions. Think about how various projects might impact security. Sometimes you’ll hear about an upcoming project that allows you to conveniently enhance security as part of a larger project. Reach out to teams that are working on interesting projects to learn more.
Your security knowledge is an asset! Building product security features correctly requires context that you likely have, which others in engineering may not. This is a great way for you to get involved in a project during the ideation phase.
In a B2B environment, security features can make a big difference to customers. Make sure you put on your product manager hat and look to see if customers are already asking for features you want to build. These requests may be in a ticketing system or in customer security questionnaires. If possible, tie these potential features to current or projected revenue.
Pick something you think is worthy of fixing, but is a lower priority in the engineering or security backlog. Look through your Jira and your bug bounty backlogs to find good candidates.
Ask people in security or engineering if they have some recommendations in mind of what to fix first, be honest about your current level of software engineering experience to help them pick an appropriate starting point.
Understand the changes you’re implementing and learn about adjacent applications and services.
Where is this component used?
How well-tested is the existing implementation?
Do you understand what this part of the app is doing and why?
Are there some good PRs or tests you should be looking at and emulating?
Clearly articulated pull request summaries, code comments, and documentation are incredibly valuable. Engineers love reviewing well-documented and well-tested code
Don’t be afraid to make mistakes, but make sure you learn from them.
See your features through past launch, don’t just build and leave. If there are launch failures, why? Attend any post mortems.
Measure the success of what you built:
Were metrics established before the project useful?
How many customers are using your feature?
Did you notify the customers that had requested the feature?
Is the go-to-market team aware of what your team has just built?
Follow your adopted team's developer culture and practices.
When joining a team for a project try to integrate as deeply as possible:
Schedule a recurring 1:1 with the manager
Review their Jira board and planning docs
Ask about documentation you can check out in advance
Join their Slack channel
Attend their planning meetings
Participate in standup
Attend events they invite you to
If there are security tasks to be done, make sure you are completing them from the perspective of the developer (not a security engineer). Use this experience to iterate on security requirements to make them more developer-friendly while maintaining an appropriate security standard for your company.
Build empathy with other parts of the business, and use this to iterate on current security requirements.
If you learn something useful, teach it to others on your team and document it. For example, if you learn something about the architecture of a given system, teach your security colleagues this context so they can make practical recommendations that won't cause unnecessary friction.
Look for opportunities to eradicate classes of vulnerabilities through secure libraries. Make things easy, while still maintaining an appropriate level of security for your organization
Recognize your coworkers. Take the time to drop their manager a note or offer to write a review for them
Our annual look at how attitudes, preferences, and experiences with personalization have evolved over the past year.