Skip links
Team standup meeting looking at security audit report on TV screen

Building a Security-First Development Culture

“Security is everyone’s responsibility” is a platitude that produces no behavioral change. I have heard it at every company I have worked at, and at most of them, security remained the security team’s problem (if there was a security team) or nobody’s problem (if there was not). Building a culture where developers actually think about security during their daily work requires specific, structural changes to processes, tooling, and incentives. After two years of deliberate effort at Harbor Software, here is what worked, what did not, and how we measure whether the culture is actually changing.

Article Overview

Building a Security-First Development Culture

6 sections · Reading flow

01
Start With the Developer Experience, Not the…
02
Security Champions, Not Security Gatekeepers
03
Threat Modeling: Lightweight and Integrated
04
Automated Guardrails
05
Training That Does Not Suck
06
Measuring Culture Change

HARBOR SOFTWARE · Engineering Insights

Start With the Developer Experience, Not the Policy

The standard approach to security culture is top-down: write a security policy, mandate annual security training, require security sign-off on every release. This approach fails because it treats security as a gate that slows down delivery rather than a capability that improves quality. Developers experience security as friction, and humans route around friction. The security review becomes a checkbox, the annual training becomes a tab-out-and-click-next exercise, and the security policy becomes a document nobody reads.

We took the opposite approach: make the secure path the easiest path. If the default database query method is parameterized (and raw queries require a deliberate escape hatch with a specific function name like unsafeRawQuery), developers will use parameterized queries by default not because they care about SQL injection but because it is less typing. If the standard HTTP client automatically validates TLS certificates and follows redirects safely, developers do not need to think about certificate pinning or open redirect vulnerabilities. The secure behavior happens by default, and insecure behavior requires explicit opt-in.

Concrete examples from our codebase:

// Our database wrapper makes the secure path the default
import { db } from '@harbor/db';

// Default: parameterized query (safe)
const users = await db.query(
  'SELECT * FROM users WHERE email = $1', [email]
);

// Must explicitly opt into raw queries (name signals danger)
const results = await db.unsafeRawQuery(
  `SELECT * FROM ${tableName}`
);
// ^^^^^^^^^^^^^^^^^ Code review bots flag this automatically

// Our HTTP client wrapper
import { httpClient } from '@harbor/http';

// Default: validates TLS, follows redirects only to same origin
const response = await httpClient.get(
  'https://api.example.com/data'
);

// Must explicitly opt into dangerous behaviors
const response = await httpClient.get(url, {
  allowInsecureTLS: true,
  allowCrossOriginRedirects: true,
});

The naming convention is deliberate: dangerous options have names that are long, explicit, and slightly embarrassing to write. No one wants to submit a PR with allowInsecureTLS: true because the reviewer will ask why. This is a form of social pressure that reinforces secure defaults without requiring anyone to remember a policy document. We also have a Semgrep rule that flags any usage of unsafe or allowInsecure methods and adds a required security reviewer to the PR.

We extended this principle to our entire internal library ecosystem. Our authentication middleware is configured to require authentication by default; routes that should be public must be explicitly marked with @public. Our logging library automatically redacts fields that match common PII patterns (email addresses, phone numbers, credit card numbers); logging sensitive data requires calling logSensitive() with a justification string that appears in the audit log. The pattern is consistent: safe by default, unsafe requires an explicit, auditable opt-in.

Security Champions, Not Security Gatekeepers

A “security champion” is an engineer on each team who has deeper security knowledge and serves as the first point of contact for security questions. This model works vastly better than a separate security team that reviews everything, for three reasons:

  1. Domain context. A security champion on the payments team understands the payment domain. They know which code paths handle sensitive data, which third-party services have access to PII, and where the authentication boundaries are. An external security reviewer lacks this context and either misses domain-specific risks or wastes time asking basic questions about the system architecture.
  2. Continuous presence. A security champion reviews code daily as part of normal code review. They catch security issues in real-time, during development, when fixes are cheap. An external security review happens quarterly (if you are lucky), catches issues after the code is deployed, and produces a report that sits in a backlog for months. By the time anyone addresses the findings, the codebase has moved on and the fixes require significant rework.
  3. Cultural influence. A security champion is a peer, not an authority. When they raise a security concern in a code review, it carries the weight of a teammate’s judgment, not the weight of an institutional mandate. This is more persuasive and more sustainable. People change behavior when their peers model different behavior, not when a policy document tells them to.

Our security champion program is lightweight. Each champion gets:

  • 4 hours per month dedicated to security learning (reading CVE reports, taking courses on platforms like PortSwigger Web Security Academy, experimenting with security tools)
  • Access to a shared security channel where champions discuss findings and share knowledge
  • A checklist of security-specific items to verify during code review (authentication checks, input validation, output encoding, error handling, logging of sensitive data)
  • Quarterly lunch-and-learn sessions where champions present a security topic to the full team

The total investment is approximately 4 hours per champion per month. We have 3 champions across 8 engineers. The ROI is measured by the number of security issues caught during code review versus the number caught in production or by external audits. In the 12 months before the champion program, we caught 4 security issues in code review. In the 12 months after, we caught 23. Production security incidents dropped from 5 to 1 in the same period. The one production incident was a novel attack vector that none of our automated tools or human reviewers had anticipated, which is exactly the kind of incident you hope to have: rare, novel, and quickly detected.

Threat Modeling: Lightweight and Integrated

Traditional threat modeling (STRIDE analysis, data flow diagrams, formal threat matrices) is thorough but heavy. A full STRIDE analysis for a medium-complexity feature takes 4-8 hours and requires a security specialist to facilitate. For a small team shipping features weekly, this cadence is impractical. The result is that threat modeling either does not happen at all, or it happens once during initial design and is never updated as the system evolves.

We use a lightweight threat modeling approach that integrates into the normal planning process. When a feature is designed, the author answers four questions in the design document:

## Security Considerations

1. **What data does this feature handle?**
   - User email addresses (PII)
   - Payment method tokens (sensitive)
   - Purchase history (business data)

2. **What can go wrong?**
   - Unauthorized access to another user's purchase history
   - Payment token leaked in logs or error messages
   - Rate limiting bypass allows enumeration of user emails

3. **What are we doing about it?**
   - Row-level security on purchase_history table
   - Payment tokens excluded from logging via field-level redaction
   - Rate limiting on the lookup endpoint (100 req/min per IP)

4. **What are we NOT doing, and why?**
   - Not encrypting purchase history at rest (low sensitivity,
     high performance cost, database-level encryption covers
     compliance requirements)
   - Not implementing CAPTCHA on lookup (rate limiting is
     sufficient for current traffic levels; we will add CAPTCHA
     if rate limiting proves insufficient)

The fourth question is the most important. It forces the author to explicitly acknowledge the risks they are accepting and articulate why. This prevents security gaps from being accidental (“I did not think about it”) and makes them deliberate (“I considered it and decided the risk is acceptable because…”). Deliberate risk acceptance is fine. Accidental risk acceptance is a vulnerability waiting to be exploited.

These four questions take 15-30 minutes to answer for a typical feature. The security champion reviews the answers during the normal design review and pushes back if the risk assessments are unrealistic or if mitigations are missing. Over time, engineers internalize the questions and start thinking about them during design rather than as an afterthought. That internalization is the real goal: you want engineers who instinctively think “what can go wrong?” when they design a feature, not engineers who fill out a security template because the process requires it.

Automated Guardrails

Culture change is slow and fragile. People forget, get busy, cut corners under deadline pressure. Automated guardrails catch the things that cultural norms miss. They are the safety net under the high wire.

Our automated security pipeline runs on every pull request:

  1. Dependency scanning (Snyk): Fails the build if a direct dependency has a known critical vulnerability with a published fix. Does not fail on transitive dependencies (too noisy) but adds a warning comment to the PR.
  2. Static analysis (Semgrep): Custom rules tailored to our codebase. We maintain a ruleset of approximately 40 rules covering our most common vulnerability patterns: unparameterized queries, missing authentication middleware, unsafe deserialization, hardcoded secrets patterns, insecure cryptographic configurations, and usage of deprecated security APIs.
  3. Secret scanning (Gitleaks): Scans the diff for anything that looks like a secret: API keys, passwords, private keys, connection strings. Fails the build if a potential secret is detected. We maintain an allowlist for false positives (test fixtures, documentation examples).
  4. VibeGuard scan: Our own AI-powered scanner runs on PRs that modify security-critical code paths (authentication, authorization, payment processing, data export). This adds 2-3 minutes to the CI pipeline but catches the subtle vulnerabilities that rule-based scanners miss.
# .github/workflows/security.yml
name: Security Checks
on: [pull_request]

jobs:
  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Dependency Scan
        uses: snyk/actions/node@master
        with:
          args: --severity-threshold=critical
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}

      - name: Static Analysis
        uses: returntocorp/semgrep-action@v1
        with:
          config: .semgrep/
          generateSarif: true

      - name: Secret Scanning
        uses: gitleaks/gitleaks-action@v2
        with:
          config-path: .gitleaks.toml

      - name: VibeGuard Analysis
        if: contains(
          github.event.pull_request.labels.*.name,
          'security-critical'
        )
        run: |
          vibeguard scan 
            --diff-only 
            --base=${{ github.event.pull_request.base.sha }} 
            --output=sarif 
            > vibeguard-results.sarif

The automated pipeline catches approximately 70% of the security issues we identify. The remaining 30% are business logic vulnerabilities and design-level issues that require human judgment. The automation frees the security champions to focus on that 30% rather than spending their time flagging missing input validation that a Semgrep rule could catch.

Training That Does Not Suck

Annual compliance-driven security training (“click through 60 slides about phishing”) is universally despised and has near-zero impact on developer behavior. A study by SANS found that compliance-driven training changes behavior for approximately 2 weeks after the training, then returns to baseline. Effective security training is hands-on, specific to the team’s technology stack, and delivered in small doses over time.

What we do instead of annual training:

Monthly security lunch-and-learns (30 minutes). One engineer presents a recent CVE, a vulnerability they found in code review, or a security tool demo. These are peer-taught, low-formality sessions. Recent topics: “How the xz-utils backdoor worked and what it means for our CI pipelines,” “Walkthrough of a real JWT validation bug I found in our auth service,” “Demo of using Burp Suite to test our API for IDOR vulnerabilities.” The sessions are recorded for engineers who cannot attend live.

Capture-the-flag exercises (quarterly, 2 hours). We set up a deliberately vulnerable version of our application and give engineers 2 hours to find and exploit vulnerabilities. This is hands-on, competitive (we keep a leaderboard), and teaches attack techniques that make engineers better defenders. The most impactful learning happens when an engineer successfully exploits a vulnerability pattern that exists in their own code. The realization is visceral in a way that slides cannot replicate.

Post-incident learning reviews (as needed). After every security-relevant incident, the postmortem includes a section on “what would have prevented this?” and the answer becomes a focused training topic for the next lunch-and-learn. This ensures training is relevant to actual risks, not theoretical ones. Real incidents have an emotional weight that hypothetical scenarios lack, which makes the lessons stick longer.

Measuring Culture Change

“Culture” is fuzzy. Metrics make it concrete. We track five indicators of security culture health:

  • Security issues found in code review vs. production: A healthy ratio is 10:1 or better (10 issues caught in review for every 1 that reaches production). Our current ratio is 23:1.
  • Time to remediate vulnerabilities: Critical vulnerabilities should be fixed within 48 hours of discovery. Our median is 18 hours for critical, 5 days for high, 14 days for medium.
  • Percentage of design documents with security considerations section: Target 100%. Current: 89%. The missing 11% are small changes that the author judged did not have security implications.
  • Security champion engagement: Measured by number of security-specific code review comments per month. Trending upward from 8/month to 19/month over 12 months.
  • Automated scan findings per PR: Trending downward from 2.3 findings per PR to 0.8 findings per PR over 12 months. This indicates that developers are writing more secure code by default, not just fixing issues after automated tools flag them.

Beyond the quarterly CTF exercises, we maintain a vulnerability hall of fame: an internal wiki page that documents every real vulnerability found by a team member, whether in our code, a dependency, or a client system. Each entry includes the vulnerability type, how it was discovered, how it was fixed, and what we learned. The hall of fame serves multiple purposes: it recognizes the engineers who find security issues (positive reinforcement is more effective than negative consequences), it creates a searchable library of real vulnerabilities for training purposes, and it builds institutional memory about the specific vulnerability patterns that affect our technology stack.

We also run a quarterly security metrics review where the entire team reviews the five security indicators together. This review takes 30 minutes and serves as a forcing function for transparency. When the automated scan findings per PR metric increased from 0.8 to 1.2 in one quarter (going the wrong direction), the team discussion identified the cause: we had onboarded three new client projects with legacy codebases that had not been through our security hardening process. The solution was to run a dedicated security hardening sprint on each new project within the first two weeks of onboarding, bringing the code up to our baseline security standards before regular development work began. This reduced the automated findings on new projects from an average of 4.1 per PR to 1.3 per PR within one month.

The most underrated aspect of security culture is making security work visible. Engineers who fix a security vulnerability in a code review rarely get recognized for it because the fix is invisible: it prevents a bad thing from happening, and prevented bad things are not visible. We address this by including security contributions in our quarterly engineering reviews. The security champions track security-related code review comments, vulnerability findings, and security tool improvements, and these contributions are weighted equally with feature work in performance evaluations. When security work is recognized and rewarded, engineers invest in developing security skills. When it is invisible and unrewarded, they optimize for shipping features faster, which is the opposite of a security-first culture.

Building a security-first culture is a multi-year effort. Two years in, we are not done. But the trajectory is measurable and positive. The key insight is that culture change is not about telling people to care about security. It is about making security the default, giving people the tools and knowledge to do it well, and measuring whether the behaviors are actually changing. Everything else is theater.

Leave a comment

Explore
Drag