Least privilege everywhere: patterns, pitfalls and fixes

Least privilege everywhere: patterns, pitfalls and fixes. Give every user, service and key only what it needs.

Most security breaches don't start with zero-day exploits—they start with overpermissioned accounts and stale access. Least privilege everywhere is the foundational principle that every user, service, and API key should have only the minimum permissions needed to do their job, no more. Without this discipline, a compromised credential or disgruntled team member can spiral into a full data loss incident.

Why least privilege matters in production

We've all inherited systems where a developer has root access to production, or a service account can read and write everything in the database. It feels efficient—fewer IAM rules to manage, fewer permission denials during development. Then someone's laptop gets stolen, or a junior engineer accidentally runs a destructive script in the wrong environment, and suddenly that overpermissioned account becomes the entry point for a cascading failure.

The cost of least privilege isn't high. Modern platforms (AWS IAM, Kubernetes RBAC, PostgreSQL roles) make fine-grained access control cheap and auditable. The cost of not implementing it—forensics, compliance penalties, customer trust—is far higher.

Separate roles for reading and writing

Start by splitting every service account into two: one for reads, one for writes. This simple rule stops a lot of damage. If a service crashes and starts writing garbage data, you can rotate the write key without breaking the read-heavy parts of your system. If the read-only account leaks, an attacker gets visibility but can't modify state.

In a typical microservices setup, we provision a read-only role for metrics and logging pipelines, and a separate write-only role for the order-processing service that publishes events. Here's how that looks in AWS:

// IAM policy for read-only metrics collector
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:Query",
        "dynamodb:Scan"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:123456789:table/Orders"
    }
  ]
}

// IAM policy for write-only order service
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:PutItem",
        "dynamodb:UpdateItem"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:123456789:table/Orders"
    },
    {
      "Effect": "Allow",
      "Action": [
        "sns:Publish"
      ],
      "Resource": "arn:aws:sns:us-east-1:123456789:order-events"
    }
  ]
}

In PostgreSQL, this translates to separate role definitions:

-- Create read-only role
CREATE ROLE metrics_reader;
GRANT CONNECT ON DATABASE app_db TO metrics_reader;
GRANT USAGE ON SCHEMA public TO metrics_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO metrics_reader;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO metrics_reader;

-- Create write role for orders
CREATE ROLE order_service;
GRANT CONNECT ON DATABASE app_db TO order_service;
GRANT USAGE ON SCHEMA public TO order_service;
GRANT INSERT, UPDATE ON TABLE orders TO order_service;
GRANT SELECT ON TABLE orders TO order_service;  -- needed to read back what was inserted

Notice that order_service gets SELECT only on the orders table—the minimum needed to validate writes. It doesn't get DELETE or TRUNCATE. It doesn't get access to the payments or users tables. A compromised order-service credential can't delete user accounts or stripe keys.

Short-lived credentials for services

Long-lived credentials are the nightmare scenario: a key checked into git five years ago, still valid, still powerful, waiting to be discovered. Short-lived credentials—tokens that expire in minutes or hours—dramatically shrink the blast radius.

Most platforms handle this now. AWS STS lets you assume a role and get temporary credentials valid for 15 minutes to 12 hours. Kubernetes mounts service account tokens that can be made short-lived. PostgreSQL can run with temporary passwords or client certificates that rotate daily.

Here's a NestJS service that refreshes AWS credentials before they expire:

import { STSClient, AssumeRoleCommand } from "@aws-sdk/client-sts";

export class CredentialManager {
  private stsClient = new STSClient();
  private cachedCredentials: any = null;
  private expiresAt: Date | null = null;

  async getCredentials() {
    // Refresh if expired or expiring in next 5 minutes
    if (!this.cachedCredentials || this.expiresAt! < new Date(Date.now() + 5 * 60_000)) {
      const result = await this.stsClient.send(
        new AssumeRoleCommand({
          RoleArn: process.env.AWS_ROLE_ARN,
          RoleSessionName: `order-service-${Date.now()}`,
          DurationSeconds: 3600, // 1 hour
        })
      );

      this.cachedCredentials = result.Credentials;
      this.expiresAt = result.Credentials?.Expiration;
    }

    return this.cachedCredentials;
  }
}

For databases, rotate service account passwords monthly or weekly. For API keys, set expiration dates at provisioning and log a calendar reminder. If your platform doesn't support expiry, it's a reason to upgrade.

Review access regularly

Access creep is invisible until it's a disaster. A service that started needing read-only access to the orders table now has read-write to five tables and admin on the compute cluster—nobody remembers when or why. A quarterly or semi-annual access review catches this.

Make reviews systematic. Export all IAM policies, all database role grants, all Kubernetes ClusterRoles, and have the team sign off: "This service needs SELECT on orders, INSERT on events, and nothing else." Automate the export:

// Script to audit AWS IAM access for a service
import { IAMClient, ListAttachedRolePoliciesCommand, GetPolicyVersionCommand } from "@aws-sdk/client-iam";

async function auditServiceRole(roleName: string) {
  const iam = new IAMClient();

  const attached = await iam.send(
    new ListAttachedRolePoliciesCommand({ RoleName: roleName })
  );

  for (const policy of attached.AttachedPolicies || []) {
    console.log(`\n=== ${policy.PolicyName} ===`);
    // Fetch and pretty-print the policy document
    const version = await iam.send(
      new GetPolicyVersionCommand({
        PolicyArn: policy.PolicyArn,
        VersionId: "v1", // simplified; get actual version in production
      })
    );
    console.log(JSON.stringify(JSON.parse(decodeURIComponent(version.PolicyVersion?.Document!)), null, 2));
  }
}

auditServiceRole("OrderServiceRole");

For databases, query the system catalogs:

-- PostgreSQL: list all grants for a role
SELECT grantee, privilege_type, table_name
FROM role_table_grants
WHERE grantee = 'order_service'
ORDER BY table_name, privilege_type;

Document the results in a spreadsheet or wiki. Have the service owner sign off monthly. If they can't justify why a permission exists, remove it.

Remove access when people change roles

This is the one we mess up most often. A senior engineer moves from backend to DevOps. They still have their old database password. They still have SSH keys on the load balancers. A year later, they've moved to another company and still have those credentials.

Offboarding must be automated. When someone's Active Directory or Okta account is deprovisioned, all downstream access revocation should fire automatically:

// Example: webhook handler for user deprovisioning
import { Router } from "express";
import { execSync } from "child_process";

const router = Router();

router.post("/okta-deprovisioning", async (req, res) => {
  const { userId, email } = req.body;

  try {
    // Revoke AWS access
    await revokeAWSAccess(email);

    // Drop database users
    execSync(`psql -U postgres -c "DROP ROLE IF EXISTS ${email}_read; DROP ROLE IF EXISTS ${email}_write;"`);

    // Remove from Kubernetes RBAC
    execSync(`kubectl delete rolebindings -l user=${email} --all-namespaces`);

    // Revoke SSH keys
    await revokeSshKeys(email);

    console.log(`Deprovisioned ${email}`);
    res.json({ status: "ok" });
  } catch (err) {
    console.error(`Deprovisioning failed for ${email}:`, err);
    res.status(500).json({ error: "Deprovisioning failed" });
  }
});

async function revokeAWSAccess(email: string) {
  // Use AWS API to detach all policies from this user/role
}

async function revokeSshKeys(email: string) {
  // Trigger SSH key rotation across servers
}

export default router;

Keep an audit log of all access changes. When was a key issued, rotated, revoked? Who approved it? This log is your forensic record if something goes wrong.

Common mistakes

• Granting overly broad wildcards: Policies like "Action": "*" or GRANT ALL are convenient during development but should never exist in production. Review every policy before merging.
• Shared accounts: Multiple people using a single service account makes auditing impossible. Each principal (person, service, key) should have its own identity. Use role assumption instead.
• Forgetting to revoke during handovers: Integrate access revocation into your offboarding checklist. Have someone verify it actually happened.
• Long-lived credentials in CI/CD: If your GitHub Actions workflow uses a static AWS key, rotate it monthly and use OIDC federation (GitHub Actions → AWS STS) where possible.
• Skipping database-level access control: Even if your application layer is secure, database roles should still enforce least privilege. Assume the application server is compromised.

Key takeaways

• Split every service account into read-only and write-only roles. Reduce blast radius.
• Use short-lived credentials everywhere. Rotate long-lived keys monthly or weekly.
• Review all access grants quarterly. Make it part of your release process or a scheduled audit.
• Automate offboarding. When someone leaves a team or the company, their access should vanish within minutes.
• Audit trails matter. Log every credential issuance, rotation, and revocation for forensics.
• Least privilege is not a one-time setup—it's ongoing discipline. Treat it like testing or code review.