<![CDATA[Stories by Peter Butkovic on Medium]]> https://medium.com/@butkovic?source=rss-74fc69b20a6f------2 https://cdn-images-1.medium.com/fit/c/150/150/1*JdWtwDmk55TCOnxC3El6rA.jpeg Stories by Peter Butkovic on Medium https://medium.com/@butkovic?source=rss-74fc69b20a6f------2 Medium Thu, 05 Nov 2026 18:47:49 GMT <![CDATA[Devless: Turning GitHub Issues into Pull Requests with Amazon Bedrock]]> https://medium.com/@butkovic/devless-turning-github-issues-into-pull-requests-with-amazon-bedrock-02958ddc6ba7?source=rss-74fc69b20a6f------2 https://medium.com/p/02958ddc6ba7 Tue, 20 Oct 2026 16:02:44 GMT 2026-10-20T16:06:52.328Z Architecture, implementation trade-offs and operational boundaries of a self-hosted AI coding agent on AWS.

Objective and scope

My objective was to evaluate an issue-driven development workflow in which team members can request changes through GitHub without operating a coding agent directly.

Desired workflow

The agent should generate code, open a review-ready pull request and address review comments. Human review and repository checks remain the approval boundary; requesting a change must not imply permission to merge it.

Scope and design constraints

Hosted coding agents already support this interaction model. I built this proof of concept to evaluate self-hosted orchestration on AWS with Amazon Bedrock, rather than to establish a replacement for hosted products.

The design priorities were asynchronous processing of long-running coding tasks, configurable models and repository checks, reviewable infrastructure, and reduced idle worker compute. Self-hosting provides control over those choices but also makes infrastructure, permissions and failure recovery the operator’s responsibility.

The result is Devless, open source at github.com/typekpb/devless, with a demo target repository at github.com/typekpb/devless-demo.

Devless is a proof of concept, not a production service.

Solution options and rationale

Devless

  • Setup: AWS bootstrap, GitHub App and Terraform.
  • Cost: AWS infrastructure and model usage.
  • LLM choice: Configurable through the coding backend.
  • Customization: Own orchestrator, prompts and tools.
  • Data boundary: AWS processing plus GitHub and configured tools.
  • Operations: Operator-managed.

GitHub Copilot coding agent

  • Setup: GitHub configuration and repository access.
  • Cost: Current GitHub plan and usage pricing.
  • LLM choice: Supported GitHub models and settings.
  • Customization: Product-provided controls.
  • Data boundary: GitHub service and configured integrations.
  • Operations: Provider-managed.

A hosted agent is the simpler starting point when avoiding worker infrastructure is the priority. Self-hosting was appropriate for this experiment because control over AWS deployment, model integration and orchestration was the subject of the evaluation, not because operating an agent is inherently preferable.

Architectural trade-offs across AWS Well-Architected pillars

Operational Excellence — OPS05-BP01: Use version control. Terraform keeps infrastructure changes reviewable alongside application code and prompts, while issue comments and PRs keep work visible in the existing review workflow. Self-hosting still means maintaining images, dependencies and prompts; version control does not remove that work.

Reliability — REL04-BP02: Implement loosely coupled dependencies. SQS separates webhook acceptance from slow model calls, and a dead-letter queue provides a failure path. GitHub does not have to wait for code generation, but retries can repeat work: queueing alone does not guarantee exactly-once code changes or PR creation.

Cost Optimization — COST09-BP03: Supply resources dynamically. Scaling Fargate to zero removes worker compute charges while no tasks run, at the cost of startup delay. Model retries, the scale-in waiting period and other infrastructure still contribute to the bill.

Security — SEC02-BP02: Use temporary credentials. GitHub App tokens and IAM roles avoid distributing long-lived user credentials, following IAM guidance on temporary credentials for workloads. The GitHub App private key and webhook secret still need protection and rotation. Temporary credentials do not make generated code safe; repository access and merge controls remain important boundaries.

Solution architecture and event flow

The architecture separates webhook ingestion from code generation through an asynchronous queue:

  1. Amazon API Gateway exposes POST /webhook for GitHub events: issues, issue comments, PR reviews and PR review comments.
  2. AWS Lambda (webhook_handler) validates the GitHub HMAC signature, ignores bot senders (so the agent never triggers itself), and puts a task on a queue.
  3. Amazon SQS buffers the tasks. The current redrive policy sets maxReceiveCount = 1. Unprocessed messages can move to a dead-letter queue after that receive allowance. A second Lambda (dlq_processor) then posts a failure comment on the issue, so failures are visible to users.
  4. Amazon ECS on AWS Fargate runs the orchestrator container, with the image stored in Amazon ECR. Amazon CloudWatch alarms on ApproximateNumberOfMessagesVisible scale the state-machine service from 0 to 1 and back, with a waiting period before scale-in.
  5. Amazon Bedrock provides the models: the demo config uses Claude Sonnet for Aider; the separate Strands factory defaults to Haiku. The ECS task role is granted bedrock:InvokeModel, so there is no Bedrock API key to manage.
  6. AWS Systems Manager Parameter Store holds the secrets, and Amazon S3 stores issue context between runs.

Webhook ingestion and validation

The webhook Lambda validates the signature, routes the event and enqueues it. The complete validation function from lambda/webhook_handler/handler.py uses Python's standard-library hashlib and hmac imports:

def validate_signature(payload: bytes, secret: str, header: str) -> bool:
    if not header or not header.startswith("sha256="):
        return False
    expected = "sha256=" + hmac.new(
        secret.encode(), payload, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, header)

This follows GitHub’s webhook validation guidance: compute HMAC-SHA256 over the unmodified payload and compare in constant time. Validate before parsing or acting on the delivery.

Message delivery and failure handling

For accepted events, the handler returns 202 Accepted only after SQS accepts the message, without waiting on the LLM. SQS buffers work and separates the two components, but the retry budget is a configuration choice. The complete task-queue resource in terraform/modules/sqs/main.tf makes that choice explicit:

resource "aws_sqs_queue" "main" {
  name                       = "${var.project_name}-tasks"
  visibility_timeout_seconds = 1200

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dlq.arn
    maxReceiveCount     = 1
  })
}

That is a PoC setting, not my production recommendation. AWS recommends allowing sufficient retries, and the 20-minute visibility timeout must cover processing or be extended with a heartbeat, as described in the SQS visibility-timeout guidance. The current state-machine consumer does not extend it. Duplicate-safe operations and a tested redrive procedure are still needed.

Worker capacity and scale-in behavior

In terraform/modules/ecs/main.tf, the state-machine service scales between zero and one worker; scale-in waits for 30 one-minute periods with no visible messages. A received message becomes invisible while it is still being processed, so an empty visible queue is not proof that a worker is idle. Long jobs need in-flight-aware shutdown protection. These scaling policies apply to the state-machine service; the Strands service starts disabled and must be enabled separately.

Implementation decisions

Code generation and validation

I reused Aider rather than building a code-editing loop. It integrates with Bedrock (bedrock/<model-id>) and supports auto-test: running a configured test command after edits and attempting repairs within a bounded reflection loop. This keeps repository editing separate from event handling and orchestration.

Once the GitHub App and webhook are configured, each target repository supplies its model and checks in one file. This example is the demo repository’s configuration:

# .devless/config.yml
model: bedrock/eu.anthropic.claude-sonnet-4-20250514-v1:0
test_command: npm ci && npx vitest --run && npm run build
max_iterations: 10

The model IDs and versions shown here record the proof-of-concept configuration, not a standing recommendation. Model capabilities, pricing and availability evolve, so the specific choice may change. The underlying concepts — separate orchestration from code editing, validate generated changes and keep human approval explicit — do not depend on those versions. Before reusing this setup, check current Bedrock availability and evaluate candidate models on representative repository tasks.

Put build, lint and tests into test_command. Aider treats a failing build like a failing test. The state-machine implementation checks for a commit and a non-failing test outcome before opening a PR. The agent-based version follows workflow instructions instead; that is not an equivalent hard guarantee. In either version, repository CI and review must remain the merge gate.

The code-editing setup in orchestrator/src/aider_runner.py shows what is delegated to Aider (excerpt inside run_aider):

coder = Coder.create(
    main_model=model_obj,
    io=io_obj,
    fnames=[],
    auto_test=True,
    test_cmd=test_cmd,
    auto_commits=True,
)
coder.max_reflections = 10  # Allow up to 10 iterations to fix test/build failures

Two limits matter here. This implementation hardcodes ten reflections rather than using the configured max_iterations. Its success check accepts test_outcome is not False, which also accepts None when tests did not run. For a production gate, I would require an explicit passing result and independently enforce required status checks and reviews through branch protection. Tests help assess a change; an agent's success flag must not replace those controls.

Orchestration: explicit state machine or agent-directed workflow

I built the orchestrator twice to compare a deterministic design with an agentic one.

Deterministic orchestration with a Python state machine

Python code controls the flow through explicit states: pending → cloned → agent_loop → pr_created → complete. Any failure resets the task to pending.

  • Pros: deterministic, easy to debug.
  • Cons: rigid. Every new workflow needs code changes. About 800 lines.

Agent-directed orchestration with Strands Agents and skills

The second version uses Strands Agents with a BedrockModel. The model decides the flow. It follows a SYSTEM.md prompt and a set of SKILL.md files (workflow, github, git, code), and it acts through a small set of tools:

  • GitHub: get_issue, post_comment, create_pull_request, get_pull_request
  • Git: clone_repository, create_branch, checkout_branch, push_branch
  • Code: read_project_config, run_aider, list_files, read_file

The factory in orchestrator-agent/agent.py configures the orchestration model and available tools:

def create_agent(model_id: str = "anthropic.claude-3-haiku-20240307-v1:0") -> Agent:
    model = BedrockModel(
        model_id=model_id,
        region_name="us-east-1",
    )

    return Agent(
        model=model,
        system_prompt=load_system_prompt(),
        tools=ALL_TOOLS,
    )

The outer Strands agent and inner Aider coding loop have separate model settings. Deployment location, orchestration-model Region and coding-model routing all need checking against residency requirements.

For changes within the existing tool set, the workflow can be adjusted in Markdown. Those files still need shipping with the application; a new capability may also need tool code. The original comparison was roughly 800 lines for the state machine and 400 for the agent version, not a measure of equivalent reliability.

State machine

  • Control flow: Hardcoded Python.
  • Lines of code: Roughly 800.
  • Adding workflows: Code changes.
  • Error handling: try/except.
  • Predictability: Deterministic control flow.
  • Flexibility: Rigid workflow.

Agent-based

  • Control flow: LLM guided by SKILL.md.
  • Lines of code: Roughly 400.
  • Adding workflows: Edit SKILL.md within the existing tool set.
  • Error handling: Agent decides how to respond.
  • Predictability: May vary.
  • Flexibility: Adapts the workflow.

Security architecture and trust boundaries

An agent that can push code is a high-value target. These are the boundaries I used, and the gaps I would close before production:

  • GitHub App instead of personal access tokens. Installation tokens are short-lived and scoped to the selected repositories (Contents, Issues and Pull requests read/write; Checks read-only). There are no PATs to rotate.
  • Keep tokens out of prompts. The orchestrator passes the token to init_github() and init_git(). Tools hold it outside the prompt and perform authenticated operations. That is the intended boundary, not proof that exceptions, subprocess output or logs cannot leak it; those paths need review too.
  • IAM for Bedrock. The Fargate task role grants access to the model, with no keys stored.
  • OIDC for CI/CD. GitHub Actions assumes an IAM role through OIDC federation, so the pipeline holds no static AWS keys. The current trust policy allows repo:${var.github_actions_org}/*:*. Before production, I would narrow that to the deploying repository and branch or protected environment, following GitHub's AWS OIDC guidance.
  • HMAC on every webhook, compared in constant time. This authenticates the delivery, not the issue author’s authority to run code. The handler filters bots but has no sender-authorization gate. Public issues and comments are untrusted input; production needs an explicit authorization policy before enqueueing work.
  • Review before merge. PRs provide the review boundary. The repository also describes auto-merge after review and checks; branch protection must enforce whichever policy the target repository requires.

Keeping credentials outside prompts is a useful separation, not a sandbox. Repository code, dependency installation and test_command execute in the worker, which also has credentials and an AWS task role. I would isolate that execution and reduce credential exposure before accepting untrusted work. The current Bedrock IAM policy also permits broad model and inference-profile resources across Regions; IAM least-privilege guidance calls for narrowing those permissions to the workload's actual needs. These are hardening tasks, not claims that the PoC already meets every Security pillar practice.

Deployment and operating cost

The whole stack is Terraform modules: api_gateway, lambda, sqs, ecs, ecr, iam, s3, secrets, vpc. One terraform apply deploys everything. After the first bootstrap, every push to main runs terraform apply, builds and pushes the image to ECR, and redeploys ECS.

The reported running cost for this PoC was about $20/month. Treat that as an observation for this experiment, not a quote for another workload: model choice, token volume, retries, networking and worker runtime change the total.

Evaluation and production-readiness gaps

The PoC produced two implementations of the same issue-to-PR workflow and a demo repository for exercising it. The useful comparison was not just code size: explicit states made the sequence easy to inspect, while the agent version moved more workflow decisions into instructions.

This was not a controlled benchmark of completion rate, security or cost per accepted change. Before treating it as a service, I would test duplicate deliveries, interrupted jobs, concurrent work on the same repository, malicious issue content and recovery after a partially completed push. Passing generated tests alone is not evidence that a change meets the request.

Architectural lessons and next steps

What worked

  • Aider, which is battle-tested.
  • GitHub App auth, which removes PATs.
  • Bedrock’s IAM-based authentication, alongside the required model and Region configuration.
  • SQS with a DLQ plus scale-to-zero Fargate, a straightforward asynchronous backbone, with retry and in-flight shutdown behavior still needing hardening.

Model choice matters; versions are a snapshot. In this experiment, Claude 3 Haiku got stuck on a simple TypeScript error, an unused import. It kept adding import React instead of removing it. Sonnet fixed it on the first try. That observation is not a ranking of later model generations. The lasting lesson is to evaluate task completion, latency and total cost, including repair iterations, rather than token price alone.

For capability categories, Anthropic’s model overview distinguishes models aimed at high-volume, latency-sensitive tasks (Haiku), a balance of speed and intelligence (Sonnet), and long-running agentic coding and knowledge work (Opus). These are useful selection categories, not fixed capability guarantees or an exhaustive list as the lineup evolves. Evaluate the orchestration and coding models separately: reliable tool selection and multi-file code repair are different demands.

Key insight. Moving workflow decisions into an agent reduced orchestration code in this experiment, but did not preserve deterministic control automatically. Tools constrain available actions; they do not guarantee the order or judgment of those actions. I would use explicit checks for non-negotiable gates and an agent where flexibility earns its cost. The Haiku/Sonnet result is a reason to evaluate models on representative tasks, not a universal ranking.

This distinction follows Anthropic’s Building effective agents: workflows follow predefined code paths, while agents let the model direct the process and tool use. Its advice to start with the simplest approach and add complexity only when task performance justifies the latency and cost applies beyond any particular model version.

Workflow suitability

Comparison:

  • Vibe coding: Instant feedback, best suited to quick prototypes by developers.
  • Spec-driven development (e.g. Kiro): Upfront planning, best suited to greenfield projects involving architects and PMs.
  • GitHub issues (Devless): Slower feedback while waiting for a PR, best suited to existing projects where non-developers and PMs request changes.

The feedback loop is slower, but the issue-driven approach lets non-developers request features in a channel they already use. A natural next step is streaming Aider output into issue comments so users can follow progress.

Source code and implementation details

The AWS deployment was temporary and is no longer running. The following repositories supplied the implementation details for this post; their main branches remain available for inspection:

References

]]> <![CDATA[From Aurora to DynamoDB: Cutting Database Costs by 90% with DynamoDB and S3]]> https://medium.com/@butkovic/from-aurora-to-dynamodb-cutting-database-costs-by-90-with-dynamodb-and-s3-e5bf2999a463?source=rss-74fc69b20a6f------2 https://medium.com/p/e5bf2999a463 Mon, 19 Oct 2026 08:06:07 GMT 2026-10-19T08:09:44.996Z How we moved a set of Spring Boot microservices off Amazon Aurora PostgreSQL, guided by the AWS Well-Architected Framework, and what we learned along the way.

The setup

As part of a long-term client engagement, I was responsible for the design, delivery and ongoing operation of a system that generates business-critical sales documents (PDFs) through a REST API (OpenAPI). It ran as Spring Boot microservices on Amazon ECS (Fargate) in private subnets. API Gateway sat in front, with PrivateLink and Network Load Balancers behind it. Supporting services included S3, SQS, Secrets Manager, KMS, AWS Systems Manager Parameter Store and Amazon CloudWatch.

All persistence went to Amazon Aurora (PostgreSQL-compatible). When we looked at how the data was used, we found:

  • Most of it was key-value data.
  • Large objects such as integration responses and generated binaries (Base64-encoded) were stored in TEXT columns.

In other words, we were using a relational database as an expensive key-value store with blobs attached. Starting from the access patterns rather than from the schema is also the first step AWS recommends for NoSQL design in DynamoDB.

Original setup

Before the migration: Aurora held both application data and the Quartz JDBC job store.

Why we migrated

No single event forced the move. The motivation came from the database upgrades we had already been through, and from knowing more of them would follow:

  • PostgreSQL 10.x reached end of life, so we had to do a major engine upgrade to 11.x. Aurora publishes these dates in the Aurora PostgreSQL release calendar, so the next one was always coming.
  • Aurora Serverless v1 was retired, so we had to move to Serverless v2 or Provisioned. For our load, Provisioned (db.t4g.medium for DEV, db.t4g.large for INT/PROD) came out cheaper than Serverless v2.
  • Serverless v2 couldn’t scale to zero at the time, unlike v1. That made the low-traffic DEV and INT environments costly.
  • Each upgrade meant downtime and manual steps, both for the major version upgrade and for the move from Serverless v1 to Provisioned.

Each upgrade cost us effort and downtime but didn’t make the system any better. Since the next upgrade would only be a matter of time, we asked whether it should be a relational database at all.

Choosing a migration strategy

AWS describes seven common migration strategies, the “7 Rs”: retire, retain, rehost, relocate, repurchase, replatform and refactor. They range from “leave it as is” to “redesign it for the cloud”:

  • Rehost (“lift and shift”): move the workload unchanged. Fastest, lowest risk, fewest cloud benefits.
  • Replatform (“lift and reshape”): small optimizations without changing the core architecture, e.g. moving a self-managed database to a managed one.
  • Refactor / re-architect: change the architecture and code to use cloud-native services. Most effort, biggest payoff in cost, scalability and operations.

A common pattern, also recommended in AWS guidance, is migrate first, modernize later. Rehost or replatform to get into the cloud quickly, then refactor once the team knows the workload and its real access patterns in production.

That is the path the system followed. It had already been moved to AWS with a managed relational database. Several years of production data then showed how the data was actually used: mostly key-value. Instead of repeating the upgrade cycle, we picked refactor: we replaced the relational model with DynamoDB and S3, and dropped custom code where managed features could do the job.

Aurora PostgreSQL vs DynamoDB

AWS recommends choosing a database by data model and access pattern, not by habit (see the AWS decision guide for choosing a database). For our workload the comparison looked like this:

FeatureAurora PostgreSQLDynamoDBTypeRelational (SQL)NoSQL (key-value & document)SchemaFixedSchema-lessScalingVertical, plus read replicasAutomatic horizontal scalingStrengthACID, complex queriesHigh-throughput key-based accessCost modelPay per instance (provisioned/serverless)Pay per requestManagementScaling and upgrades are your jobFully managed, serverless

Deciding with the Well-Architected Framework

We didn’t want to pick on gut feeling, so we scored the decision against the AWS Well-Architected Framework pillars.

Cost Optimization. “Adopt a consumption model” and “Measure overall efficiency.” DynamoDB on-demand costs almost nothing when idle, while our Aurora instances ran all the time. DEV and INT saw very little traffic.

Performance Efficiency. “Democratize advanced technologies” and “Consider mechanical sympathy”, which explicitly says to consider data access patterns when selecting a database. DynamoDB scaled read and write capacity with demand automatically.

Operational Excellence and Security. “Automate security best practices” and “Keep people away from data.” Access went entirely through IAM. ECS tasks assumed a role, so there were no database credentials to rotate. Read-only users could browse data in the AWS Console under IAM policies.

Sustainability. “Maximize utilization” and “Use managed services.” A managed, consumption-based service has no idle instances burning energy.

Reliability. We no longer needed maintenance windows for major engine upgrades.

The numbers

Aurora made up 22% of the whole AWS bill on PROD. Our estimate for the same workload on DynamoDB across DEV, INT and PROD came out about 90% lower than the Aurora cost.

Challenges and how we solved them

Updated setup

After the migration: DynamoDB stores metadata and S3 keys; S3 stores large payloads. TTL, lifecycle expiration, EventBridge and SQS replace the Quartz jobs.

1. The 400 KB item size limit

A DynamoDB item can’t exceed 400 KB, and some of our payloads were far bigger.

Solution: we stored integration response payloads and binary content (images, PDFs) in Amazon S3, and kept only the metadata and the S3 key in DynamoDB. This is safe because S3 provides strong read-after-write consistency: an object is readable as soon as the write returns. This is the pattern AWS itself recommends in best practices for storing large items. Writing to S3 first and DynamoDB second means a reader never finds a pointer to an object that doesn’t exist yet.

// simplified: real code adds error handling and content-type
String key = "payloads/" + id;
s3.putObject(b -> b.bucket(bucket).key(key), RequestBody.fromBytes(pdf));
dynamo.putItem(b -> b.tableName(table).item(Map.of(
    "id",     AttributeValue.fromS(id),
    "s3Key",  AttributeValue.fromS(key),
    "status", AttributeValue.fromS("GENERATED"))));

DynamoDB or S3: what goes where

The split wasn’t arbitrary. DynamoDB and S3 are built for different shapes of data:

AspectDynamoDBAmazon S3Unit of storageItem, max 400 KBObject, up to terabytesAccessBy key, plus secondary indexes and conditional updatesBy key or prefix onlyLatencySingle-digit millisecondsHigher per request, built for throughputStorage priceRoughly 10x more per GBRoughly 10x less per GBBest forSmall, frequently read and updated recordsLarge, write-once blobsExpiryTTL per itemLifecycle rules per prefix or tag

Prices are region-dependent. See DynamoDB pricing and S3 pricing.

Our rule followed from that table. Anything we query or update, like IDs, status and timestamps, went to DynamoDB. Anything we only store and fetch whole, like PDFs, images and raw integration responses, went to S3. A PDF is written once and read occasionally. Keeping it in DynamoDB would mean paying premium per-GB storage and read capacity for bytes we never filter on. This is Well-Architected best practice PERF03-BP01, use a purpose-built data store. It asks you to document size, access pattern, latency and retention for each data type, and lists “you stick to one data store because there is internal experience” as an anti-pattern. That’s a good reminder that familiarity alone isn’t a reason to keep everything in one database.

AWS’s large-items guidance also lists the costs of this pattern, and they’re worth planning for:

  • No transactions span DynamoDB and S3. Writing S3 first means a failure leaves, at worst, an orphaned object, never a dangling pointer. A lifecycle rule or a periodic cleanup handles the orphans.
  • Link back from the object. AWS suggests storing the item’s primary key as S3 object metadata, which makes maintenance and orphan detection easier.
  • Two expiry mechanisms. If items expire with DynamoDB TTL, the matching objects need an S3 lifecycle rule with the same retention, or storage keeps growing.

2. SQL aggregations

One service did advanced aggregations in SQL, and DynamoDB has no GROUP BY.

Solution: the data set was small enough to fit in container memory, so we rewrote the aggregations in Java. This also made them faster, since in-memory processing beats a database round trip. For larger data sets this wouldn’t scale. There, AWS suggests materialized aggregations, for example updated through DynamoDB Streams and AWS Lambda.

3. Data migration

We considered three options:

We chose the manual CSV path because the data volume was small. The DynamoDB import from S3 feature handles the CSV-to-table step without custom code. It doesn’t consume write capacity on the table. Note that it always creates a new table, so the table definition is part of the import rather than created up front. DMS would have been the right choice for large volumes, or for near-zero downtime with ongoing replication.

4. Quartz scheduler with a SQL job store

We used Quartz with a JDBC job store for three jobs. Without a SQL database, each one needed a new home.

Periodic cleanup of data older than X days. We replaced this with DynamoDB TTL and S3 lifecycle expiration. No cleanup code was left to maintain. TTL deletes don’t consume write throughput. They aren’t instant, though: AWS documents that expired items are typically removed within a few days, so reads have to filter out items whose expiresAt is in the past.

long expiresAt = Instant.now().plus(Duration.ofDays(retentionDays)).getEpochSecond();
item.put("expiresAt", AttributeValue.fromN(Long.toString(expiresAt)));
resource "aws_dynamodb_table" "documents" {
  name         = "documents"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "id"
attribute {
    name = "id"
    type = "S"
  }
  ttl {
    attribute_name = "expiresAt"
    enabled        = true
  }
}

Daily statistics exported to Amazon CloudWatch. We replaced the cron job with an Amazon EventBridge scheduled rule. The rule sent a message to SQS, and the service consumed it. Going through SQS rather than calling the service directly kept the trigger decoupled and retryable. For new projects AWS now recommends Amazon EventBridge Scheduler over scheduled rules.

Detecting stuck business transactions. If PDF generation stalled, the client still had to be notified through its callback URL. We replaced the polling cron with an SQS delay queue. When a transaction started, the service sent a delayed message. When the message became visible, the consumer checked whether the transaction had finished. If it hadn’t, the consumer fired the callback. There was no polling and no scheduler state. SQS delays are capped at 15 minutes. For longer timeouts, a one-time schedule in EventBridge Scheduler does the same job.

sqs.sendMessage(b -> b.queueUrl(timeoutQueueUrl)
    .messageBody(transactionId)
    .delaySeconds(900)); // SQS max delay is 15 minutes

Backups

StrategyHow it worksBest forOn-demand backupManual or scheduled full snapshot, no performance impactRegular backups, snapshots before major changesPoint-in-time recovery (PITR)Continuous; restore to any second in the last 35 daysAccidental deletes or writes

We went with scheduled daily on-demand backups (via AWS Backup), which matched the existing backup policy. This follows the Reliability pillar best practice REL09-BP03, perform data backup automatically. The same section adds REL09-BP04: restore backups periodically to verify them. Turning on PITR is the next step if the recovery point objective tightens.

Bonus: Aurora backed up the whole cluster, temporary data included. With DynamoDB, backups are configured per table. We didn’t back up tables whose data could be rebuilt from external systems, which meant less storage and lower cost.

Results

  • We migrated all three microservices to DynamoDB and S3 in PROD.
  • Data migration took a couple of hours rather than the expected one hour or less.
  • Along the way we removed deprecated data and simplified the code.

Refactoring, not just a datastore swap

We redesigned the data access around key-value lookups and S3 instead of swapping the database driver. To check that the refactor didn’t trade cost for performance, we measured before and after:

  • Same scenario, same environment. We ran an identical load test against the old and new versions, so the only variable was the data layer.
  • Percentiles, not averages. We compared median, 95th percentile and max response times. Tail latency is what users notice.
  • Repeatable. The test was scripted, so we reran it after each microservice migration.

This followed the Performance Efficiency pillar of the Well-Architected Framework, which recommends using load testing and metrics to choose and validate architecture changes.

Lessons learned

  1. DynamoDB fits key-value access well. If your SQL tables are mostly SELECT * WHERE id = ?, you're paying for features you don't use.
  2. Put files in S3, not in the database. Strong read-after-write consistency makes the S3-pointer pattern safe.
  3. Make the business case explicit. The migration took a couple of months of effort. Weighing short-term engineering cost against long-term AWS bill savings is what got it approved.
  4. Write tests first. Unit and integration tests caught regressions throughout the rewrite.
  5. Prefer managed features to custom code. TTL, EventBridge and SQS delay queues replaced scheduler code that we used to maintain ourselves. Find the right balance between custom implementation and what the cloud already provides.

References

]]> <![CDATA[AWS SAM based Lambda development with Localstack (on MacOs and Podman)]]> https://medium.com/@butkovic/aws-sam-based-lambda-development-with-localstack-on-macos-and-podman-1471de10598e?source=rss-74fc69b20a6f------2 https://medium.com/p/1471de10598e Fri, 04 Nov 2022 09:24:53 GMT 2022-11-04T09:25:58.905Z Overview

Serverless is a hot topic in the cloud world. AWS primary answer to serverless computing was Lambda. And the recommended way to develop AWS Lambda is the usage of the AWS Serverless Application Model (a.k.a SAM).

Local development of the Lambda functions using SAM and Localstack is not streightforward. For my case is the additional complexity is introduced by the Podman usage on MacOs.

The aim of this post is to give you an overview of the needed steps that you might want to follow for you very own lambda development adventures.

Enjoy.

Demo application architecture

Demo application architecture is depicted on the following image:

Image source: https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-dynamo-db.html

Github sources

Github repo with initialized project is available at:

GitHub - typekpb/blog-sam-app-demo

Please note, it contains the bootstraped project adapted for the localstack/AWS Cloud usage.

Install the tooling

  1. Install the AWS SAM cli on MacOS via Homebrew (as per official instructions):
brew tap aws/tap
brew install aws-sam-cli

2. Init + start the podman machine for the SAM usage (following the instructions):

sudo /opt/homebrew/Cellar/podman/*/bin/podman-mac-helper install
podman machine init --cpus=4 --disk-size=60 --memory=8096 -v $HOME:$HOME
podman machine set --rootful
podman machine start
alias docker='podman'

3. Setup localstack AWS profile locally:

aws configure --profile localstack
AWS Access Key ID [None]: dummy
AWS Secret Access Key [None]: dummy
Default region name [None]: eu-central-1
Default output format [None]: json

4. AWS SAM local cli installation is required, until the issue: https://github.com/aws/aws-sam-cli/issues/1194 would be resolved.

To do so use pip (as per official instructions):

pip3 install aws-sam-cli-local

Install NoSQL Workbench as DynamoDB client (via Homebrew instructions):

brew install --cask nosql-workbench

Run localstack

Clone the git repository of the sample app and run localstack via podman-compose:

git clone https://github.com/typekpb/blog-sam-app-demo.git
cd blog-sam-app-demo/
podman-compose down && podman-compose up

The docker-compose.yaml file contents follow:

version: "3.7"
services:
  localstack:
    image: localstack/localstack:latest
    environment:
      - AWS_DEFAULT_REGION=eu-central-1
      - EDGE_PORT=4566
    ports:
      - '4566:4566'

Bootstrap the SAM project

The following options are to be taken during the guided init process:

sam init
You can preselect a particular runtime or package type when using the `sam init` experience.
Call `sam init --help` to learn more.
Which template source would you like to use?
        1 - AWS Quick Start Templates
        2 - Custom Template Location
Choice: 1
Choose an AWS Quick Start application template
        1 - Hello World Example
        2 - Multi-step workflow
        3 - Serverless API
        4 - Scheduled task
        5 - Standalone function
        6 - Data processing
        7 - Infrastructure event management
        8 - Serverless Connector Hello World Example
        9 - Multi-step workflow with Connectors
        10 - Lambda EFS example
        11 - Machine Learning
Template: 3
Which runtime would you like to use?
        1 - dotnet6
        2 - dotnetcore3.1
        3 - nodejs16.x
        4 - nodejs14.x
        5 - nodejs12.x
Runtime: 3
Based on your selections, the only Package type available is Zip.
We will proceed to selecting the Package type as Zip.
Based on your selections, the only dependency manager available is npm.
We will proceed copying the template using npm.
Would you like to enable X-Ray tracing on the function(s) in your application?  [y/N]:
Project name [sam-app]: sam-app-demo
Cloning from https://github.com/aws/aws-sam-cli-app-templates (process may take a moment)
-----------------------
    Generating application:
    -----------------------
    Name: sam-app-demo
    Runtime: nodejs16.x
    Architectures: x86_64
    Dependency Manager: npm
    Application Template: quick-start-web
    Output Directory: .
Next steps can be found in the README file at ./sam-app-demo/README.md
Commands you can use next
    =========================
    [*] Create pipeline: cd sam-app-demo && sam pipeline init --bootstrap
    [*] Validate SAM template: cd sam-app-demo && sam validate
    [*] Test Function in the Cloud: cd sam-app-demo && sam sync --stack-name {stack-name} --watch

Adapt generated sources

Following were the areas I had to update the generated sources:

  • template.yaml — to have it running on localstack
  • lambda js files — to enable local dynamodb usage (introducing custom env var: LOCAL_DYNAMODB as well as some enhancements to error logging

The documented version is available in the referred github repo.

Build the project

Change to project dir:

cd sam-app-demo

and build the project itself:

sam build
Building codeuri: .../sam-app-demo runtime: nodejs16.x metadata: {} architecture: x86_64 functions: getAllItemsFunction, getByIdFunction, putItemFunction
Running NodejsNpmBuilder:NpmPack
Running NodejsNpmBuilder:CopyNpmrcAndLockfile
Running NodejsNpmBuilder:CopySource
Running NodejsNpmBuilder:NpmInstall
Running NodejsNpmBuilder:CleanUpNpmrc
Running NodejsNpmBuilder:LockfileCleanUp
Build Succeeded
Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml
Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {stack-name} --watch
[*] Deploy: sam deploy --guided

Option 1 (WIP): Running full stack in Localstack

This option provisions all the needed resources in the localstack.

Unfortunatelly, this way is not working as expected as of now. I didn’t convince Lambda to call DynamoDB within the localstack yet (despite following the advices, post should be updated, once this works as expected).

For local testing, the deploy to localstack is needed (which takes care of the all the needed resources). Run the following command to deploy (e.g. to region eu-central-1):

samlocal deploy --resolve-s3 --stack-name sam-app-demo --capabilities CAPABILITY_IAM --no-fail-on-empty-changeset --region eu-central-1
        Creating the required resources...
        Successfully created!
Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-eb5effb8
                A different default S3 bucket can be set in samconfig.toml
                Or by specifying --s3-bucket explicitly.
Uploading to 3acbdb74d3929867f649b7505c9d4f43  10614190 / 10614190  (100.00%)
File with same data already exists at 3acbdb74d3929867f649b7505c9d4f43, skipping upload
File with same data already exists at 3acbdb74d3929867f649b7505c9d4f43, skipping upload
Deploying with following values
        ===============================
        Stack name                   : sam-app-demo
        Region                       : eu-central-1
        Confirm changeset            : False
        Disable rollback             : False
        Deployment s3 bucket         : aws-sam-cli-managed-default-samclisourcebucket-eb5effb8
        Capabilities                 : ["CAPABILITY_IAM"]
        Parameter overrides          : {}
        Signing Profiles             : {}
Initiating deployment
=====================
Uploading to 22e245ed08e7e1c6d8e4197df4c59a84.template  2969 / 2969  (100.00%)
Waiting for changeset to be created..
CloudFormation stack changeset
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation                                    LogicalResourceId                            ResourceType                                 Replacement
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Add                                        putItemFunctionApiPermissionProd             AWS::Lambda::Permission                      N/A
+ Add                                        putItemFunction                              AWS::Lambda::Function                        N/A
+ Add                                        getAllItemsFunction                          AWS::Lambda::Function                        N/A
+ Add                                        getByIdFunctionRole                          AWS::IAM::Role                               N/A
+ Add                                        SampleTable                                  AWS::DynamoDB::Table                         N/A
+ Add                                        getAllItemsFunctionApiPermissionProd         AWS::Lambda::Permission                      N/A
+ Add                                        getAllItemsFunctionRole                      AWS::IAM::Role                               N/A
+ Add                                        putItemFunctionRole                          AWS::IAM::Role                               N/A
+ Add                                        getByIdFunction                              AWS::Lambda::Function                        N/A
+ Add                                        ServerlessRestApiProdStage                   AWS::ApiGateway::Stage                       N/A
+ Add                                        ServerlessRestApiDeploymenta4d359a69a        AWS::ApiGateway::Deployment                  N/A
+ Add                                        ServerlessRestApi                            AWS::ApiGateway::RestApi                     N/A
+ Add                                        getByIdFunctionApiPermissionProd             AWS::Lambda::Permission                      N/A
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Changeset created successfully. arn:aws:cloudformation:eu-central-1:000000000000:changeSet/samcli-deploy1667488518/14b1b688
2022-11-03 16:15:18 - Waiting for stack create/update to complete
CloudFormation events from stack operations (refresh every 0.5 seconds)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                               ResourceType                                 LogicalResourceId                            ResourceStatusReason
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
REVIEW_IN_PROGRESS                           AWS::CloudFormation::Stack                   sam-app-demo                                 -
CREATE_IN_PROGRESS                           AWS::CloudFormation::Stack                   sam-app-demo                                 -
CREATE_COMPLETE                              AWS::CloudFormation::Stack                   SampleTable                                  -
CREATE_COMPLETE                              AWS::CloudFormation::Stack                   getAllItemsFunctionRole                      -
CREATE_COMPLETE                              AWS::CloudFormation::Stack                   putItemFunctionRole                          -
CREATE_COMPLETE                              AWS::CloudFormation::Stack                   ServerlessRestApi                            -
CREATE_COMPLETE                              AWS::CloudFormation::Stack                   putItemFunction                              -
CREATE_COMPLETE                              AWS::CloudFormation::Stack                   getAllItemsFunction                          -
CREATE_COMPLETE                              AWS::CloudFormation::Stack                   getByIdFunctionRole                          -
CREATE_COMPLETE                              AWS::CloudFormation::Stack                   getAllItemsFunctionApiPermissionProd         -
CREATE_COMPLETE                              AWS::CloudFormation::Stack                   getByIdFunction                              -
CREATE_COMPLETE                              AWS::CloudFormation::Stack                   ServerlessRestApiDeploymenta4d359a69a        -
CREATE_COMPLETE                              AWS::CloudFormation::Stack                   getByIdFunctionApiPermissionProd             -
CREATE_COMPLETE                              AWS::CloudFormation::Stack                   putItemFunctionApiPermissionProd             -
CREATE_COMPLETE                              AWS::CloudFormation::Stack                   ServerlessRestApiProdStage                   -
CREATE_COMPLETE                              AWS::CloudFormation::Stack                   sam-app-demo                                 -
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CloudFormation outputs from deployed stack
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key                 WebEndpoint
Description         API Gateway endpoint URL for Prod stage
Value               https://jm9nvoft0x.execute-api.amazonaws.com:4566/Prod/
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Successfully created/updated stack - sam-app-demo in eu-central-1

Please note, the API Gateway URL is shown as the Value output. In my case it was: https://jm9nvoft0x.execute-api.amazonaws.com:4566/Prod/

As per official docs, that state that the URL format available is: http://localhost:4566/restapis/<apiId>/<stageId>/_user_request_/<path>

I can now test running Lambda locally using e.g. curl command (as mentioned above, I’m facing error here and need deeper investigation on the topic):

curl http://127.0.0.1:4566/restapis/jm9nvoft0x/Prod/_user_request_/
{"message":"The security token included in the request is invalid.","code":"UnrecognizedClientException","time":"2022-11-04T09:11:20.983Z","requestId":"2993H6FK7HRL42GIVAVIENCQ4NVV4KQNSO5AEMVJF66Q9ASUAAJG","statusCode":400,"retryable":false,"retryDelay":36.22586241543885}

You can use the aws cli to execute the commands, e.g. to see the cloudformation stacks, use:

aws --no-sign-request --endpoint-url=http://localhost:4566 --region eu-central-1 cloudformation list-stacks

Option 2: Running using SAM CLI locally (with DynamoDB only in Localstack)

As a prerequisite, we need to provision the DynamoDB in the Localstack. As there seems to be no simple way to provision only a subset of resources (namely DynamoDB only) via SAM cli, let’s take the lazy way and deploy everything. We’ll run the same command we referred to in the previous section:

samlocal deploy --resolve-s3 --stack-name sam-app-demo --capabilities CAPABILITY_IAM --no-fail-on-empty-changeset --region eu-central-1

Set the DynamoDB table name in the env.json file, in my case the updated contents is:

{
  "getAllItemsFunction": {
    "SAMPLE_TABLE": "sam-app-demo-SampleTable-6100dad0",
    "LOCAL_DYNAMODB": true
  },
  "getByIdFunction": {
    "SAMPLE_TABLE": "sam-app-demo-SampleTable-6100dad0",
    "LOCAL_DYNAMODB": true
  },
  "putItemFunction": {
    "SAMPLE_TABLE": "sam-app-demo-SampleTable-6100dad0",
    "LOCAL_DYNAMODB": true
  },
}

Then start the API locally, by running:

sam local start-api --env-vars ./env.json
Mounting getAllItemsFunction at http://127.0.0.1:3000/ [GET]
Mounting getByIdFunction at http://127.0.0.1:3000/{id} [GET]
Mounting putItemFunction at http://127.0.0.1:3000/ [POST]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2022-11-03 13:35:57  * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)

Afterwards, feel free to use curl for accessing the APIs (as advised in the start-api output). To retrieve all the DyanmoDB items or specific one run:

curl http://127.0.0.1:3000/

To insert new entry to DynamoDB, run:

curl -X POST -H "Content-Type: application/json" -d "{\"id\": \"id1\",\"name\": \"name1\"}" localhost:3000/

And fetch it afterwards all:

curl localhost:3000
[{"name":"name1","id":"id1"}]

or specific one only:

curl http://127.0.0.1:3000/id1
{"name":"name1","id":"id1"}

Please note, this option also supports remote debugging, once using flag -d <port_number> (e.g. for remote debuggin port 5858):

sam local start-api -d 5858 --env-vars ./env.json

Option 3: Running in AWS Cloud

  1. Build using command:
sam build

2. Deploy the stack to AWS account (assuming your profile is configured with name: sam-app-demo and you deploy to eu-central-1 region):

sam deploy --profile sam-app-demo --resolve-s3 --stack-name sam-app-demo --capabilities CAPABILITY_IAM --no-fail-on-empty-changeset --region eu-central-1

3. To test that endpoint works:

curl https://<api id>.execute-api.eu-central-1.amazonaws.com/Prod/

Connecting to DynamoDB running Localstack from NoSQL Workbench

  1. Open NoSQL Workbench
  2. In the Operation builder click: +Add connection button
  3. Switch to DynamoDB local tab
  4. Fill out the Connection name and Port number (4566 in our case), see:

Contents of the table sam-app-demo-SampleTable-* are shown afterwards:

(Optional) Generating OpenAPI definitions

OpenAPI is these days a standard for the documentation of the REST APIs.

As the API Gateway is capable of generating it automatically, let’s go for it, by running (please replace <api-id> with value relevant for your API):

aws apigateway get-export --parameters extensions='apigateway' --rest-api-id <api-id> --stage-name Prod --export-type oas30 --profile sam-app-demo oas30.json

The output is stored in local file: oas30.json. Please note, this option doesn’t work in the localstack, so you’d need AWS Cloud env to make it happen.

Conclusion

SAM offers an amazing tooling for the Lambda development and the related resources provisioning. It took me a while to find the way to overcome the issues for the local setup, so I hope you find the sum-up useful for your Lambda jump-start.

]]> <![CDATA[Visualizing circuit breaker behaviour with kiali addon for Istio]]> https://medium.com/@butkovic/visualizing-istios-circuit-breaker-with-kiali-cee45eaabc4d?source=rss-74fc69b20a6f------2 https://medium.com/p/cee45eaabc4d Sun, 25 Sep 2022 11:48:36 GMT 2022-09-25T11:49:33.831Z Let’s have a look at the option to play around with the circuit breaker pattern locally with the Istio service mesh and kind created kubernetes cluster. For the visual part we’ll use kiali addon.

As I’m running Podman, whose setup I’ve documented in: Favoring Podman over Docker Desktop, I won’t describe that in more detail, except the Podman machine initialization arguments.

The book

Circuit breaker pattern itself is nicely described in the book: Release it! by Michael T. Nygard, which I can only recommend for reading.

Source: https://www.amazon.com/Release-Design-Deploy-Production-Ready-Software/dp/1680502395

Local environment

Local environment setup consists of the information mostly documented on the: https://istio.io/latest/docs/setup/platform-setup/kind/.

Podman installation

The process is described in my other Medium article: Favoring Podman over Docker Desktop. There is one step, that is specific to kind & Istio setup, namely the Podman machine resources demand. Based on my experience following works:

podman machine init --cpus=2 --disk-size=60 --memory=8192 -v $HOME:$HOME
podman machine start

In case you’ve already started the podman machine, run the following commands as well:

podman machine stop
podman machine rm

Istio installation

Homebrew is my favourite package manager for MacOS. Let’s use it to install istio:

brew install istioctl

kubectl installation

We’ll also need kubectl command line tool, install it with:

brew install kubernetes-cli

Cluster Creation

Following the official docs for istio kind installation:

kind create cluster --name istio-testing

To setup the cluster config, run:

# to see the available contexts
kubectl config get-contexts
kubectl config use-context kind-istio-testing

Install Istio on k8s cluster

istioctl install --set profile=demo -y
kubectl label namespace default istio-injection=enabled

Install sample app

For detailed instructions see: https://istio.io/latest/docs/tasks/traffic-management/circuit-breaking/#before-you-begin

There is a httpbin app available at Istio GitHub, so you’d need to clone it first:

git clone https://github.com/istio/istio.git
cd istio
kubectl apply -f samples/httpbin/httpbin.yaml

Install circuit breaker via istio destination rule:

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
    outlierDetection:
      consecutive5xxErrors: 1
      interval: 1s
      baseEjectionTime: 3m
      maxEjectionPercent: 100
EOF

As documented in official instructions:

maxConnections: 1 and http1MaxPendingRequests: 1. These rules indicate that if you exceed more than one connection and request concurrently, you should see some failures when the istio-proxy opens the circuit for further requests and connections.

Install client app:

kubectl apply -f samples/httpbin/sample-client/fortio-deploy.yaml

kiali dashboard

To install it, run:

kubectl apply -f samples/addons
kubectl rollout status deployment/kiali -n istio-system

To open it in the browser, run:

istioctl dashboard kiali

Test time: single call

Let’s first run the single request and see how this affects stats in kiali:

export FORTIO_POD=$(kubectl get pods -l app=fortio -o 'jsonpath={.items[0].metadata.name}')
kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio curl -quiet
 http://httpbin:8000/get

In the Graph section (having last 5 mins set), we see the fortio pod contacting the httpbin service:

We can see the traffic stats on bottom right from fortio by clicking the pod:

As well as much more detailed in Applications -> Outbound Metrics view (of fortio):

Test time: 3000 calls performed serially

Now let’s see what happens when we test 3000 calls executed in a single thread:

$ kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 1 -qps 0 -n 3000 -loglevel Warning http://httpbin:8000/get
07:46:05 I logger.go:134> Log level is now 3 Warning (was 2 Info)
Fortio 1.38.0 running at 0 queries per second, 2->2 procs, for 3000 calls: http://httpbin:8000/get
Starting at max qps with 1 thread(s) [gomax 2] for exactly 3000 calls (3000 per thread + 0)
Ended after 15.730691105s : 3000 calls. qps=190.71
Aggregated Function Time : count 3000 avg 0.0052430763 +/- 0.0009012 min 0.004083992 max 0.022384591 sum 15.7292288
# range, mid point, percentile, count
>= 0.00408399 <= 0.005 , 0.004542 , 44.23, 1327
> 0.005 <= 0.006 , 0.0055 , 89.10, 1346
> 0.006 <= 0.007 , 0.0065 , 96.80, 231
> 0.007 <= 0.008 , 0.0075 , 98.73, 58
> 0.008 <= 0.009 , 0.0085 , 99.30, 17
> 0.009 <= 0.01 , 0.0095 , 99.63, 10
> 0.01 <= 0.011 , 0.0105 , 99.77, 4
> 0.011 <= 0.012 , 0.0115 , 99.83, 2
> 0.012 <= 0.014 , 0.013 , 99.87, 1
> 0.014 <= 0.016 , 0.015 , 99.90, 1
> 0.016 <= 0.018 , 0.017 , 99.93, 1
> 0.018 <= 0.02 , 0.019 , 99.97, 1
> 0.02 <= 0.0223846 , 0.0211923 , 100.00, 1
# target 50% 0.00512853
# target 75% 0.00568574
# target 90% 0.00611688
# target 99% 0.00847059
# target 99.9% 0.016
Error cases : no data
# Socket and IP used for each connection:
[0]   1 socket used, resolved to [10.96.157.177:8000] connection timing : count 1 avg 0.000122462 +/- 0 min 0.000122462 max 0.000122462 sum 0.000122462
Connection time (s) : count 1 avg 0.000122462 +/- 0 min 0.000122462 max 0.000122462 sum 0.000122462
Sockets used: 1 (for perfect keepalive, would be 1)
Uniform: false, Jitter: false
IP addresses distribution:
10.96.157.177:8000: 1
Code 200 : 3000 (100.0 %)
Response Header Sizes : count 3000 avg 230.00267 +/- 0.05157 min 230 max 231 sum 690008
Response Body/Total Sizes : count 3000 avg 824.00267 +/- 0.05157 min 824 max 825 sum 2472008
All done 3000 calls (plus 0 warmup) 5.243 ms avg, 190.7 qps

We see from the log, that all passed with HTTP code 200 => OK.

Nothing unusual on a kiali graph:

Test time: 3000 calls performed in 2 parallel threads

Performing 3000 calls in 2 parallel threads:

$ kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 2 -qps 0 -n 3000 -loglevel Warning http://httpbin:8000/get
08:04:18 I logger.go:134> Log level is now 3 Warning (was 2 Info)
Fortio 1.38.0 running at 0 queries per second, 2->2 procs, for 3000 calls: http://httpbin:8000/get
Starting at max qps with 2 thread(s) [gomax 2] for exactly 3000 calls (1500 per thread + 0)
08:04:18 W http_client.go:922> [0] Non ok http code 503 (HTTP/1.1 503)
08:04:18 W http_client.go:922> [0] Non ok http code 503 (HTTP/1.1 503)
08:04:18 W http_client.go:922> [0] Non ok http code 503 (HTTP/1.1 503)
08:04:18 W http_client.go:922> [0] Non ok http code 503 (HTTP/1.1 503)
08:04:19 W http_client.go:922> [0] Non ok http code 503 (HTTP/1.1 503)
08:04:19 W http_client.go:922> [0] Non ok http code 503 (HTTP/1.1 503)
08:04:19 W http_client.go:922> [0] Non ok http code 503 (HTTP/1.1 503)
08:04:19 W http_client.go:922> [1] Non ok http code 503 (HTTP/1.1 503)
08:04:19 W http_client.go:922> [0] Non ok http code 503 (HTTP/1.1 503)
08:04:20 W http_client.go:922> [0] Non ok http code 503 (HTTP/1.1 503)
08:04:21 W http_client.go:922> [0] Non ok http code 503 (HTTP/1.1 503)
08:04:21 W http_client.go:922> [1] Non ok http code 503 (HTTP/1.1 503)
08:04:24 W http_client.go:922> [1] Non ok http code 503 (HTTP/1.1 503)
08:04:25 W http_client.go:922> [1] Non ok http code 503 (HTTP/1.1 503)
08:04:25 W http_client.go:922> [1] Non ok http code 503 (HTTP/1.1 503)
08:04:28 W http_client.go:922> [1] Non ok http code 503 (HTTP/1.1 503)
08:04:28 W http_client.go:922> [1] Non ok http code 503 (HTTP/1.1 503)
08:04:28 W http_client.go:922> [1] Non ok http code 503 (HTTP/1.1 503)
08:04:28 W http_client.go:922> [0] Non ok http code 503 (HTTP/1.1 503)
08:04:29 W http_client.go:922> [1] Non ok http code 503 (HTTP/1.1 503)
08:04:29 W http_client.go:922> [0] Non ok http code 503 (HTTP/1.1 503)
08:04:30 W http_client.go:922> [1] Non ok http code 503 (HTTP/1.1 503)
Ended after 12.208230606s : 3000 calls. qps=245.74
Aggregated Function Time : count 3000 avg 0.0081280476 +/- 0.00139 min 0.000258827 max 0.025953012 sum 24.3841427
# range, mid point, percentile, count
>= 0.000258827 <= 0.001 , 0.000629414 , 0.60, 18
> 0.001 <= 0.002 , 0.0015 , 0.73, 4
> 0.004 <= 0.005 , 0.0045 , 0.90, 5
> 0.005 <= 0.006 , 0.0055 , 1.27, 11
> 0.006 <= 0.007 , 0.0065 , 4.70, 103
> 0.007 <= 0.008 , 0.0075 , 54.90, 1506
> 0.008 <= 0.009 , 0.0085 , 88.73, 1015
> 0.009 <= 0.01 , 0.0095 , 95.57, 205
> 0.01 <= 0.011 , 0.0105 , 97.47, 57
> 0.011 <= 0.012 , 0.0115 , 98.27, 24
> 0.012 <= 0.014 , 0.013 , 99.30, 31
> 0.014 <= 0.016 , 0.015 , 99.67, 11
> 0.016 <= 0.018 , 0.017 , 99.73, 2
> 0.018 <= 0.02 , 0.019 , 99.83, 3
> 0.02 <= 0.025 , 0.0225 , 99.97, 4
> 0.025 <= 0.025953 , 0.0254765 , 100.00, 1
# target 50% 0.00790239
# target 75% 0.00859409
# target 90% 0.00918537
# target 99% 0.0134194
# target 99.9% 0.0225
Error cases : count 22 avg 0.00069907759 +/- 0.000328 min 0.000258827 max 0.001549756 sum 0.015379707
# range, mid point, percentile, count
>= 0.000258827 <= 0.001 , 0.000629414 , 81.82, 18
> 0.001 <= 0.00154976 , 0.00127488 , 100.00, 4
# target 50% 0.000694811
# target 75% 0.000934602
# target 90% 0.00124739
# target 99% 0.00151952
# target 99.9% 0.00154673
# Socket and IP used for each connection:
[0]  13 socket used, resolved to [10.96.157.177:8000] connection timing : count 13 avg 6.7982923e-05 +/- 2.743e-05 min 3.0694e-05 max 0.000127744 sum 0.000883778
[1]  11 socket used, resolved to [10.96.157.177:8000] connection timing : count 11 avg 0.00042713355 +/- 0.001066 min 4.9292e-05 max 0.003794291 sum 0.004698469
Connection time (s) : count 24 avg 0.00023259363 +/- 0.0007438 min 3.0694e-05 max 0.003794291 sum 0.005582247
Sockets used: 24 (for perfect keepalive, would be 2)
Uniform: false, Jitter: false
IP addresses distribution:
10.96.157.177:8000: 24
Code 200 : 2978 (99.3 %)
Code 503 : 22 (0.7 %)
Response Header Sizes : count 3000 avg 228.34033 +/- 19.63 min 0 max 231 sum 685021
Response Body/Total Sizes : count 3000 avg 819.75167 +/- 49.74 min 241 max 825 sum 2459255
All done 3000 calls (plus 0 warmup) 8.128 ms avg, 245.7 qps

Results in 0.7% failure rate.

Kiali graph shows:

Test time: 3000 calls performed in 3 parallel threads

To demo the limitation, I had to use 3 parallel threads, where the throttling really showed off:

$ kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 3 -qps 0 -n 3000 -loglevel Warning http://httpbin:8000/get
08:19:26 I logger.go:134> Log level is now 3 Warning (was 2 Info)
Fortio 1.38.0 running at 0 queries per second, 2->2 procs, for 3000 calls: http://httpbin:8000/get
Starting at max qps with 3 thread(s) [gomax 2] for exactly 3000 calls (1000 per thread + 0)
08:19:26 W http_client.go:922> [0] Non ok http code 503 (HTTP/1.1 503)
08:19:26 W http_client.go:922> [0] Non ok http code 503 (HTTP/1.1 503)
08:19:26 W http_client.go:922> [2] Non ok http code 503 (HTTP/1.1 503)
08:19:26 W http_client.go:922> [2] Non ok http code 503 (HTTP/1.1 503)
08:19:26 W http_client.go:922> [2] Non ok http code 503 (HTTP/1.1 503)
08:19:26 W http_client.go:922> [2] Non ok http code 503 (HTTP/1.1 503)
08:19:26 W http_client.go:922> [2] Non ok http code 503 (HTTP/1.1 503)
08:19:26 W http_client.go:922> [2] Non ok http code 503 (HTTP/1.1 503)
08:19:26 W http_client.go:922> [2] Non ok http code 503 (HTTP/1.1 503)
08:19:26 W http_client.go:922> [1] Non ok http code 503 (HTTP/1.1 503)
08:19:26 W http_client.go:922> [2] Non ok http code 503 (HTTP/1.1 503)
08:19:26 W http_client.go:922> [2] Non ok http code 503 (HTTP/1.1 503)
08:19:26 W http_client.go:922> [2] Non ok http code 503 (HTTP/1.1 503)
...
Error cases : count 1957 avg 0.001547909 +/- 0.001475 min 0.000185792 max 0.017581766 sum 3.02925787
# range, mid point, percentile, count
>= 0.000185792 <= 0.001 , 0.000592896 , 49.41, 967
> 0.001 <= 0.002 , 0.0015 , 78.18, 563
> 0.002 <= 0.003 , 0.0025 , 89.22, 216
> 0.003 <= 0.004 , 0.0035 , 93.87, 91
> 0.004 <= 0.005 , 0.0045 , 96.01, 42
> 0.005 <= 0.006 , 0.0055 , 97.65, 32
> 0.006 <= 0.007 , 0.0065 , 98.77, 22
> 0.007 <= 0.008 , 0.0075 , 99.39, 12
> 0.008 <= 0.009 , 0.0085 , 99.59, 4
> 0.009 <= 0.01 , 0.0095 , 99.69, 2
> 0.01 <= 0.011 , 0.0105 , 99.74, 1
> 0.011 <= 0.012 , 0.0115 , 99.80, 1
> 0.012 <= 0.014 , 0.013 , 99.90, 2
> 0.014 <= 0.016 , 0.015 , 99.95, 1
> 0.016 <= 0.0175818 , 0.0167909 , 100.00, 1
# target 50% 0.00102043
# target 75% 0.00188943
# target 90% 0.00316813
# target 99% 0.00736917
# target 99.9% 0.014086
# Socket and IP used for each connection:
[0] 456 socket used, resolved to [10.96.157.177:8000] connection timing : count 456 avg 0.00012040605 +/- 0.0004198 min 2.3292e-05 max 0.005671351 sum 0.054905161
[1] 870 socket used, resolved to [10.96.157.177:8000] connection timing : count 870 avg 8.3235437e-05 +/- 0.0003149 min 2.5729e-05 max 0.00726584 sum 0.07241483
[2] 633 socket used, resolved to [10.96.157.177:8000] connection timing : count 633 avg 9.3076908e-05 +/- 0.0002951 min 2.1396e-05 max 0.004866478 sum 0.058917683
Connection time (s) : count 1959 avg 9.5067725e-05 +/- 0.0003368 min 2.1396e-05 max 0.00726584 sum 0.186237674
Sockets used: 1959 (for perfect keepalive, would be 3)
Uniform: false, Jitter: false
IP addresses distribution:
10.96.157.177:8000: 1959
Code 200 : 1043 (34.8 %)
Code 503 : 1957 (65.2 %)
Response Header Sizes : count 3000 avg 80.106333 +/- 109.7 min 0 max 231 sum 240319
Response Body/Total Sizes : count 3000 avg 443.83267 +/- 277.8 min 241 max 825 sum 1331498
All done 3000 calls (plus 0 warmup) 5.367 ms avg, 458.9 qps

Finally some higher error rate (65%) caused by the istio destination route settings.

Kiali graph shows:

Sum up

Combination of kind + Istio seems like a perfect way to quickly bootstrap and test the service mesh capabilities.

Moreover, kiali offers pleasant visualizations, giving insights to cluster components and their interactions.

The combination of these tools seems like a perfect fit for the demos and PoCs.

]]> <![CDATA[Favoring Podman over Docker Desktop]]> https://medium.com/@butkovic/favoring-podman-over-docker-desktop-33368e031ba0?source=rss-74fc69b20a6f------2 https://medium.com/p/33368e031ba0 Tue, 22 Mar 2022 08:21:57 GMT 2022-10-11T10:00:08.986Z Since the announcement, that docker desktop won’t be available for free for the bigger organizations, I’ve been looking for the open source alternative with smooth migration option.

Podman did a great job in delivering the docker-compatible experience. The major pain point was the lack of volume mounts. Since the podman 4.x release has the gap been finally closed.

I’m used to run my dev stack on docker-compose. In general it prevents a need for manual actions or custom scripts to be written for provisioning whatever your runtime depends on, e.g. database, message broker, or even external service mockups. Podman-compose project provides an alternative in the podman world. Let’s have a look on short demo on usage of these. Steps are adapted to MacOS, as that is the OS of choice for my laptop.

Installation

Brew installs the podman (https://formulae.brew.sh/formula/podman) and podman-compose (https://formulae.brew.sh/formula/podman-compose):

brew install podman
brew install podman-compose

Upgrade

If you need to upgrade the existing installations, for podman use:

brew upgrade podman

and for podman-compose use:

pip3 install --upgrade podman-compose

Podman machine setup

In podman we first need to create and run podman machine. Then we’re free to run podman or podman-compose commands.

Let’s say, we’re interested in:

  • option to mount the $HOME directory (or any of it’s subdirectories) to containers,
  • reserving 4 CPUs, 60GB of disk space and 6GB of memory for podman machine.

Let’s create the podman machine using:

podman machine init --cpus=4 --disk-size=60 --memory=6096 -v $HOME:$HOME

and start it afterwads:

podman machine start

To see if it runs, let’s use:

podman machine ls

Podman usage

Now let’s see whether we’re able to run the container with volume mount.

Let’s first run sample (ubuntu) container without mount and see what is present in /opt dir:

podman run ubuntu ls /opt

No output indicates, that /opt dir is empty in the container.

Now let’s try with volume mount option:

mkdir test-dir
touch test-dir/test-file.txt
podman run -v ./test-dir:/opt ubuntu ls /opt
test-file.txt

We see that mounted dir with it’s content is accessible in container now!

Podman compose usage

Now let’s give a podman compose a try for the similar case.

Having docker-compose.yaml file:

version: "3.9"
services:
  test:
    image: ubuntu
    volumes:
    - ./test-dir:/opt
    command: ls /opt

Running (only relevant output shown):

podman-compose up
...
test-file.txt
exit code: 0

we can see, that volume mount worked!

]]>