My LFX Mentorship Journey
From Watching a Chaos Engineering Livestream to Building Dynamic Discovery in krkn-ai
apiVersion: lfx.krkn.io/v1
kind: Mentee
metadata:
name: Mrinal
program: LFX Term 2
project: krkn-ai
spec:
kubernetes:
knowledge: "almost none"
chaosEngineering:
knowledge: "discovered on a CNCF livestream"
openSource:
experience: "just getting started"
motivation:
- curiosity
- learning
- building something useful
- working with great engineers :)
objective:
- "Understand krkn-ai"
- "Contribute to open source"
- "Make discover smarter"
status: "accepted"
finalState:
kubernetes: "much better"
chaosEngineering: "much better"
openSource: "now an experienced contributor"
krkn-ai: "dynamic discovery"One December night in 2025, I started watching a CNCF livestream on Chaos Engineering. My earlier expectation from the stream was based on my semester course on nonlinear dynamics, where we were first introduced to chaos theory. We were taught how a butterfly flapping its wings in Delhi could theoretically cause a tsunami in Mumbai.
I found that the stream was different from our course, but the underlying idea was similar. In the stream, Chaos Engineering was introduced as the deliberate injection of failures into systems to build resilience.
This was my first dopamine hit, and my curiosity about it started increasing.
In mid-January, I found out that the same organization provides an open-source mentorship program through which we can directly interact with highly professional engineers in this field.
Since open source was new to me, I started exploring how this wonderful community works, and this was my second dopamine hit.
I made the decision that I would try my best for this opportunity.
I first thought of getting some experience with OSS and contributions, and then I would apply for the position. I targeted Term 2 of the LFX Mentorship Program and I was selected.
Why krkn-ai
You would have got some idea about why I applied for the mentorship program. Here I would like to explain my interest in krkn-ai.
I already knew about kraken through the stream, but I didn’t know how it worked. When the problem statement was released, I read it and found out that this would be the most interesting work that I could have the opportunity to work on.
Since I had already explored algorithms, ML research, and finance in my first three years of college, I had never worked with cloud-native tools, and I always had it in my mind that someday I would explore them.
I am driven by my curiosity and rational mindset. I thought that if I could enhance krkn’s discover command, this work would benefit a larger community.
This motivated me even more:
- It’s a great learning curve for me.
- This work would benefit a larger community.
- I would be able to work with great Red Hat engineers :) which was something I had never imagined.
It was equally challenging for me because this problem required specialized expertise in Kubernetes development, which I did not have.
So I was equally nervous.
Application Process
I began by learning how these systems worked, including OS-level concepts and Kubernetes fundamentals. I then started building krkn-ai through its documentation. Once I had a basic understanding of the system, I began replicating how a user would interact with it.
When I became familiar with it, I started researching and reproducing the issue to make notes.
I dug deep and was able to come up with a few rationales, and I also shared some of them with my mentor during the pre-application period on GitHub.
I made a proposal around it, and to my surprise, I was shortlisted for the interview. I never thought I would come this far.
I started preparing for the interview, getting more into kraken and solidifying my approach.
The Longest Night of My Life
I was very nervous about how things would go down.
Hopefully, the interview went well. I was able to answer some questions and was also stuck on a few.
This was my first interaction with my mentor, and one thing I was very sure about was that working with him would be a great experience since he walked me through some concepts during the interview like a teacher, without expecting that I should already know them.
The Great Day
I was sure that there was a 50–50 chance that I would make it. It was coming from my inner belief.
Hurray! I was selected for Term 2.
The first thing I did was send a gratitude message to my mentor for believing in me and motivating me to give 200% to this opportunity.
How krkn-ai Works
Krkn-ai uses a genetic algorithm to generate chaos scenarios for a Kubernetes/OpenShift cluster.
To know more about how things work inside, you should refer to this excellent documentation: krkn-ai Documentation
The Problem
krkn-ai discover connects to a Kubernetes/OpenShift cluster and produces a starter krkn-ai.yaml.
In short, krkn-ai discover parses the kubeconfig and filter flags. It uses the discover function to collect cluster components such as namespaces, pods, services, nodes, etc.
The result is passed to render krkn-ai.yaml, but it kept health checks commented out, used a hardcoded fitness query, and applied static scenario enable flags.
This YAML file is used by the run command to generate experiments.
The problem was that the generated configuration was based on static assumptions, while the cluster itself could be very different.
A configuration that makes sense for one cluster may not make sense for another.
That is where the idea of dynamic, cluster-aware discovery came in.
My Approach
I planned to keep the implementation close to the existing discover flow instead of creating a separate discovery system.
For this project, I addressed the issue by checking LoadBalancer Services and finding Prometheus when possible.
During the reproduction of the issue, one thing I noticed was that run command uses ScenarioFactory.generate_valid_scenarios, which already knows which scenarios can run against a given ClusterComponents object.
We used that to determine scenario eligibility.
The outcome is an improved dynamic discover path that makes it easier to start a first chaos experiment and reduces configuration errors.
For the fitness query, I thought of maintaining a catalog of template queries and having the best query be emitted from it.
This idea was also appreciated by my mentor since he also had this kind of approach in his mind.
My PRs
1) --save-strategy
My first major PR was a little interesting and would actually help after the completion of the dynamic discovery project.
The new control was --save-strategy, which decides what happens when krkn-ai.yaml already exists:
- overwrite — replace the existing file. This preserves the current behavior.
- skip — leave the existing file unchanged.
- merge — combine the existing config with the newly discovered config.
This gave users more control over how discovery behaves when a configuration already exists.
2) Dynamic Scenario Enablement
The second major PR was about transitioning static scenario enablement into dynamic cluster-aware scenarios.
Scenario classes know their requirements, which are used during the run experiment, so we used that logic in discover.
Initially, we passed all discovered cluster components, and the scenario function filtered out which scenarios were eligible based on the discovered components.
Instead of assuming which scenarios a cluster could run, discovery could now look at the cluster and determine what was actually possible.
3) Dynamic Health-Check Discovery
The third major PR was about health-check discovery.
Before, health-check URLs for fitness scoring had to be manually specified.
So instead, we reused what was already declared on the cluster.
For each LoadBalancer service, we look at the readiness or liveness probe already defined on the pod behind it, and build the health-check URL from that.
We also validate reachability at discovery time, so a dead recommendation surfaces immediately rather than in the middle of a run.
Those discovered URLs then feed into the fitness function, which scores them during chaos runs.
4) Dynamic Fitness-Function Recommendations
The fourth major PR was about dynamic fitness-function recommendations.
It followed the same pattern again: scenario enablement discovers which chaos experiments are possible, health checks discover what to watch, and fitness function recommendations discover what to measure.
We maintained a standard catalog of PromQL fitness queries covering availability, resources, nodes, etc.
Get Mrinalchaturvedi’s stories in your inbox
Join Medium for free to get updates from this writer.
At discovery time, we test each one against the cluster.
If the metrics it needs aren’t available, we skip it.
If they are available, we run it once to make sure it returns a single value, not something that will break the score later.
Queries that use a namespace get repeated for each namespace we found.
Whatever is left after this gets its weight balanced against the rest, so the fitness score reflects only checks that actually work on this cluster.
5) Improving the Fitness-Query Catalog
The fifth major PR was about improving the catalog.
For this work, I had zero experience since I had never worked on real production setups where I had to decide which queries people typically use.
As usual, my mentor really helped me a lot during this week.
I followed his suggestions and came up with some pretty good additions.
We added 16 fitness queries to the catalog, focused on etcd and Kubernetes.
6) Documentation
One thing I really learned during this mentorship, as well as in the open-source community, is that there is much more importance placed on good documentation.
The first thing a developer sees is documentation.
So my next work was around authoring good documentation that would be less verbose and simple enough for users to understand.
I documented all the major feature changes that were brought up during this time.
I did documentation for both the website as well as the local repository.
7) Dashboard
Since discovery now dynamically decides scenarios, health checks, and fitness functions per cluster, the dashboard’s static assumptions no longer matched what was actually running so it needed to be refactored to catch up.
I added a dedicated Fitness tab so users can see which fitness queries were recommended/enabled for their cluster and how each one scored across generations, instead of that being invisible.
Alongside it, I fixed several layout misalignments that surfaced.
The dashboard now reflects the same cluster-aware, dynamic-first model as the rest of the pipeline.
8) Dynamically Sizing the Genetic Algorithm
My work during the final week was very interesting and challenging for me.
The problem was to dynamically size the genetic algorithm configuration, i.e. the generation and population counts.
I researched some foundational papers on how we can determine optimal GA population sizes and also searched through some popular Chaos Engineering frameworks that use, and do not use, GA as a core algorithm for chaos setups.
This work resulted in PR #472.
What I Learned
This was one of the best experiences of my professional life and was the perfect learning curve.
I learned how to deliver in a competent environment.
I enhanced my understanding of how Kubernetes works and how Prometheus works in a production environment.
I learned how genetic algorithms can be used in an environment where no one might think of using them.
I learned how to strive for good work and also how to prepare before starting any work.
I learned how to reach out to my mentor when I was stuck, and how to handle all the controlled chaos life could throw at me while enhancing krkn’s chaos.
The Weekly Sync Meetings
My mentor was one of the best parts of my mentorship journey.
He played a crucial role in shaping my understanding of the technical aspects of the project and, equally importantly, the culture of open source development. His patience in answering my questions, willingness to guide me through challenges, and encouragement to explore different approaches made the entire experience much more meaningful.
This collaborative environment extended beyond the technical work, and he has a place in my heart since this was my first mentorship experience.
Things I Learned From My Mentor
1. Being a Professionally Sound and Competent Engineer
He is a very professionally sound and competent engineer. His experience and rational approach to problem solving are remarkable and inspiring.
His solutions were often like what we would call “book moves” in chess — the kind of solution that feels obvious once you see it, but is difficult to arrive at yourself.
A single instance I can remember is when I was stuck on how we should rank the best fitness queries.
He instantly came up with an approach that completely amazed me.
I know he is not particularly inclined towards recommendation systems, but I had little knowledge about them, and his solution felt like it came from a pretty experienced recommendation systems engineer.
2. How to Be Patient and Polite
He is extremely calm in different scenarios.
As a maintainer of a good community organization, I have never seen one instance of him losing his composure.
In a world of AI, where there are many PRs that are obviously spam, handling them calmly is a great feat.
3. Why Would Anyone Clean My Mess?
My mentor does that for me.
This is not about one or two instances. He cleaned up or improved my work himself without troubling me.
I still can’t digest why someone would do that for anyone.
4. Never Hesitate to Ask Questions
I never had a second thought about asking even an obvious question.
His doors were always open whenever I wanted to reach out.
I sometimes felt that I might be disturbing him with stupid questions, but he always acknowledged them and motivated me to work even more.
And for that, I will always be grateful.
Impact
The most important outcome of this work was moving krkn-ai discover from a static configuration toward a cluster-aware, dynamic discovery process.
I measured the changes on a live cluster:
Before this work, krkn-ai generated the same configuration for every cluster: four hardcoded scenarios, one hardcoded Prometheus query, and a commented-out block of sock-shop health-check URLs.
Discovery now derives all three from the cluster in front of it.
Future Plans
Participating in the LFX Mentorship Program with the krkn community was a transformative experience that expanded my horizons and solidified my passion for open source development.
This experience has enriched my technical knowledge and inspired me to continue my journey as a proactive contributor to the open source ecosystem.
I would definitely stick with kraken and hope to make meaningful contributions in any form.
We have some good research ideas on things that we can improve.
My next target is to do some research and try to add new ideas and contributions to the organization.
Acknowledgments
I would like to express my gratitude to my mentor, Rahul Shetty, for mentoring me through LFX, and also to the krkn community for being so inclusive and friendly.
It has been one of the best professional experiences that I have had this year.
Learning about new tools and technologies, and working on an open-source project along with experienced developers, has been a great experience for me.
I would love to continue working with kraken, improving my project, and contributing wherever I feel I can.
Connect & Resources
GitHub: @mrinalchaturvedi27
Email: mrinal.chaturvedi27@gmail.com
krkn-ai:
- https://krkn-chaos.dev/
- https://github.com/krkn-chaos/krkn-ai
- https://github.com/krkn-chaos/krkn-ai/issues/364 (Tracking Issue)
Some Impactful Contributions:
PR #375 — save-strategy for discover
PR #381 — Dynamic scenario enablement
PR #396 — Dynamic health-check discovery
PR #406 — Dynamic fitness-function recommendations
PR #456 — etcd + Kubernetes fitness query catalog
PR #471 — Dashboard per-query fitness breakdown
PR #472 — GA sizing from cluster targets
