This conceptual paper introduces the MVH Arc — a framework describing the trajectory by which high-performing professionals, particularly in the technology sector, migrate from being Most Valuable Humans to Minimum Viable Humans. Drawing on peer-reviewed research, WHO data, and industry-specific burnout studies, it argues that the human capabilities being systematically eroded by overwork culture are precisely those that the next decade of AI-augmented work will demand most. The framework is not academic in origin. It is observed. The author writes from the middle of it. It is a conceptual framework presentation at best which can further be explored for realistic empirical studies.
1. The Observation
My LinkedIn feed on a Tuesday morning this week contained 5-10 promotion announcements, 10+ certifications on various AI technologies and two obituaries.
From confetti graphics to AI certification badges to phrases like “excited to announce” to the photograph of the person in early forties perhaps was celebrated for promotion in another cycle.
Nobody connected the two posts. Nobody ever does unless you are the one closely affected.
2. The Connect
This article is an attempt to connect them (submitting that corelation is not causality), not through grief, but through a framework. A framework that I am naming the MVH Arc, and which I am documenting here with a timestamp, because I believe the next decade will make it necessary.
So let me state something controversial. You can call it my journal if you think it is untrue.
“The most valued professional is often the least “valuable” person in the room — and increasingly, the least likely to survive the decade in one piece.
If you’ve spent years becoming “indispensable” at work, this is probably happening to you - Senior ICs transitioning to leadership, managers in delivery pressure environments and other high performers gunning to be partners
3. Is this Anecdotal?
Is the crisis merely anecdotal? — Maybe not.
If overwork is a civilisational problem for the middle aged professional, the technology industry is its leading edge.
The sector that built the tools for always-on productivity has applied those tools most aggressively to its own people.
As for me, I didn’t notice this shift happening. It didn’t feel like a drop. It felt like growth. More responsibility. Bigger projects. Higher stakes. But somewhere along the way, I optimized everything except being human.
4. Evidence
65% of engineers reported experiencing burnout in the past year. The 2025 LeadDev Engineering Leadership Report found only 21% of engineering leaders could be categorised as genuinely healthy — meaning the vast majority are operating in some state of depletion. [1]
67% of IT professionals report experiencing burnout, with System Administrators and DevOps Engineers recording the highest stress levels of any technical role.[2]
Perhaps the most clinically precise description of what is happening in the sector comes from a 2025 study of technology founders by CEREVITY, which introduced the term shadow burnout — defined as persistent exhaustion, cynicism, and reduced efficacy hidden behind continued high performance. The study found that 73% of surveyed founders met the criteria, with 68% actively concealing mental health struggles from investors and board members. [5]
5. My Interpretation
I may argue this is true for high performing professionals within many organizations, often working with high agency and semi-enterpreneurial or highly technical expertise often having to take decisions with platform implications. They are often in AI-induced technostress as mentioned in the Cerevity article, constantly under pressure to know it all, show it all, guide it all and what not!
Every other project and product team will have such an individual often attributed for visionary skills and remarkable turnarounds. Compressed learning and delivery cycle may eventually take a toll on these type of individuals. They will go broke and become less of a human, degrading in parts, if not in full.
6. The Framework Introduction
The Framework: From MVH to MVH – The just career arc for technologists/high performers
We perfected the MVP lingo — Minimum Viable Product. Ship fast. Ship lean. Learn and iterate later.
And the other MVP – Most Valuable Professional became its greatest achievement — certifications, promotions, AI badges, always on, always delivering.
But somewhere between the third promotion and the fifth burnout, the Most Valuable Professional became the Minimum Viable Human.
Functional. Deployable. Impressive on paper. Running on fumes and external validation..
Let me call this progression as
MVH Arc – Most Valuable Human to Minimum Viable Human
The most expensive journey you'll ever take. And the destination and the story arc are not really worth it. Let's admit that not all of are Mr. Robot from the series changing the world through anarchy.
You know this person. You might be this person.
They show up. They deliver. They’re always on. But ask them the last time they laughed without checking their phone, sat with a friend without half a mind on a deadline, or simply did nothing — and watch them go quiet.
They haven’t disappeared. They’ve just been… optimized.
7. The Irony
The decade ahead won’t be won by the most optimized human in the room. That’s for the agent to be.
It will be won by the most human human in the room.
People with judgment. Empathy. Presence. The ability to sit with another human being and actually see them. That is the scarce resource AI cannot manufacture.
These are what AI cannot replicate. And we are systematically destroying them in ourselves, in the name of growth.
The decade ahead desperately needs Most Valuable Humans and less of MVPs.
8. Where am I?
I’ll be honest because this post deserves honesty. I have never been the most valued human in my own evaluation.
I have measured myself in output, in delivery, in whether I was useful enough. The professional scorecard was always full. The human one I kept meaning to get to that. This post isn’t wisdom from the other side. It’s a note from the middle of it while the change is in progress.
Don’t let career growth outrun identity stability. Be identifiable in the end.
Drift & yet remain purposeful, like a solid, driftwood.
10. In conclusion
Most people don’t burn out. They slowly stop being fully human. And nobody notices — because the system rewards it.
The rat race doesn’t announce itself as dangerous. It just asks for a little more.
One more late night. One more certification. One more quarter.
FOMO does the rest because everyone else seems to be sprinting within the same cage. So stopping feels like failing until we faint.
But the most valuable thing you can be in the next ten years isn’t the most optimized professional in the room.
It’s a whole person who showed up. Someone worth knowing, not just worth hiring.
11. Remember
You are not a product. Stop shipping yourself into the ground.
There’s a better GitVersion of you out there, imperfect and unoptimized. You are not a costly GPU!
Know your life’s pace before you outpace.
Don’t be “Certified” Dead, at some point we all will anyway be one!
Don’t go from most valued to minimum viable human.
~ From one MVH professional to another ~
References
Wellhub. (2025). U.S. Work-Related Stress in 2025: Key Stats & Solutions.wellhub.com
DORA / Google Cloud. (2024). State of DevOps Report 2024. As cited in IT Support Group, 2025. thisisanitsupportgroup.com
State of Engineering Management Report (2024) as cited in CIO Magazine (2025); LeadDev. (2025). Engineering Leadership Report 2025.leaddev.com
Ketamine MC. (2026). Is the Tech Industry Facing a Mental Health Crisis?ketamc.com
CEREVITY. (2025). Shadow Burnout Among California Tech Founders: The Hidden Crisis Behind High Performance. Survey of 127 founders conducted January–December 2025. cerevity.com
Loaiza, I. & Rigobón, R. (2025). The EPOCH Framework. MIT Sloan School of Management. As cited in Futureneers, 2025. futureneers.io
EY. (2026). Redesigning Work Around Human Skills in the Age of AI.ey.com
From Paper to production story of Human + AI – How I used LLMs as a domain soundboard, data validator, and simulation builder to manage my apartment’s sewage treatment plant!
The Setup
For the last two years, I have been voluntarily managing the sewage treatment plant of my residential apartment complex. Not because I am a wastewater engineer – “I am not” but because nobody else on the committee had the bandwidth (or so they told), and I had the curiosity to question things as I stayed closer to its vicinity . Also, managing STP was brutally hard, this space was notorious with frequent mails, shouting fixes and what not! So, in a way, this was also forced upon me. In hindsight, it was a blessing in disguise for me to learn something new and apply technology meaningfully.
What started as “just keep it running” slowly turned into something more complex. The plant was showing signs of overload during monsoons. The operator (humble plumber) said it was “always like this.” The trigger point was the helplessness of our plumber when he said, “There is so much water and you tell me what to do with it!”. The previous committee had limited documentation. The logs were inconsistent. The regulatory filings used assumptions that did not match what I was observing on the ground.
I had questions. How much water are we actually consuming? Where does it go after use? Why does the STP overflow seasonally? Is rainwater getting into the sewage line? Are we even compliant?
I had no budget to hire a consultant. What I had: two years of manually collected operational data, sensor readings from a couple of installed meters, hand-drawn site maps, commissioning documents from regulatory authorities, and field knowledge from our plumber who has been maintaining the system for years. Given my background (next segment), I had ensured that in my two year tenure, we approach this data driven, if manually collected. So, in 2 years, I had this body of work to assimilate Very Important
And I had access to something that did not exist two years ago – large language models that could reason across domains. In its absence I was a bit lost manually doing this number crunching and modeling.
My Background (and the Gap)
My day job involves enterprise platforms and Industrial IoT. I have spent years working with sensor data, systems architecture, data pipelines, and building solutions that connect operational technology with information technology. So the analytical side – collecting data, structuring it, building models, thinking in systems – came naturally to me.
What I lacked was the wastewater domain knowledge. I did not know the standard hydraulic retention times for an aeration tank. I could not tell you what MLSS levels indicate healthy biology. I had no reference for what percentage of domestic water typically becomes sewage in an Indian residential context.
I was genuinely curious: could an LLM fill that gap? Not replace an expert, but act as a domain soundboard – the way a knowledgeable colleague would if I were brainstorming at a whiteboard?
What I Did
I spent roughly two weeks in deep collaboration with an LLM (primarily Claude, with some use of GPT for cross-validation and M365 Copilot for enterprise document workflows). Here is what that collaboration looked like:
Phase 1 – Data Collection and Structuring
I fed in two years of manual logs, sensor data exports, hand-drawn P&ID diagrams, regulatory documents, and field observations. The LLM helped me structure this into a normalized format – a 17-worksheet operational workbook covering asset inventories, maintenance schedules, daily logs, chemical dosing records, and process parameters.
This alone would have taken me weeks manually. With the LLM, it took about two days of iterative back-and-forth.
Phase 2 – Water Balance Analysis
This is where things got interesting. I needed to understand the full water balance: how much comes in, what percentage becomes sewage, how much is treated and reused, and critically – how much “extra” water was entering the STP that should not have been there.
I used the LLM as a calculation partner. I would share sensor readings, it would build the formula, I would cross-check against field observations, and we would iterate. The LLM caught a calculation error that had been sitting in our records for over a year – a “ghost” inflow number that previous analyses had been trying to explain with increasingly complex theories. Turns out it was a simple arithmetic mistake in the original model. We had been chasing a problem that did not exist. The fact that consumption and efficiency in some other places increased our fundamental inflow was something we all had neglected!
Nose on the wall, ha!
The corrected water balance revealed the real issue: rainwater was entering the sewage system through pathway floor drains during monsoons – a classic Inflow and Infiltration (I&I) problem. A physical site walk confirmed what the data model predicted. LLM suggested all interventions even the ones I knew and said what should I do to find it given I had fed detailed flow diagram.
Phase 3 – Scenario Modeling
Once the base model was solid, I asked the LLM to help me build seasonal what-if scenarios. What happens in dry season vs monsoon? What if we seal the infiltration sources? What if residents reduce consumption by 10%? What if we install dual-flush systems?
Seven scenarios, each with cost projections and capacity impact analysis. The kind of analysis that would typically require a consulting engagement.
Phase 4 – Deliverables for the Committee
The raw analysis is useless if the committee cannot understand it. I needed to translate engineering data into plain language that non-technical residents could follow. The LLM helped me produce:
A committee-ready briefing document with visual summaries
An investigation checklist for the next committee to pick up where I left off
Cost projections for both completed work and future actions
An executive summary for the annual general body meeting
Phase 5 – The Simulation App
This was the part that surprised me most. I specified the requirements for a desktop-based water balance simulator in about an hour – sliders for consumption patterns, checkboxes for seasonal conditions and infrastructure fixes, a live capacity gauge, animated flow diagrams showing water movement through the system.
Here is the simulator in action – a compressed 24-hour cycle showing how each tank behaves across a full plumber shift and overnight:
Watch the EQ tank fill overnight when the pump is off, and drain during the day. The DO (Dissolved Oxygen) gauge crashes during the 2-hour blower gaps and recovers when the next blower kicks in. Every tank has independent physics – I caught a bug where the aeration tank was rising after the pump stopped. It should drop. Getting this right required understanding what’s pumped vs. gravity, what’s continuous vs. scheduled.
The LLM (Claude Opus + Claude Code) generated the entire frontend (React-based, standalone, runs on a laptop with no internet). I focused on the data model and validation; it built the visualization and interaction layers. The result is a working proof-of-concept that I can demonstrate live at the general body meeting – residents can see in real time how sealing a drain or reducing consumption affects the plant’s capacity.
Bonus: Phase 6: The Hidden Cost Cascade
Every committee tracks the Cauvery (Water) bill. Almost nobody tracks what that water costs after it goes down the drain…
Blower overload — more water = higher backpressure at diffuser depth. O₂ margin shrinks from +90 to +6 kg/day
Pump wear — RSLP, FFP run 30-75% more hours than designed. Motor replacements 2x faster
Every extra KLD costs ~₹Lakhs/year in hidden STP operations cost! The upstream-downstream link is invisible by default. No maintenance ledger connects the Cauvery bill to the STP electricity bill to the pump replacement cost. AI helped me trace the full chain – from one extra shower to ₹5X,XXX/year in hidden STP costs. That single insight changed the committee’s conversation from “STP is fine” to “we need to understand our water budget.”
Where AI Genuinely Surprised Me – Hypothesis to Theory to Application!
Let me be specific about where the LLM added value that I could not have replicated on my own in the same timeframe:
As a domain soundboard. I would describe a symptom (“the EQ tank level rises overnight even when no one is using water”) and the LLM would walk me through possible causes – infiltration, groundwater seepage, leaking valves – with diagnostic steps for each. It was like having a knowledgeable colleague available at midnight. [ This was totally outside work and hence often really midnight 😉 ] This all happened because I had seen this pattern once before as well. I had earlier used an LLM to think through a soundproofing problem around the plant – target noise reduction, material options, procurement considerations, and how to measure whether the intervention actually worked. That gave me early confidence that AI could be useful as a practical soundboard when expert access is limited.
As a cross-validator. Every assumption I made – population estimates, per-capita consumption norms, return rates – the LLM challenged with references to Indian standards (IS:1172, NBC guidelines). It forced me to justify my numbers, which made the final analysis much more robust.
As an error catcher. The “ghost inflow” story I mentioned earlier. I had accepted a number from a previous analysis without questioning it. The LLM, working from first principles, flagged the inconsistency. That single correction changed the entire narrative of what was wrong with our STP.
As a visualization builder. I know data and backend systems. I do not build React frontends for a living. The LLM bridged that gap completely – generating production-quality UI code from a natural language specification. My role was to validate the math and the user experience, not to write CSS.
What I Learned (The Hard Way)
These are not theoretical observations. Each one cost me time, rework, or a moment of “wait, what just happened?”
1. LLMs fail brilliantly. When the context window overflows mid-reasoning, they do not make small errors. They silently drop entire sections of work – worksheets, formulas, validation steps – without any indication that something is missing. I lost a complete worksheet once because the model ran out of context and simply did not include it in the output. No error message. No warning. Just gone. I had to start off from 10 versions behind.
The fix: validate after every step. Not at the end. Every step. Do step 4 to avoid it.
2. Guardrails matter more than specifications. Especially the negative ones. Telling the model “do not modify any sheet other than the one we are currently reviewing” was more impactful than detailed instructions about what to do. If you have worked in enterprise architecture, this will feel familiar – constraints define the system more than features do – Read NFRs.
3. Challenge the LLM like you would challenge a junior analyst. Say “I think you made a mistake here – check this again.” Nine times out of ten, it recalculates, finds the error, corrects it, and – here is the key – internalizes that correction for the rest of the session. This is essentially what Judge LLMs need to do systematically: validate, challenge, and correct after every reasoning step.
4. Decompose ruthlessly. Break the problem into units small enough to validate independently and retract if wrong. If you cannot verify a step on its own, it is too large. I learned this after a batch update corrupted data across multiple sheets because I had asked the model to do too much in one pass.
5. The human is still the architect. The LLM accelerated everything by an order of magnitude. But the domain intuition – noticing that the plumber’s observation about “monsoon overflow” correlated with the pathway drain locations, connecting that to the sensor data anomaly, deciding to physically walk the site to confirm. While I owned this “chain of reasoning”, AI made it faster to act on those insights and critique my iniital observations. It also made mistakes but recovered eventually with right prompts.
The Bigger Takeaway
The gap between “enterprise technology professional” and “community problem solver” is thinner than we think. The same skills – systems thinking, data triangulation, stakeholder communication, iterative design, knowing when to trust the model and when to trust the field – apply whether you are architecting a cloud platform or optimizing a treatment plant.
What has changed is the accessibility of domain expertise. Two years ago, I would have needed a wastewater consultant to do what I did in two weeks. Not because the consultant knows something fundamentally unknowable – but because the knowledge was locked behind specialization. LLMs are flattening that barrier. Not eliminating the need for expertise, but making it accessible to adjacent professionals who bring complementary skills. This goes vice versa for a domain expert to build simulation software with LLMs and not developers.
What is Next
I am exploring what I can open-source from this effort. At minimum, operational templates that any apartment committee can adapt – asset registers, maintenance checklists, log formats, handover documents. Perhaps a longer technical write-up on the water balance methodology. And if there is interest, the simulation app framework itself.
If you are managing infrastructure in your residential community – or thinking about how AI applies to problems outside your core domain – I would be happy to exchange notes.
Disclaimer: I am not a wastewater management expert. I am an enterprise architect with an Industrial IoT background who enjoys solving real-world problems and connecting dots across domains. The best way to learn a tool is to point it at a problem that matters to you.
Musings – While I have been working on IoT related projects, there was a lot buzz around sustainability esp. Sustainability and the role of IoT in it. Thanks to my organization’s imperatives of learning, reaching a level 100 was not a big deal. During that time, I realized there are broadly two branches – Sustainability in Technology and Sustainability with Technology, my work being in the latter. While researching on this topic, along with my friend and colleague Roopa Shivani (Data Architect), we took a slight digression to investigate the literature on Green(er) Software (Sustainability in Technology – using Technology optimally to solve problems and not use everything at disposal even when it is not needed). Below is the material from the outcome – our point of view based on publicly available material on this topic.
Disclaimer: My company is part of Green Software Foundation which leads the sustainability in tech branch of industry evolution, thought leadership and standardization. This is an independent article without any proprietary information or sponsorship done at our spare times.
Abstract
In the era of moving on-premises systems to cloud based ecosystems, the basic ask from customers is about scaling the existing systems to a greater number of available resources with ease. This is aided by the seamless hardware expansion on demand, by thoughtlessly increasing the computational power in order to get the right outcomes. Our hardware systems have undergone many evolutions to accommodate high performing computational engines which obviously consume higher energy with GPUs and parallel processing. Increasing software workloads like bitcoin mining, Deep learning use cases with the need for better hardware amplifies the problem multifold. Our collective objective to remain planet friendly is to look at the opportunities where energy consumption can be reduced by designing optimum hardware and software. We look at software here as part of the below paper.
Introduction
In 2018, online video viewing generated more than 300 MtCO2, i.e., as much greenhouse gas as Spain emits: 1% of global emissions. [1] [2]
The greenhouse gas emissions of VoD (video on demand) services (e.g. Netflix and Amazon Prime) are equivalent to those of a country like Chile (more than 100 MtCO2eq/year, i.e. close to 0.3% of global emissions), the country hosting the COP25 in 2019[1][2]
The lead scientist, Rabih Bashroush, calculated that five billion downloads and streams clocked up by the song Despacito, released in 2017, consumed as much electricity as Chad, Guinea-Bissau, Somalia, Sierra Leone and the Central African Republic put together in a single year.[1][3]
Training a single natural language processing model produces as much CO2 as 315 return flights from New York to San Francisco.[1] [4]
The conservative studies estimate that the information and communication technology sector accounts for 4% of electricity consumption and 1.4% of carbon emissions worldwide .[1] [5]
The above stated and many similar studies highlight the importance of designing workloads that respect the energy for computation vs the actual need. One important concept that comes to the table is the role of Green Software which can also be read as sustainable coding. This emerging discipline puts forth the basic principles where we use the right language for the right task at hand and run it efficiently. Can this reduce the energy emissions and make our planet greener?
Green Software and Sustainable Coding
Green or sustainable software is developed and run in a way that ensures maximum energy efficiency and has minimal or no impact on the environment. There are two broad ways of looking at software in the context of sustainability – software as part of the climate problem and software as part of the climate solution. Building green software and doing it at scale requires the creation of a trusted ecosystem of people, standards, tools and best practices. The mission of Green Software Foundation is exactly that. [5]
The 8 principles of Green Software Engineering [6]
Carbon: Build applications that are carbon efficient.
Electricity: Build applications that are energy efficient.
Carbon Intensity: Consume electricity with the lowest carbon intensity.
Embodied Carbon: Build applications that are hardware efficient.
The answer lies in the two philosophies of sustainable software:
Everyone Has a Role in Fighting Global Warming and Climate Change
In this connected world every drop counts. To achieve our climate goals on a global scale, we need as many hands-on deck as possible. People with skills across industry, engineering and design are needed to create both conventional and unconventional solutions. So every sustainable line of code makes an impact.
Sustainability Alone Is Enough to Make Your Efforts Worthwhile
Sustainable software applications are often cheaper, better-performing, and more resilient than their conventional counterparts. However, the main reason to invest in green technologies should be sustainability itself. Everything else should be considered as bonus.
While it is a given that green software engineering is for reducing carbon emissions, like every project it needs to have its own business outcomes without which any attempt at this field will die within organizations.
Some of its advantages include:
Simpler architecture. Sustainable application architecture has fewer interdependencies. As a result, green software systems tend to be more straightforward and energy efficient. Case in point is moving to a simpler language or reducing parallel processing there by keeping the processor load low, using simpler languages and libraries.
Faster computing speed. Simpler systems make for faster software. Single threaded powerful workhorses are sufficient if the business outcome is not affected.
Low resource usage. Green systems use fewer resources, which translates into long-term cost savings. Either at local laptop or cloud level, resources can be reduced if we are reducing massive parallelism, consistency management, synchronization etc.
Brand resilience. Consumers today care about the integrity and social responsibility of their service providers. By raising awareness about environmental issues and reducing your carbon footprint through green software development, you can foster brand loyalty and resilience.
A Green Software & coding handout for developers
The next obvious question would be how shall we incorporate this into our software lifecycle. Often, we look at opportunities in efficient software development during implementation. The design phase is an outcome of several analysis at depth and necessary technical guard rails have to be added during that phase, if you are serious about green software. PoCs to finalize the design and implementation should ensure best coding practices. Most of the application development functional requirement does not target lowest energy consumption (which can be at times a hidden non-functional requirement). This presents the need of targeting the principles and objectives of Green Software right from the beginning phase of software development keeping it as one of the main requirements.
What handout could we give to our architects, leads and developers to ensure they choose the right software solution respecting the functional and non-functional requirements ?
Let us look at it from SDLC perspective – whether it is waterfall/staggered/agile elements of the below ideas should go into respective areas cyclically to ensure adherence to sustainability.
Requirement phase
Coding language of choice
There are a lot of scholarly articles comparing the coding language of choice which says C++ and C being the optimal and the likes of python, typescript being the least sustainable languages. Whereas many of the image AI libraries are in python on the contrary. Hence, during the requirement phase, choosing the right language is a tricky problem for the architect or developer!
One may need to weigh in many people factors like developer availability, developer’s preference, ease of maintenance, organization’s IT priority. There is a lot of dependency between green and brown software coding languages based on their developer friendliness and organization choice [11]. Let us not confuse the green and brown here with sustainability. In this context, green is about a developer preferred language which is often used like C#, Scala or Python versus brown like C, C++, Java etc. This ensures that most of the available libraries, source code samples on GitHub are all on these languages making it even more difficult for future developers to choose another sustainable language.
Sustainable programming languages battle the above developer choices. Architects often yield to the developer affinity and organizational choice to manage project execution speed.
If we are ready to go beyond that, then we can look at green coding and software.
Here the factors to measure are time, energy and memory [12]. Depending on whether we are working on a massively parallel financial software, scheduled batch analytics of a maintenance report, an IoT device application collecting and streaming data or a machine learning algorithm, our yardsticks or weightage between the 3 factors will vary. The reference [12] details the analysis to compare the languages choosing 2 among the factors and all the 3 together, where programming language C seems to the safest choice.
But we all know that C doesn’t fit into those full stack developer friendly languages. It is important that we look at application sustainability [14] deeper than the coding languages. Application workload may influence the language choice or have a larger sway on the factors like the IoT device which has low memory and need to optimize battery usage. So the jury is out there until we find the holy grail and middle path between green language for sustainability and developer love. The data here provide an initial attempt at choosing a better option.
In the context of this, what is important is to have a guidance that aligns well with good coding practices to optimize space and time complexity (that is the business case for green coding to ensure that we at least maintain the right coding practices which improves the 3 factors articulated here based on application or workload scenario).
Batch I/O operations to reduce frequent I/O access
Full readof data from filesystems to perform next operations should be done if the data size is less and within the memory capacity limits. This prevents accessing filesystem multiple times for each row/column. This will store all data in memory. All operations will then be performed in memory. If the input filesystem data is huge, full read may lead to out of memory exceptions. So ideal case would be to know the memory limits and decide whether to do a full read or read in batches which will still avoid reading every row every single time. If the scenario is real time streaming, verify if micro batch operations fulfill the requirement rather than going for line by line reading. Can this read be done on-prem without sending data to public internet reducing energy in transit?
Batch read/write records in database based on partitions if exist or Select Filter Query
Micro batch operations as well can be performed for near real time operations
Caching the objects not changing frequently. Caching is retaining not so frequently updated objects in memory. Although this prevents from accessing I/O frequently, it ends up adding cache related complexity and custom cache software.
Memory Consumption optimization
Type Sizing – One of the deciding factors of memory consumption is the size of the data type defined in the program.The smaller data type uses less memory. Despite the previous statement being so obvious, many standard programs have the assignments of integer data type while the actual value never expected to cross the boundary of size of short data type.
Analyzing the possible lowest and highest range of the actual value will help in choosing the right data type. The possibility of only non-negative(positive) values will further help in reducing the size allocation by 50%. For example, if the value is not going to exceed 32767 and cannot fall below -32768, then it is safe to declare the data type short than integer. If the value is only expected to be positive, then it can further cut short the allocation by declaring as unsigned short.
Making Garbage Collection faster
In case of .Net, when a new object is created, it is moved to Generation 0 of Garbage collection. The object keeps promoted to next subsequent higher Generations 1 and 2 if the object is still referenced. Objects in generation 0 are collected frequently and efficiently. Generation 0 holds very small memory in heap and thus least expensive while Gen 2 is the most expensive which includes the maximum memory space and is also called as Large Object Heap.
.Net exposes collect method of GC which forces the system to try to reclaim the maximum amount of available memory. All objects, regardless of how long they have been in memory, are considered for collection; however, objects that are referenced in managed code are not collected. However, The best practice is to not to call the GC.Collect() explicitly. Referencing object multiple times is expensive compared to creating objects as it keeps moving to higher generation when the existing gen level is getting garbage collected. The best way to prevent the objects moving from Gen 0 to Gen 2 is to prevent referencing objects frequently.The exception to this remark is the most used design pattern Singleton where object forever stays in memory however the intent of using singleton is for least expensive utility functions like logging, caching, database connections etc.
Weak References
The garbage collector cannot collect an object in use by an application while the application’s code can reach that object. The application is said to have a strong reference to the object. A weak reference permits the garbage collector to collect the object while still allowing the application to access the object. A weak reference is valid only during the indeterminate amount of time until the object is collected when no strong references exist. When you use a weak reference, the application can still obtain a strong reference to the object, which prevents it from being collected. However, there is always the risk that the garbage collector will get to the object first before a strong reference is reestablished.
Weak references are useful for objects that use a lot of memory but can be recreated easily if they are reclaimed by garbage collection.
There are multiple ways to detect memory leakage either using profiler, task manager or diagnostic tool window. The objective here is to highlight the prime sources which may cause memory leak.
The best practice is to use the using statement which in turn uses a try / finally statement behind the scenes, where the Dispose method is called in the finally clause.
Unregistering the user defined event handler from the event by implementing IDisposable.
Any type of caching mechanism can easily cause memory leaks. By storing cache information in-memory, eventually, it will fill up and cause an OutOfMemory exception. The solution can be to periodically delete older caching or limit caching size.
Efficient Data Structures, selecting less energy-greedy data structures
The mutable collection types in lower in algorithmic complexity to their corresponding immutable counterparts. Often immutable collection types are less performant but provide immutability – which is often a valid comparative benefit.
Approximate Programming, to reduce unnecessary precision of computations
Many applications perform computations targeting higher accuracy than required in actual which results in more consumption of resources. The objective of Approximate Programming is to deliberately reduce accuracy to save memory and energy.
In case of computations, often the double or decimal precision point is used where floating precision can just be sufficient. The below table shows the precision ranges of datatypes.
C# type/keyword
Approximate range
Precision
Size
float
±1.5 x 10−45 to ±3.4 x 1038
~6-9 digits
4 bytes
double
±5.0 × 10−324 to ±1.7 × 10308
~15-17 digits
8 bytes
decimal
±1.0 x 10-28 to ±7.9228 x 1028
28-29 digits
16 bytes
The decimal type is appropriate when the required degree of precision is determined by the number of digits to the right of the decimal point. Such numbers are commonly used in financial applications, for currency amounts.
Design Phase
Design phase efficiency depends on the deep understanding on requirement, with the ability to design the architecture framework with loosely coupled rules and abstracting the low-level information which should be flexible to extended at later steps. Define methodology to measure, report and correct deviations using platform logs.
.Net has defined reusable design patterns for multiple use cases which can be used in combination to solve the problem.
Infrastructure Design
Create infrastructure that is sufficient and scalable instead of over provisioning. In the case, of migration or optimization of existing infrastructure where workloads are run, there are tools and assets from vendors and system integrators to scan, plan and to optimize. if we need to run workloads at the edge and then cloud separately, then plan for it.
System design and platform design should be important for architects and product managers. Poor design may cause increase in energy consumption, say an UI polling a backend continuously using old world unoptimized APIs without improper caching or not co-locating servers when there is regular access needed. Such improper design can lead to lot of changes and platform refactoring later. Frequent calls over network for redundant data, an improper back-end design to represent data could be major blockers in impacting the performance and energy consumption. The design should also consider the nature of end user need and the domain criticality. For example, if the data at end user tool is not critically monitored then the use of real time pipelines can just be overkill and overutilization of resources, a simple one-time (2-3 times a day) batch process can handle the data updates instead running them every second/minute.
The experiments performed by Sahin et al. [2012] and Bunse et al. [2013] performed empirical studies where they compared the energy consumption of selected patterns. Their observation indicated that Particularly the Flyweight, Mediator and Proxy patterns resulted in energy savings when applied on selected applications, while the Singleton and Decorator pattern tremendously increased energy consumption.
Energy Optimization and Run Time Performance of Design Patterns & Data Structures
Bunse et al. focused on evaluating the energy consumption and run-time performance impact of design patterns on Android applications. The authors observed an increase in both energy consumption and execution time after applying six out of the 23 design patterns (Facade, Abstract Factory, Observer, Decorator, Prototype, and Template Method) on selected applications.
Structural changes in the design patterns can lead to significant energy optimizations as presented by Noureddine and Rajan [2015]. Their experiment revealed the Observer and Decorator as the most energy-greedy design patterns. The authors transformed the existing source code, by reducing the number of created objects and function calls and accomplished important energy savings. Specifically, after optimizing the Observer and Decorator design patterns, the authors reduced the applications’ energy consumption by 10%, on average.
Choosing the right data structure is also as important as design patterns considering some data structures are costlier than the others without any apparent additional benefits in terms of number of graph traversals versus a dictionary versus a map.
Implementation Phase
Class Libraries Performance
Avoid the need for synchronization, if possible. This is especially true for heavily used code. For example, an algorithm might be adjusted to tolerate a race condition rather than eliminate it. Unnecessary synchronization decreases performance and creates the possibility of deadlocks and race conditions.
Make static data thread safe by default.
Do not make instance data thread safe by default. Adding locks to create thread-safe code decreases performance, increases lock contention, and creates the possibility for deadlocks to occur. In common application models, only one thread at a time executes user code, which minimizes the need for thread safety. For this reason, the .NET class libraries are not thread safe by default.
Avoid providing static methods that alter static state. In common server scenarios, static state is shared across requests, which means multiple threads can execute that code at the same time. This may cause threading bugs which is highly energy consuming. Consider using a design pattern that encapsulates data into instances that are not shared across requests. Furthermore, if static data are synchronized, calls between static methods that alter state can result in deadlocks or redundant synchronization, adversely affecting performance. [11]
Parallel Programming
While parallel programming was in the cautionary list earlier, it is needed in certain other areas. This may sound contradicting but certain services on cloud are built for parallel workloads natively like Azure Data factory from Microsoft.
Data Level Parallelism
The data level parallelism is performed to run the same action on divided several chunks of data simultaneously instead either doing sequentially or running all in one go. The best approach is to divide the huge data and assign it to multiple threads/tasks to perform the action simultaneously.
Instruction Level Parallelism
While above the data is divided to perform same action, here actions are broken into parallel steps to speed up the total execution time. Here, one of the best examples can be ETL pipelines or any pipeline where the instructions like reading the data from multiple source systems can be run in parallel and different transformations based on nature of data can also be run on parallel than running as sequential steps.
Thread/Task Level Parallelism
Parallel execution of actions either my threads or Tasks are now a days over burned with the presence of easily configurable available high-end machines in the cloud. The number of threads using Thread pool or Task Parallel Library should be validated with the available configuration to benchmark the optimal numbers.
AI – Data, Storage, Training ML-Ops Cycle
Storage is cheap and getting cheaper. PaaS software gives the necessary cloud designs of replication, availability and the likes. Prebuilt algorithms to clean, learn and re-learn aided by DevOps help the MLOps run day in and day out without much value. Design should incorporate the need of the hardware infrastructure to run this MLOps and optimize it. As of today, a lot of energy is wasted in trying out newer algorithms for minimal % gains from monetary value and continuously running it.
Source Code Analysis
This is a technique by which tools detects the defects and vulnerability in software before deployment. There are many profilers or analyzer tools which efficiently detect these aspects of software. The objective is to target the following (roughly falls in performance improvement which affects energy consumption)
Unreachable code detection: Multiple such block in entire software results in blocking more unused storage.
Unused Variables: The cases where variables are declared and assigned with default value in beginning however it’s not called in the entire program. For example, defining double, string, long type of datatypes results in higher consumption of memory if not used at later stages.
Unused Functions: The functions defined but never called by any module of software.
Better code usage recommendation: The absence of efficient disposal of unmanaged resources will result in memory consumption when it is no longer needed.
There are many rules which identifies the vulnerabilities and defects as out of box feature of these tools. These tools also help in identifying the issues in code if go unnoticed.
Application Deployment
The deployment of software in dev, stage and production environment could lead to more energy consumption especially in prod if the dev ops pipeline is not well designed. Since the software is already running in 3 environments which means 3 times, the energy consumption is already high.
Evaluate how many environments we really need for a platform: while you need multiple environments could 2 environments, DevOps and a human in loop be an efficient solution as compared to 3-4 environments and end to end automation.
Minimum bugs in subsequent stages: Test Coverage should be more emphasized and automated test runs should be encouraged to detect the bugs faster and majority of bugs in dev itself. The most common practice is to keep dev environment with low power resources compare to stage and stage can be still low compared to prod environment. Hence the bugs leak into production. This repeats the test runs in all previous stages, thus the new deployment and test in production stage.
Automating the monitoring dashboards: This could be built in feature of cloud provider or third-party tool configurable which will help in monitoring overall all resources usage and help in identifying the bottlenecks.
Clean up deprecated Processes/Jobs: The development environment at times comprises of multiple combinations of tasks that are running jobs which will not be used later. It is necessary to identify the additional resources created for temporary purpose and clean them up to avoid space and energy consumption. This is application code repositories which are built even after refactoring or cleaning up.
Testing & Verification Phase
Framework, Tools and Benchmarking
The framework and tools enable the testing phase execution at ease. The benchmarking is one of the initial levels of agreement where the project decides the various KPIs to benchmark the software for its release to production. The best practice helps in defining the goals and targeting right from the initial phase.
The automated test framework development at unit test level helps in detecting the bugs at functional level and exposing the majority bugs. This reduces the number of repetitions of executions and deployment at stage or production level. The benchmarking here could be the defect leakage in subsequent stages. For example, the project team can define one of the KPI as no more than 10% of defects leakage to subsequent stage as a standard.
The code coverage tools help in detecting the efficiency of code coverage and hence reduces the additional code, memory and storage capacity. The benchmarking in percentage of code coverage sets the goal in defining the efficiency of the code. For example, the project team can set minimum 70% of code coverage for the project.
The memory and performance tools help in detecting the memory and performance imprint and hence taking actions on saving the energy.
The Load Test analyzers can be run locally and in cloud to understand the load usage of resources in that environment.
But the important point is to move away from aimless end to end automation to human monitored, AI/ML based automation in the DevOpsworld else we will be running large verification cycles without any major gains and with large energy expenditure.
Conclusion
There may be a lot more viewpoints out there. These are something that authors felt is a good starting point to debate on this. The above may seem like a summary of best coding guidelines from last decade. Partly that is true. We knew this.
Commonsense design, coding and management may make our software more carbon efficient.
To conclude, the authors here believe that soon enough the software editors like visual studio with plugins will be available to give options and suggestions to coders, much like Grammarly and Microsoft word suggestions to authors like us, to use an alternate expression or data structure based on the engineering and domain related benchmarks. Add to that an expert’s careful consideration of the technical requirement and business need behind it – functional and non-functional.
[10] Software Development Life Cycle for Energy-Efficiency: Techniques and Tools by STEFANOS GEORGIOU, Athens University of Economics and Business, Singular Logic S.A. STAMATIA RIZOU, Singular Logic S.A. DIOMIDIS SPINELLIS, Athens University of Economics and Business
Somewhere during the first half of 2017, our team was experimenting with Unity 3D based applications for Field Service maintenance scenarios. It brought together many technologies, some of which were in its nascency. Solution contained Mixed Reality application on HoloLens with a Digital Twin of a machine, AI to detect the machine, dynamic data rendering, contextual IoT data based on that machine, Skype calling with annotations, multiple people and multi device collaborations, action system to take in voice and gesture commands, integration with Microsoft D365 and warranty management on blockchain.
Quite over the top you may say. This was much before the Microsoft’s out of the box Field Services application like Remote Assist was released. And, my role was to ensure that we harmoniously blend the latest and greatest technologies to create a useful solution for Remote maintenance and field services.
Each of the technologies we had brought together to solve complex maintenance issues were independently interesting and collectively captivating. We went ahead, created a solution with some quirky nuances and applied for a patent during October of 2017.
It was an idea whose time had not come. It would later be called as “Metaverse“!
American author Neal Stephenson in his 1992 science fiction novel Snow Crash first called the concept in which humans interact with each other in a virtual reality based world, the next internet aka “metaverse”.
We understood that it is one thing to do a PoC and thought leadership exercise but totally different ball game to bring so many technology mashups without confusing the user and having real life adoption.
So what was that we built vs what are we talking about now!
For that, I guess you would need to recall the narrative we had about Internet of things a few years back. We always had sensors at home and industries, various protocols, ability to issue commands, programmable devices etc. What was so different about Internet of Things anyway? The common explanation was, it was the maturity of those new and independent technologies to interoperate, collect, ingest, make sense, predict, prevent, act, through edge and cloud, and the scalability of this platform. This overall maturity and the cost benefits, enabled new use cases to be envisioned. Today, we are ok with sharing our heart beat with technology companies, and as operators of manufacturing plants ready to share sensor heart beats (and data) with platform vendors, securely, in the hope that it will help predict and prevent failures!
Argument is the same.
While the technologies we used in our patent pending innovation of 2017 were similar to what we see today, all of them have attained a higher level of maturity. Massively parallel multi player video games on virtual realities, GPU enhancement, web3, NFTs, collaboration stack, secure data shareability, avatars, robust decentralized data systems, advance networks, 5G speeds, AI generated personas and deep learning and the likes, lower cost of entry, all come together, to bring that “new brave world”, where we can exist in a parallel world – both professionally and personally.
Metaverse Component Model
The different components have come under an easier name to tell the concept to everyone. But,
“We were learnt the difference between a Metaverse vs metAverse Application!”
the hard way.
The less spoken magic word is maturity of technology and experience. Fast forward to today, these 2 may seem commonsense terms, considering the cycles of websites, apps, OSes, mobile devices, tablets that we have been through. But, come metaverse, we will be grappling with another maturity curve where this will be repeated.
Before 2017:
I have been leading mobile applications and its experiences over a variety of platform building applications for enterprise clients. Starting off with Symbian and Qt apps, I transitioned to Microsoft, Android, iOS app store world. This journey would roughly be like the below graph where we tried unifying technology for development but still had native platforms to competing to create rich apps. Soon, we had too many applications fighting for our constant attention. Enterprise users were struggling to find their apps amidst the BYOD era. Some had work and personal phones separated out to manage this distraction. Eventually, we needed a single phone but then people started asking application aggregation on phones to avoid noise and security. In a few years, we discovered new features that could be augmented in phones – Gamification (one of our earlier patent for mobile gamified experiences), Siri, Cortana, assistants, Augmented Reality, advanced sensors on phones and we had a gamut of deep applications which were irreplaceable. Most became fancy features, until we realized that Application augmentation needed to have an experience that is worth the recall or monitory value. Thus finally we started looking for the holy grail of experience continuum, where there is smooth handoff between new features and new devices from the existing.
Note: Meanwhile, gamers kept playing multi player games on all form factors 🙂
This realization perhaps gave me Experience Aversion towards ill conceived continuums. I believe that is what we have to understand in this new wave as well. Else, our rush to create metaverse experiences will rather become aversion for the same experiences we fondly construct, making it MetAverse!
Experience abstraction and designing the continuum..
My experience building the Field Services app was based on these learnings where we tried to segregate human experiences.
Sensory Experiences in general –
Visual, Audio, Olfactory (smell) and Tactile, Neural – how we are going to get augmented by these?
Visual is easy – AR/VR/MR – By now, most of us have experienced this from 360 immersive videos to VR to AR to MR across various form factors and devices, tethered and untethered experiences.
Audio – Aural AI – I still think this is much more powerful that what we have experienced so far. Why? I believe this is one of the only distractions that a human brain allows while doing another task – typing , walking, running etc. True multi tasking can be achieved when you are listening to something repetitive like classical music along with your work or something interesting like audio book while doing a chore like folding clothes. Much of my thoughts in this area has been inspired from the movie HER . At some point in future, AI agents like Siri and Alexa will break out of the existing imagination into much more pervasive systems.
Olfactory and tactile – are touch and smell based systems – While not much used, this was researched and embedded into our demonstrations as fingerprint sensors, magnetic and electric vibrations to understand cloth textures remotely.
Neural – using devices like Emotiv, sensors which could approximately understand human actions and emotions could one day help create better systems was what we had researched. We tried to create synthetic experiences based on actual experiences. While devices we had were quite evolving, our ideas were to understand inherent human behaviours during mundane, repeated tasks like electronics repair.
These design experiences had to be compared in the context of below 4 use case archetypes
Product Design
Maintenance Jobs
Learning & Training Sessions
End Customer Experience
for every single persona, be it design engineer/s in a product design team working on a CAD model OR a maintenance staff learning about part disassembly and assembly to build muscle memory OR a customer experiencing a custom dress before an in-store experience.
Put all of these together and you will get a matrix like the one below.
Such objective vision of subjective experiences, helped us understand which experiences are good during what phase. what may cause more distraction, helping us understand the VR and MR’s role in Designing products, Training employees whereas VR and AR’s relevance for (B2C) customer experiences.
I suggest a Metaverse enthusiast to really understand the nature of the application, personas and the human experiences really well to create meaningful experiences. The technology stack that define a metaverse application like VR, GPUs, networking devices, AI, web3 etc. will mature eventually to create that seamless continuum. This seamless continuum is a 3 Dimensional – Experience continuum, Application continuum and Technology continuum.
Words of Caution In our pursuit to create a superior metaverse application which is not experience averse, we shouldn’t push our unsuspecting users into a dystopian hyperreality.
Trilok R | September 2022
Next: Industrial Metaverse, Digital Twin, Virtual Twin – Beyond the narrative..
LogRCubed, A Whole Brain & A Few Ciphers 