Search tools, AI agents, MCP servers
Submit a tool

Search iThub

Search tools, categories and collections. Type at least two characters.

Explore all tools
↑ ↓ to explore resultsEnter to open · Esc to close

Monitoring & Observability

Empirical software engineering in practice: making engineering decisions from evidence

Empirical software engineering in practice: the evidence ladder, SLOs, feature flags, A/B test pitfalls, benchmarks, DORA and SPACE, with a worked example.

iThub Insights cover: Explainer, Empirical software engineering in practice, with the iThub logo tile
On this page
  1. What does empirical software engineering mean in practice?
  2. How does Dave Farley frame engineering as learning?
  3. What counts as evidence in software engineering?
  4. Which evidence answers which engineering question?
  5. How do you run an engineering experiment?
  6. Why should you instrument before you change anything?
  7. How do small batches and fast feedback make experiments cheaper?
  8. How do feature flags and canary releases reduce risk?
  9. What goes wrong with A/B tests?
  10. How do you benchmark and load test without fooling yourself?
  11. What role do postmortems and spikes play?
  12. How do you measure developer productivity without gaming it?
  13. Why keep a decision log of experiments?
  14. Worked example: will a cache reduce p95 latency?
  15. Sources

Empirical software engineering is the practice of making engineering decisions from observed evidence rather than from opinion, seniority or habit. You state what you believe will happen, design the cheapest test that could prove you wrong, measure the outcome and let the result shape the next decision. The term also names an academic research field; this guide covers the day-to-day version, where teams settle arguments about caches, rollouts and productivity with data instead of volume.

What does empirical software engineering mean in practice?

It means running the scientific method on your own system. The loop has four parts:

  • Hypothesis. A falsifiable claim, such as "moving session storage to Redis will cut p95 login latency by a fifth". "Redis is faster" is not one, because nothing could disprove it.
  • Experiment. A change made so the outcome can be attributed to it: a control group, a canary, a repeatable benchmark.
  • Measurement. Numbers collected the same way before and after, from instrumentation you trust.
  • Feedback. The result changes what you do next, including when it says your idea was wrong.

Evidence-based engineering decisions do not require a lab. Logs, metrics, traces and incident reports already hold the raw material. What is usually missing is the habit of writing the prediction down before looking at the data.

How does Dave Farley frame engineering as learning?

In Modern Software Engineering (Addison-Wesley, 2021), Dave Farley describes engineering as the practical application of science and treats software engineering as an empirical, scientific approach to finding efficient, economic solutions to practical problems (InformIT). The book reduces the discipline to two core exercises.

Optimize for learning. Its chapters cover working iteratively, feedback, incrementalism, empiricism and being experimental.

Optimize for managing complexity. Here the chapters cover modularity, cohesion, separation of concerns, information hiding and abstraction, and managing coupling.

The halves depend on each other. An experiment only tells you something when you can change one thing and hold the rest still, and in a tightly coupled system no result can be attributed to a single cause.

What counts as evidence in software engineering?

Think of evidence as a ladder: each rung costs more and supports stronger conclusions.

  1. Anecdote. "A customer said checkout felt slow." Useful for forming a hypothesis, useless for confirming one.
  2. Logs and metrics. Observational data from production. It shows what happened and when, but only correlation: a latency drop after a deploy may come from the deploy, a traffic shift or a quiet holiday.
  3. Controlled experiments. A/B tests, canaries and benchmarks with a control. Randomization or a simultaneous baseline lets you claim cause and effect.
  4. Published research. Studies across many organizations, such as DORA's program, presented in Accelerate by Nicole Forsgren, Jez Humble and Gene Kim (IT Revolution, 2018). One finding: speed and stability are not trade-offs, and top performers do well on all five DORA delivery metrics (DORA).

For questions about your own system, your own controlled experiment beats any industry study, because no paper has measured your traffic.

Which evidence answers which engineering question?

The skill is choosing the lowest rung that still answers the question honestly. Stopping lower than that produces confident mistakes.

Question Cheapest valid evidence Tool type
Is this endpoint getting slower over time? Latency percentiles over several weeks Metrics store and dashboards
Will this release break things for real users? Canary compared against the stable version at the same time Progressive delivery, feature flags
Does the new onboarding flow increase activation? Randomized A/B test with a metric and sample size fixed in advance Experimentation platform
Can the service handle twice today's peak? Load test against a production-like environment Load testing tool
Is library A faster than library B for our workload? Microbenchmark with warmup and repeated runs Benchmark harness
Is this approach technically feasible at all? Time-boxed spike Throwaway prototype on a branch
Why do incidents keep starting in the same area? Contributing factors tagged across postmortems Incident tracker, postmortems
Are our delivery practices improving? DORA metrics trended against the team's own history Data from version control and CI/CD
Is tooling friction slowing developers down? Developer survey plus system data Survey tool, CI and repository data
Does a new prompt give better LLM answers? Evaluation on a fixed dataset, then traced production samples LLM observability

How do you run an engineering experiment?

  1. Write down the question and the decision it feeds. If no result would change what you do, skip the experiment.
  2. State a falsifiable hypothesis. Name the metric, the direction and the threshold that counts as success.
  3. Measure the baseline. Record the current value and its normal variation. A change smaller than everyday noise cannot be detected.
  4. Pick the cheapest valid evidence. Use the table above as a starting point.
  5. Set guardrails and stopping rules up front. Which metrics must not get worse, when to abort, and how long it runs or how much data it needs.
  6. Run it with a control. Compare against a baseline measured at the same time, not last month's numbers.
  7. Analyze against the threshold you set. Check the whole distribution and key segments.
  8. Decide, record and clean up. Log the outcome, remove temporary flags and set a date to re-check.

Why should you instrument before you change anything?

Without a baseline, every result is a story. Instrument first, and measure what users experience.

Google's SRE book gives the vocabulary (Google SRE book, chapter 4). A service level indicator (SLI) is "a carefully defined quantitative measure of some aspect of the level of service that is provided". A service level objective (SLO) is a target value or range for an SLI. The chapter warns against choosing too many indicators and prefers percentiles to averages, because most metrics are better thought of as distributions.

In practice, record latency as histograms rather than pre-computed averages, so you can calculate p95 and p99 later; percentiles from different hosts cannot simply be averaged. Prometheus histograms, long-term storage in VictoriaMetrics and Grafana dashboards cover most stacks, and our Grafana vs Datadog comparison helps if you are still choosing. Add tracing where you need to know where time goes inside a request. For LLM features, Langfuse records traces and runs evaluations against datasets. More options sit in the monitoring and observability category.

How do small batches and fast feedback make experiments cheaper?

Every experiment costs a cycle of change, deploy and observe. Shorter cycles mean more affordable experiments and results that are easier to attribute.

DORA's guidance is to slice work so it can be finished in hours to a couple of days, and to treat a batch that takes longer than a week to complete and check as too big (DORA). When one small change ships and a metric moves, there is one suspect. When fifty changes ship together, there are fifty.

If changes queue for days in review or manual testing, the feedback loop is only as fast as that queue. The theory of constraints explains how to find the stage that sets the pace.

How do feature flags and canary releases reduce risk?

Feature flags separate deploying code from releasing behavior, which turns many releases into experiments. Pete Hodgson's article on martinfowler.com sorts them into four categories (Feature Toggles):

  • Release toggles Hide unfinished code and should be short-lived.
  • Experiment toggles Route each user consistently into a cohort for A/B testing.
  • Ops toggles Let operators switch off or degrade a feature, including long-lived kill switches.
  • Permissioning toggles Decide which users get a feature, sometimes for years.

Hodgson also warns that toggles carry a cost and tend to multiply, so treat them as inventory and plan each one's removal when you create it.

A canary release, in Danilo Sato's description, rolls a new version out to a small subset of users first, and rollback means routing those users back to the old version (Canary Release). For empirical work, compare the canary with the stable version over the same time window, so traffic patterns affect both equally. Progressive delivery automates the ramp: traffic share grows step by step while guardrail metrics stay healthy. Argo includes Argo Rollouts for this on Kubernetes, while GrowthBook and PostHog combine flags with experiment analysis. See the feature flags and experimentation category for more.

What goes wrong with A/B tests?

Online controlled experiments give the strongest evidence and are easy to misuse. Trustworthy Online Controlled Experiments by Ron Kohavi, Diane Tang and Ya Xu (Cambridge University Press, 2020) is the standard reference (Cambridge University Press). Common pitfalls:

  • No sample size calculation. Decide the smallest effect worth detecting, then compute how many users you need. An underpowered test ends in "no significant difference", which is not evidence of no difference.
  • Peeking. Checking a fixed-horizon test every day and stopping at the first significant p-value inflates the false positive rate. Either wait for the planned sample or use a sequential method designed for continuous monitoring.
  • Too many metrics. Test twenty metrics and one will probably look significant by chance. Pick one primary metric in advance.
  • Novelty and primacy effects. Users may click a new design because it is new, or avoid it because it is unfamiliar. Plot the effect over time and run the test long enough for it to settle.
  • Sample ratio mismatch. If a planned 50/50 split arrives noticeably uneven, assignment or logging is broken and the result cannot be trusted.

The book's chapter on trustworthiness rests on Twyman's law: a surprising result is more often an error than a discovery.

How do you benchmark and load test without fooling yourself?

Performance numbers are easy to produce and hard to trust. A few rules help:

  • Warm up first. JIT compilers, caches and connection pools make early runs slower. Discard them, or treat cold start as its own question.
  • Repeat and report variance. Run each variant many times, alternate the order and show the spread. Two single runs prove nothing.
  • Report percentiles, not averages. Give p50, p95, p99 and the maximum, ideally with the full histogram. Users feel the tail.
  • Control the environment. Same hardware, same data size, no other jobs on the machine.
  • Model the arrival rate. A load generator that waits for each response before sending the next request slows down when the system stalls, which hides the worst latency. Gil Tene named this coordinated omission. Arrival-rate executors in tools such as k6 keep sending requests at a fixed rate instead.

Related tooling sits in the testing and QA category.

What role do postmortems and spikes play?

Postmortems are data. The SRE book's postmortem chapter asks teams to focus on the contributing causes of an incident "without indicting any individual or team" and to agree on triggers in advance, such as user-visible degradation beyond a threshold or any data loss (Google SRE book, chapter 15). Blamelessness protects evidence quality, because people who fear blame edit the record. Tag each postmortem with contributing factors such as config change, missing alert or capacity, and after a year you have a dataset showing where reliability work pays off. Incident tooling like that in our Better Stack overview supplies the timeline.

Spikes answer feasibility questions. A spike, a term from Extreme Programming, is a short, time-boxed prototype built to learn something, then thrown away. It fits questions like "can this library handle our auth flow at all?" Write the question first, stop when the time box ends and record the answer, even "we could not tell".

How do you measure developer productivity without gaming it?

Productivity is where opinion is loudest and measurement most dangerous.

DORA metrics measure delivery: change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate. DORA itself cautions against turning them into goals, citing Goodhart's law, and against comparing very different applications; the aim is for a team to improve over time, not to compete (DORA).

SPACE, from Nicole Forsgren, Margaret-Anne Storey and colleagues, argues that developer productivity cannot be measured by a single metric or dimension (ACM Queue). Its five dimensions are satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow. The usual guidance is to combine metrics from at least three of them, including how developers themselves describe their work.

Goodhart's law comes from economist Charles Goodhart's 1975 work on monetary policy; the popular wording, "when a measure becomes a target, it ceases to be a good measure", is anthropologist Marilyn Strathern's. Reward commit counts and you get more, smaller commits. Use these metrics to find problems in the system, never to rank individuals.

Why keep a decision log of experiments?

Experiments that are not written down get repeated, misremembered or quietly reversed; Kohavi, Tang and Xu give institutional memory its own chapter. A Markdown file in the repository is enough. Each entry should hold:

  • date, owner and the question being asked
  • the hypothesis, metric and threshold, written before the experiment ran
  • what was done, for how long, with which control
  • the result, including inconclusive and negative ones
  • the decision taken and a date to revisit it

Negative results are the most valuable entries, because they stop the next team from retrying an idea that already failed under the same conditions.

Worked example: will a cache reduce p95 latency?

This example is hypothetical. The team, service and outcomes are invented to show the reasoning; they are not measurements.

A team runs a product catalog API. Users report slow product pages. One engineer wants a cache in front of the catalog database; another blames image processing. Neither has evidence.

Question. Should the team build a cache for catalog reads this sprint, or work on image processing?

Instrument. The team adds tracing to the product endpoint. In this scenario, traces show the catalog query is the largest span in the slowest requests, while image processing barely registers.

Hypothesis. "A read-through cache with a five-minute expiry on catalog queries will reduce p95 latency of the product endpoint by at least 25 percent over one week of normal traffic, without raising the error rate or serving prices older than five minutes." The threshold is the team's own bar for the work being worth it.

Cheapest valid evidence. A local benchmark cannot predict the hit rate, which depends on real request patterns. Replaying sampled production requests in a load test gives an estimate; a canary gives the answer.

Guardrails. Error rate stays within the SLO, cache memory stays under its limit and an ops toggle can disable the cache instantly.

Run. The cache ships behind a flag, enabled on a small share of instances, and the team compares p95 and p99 against stable instances over the same week.

Result. On popular products the canary beats the threshold. Across all traffic it falls short, because many requests carry per-user query parameters that make every cache key unique. Worse, p99 rises when popular keys expire and many requests miss at once, a cache stampede.

Decision. Partial confirmation: the cache helps where requests repeat. The team normalizes cache keys, adds request coalescing for misses, logs the result and reruns the canary. Without the canary, the stampede would have surfaced later as an unexplained incident.

Sources

Frequently asked questions

Is empirical software engineering the same as data-driven engineering?

The two overlap. Data-driven engineering often means watching dashboards and reacting to what they show. The empirical approach adds a prediction made before the data arrives and a control to compare against, which protects a team from reading whatever it hoped for into a chart.

Is empirical software engineering also an academic discipline?

Yes. Researchers use controlled experiments, case studies, surveys and mining of code repositories to study how software is built, and the field has its own journals and conferences. Practitioners can borrow its methods, especially careful study design, without publishing anything.

Does a small team need an experimentation platform?

Rarely at first. Metrics you already collect, a flag stored in configuration and a decision log in the repository cover most questions. A dedicated platform starts to pay off when several product experiments run at once and the statistics need to be handled the same way every time.

How long should an A/B test run?

Until it reaches the sample size calculated before launch, and for whole weekly cycles so that weekday and weekend behavior are both represented. Ending a test early because the result looks good is the most common way to ship a false positive.

What if a proper experiment is impossible or too expensive?

Use a lower rung knowingly and say so in the decision log. A staged rollout by region, a comparison with a similar service or a written expert estimate with the conditions that would change it are all better than an undocumented hunch. Favor reversible choices, so a wrong call can be undone cheaply.

Can the impact of AI coding assistants be measured empirically?

Yes, with the same method. Roll the tool out to some teams before others, compare delivery and stability metrics such as lead time and change fail rate, and survey developers about their experience. Counts of generated or accepted lines measure activity, not outcomes, and are easy to inflate.

How is an experiment log different from an architecture decision record?

An architecture decision record captures a design choice, its context and its consequences. An experiment log entry adds the hypothesis, the evidence gathered and how it was gathered. Many teams link the two, so a decision record points to the experiment that justified it.

Tools in this article

  1. EvidenceBI as code: reports written in Markdown and SQL, with an open-source CLI and the hosted Evidence StudioAnalytics & BI10 views
  2. PrometheusCNCF metrics monitoring system with a pull-based scraper, time series database, PromQL and alerting rulesMonitoring & Observability41 views
  3. VictoriaMetricsPrometheus-compatible time series database with PromQL and MetricsQL, as single-node or clusterMonitoring & Observability0 views
  4. GrafanaAGPL-3.0 dashboards, Explore and alerting for metrics, logs and traces from Prometheus, Loki and other sourcesMonitoring & Observability62 views
  5. LangfuseOpen-source LLM engineering platform for tracing, prompt management, evaluations and datasetsMLOps & LLMOps0 views
  6. ArgoOpen source Kubernetes tools for GitOps and workflowsCI/CD28 views
  7. GrowthBookFeature flags, experimentation and product analyticsFeature Flags & Experimentation0 views
  8. PostHogOpen-source product platform: analytics, session replay, feature flags, experiments and error trackingAnalytics & BI4 views
Share this article

Facts link to their official sources. Product details can change after publication: report an error or read the editorial methodology.

Emanuel DE ALMEIDA

Written by

Emanuel DE ALMEIDA

Entrepreneur · IT consultant · Software creator

Geneva-based entrepreneur, IT consultant and creator of SuniPDF and SuniCapture. Emanuel shares practical IT, cybersecurity and software insights on iThub.

Keep reading

More guides and analysis from the iThub editors.

All insights
Better Stack explained: uptime monitoring, on-call, status pages, logsWhat is Better Stack (formerly Better Uptime)? How its uptime monitoring, on-call escalation, status pages, logs and free tier work, and who it fits.Grafana vs Datadog: choosing an observability stackCompare Grafana and Datadog on hosting, metrics, logs, traces and billing models, then plan a trial using your own workload and usage.50 IT tools worth knowing as a system administratorA practical map of 50 IT tools for monitoring, security, cloud, delivery and data. Use each category to build a shortlist for your environment.