A Key Performance Indicator (KPI) is a way to measure and understand progress towards a particular result.
Certain KPIs are considered Health Metrics, which means we actively monitor them and alarm on them and take action when and if they fall out of expected range. Otherwise we may or may not have goals or initiatives associated with their outputs.
Meltano’s primary strategic goal is to achieve our vision towards product-market fit for Meltano as a DataOps OS. Our primary metric for measuring progress towards this goal is the average number of plugin types per project (APP).
We want to increase APP but it’s as much a tool to aid in prioritization as it is a metric we care about.
APP is a lagging indicator that should be interpreted in context of other KPIs and health metrics. Short-term fluctuations are expected and our goal is an increase of the KPI over the long term.
APP is calculated using CLI invocations for elt, run, test, and invoke and by categorizing each plugin into its appropriate type (i.e. Singer, dbt, Airflow, etc.).
For example, if a project executes the following commands:
meltano elt tap-github target-snowflake meltano run tap-gitlab some-map target-snowflake dbt meltano invoke airflow meltano test tap-gitlab some-map target-snowflake
APP for this project is 3 because only Singer, dbt, and Airflow plugins are used.
This definition of type is different from how we discuss plugin types in the Meltano documentation. In the above example, tap-gitlab, some-map, and target-snowflake are 3 different plugin types (Extractor, Mapper, and Loader, respecitvely), but for this metric they are all Singer-based plugins.
Since we have 5 tracked plugin types for the purposes of APP tracking, and since there are two ways of tracking each metric, there are 10 (5x2) component product penetration KPIs:
%monthly projects with “Orchestrator” plugins usage.
%monthly projects with “Transformer” plugins usage.
%monthly projects with “Utility” plugins usage for BI/Analytics.
%monthly projects with “Utility” plugins usage (Other).
%monthly projects with “Singer” plugins usage.
As with any ratio-based KPI, the total % or average product count can go down whenever we add to the denominator disproportionately. Since our denominator is number of total users, any large influx of new users may reduce the overall percentages and APP averages. Changes in the denominator should always be considered therefor when investigating or analyzing changes over time in these metrics.
Making KPIs as internal-by-default instead of “public” (counterintuitively) keeps politics and pride out of the honest reporting of our metrics, and it supports our values of company-wide humility, ambition, and transparency. This approach allows the best and most honest internal iterations on metrics, and promotes transparency across the company by default with no need for individuals to avoid or suppress metrics which at least initially don’t look “flattering” for meltano.
Additional KPIs and health metrics like retention, overall usage, and community engagement will be added to this page as we’re able to define and measure them.
Other metrics we track are documented in the data team handbook