[Econ] Fix biophysical-to-economics bedding mapping and per-pen cost processing by matthew7838 · Pull Request #3117 · RuminantFarmSystems/RuFaS

matthew7838 · 2026-06-25T09:18:55Z

Context

Issue(s) closed by this pull request: closes #3088

What

Reworks how the economics module computes the bedding cost line item. The "Bedding requirements" mapping entry is now routed to a dedicated per-pen processor (EconomicPreprocessor._preprocess_bedding) instead of the generic quantity×price engine, which could not represent bedding correctly. Also extracts two shared helpers (_package_line_item, _warn_if_pricing_missing) so the generic path and the new path build their output through one definition.

Why

On the example freestall farm the old code produced ~$42.50/yr of bedding cost when the correct figure is ~$18,700/yr (verified end-to-end; see Test plan). Three underlying bugs, all in the shared engine:

Price never matched. A pen's bedding_name is a user config name (e.g. calf_straw), but it was compared directly against the price-file keys (e.g. straw), so no price file ever matched and every pen fell back to a $1/head placeholder.
Pens were lumped. All pens' daily animal counts were averaged into one number and multiplied by one averaged price, never pairing a pen's count with that pen's own bedding price.
Annual units ignored. Prices are dollar_per_head per year while counts are per day, and a multi-year run needs each year priced separately.

How

Mapping (mapping.py): tag the "Bedding requirements" entry with dedicated_processor: "bedding", add a bedding_type_to_file_key table and bedding_configs_path; remove the brittle hardcoded wildcard_value_map.
Processor (preprocessing.py): _preprocess_bedding resolves each pen bedding_name → bedding_configs.bedding_type → price file, matches pens by their id field (not list position), and for each calendar year computes (head-days ÷ days-in-year) × that year's price, summed over pens and years.
Robustness: pens with bedding_type "none" cost nothing; a missing price year falls back to the nearest available year; a missing FIPS county, an unloadable price file, an unknown config, or an unmapped type each emit a warning instead of silently charging $1/head.
De-duplication: both the generic engine and the bedding processor now return via a shared _package_line_item and warn via a shared _warn_if_pricing_missing, so the two paths can't drift apart.

Test plan

Run updated unit tests.
SME review.

Input Changes

input/data/EEE/economics_map.json — updated the "Bedding requirements" block to mirror mapping.py

Output Changes

The bedding line item within econ_economic_line_item_breakdown and econ_preprocessed_economic_inputs now reflects correct per-pen annual cost

Filter

{
    "direction": "landscape",
    "multiple": [
        {
            "name": "Bedding total cost ($)",
            "filters": ["econ_bedding_total_cost$"]
        },
        {
            "name": "Billed lactating head-years",
            "filters": ["econ_bedding_billed_head_years$"]
        },
        {
            "name": "Average bedding price ($/head/yr)",
            "filters": ["econ_bedding_avg_price_per_head_year$"]
        },
        {
            "name": "Average lactating head per day",
            "filters": ["number_of_animals_in_pen_.*LAC_COW$"],
            "vertical_aggregation": "average"
        }
    ]
}

…pping

bradenlimb · 2026-07-20T00:18:36Z

Hi Matthew,

I reviewed this, but the values coming out the backend still don't seem correct for me. The average bedding cost is $53.22/head/year and I believe there is 100 head of cattle. Therefore the aggregated cost in the freestall scneario should be closer in the high $40k range. It is outputting a values of $70874 for me which is also different than the ~$18,700/yr you mentioned above since it's running for 8 years.

I think something is going wrong in the preprocess step or at least how it is being reported. The total cost should equal the "biophysical_aggregate" value multiplied by the "price_aggregate" value. But currently the total cost on the "line_item_values_by_scneario" variable is equal to the "biophysical_aggregate" value

Hopefully this makes sense. If not, I'll be on the econ task force call tomorrow if you're available to discuss.

matthew7838 · 2026-07-21T14:33:14Z

Hi Matthew,

I reviewed this, but the values coming out the backend still don't seem correct for me. The average bedding cost is $53.22/head/year and I believe there is 100 head of cattle. Therefore the aggregated cost in the freestall scneario should be closer in the high $40k range. It is outputting a values of $70874 for me which is also different than the ~$18,700/yr you mentioned above since it's running for 8 years.

I think something is going wrong in the preprocess step or at least how it is being reported. The total cost should equal the "biophysical_aggregate" value multiplied by the "price_aggregate" value. But currently the total cost on the "line_item_values_by_scneario" variable is equal to the "biophysical_aggregate" value

Hopefully this makes sense. If not, I'll be on the econ task force call tomorrow if you're available to discuss.

Hi Braden, I have made another update addressing these problems. As for the first part, I've made changes reflecting only the costs of lactating cows in the pen, which averages to around 89 of them. Therefore, I'm getting around 33k as the result. Let me know how things are looking on your side.

bradenlimb

All looks great to me - thanks for fixing this!

allisterakun · 2026-07-29T14:04:12Z

            "Bedding requirements": {
                "biophysical_simulation": ["AnimalModuleReporter.report_daily_pen_total.number_of_animals_in_pen_.*"],
                "input_manager": ["animal.pen_information.*.manure_streams.0.bedding_name"],
-                "match_source": "input_manager",
-                "wildcard_value_map": {"0_CALF": "0", "1_GROWING": "1", "2_CLOSE_UP": "2", "3_LAC_COW": "3"},
+                # Handled by EconomicPreprocessor._preprocess_bedding (issue #3088):
+                # each pen's bedding price is paired with that pen's animal count
+                # and converted from the annual dollar-per-head price.
+                "dedicated_processor": "bedding",
+                "bedding_configs_path": "animal.bedding_configs",
+                # SME-confirmed: the dollar-per-head bedding prices are per LACTATING
+                # cow, so only lactating-cow pens are billed. Other pens still get
+                # physical bedding in the biophysical simulation; they are just not
+                # part of this economic line item.
+                "billable_pen_combinations": ["LAC_COW"],
+                # A pen's manure_stream bedding_name is a user config name; resolve
+                # it to the config's canonical bedding_type, then to an economics
+                # file key below. Types with no entry here (e.g. "none") cost nothing.
+                "bedding_type_to_file_key": {
+                    "sand": "sand",
+                    "sawdust": "sawdust",
+                    "straw": "straw",
+                    "CBPB sawdust": "CBPB",
+                    "manure solids": "manure_solids",
+                },
                "economics_files": {
                    "CBPB": "commodity_prices_bedding_compost_bedded_pack_dollar_per_head",
                    "manure_solids": "commodity_prices_bedding_manure_solids_dollar_per_head",
                    "sand": "commodity_prices_bedding_sand_dollar_per_head",
                    "sawdust": "commodity_prices_bedding_sawdust_dollar_per_head",
                    "straw": "commodity_prices_bedding_straw_dollar_per_head",
                },
-                "preprocessing": "average number of animals in each pen",
+                "preprocessing": "per-pen annual bedding cost: average head per year x dollar-per-head-per-year",
            },


Why the duplicate information?

allisterakun · 2026-07-29T14:12:48Z

+
+        path = item.bedding_configs_path or "animal.bedding_configs"
+        configs = self.im.get_data(path)
+        name_to_type: dict[str, Any] = {}


Suggested change

name_to_type: dict[str, Any] = {}

name_to_type: dict[str, str] = {}

any reason that we'd expect any other type besides str?

allisterakun · 2026-07-29T14:13:19Z

+        if isinstance(configs, (list, tuple)):
+            for config in configs:
+                if isinstance(config, dict) and "name" in config:
+                    name_to_type[str(config["name"])] = self._normalize_bedding_type(config.get("bedding_type"))


add error handling for if configs is not the expected type

allisterakun · 2026-07-29T14:13:30Z

+                if isinstance(config, dict) and "name" in config:
+                    name_to_type[str(config["name"])] = self._normalize_bedding_type(config.get("bedding_type"))


Same, error handling

allisterakun · 2026-07-29T14:14:28Z

+    def _normalize_bedding_type(self, raw: Any) -> Any:
+        """
+        Normalizes a bedding type into a plain value.
+
+        Parameters
+        ----------
+        raw : Any
+            The bedding type value to normalize.
+
+        Returns
+        -------
+        Any
+            ``raw.value`` when ``raw`` exposes a ``value`` attribute, otherwise ``raw``
+            unchanged.
+
+        Notes
+        -----
+        A bedding type may be a plain string (read directly from JSON input) or an
+        enum-like object exposing a ``value`` attribute. Returning the underlying
+        value in both cases lets downstream lookups always compare against strings.
+
+        """
+        return getattr(raw, "value", raw)


Suggested change

def _normalize_bedding_type(self, raw: Any) -> Any:

"""

Normalizes a bedding type into a plain value.

Parameters

----------

raw : Any

The bedding type value to normalize.

Returns

-------

Any

``raw.value`` when ``raw`` exposes a ``value`` attribute, otherwise ``raw``

unchanged.

Notes

-----

A bedding type may be a plain string (read directly from JSON input) or an

enum-like object exposing a ``value`` attribute. Returning the underlying

value in both cases lets downstream lookups always compare against strings.

"""

return getattr(raw, "value", raw)

def _normalize_bedding_type(self, raw: Any) -> str:

"""

Normalizes a bedding type into a plain value.

Parameters

----------

raw : Any

The bedding type value to normalize.

Returns

-------

Any

``raw.value`` when ``raw`` exposes a ``value`` attribute, otherwise ``raw``

unchanged.

Notes

-----

A bedding type may be a plain string (read directly from JSON input) or an

enum-like object exposing a ``value`` attribute. Returning the underlying

value in both cases lets downstream lookups always compare against strings.

"""

return getattr(raw, "value", raw)

Can we modify the logic here a bit to make sure it returns a str?

allisterakun · 2026-07-29T14:19:02Z

+    def _get_simulation_start_date(self) -> tuple[int, datetime]:
+        """
+        Returns the simulation start year and start date.
+
+        Returns
+        -------
+        tuple[int, datetime]
+            The start year of the simulation.
+            The calendar date of the first simulation day.
+
+        Notes
+        -----
+        The start date is read from ``config.start_date`` in the ``YYYY:day_of_year``
+        format (for example ``"2013:20"``) and converted to a calendar date.
+
+        """
+
+        raw = self.im.get_data("config.start_date")
+        parts = str(raw).split(":")
+        year = int(parts[0])
+        day_of_year = int(parts[1]) if len(parts) > 1 and str(parts[1]).strip() else 1
+        start_date = datetime(year, 1, 1) + timedelta(days=day_of_year - 1)
+        return year, start_date


Suggested change

def _get_simulation_start_date(self) -> tuple[int, datetime]:

"""

Returns the simulation start year and start date.

Returns

-------

tuple[int, datetime]

The start year of the simulation.

The calendar date of the first simulation day.

Notes

-----

The start date is read from ``config.start_date`` in the ``YYYY:day_of_year``

format (for example ``"2013:20"``) and converted to a calendar date.

"""

raw = self.im.get_data("config.start_date")

parts = str(raw).split(":")

year = int(parts[0])

day_of_year = int(parts[1]) if len(parts) > 1 and str(parts[1]).strip() else 1

start_date = datetime(year, 1, 1) + timedelta(days=day_of_year - 1)

return year, start_date

def _get_simulation_start_date(self) -> datetime:

"""

Returns the simulation start date.

Returns

-------

datetime

The start year of the simulation.

Notes

-----

The start date is read from ``config.start_date`` in the ``YYYY:day_of_year``

format (for example ``"2013:20"``) and converted to a calendar date.

"""

return datetime.strptime(str(self.im.get_data("config.start_date")), "%Y:%j")

The start year should be easy to get: start_year.year.

allisterakun · 2026-07-29T14:21:51Z

+            except (TypeError, ValueError):
+                continue


error handling

allisterakun · 2026-07-29T14:23:10Z

+            info_maps.append(raw_info_maps[index] if index < len(raw_info_maps) else {})
+        return values, info_maps
+
+    def _collect_pen_data(self, sim_paths: Iterable[str]) -> dict[str, dict[str, dict[str, list[Any]]]]:


Suggested change

def _collect_pen_data(self, sim_paths: Iterable[str]) -> dict[str, dict[str, dict[str, list[Any]]]]:

def _collect_pen_data(self, sim_paths: list[str]) -> dict[str, dict[str, dict[str, list[Any]]]]:

why not just list?

allisterakun · 2026-07-29T14:27:21Z

+        matched variable name (for example ``"0_CALF"``) identifies the pen. Results are
+        grouped by scenario, which is ``"baseline"`` for a single run or the run's name
+        in a multi-run comparison.
+


Can you add an Example to show what the data would look like?

allisterakun · 2026-07-29T14:33:05Z

+
+    def _get_annual_bedding_price(
+        self,
+        price_dict: Any,


Suggested change

price_dict: Any,

price_dict: dict[str, list[float | str]],

matthew7838 and others added 3 commits June 25, 2026 18:18

Initial implementations for the mapping function

8b25997

patch for preprocessing.py

0cc6a9e

Merge branch 'economics' into bedding-mapping

a38f6e5

matthew7838 self-assigned this Jun 29, 2026

matthew7838 linked an issue Jun 29, 2026 that may be closed by this pull request

[Econ] Improve biophysical to economics bedding mapping and processing #3088

Open

matthew7838 and others added 3 commits June 30, 2026 17:47

Merge remote-tracking branch 'origin/bedding-mapping' into bedding-ma…

0b10733

…pping

Merge branch 'economics' into bedding-mapping

1ceac56

Fixing more mapping logic

a4007c1

matthew7838 marked this pull request as ready for review July 10, 2026 08:39

matthew7838 changed the title ~~[Econ] Initial implementations for the bedding mapping functions~~ [Econ] [Econ] Fix biophysical-to-economics bedding mapping and per-pen cost processing Jul 10, 2026

matthew7838 changed the title ~~[Econ] [Econ] Fix biophysical-to-economics bedding mapping and per-pen cost processing~~ [Econ] Fix biophysical-to-economics bedding mapping and per-pen cost processing Jul 10, 2026

matthew7838 requested review from bradenlimb and jonahgreene01 July 10, 2026 08:48

Added comments for tmep clarification

11527d3

matthew7838 added 2 commits July 21, 2026 01:45

Fixed reporting bug

12b34d2

Update to only calculate the costs based on LAC cows

abd6c71

bradenlimb approved these changes Jul 21, 2026

View reviewed changes

matthew7838 added 3 commits July 22, 2026 17:49

improved function namings

b7441b1

cleaned up some in-line comments

ec9e179

Matched tuple return docstring style

158cd83

matthew7838 requested review from allisterakun and ew3361zh July 22, 2026 08:56

allisterakun reviewed Jul 29, 2026

View reviewed changes

	name_to_type: dict[str, Any] = {}
	name_to_type: dict[str, str] = {}

		if isinstance(config, dict) and "name" in config:
		name_to_type[str(config["name"])] = self._normalize_bedding_type(config.get("bedding_type"))

	def _collect_pen_data(self, sim_paths: Iterable[str]) -> dict[str, dict[str, dict[str, list[Any]]]]:
	def _collect_pen_data(self, sim_paths: list[str]) -> dict[str, dict[str, dict[str, list[Any]]]]:

Uh oh!

Conversation

matthew7838 commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

What

Why

How

Test plan

Input Changes

Output Changes

Filter

Uh oh!

bradenlimb commented Jul 20, 2026

Uh oh!

matthew7838 commented Jul 21, 2026

Uh oh!

bradenlimb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

matthew7838 commented Jun 25, 2026 •

edited

Loading