feat: adapt management events data schema standard and dynamic ui accordingly

[pratikpakhale]

Final Report: GSoC '24

• Student Name: Pratik Pakhale
• Organization: PEcAn
• Mentors: Henri Kajasilta, Olli Niemitalo, Nihar Sanda
• Project: Adopting Data Schema for Field Management Events

Summary

The event form, the event list, the downloads and the saved events.json now all follow the management event JSON schema (inst/extdata/management-event.schema.json). Before this PR they were hand-written in ui_structure.json and display_names.csv. Adding or changing a field now means editing the schema.

This finishes the GSoC '24 project Adopting Data Schema for Field Management Events. The original final report is in this description's edit history.

What changes

Event form

• The schema is the source for the fields, their English and Finnish labels, choices, units and required markers, and for validation (required, minimum/maximum, whole numbers).
• Events with subtypes have a subtype selector, and only the selected subtype's fields are shown:
  • fertilizer: mineral / organic
  • observation: soil / vegetation / water
• Fields with an x-ui condition are shown only when the condition is met. Hidden fields are not validated or saved.
• Array properties such as sowing, harvest and soil layers are tables with add and remove row buttons. Every row is validated.
• Totals such as total harvest yield are summed from their table when the table has values. Otherwise they can be typed in.
• Soil and canopeo photo upload works again. The two fields are now defined in the schema (x-ui form-type: fileInput).

Saved events

• Events are saved in the schema's shape: per-item values are nested in arrays, empty values are left out, and every event has a $schema key.

• Existing events.json files are upgraded when they're read:

  • the organic_material event type becomes fertilizer with the organic subtype
  • notes stored under per-event names are moved to mgmt_event_long_notes / mgmt_event_short_notes
  • flat item values (scalars or parallel vectors) become array rows
  • -99.0 missing values are dropped
  • grazing_period is split into date / end_date

  A block's file is rewritten in the new format the next time an event in that block is saved.

• Keys the schema doesn't know are kept unchanged, so no data is lost.

Event list and downloads

• The columns shown for an event type come from the schema. Tables and images aren't shown.
• The CSV export reads events the same way the app does, so legacy events are exported in the new format too. Array items become one column per item property, with the values joined by ; .
• Both downloads use the selected site and the configured data folder. Dev mode no longer hardcodes qvidja and dev/dev_events.

Cleanup

• ui_structure.json now only describes the app chrome: site and block, buttons, event list and help texts. The event form definitions and the code that built them are removed.
• Removed unused imports: attempt, ggplot2 (only an @import, never called), methods, pkgload, processx.
• Added jsonvalidate to Suggests, for the test that validates saved events against the schema.
• Removed the tracked .RData file.

Known issues

1. The bundled schema has drifted from upstream. The copy in inst/extdata/ has changed a lot since it was taken from hamk-uas/fieldobservatory-data-schemas, which hasn't changed since 2022. Saved events still set $schema to upstream main, and they don't validate against it. The changes need to go upstream, and $schema should be pinned to a tag or commit.
2. The image upload fields exist only in the bundled schema. soil_image and canopeo_image (x-ui form-type: fileInput) were added so photo upload keeps working. They need to be added upstream as well.
3. Swedish is only partly added. display_names.csv has no Swedish column, so with svenska selected the app chrome labels are blank and the event list crashes. It needs translations, or svenska should come out of the selector until they exist.
4. Event type names are capitalised differently. The form shows "Sowing" and the event list shows "sowing".

Project Reflection

As I look back on my GSoC '24 project for PEcAn, I can see that the core components of my proposal were:

1. Creating a parser for the JSON schema to generate dynamic UI elements
2. Integrating this parser into the existing Fieldactivity app

These two aspects ended up taking the majority of my time during the GSoC period. The parser development was particularly challenging, especially when it came to handling nested structures, 'oneOf' properties, and conditional rendering. I'm pleased that I was able to successfully implement the parser as a separate project, which can now generate UI elements based on the JSON schema.

However, within the allocated time for the project, I didn't have the opportunity to work on some of the additional features I had originally planned. These included:

1. Implementing the parser with the 'management-event-schema.json' instead of 'ui_structure.json'
2. Integrating the parser with the existing Fieldactivity app, including site selection and event display table

Looking back, I still see these additional features as valuable for the project's overall goals. Implementing them would make the Fieldactivity app more flexible and easier to maintain, as it would allow for dynamic form generation based on a standardized schema. This would improve compatibility with other data sources and make it easier to update the data structure in the future.

While I didn't have time to implement these extra features during the GSoC period, I believe the work I've done on the parser lays a solid foundation. This can serve as a starting point for future development, either by myself (I plan to do so) or other contributors to the PEcAn project.

In reflection, I feel that my project achieved its primary goal of creating a functional parser for the JSON schema, which is a significant step towards improving the Fieldactivity app's flexibility and maintainability. The integration and additional features remain important for the project's future development, and I believe they could be valuable contributions for the PEcAn community going forward.

Overall Experience

One of my favorite aspects of participating in Google Summer of Code (GSoC) has been the incredible exposure to real-world software development projects, especially in R. It's been a fantastic way to apply what I've learned in a practical setting and see the impact of my work.

Mentorship and Community Interaction: I absolutely value the opportunity to connect with mentors, the community and my fellow peers in PEcAn family. The guidance and support from my mentors have really helped a lot. I've learned so much about open-source culture, how to communicate and have developed valuable relationships that I believe will continue to benefit me in the future.

Flexibility and Independence: The program's flexibility is another highlight for me. Being able to work independently from anywhere has been empowering. My mentors understanding my commitments and requests was also one of the things I appreciate the most!