FarmOS Integration for Developers

Description

FarmOS is a farm management system with farm-specific data storage structures. This is useful for storing farm data, but farmOS itself is extremely unopinionated about how the data is stored. Some structures are specific (crop and area) and some are semi-flexile (materials and quantity) and some could be used in many ways (data and purpose).

We use farmOS as our primary location to store farm data collected through SurveyStack because it is has appropriate data types, it's open source, and we can put in the data from many different forms but also ensure we can get the data back out in the same way.

As such - we NEED to be opinionated about how data is stored in farmOS so we can get the data back out in a way we can expect and understand. We are able to be opinionated by

  1. using consistent Question Sets to input the data, resulting in consistent data entry and
  2. producing schemas from those Question Sets which can be assembled into conventions (a set of schemas used by a service). A convention is then used as a standard in an Ag Data Wallet (the convention is the data structure, the wallet is the place to store some actual data following that structure), providing easy transfer of data between services and a set of common Conventions by which services can agree to exchange and understand ag data.

Conventions used by SurveyStack and their Details

SurveyStack's FarmOS integration created and maintains a set of conventions for both farm level (farm name, address, etc.) and field level (tillage events, plantings, etc.) information. We have a process for building consensus around those conventions and for updating them over time. Below is a description of the conventions and the process.

  1. Establish a standard FarmOS installation and set of modules (additional bits of code).
  2. In that installation, determine the farm level data structure + norms
  3. In that installation, determine the field level data structure + norms
  4. Build a process for maintaining and updating the installation, farm level convention(s), and field level convention(s).

What FarmOS modules we use

When using SurveyStack to create FarmOS instances, we install a set of default modules to ensure that SurveyStack can communicate with that instance and the instance has the expected fields. That list of modules is:

... add list of modules here (mike) and what each one does...

Farm Profile

Common Profile profile--common

Using the Common Profile, users can enter core information about their farm, such as:

  • Contact information
  • Address & Location
  • Products & services
  • Certifications
  • Interests and specialities

This information is then used to create a farm profile in farmOS, and can be easily passed to other services in the tech ecosystem without the user having to enter it again. For example, this same information is used to build farm profiles in Hylo.

Field level data (farm activities + assets)

Field level data includes planting events and resulting assets, activities like tillage, pruning/thinning, or spraying and data input activities like lab tests or observational data.

... list of link to our list of field level schemas...

Log Categories (log_category)

Log Categories are used when results are displayed by an external service (like the Farmer Coffee Shop) to organize and compare activities with a similar broad purpose. Activities may have multiple log categories (tillage can be for tillage and for weed control for example). Here is the list of Log Categories that we use.

  • pest_disease_control
  • weed_control
  • harvest
  • termination
  • amendment
  • tillage
  • irrigation
  • seeding
  • transplanting
  • observation
  • thinning_pruning
  • information (used for ESMC, ownership info, field level info for certifications that doesn't fit elsewhere).
  • conservation_practices (used for ESMC, field activities to structurally improve the field)
  • other - unknown operations

Log Types

Log Types log_type describes a specific action. This action may have many purposes (defined by log_category), but the action itself is defined by Log Type.

  • farm_input (amendments, pest_disease_control)
  • farm_seeding (seeding)
  • farm_transplanting (transplanting)
  • farm_harvest (harvest)
  • farm_activity (...)
  • farm_observations (observations)
  • farmOS defined - single value
  • ... need to add lab test.

Input Purpose

Input Purpose provides additional information about the purpose of an event. This is often not needed as Log Category provides overlapping information, but some specific events benefit from organizing additional purposes.

  • weed_control
  • pest_disease_control
  • dessicant
  • observed_pests
  • observed_damage
  • scheduled

Convention update process

Conventions are updated starting with the Question Set Library, then (if needed) updating farmOS via a farmOS module, then (if needed) exporting the final schema from farmOS to gitlab for public use by others using this convention for their Ag Data Wallet.

This process can be seen in [the miro board here].(https://miro.com/app/board/uXjVOPt_DfA=/?share_link_id=765100301448)

... add more detail here...

FarmOS reference instances

Typically, we use a farmOS instance's publicly generated schema (available via api, like myfarm.farmos.net/api/schema) to assemble and validate schemas in SurveyStack inside apiCompose. However, in some cases we still want to create the data objects, but users may not have a farmOS instance.

surveystack-reference.farmos.net - In those cases, we maintain an instance which provides the standard schemas / conventions for SurveyStack's standard FarmOS implementation. This instance should have the most up to date version of all our required modules see section above. This is used to generate schemas for any user who doesn't have a farmOS farm but wants to utilize the SurveyStack standard convention / schemas.

In addition, we maintain an FarmOS instance used to describe typical farms and typical use in various applications.

... we should add here... we don't really have any but should create these... maybe Vic or Nat can do that as part of OT stuff...

Additional notes (need refining... this is older stuff)

So... here is how we use farmOS:

  • We add items we want to store which otherwise don't fit into the 'description' field for now, but hope to have another more general purpose location for machine-readable data in the future which won't be shown to the use in a clear way (or be editable by them at least)
  • we organize things by either planting, or by field. if organized by field, then we auto-archive all plantings immediately, so they don't show up... this makes sense when you're planting one thing at a time, or one set of things which are always managed the same. If you plant multiple things that you manage differently in a field at the same time, it's really important to track things by planting. That way, activities which are performed on one thing in the field and not another can be tracked correctly.
  • If you track by field, it also means you don't need to archive plantings, and you won't ever have a pile of unarchived plantings showing up on your lists... you're simply always choosing an area.

  • If you track by planting - then you do need to actively archive plantings - or your planting lists will become cumbersome! We discussed auto-archiving at a certain date (based on the expected harvest date upon planting), at the end of the year (always by Feb 1, for example)... but felt those were flimsy and could be wrong. So we decided to make it a simple user-driven process - at the beginning of each season, we have a single question form which asks 'which of these plantings are you still actively working with'? Then we archive everything else... this adds (a small amount of) overhead to the user, but is robust even with perennials, annuals, and other oddball cases.

  • General Mills has multiple plantings in single survey... which means I need to create multiple Planting assets in order to record multiple seedings. Seedings require a connection to a planting. Since there could be multiple plantings and using $PLANTING I could associate a seeding with the wrong planting... does it break anything? The plantings are in theory archived so it shouldn't matter.

  • Submissions don't seem to be working, and it's giving me a green success screen but not showing me the linked logs any more

TO DO NOW - archived - fix to seconds (manuel)- X check requireds- X fix planting 2 and 3 (seeding only) - X finish cover crops- add + test

planting not archived, tho is should be harvest isn't working, though the desiccant is working...add category for all logs (we can make them up)

flags

A flag is a production strategy/regimes/certification which may be applied to a planting and/or a field

  • usda_organic, non_gmo

  • tracking for certifications / external use

  • array of text

  • future: use for organic certification review (?)

material: greg's great pest spray, wheat seeds, compost - description of exact material being used (seed, amendment, spray, etc.) - array of objectsquantity: 10 oz of greg's great pest spray, 50lbs/acre wheat seeds, 800lbs compost - amount of material being applied w/ units- array of amounts/units, allows specific input of N, P, K separately, etc.

Amendments: - OK - input_purpose (reason for application), input_purpose, machine readableOK - quantity logs, quantity [], (for n, p, k, etc.) - OK - input_method, input_method (how you put it in / on)OK - material, array of objects, include name of material (greg's great pesticide no 56) - OK - material.description, json array, include classes of material

planting - crop - field (optional) - archived (optional) seeding - field

Next steps: - get utils pushed to main, so that getClean and getCleanArray work with matrix questions - check that we can leave more empty values and not break apiCompose - search all for utils. and replace functions with correct util functions