Adding attributes to assets in a subdirectory

warning

This feature is considered in a preview stage and is under active development. It can change significantly, or be removed completely. It is not considered ready for production use.

Attaching and modifying meta information on definitions such as assets is important in Dagster projects. You want to have groups, teams, owners, kinds, tags, and metadata set correctly to organize definitions and ensure that other tools and processes that rely on them correctly function.

Within a dg-driven defs project layout, you can apply metadata transformations at any point in the folder structure. This can span uses cases from ensuring that all definitions in a particular folder have a tag associating them with the same team, to more complex workflows of applying a group conditionally based on other properties of the definition.

Example

First, we can look at an existing project, where we have assets defined in a subdirectory:

dg list defs

┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions                                    ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets  │ ┏━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│         │ ┃ Key ┃ Group   ┃ Deps ┃ Kinds ┃ Description ┃ │
│         │ ┡━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│         │ │ a   │ default │      │       │             │ │
│         │ ├─────┼─────────┼──────┼───────┼─────────────┤ │
│         │ │ b   │ default │      │       │             │ │
│         │ ├─────┼─────────┼──────┼───────┼─────────────┤ │
│         │ │ c   │ default │      │       │             │ │
│         │ └─────┴─────────┴──────┴───────┴─────────────┘ │
└─────────┴────────────────────────────────────────────────┘

tree my_project/defs

my_project/defs
├── __init__.py
├── team_a
│   ├── b.py
│   └── subproject
│       └── a.py
└── team_b
    └── c.py

4 directories, 4 files

Now, we can add a component.yaml file to the my_project/defs/team_a directory with the following contents:

type: dagster.components.DefsFolderComponent

attributes:
  asset_post_processors:
    - target: "*"
      attributes:
        group_name: "team_a"

tree my_project/defs

my_project/defs
├── __init__.py
├── team_a
│   ├── b.py
│   ├── component.yaml
│   └── subproject
│       └── a.py
└── team_b
    └── c.py

4 directories, 5 files

This configuration will set the group of all assets within the directory to team_a.

Finally, we can run dg list defs again to see the new group applied to the assets in the team_a subdirectory:

dg list defs

┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions                                    ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets  │ ┏━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│         │ ┃ Key ┃ Group   ┃ Deps ┃ Kinds ┃ Description ┃ │
│         │ ┡━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│         │ │ a   │ team_a  │      │       │             │ │
│         │ ├─────┼─────────┼──────┼───────┼─────────────┤ │
│         │ │ b   │ team_a  │      │       │             │ │
│         │ ├─────┼─────────┼──────┼───────┼─────────────┤ │
│         │ │ c   │ default │      │       │             │ │
│         │ └─────┴─────────┴──────┴───────┴─────────────┘ │
└─────────┴────────────────────────────────────────────────┘

Advanced Usage

The configuration for the dagster.components.DefsFolderComponent can be more complex than the example above. You can apply multiple transforms to the assets in the subdirectory, each of which will be processed in order.

You can also target the configuration to a specific selection of assets using the target field. This field uses Dagster's Selection Syntax. All selections are evaluated against the assets defined within the subdirectory.