Skip to content

File and column descriptions from dataset-metadata.json are still not saved (after #1055) #1207

Open

Description

@x5hwuk7nc6jsudcc

Summary

File and column descriptions in dataset-metadata.json are accepted without error but never saved. This happens on datasets create, datasets version and datasets metadata --update. #447 and #248 were closed by #1055, but that PR changed only how the CLI builds the request. The server still drops the descriptions. Descriptions entered in the web editor are saved as expected.

The documentation says this should work:

  • docs/datasets_metadata.md lists resources (with path, description and schema.fields[].description) as supported for kaggle datasets create and kaggle datasets version.
  • The same page says kaggle datasets metadata --update "supports all fields mentioned above for kaggle datasets version".

Environment

  • kaggle 2.2.4, kagglesdk 0.1.37
  • Python 3.14.3, Ubuntu (GitHub Actions ubuntu-latest)
  • Authenticated with KAGGLE_API_TOKEN

Steps to reproduce

  1. Make a folder with data.csv:

    year,category,grants
    1977-78,Refugee,9326
    1978-79,Refugee,12750
    
  2. Add dataset-metadata.json beside it, following the documented format:

    {
      "title": "Metadata description test",
      "id": "<your-username>/metadata-description-test",
      "licenses": [{"name": "CC0-1.0"}],
      "resources": [
        {
          "path": "data.csv",
          "description": "Rows of the table, as CSV.",
          "schema": {
            "fields": [
              {"name": "year", "description": "Financial year.", "type": "string"},
              {"name": "category", "description": "Visa category.", "type": "string"},
              {"name": "grants", "description": "Visas granted.", "type": "number"}
            ]
          }
        }
      ]
    }
  3. Create the dataset, wait until it is ready, then upload the same files as a new version:

    kaggle datasets create -p . -u
    kaggle datasets status <your-username>/metadata-description-test   # until "ready"
    kaggle datasets version -p . -m "second version with the same resources"
    
  4. Send the same file again as a metadata update:

    kaggle datasets metadata <your-username>/metadata-description-test --update -p .
    
  5. Open the dataset page and select data.csv in the Data Explorer.

Expected

"About this file" shows "Rows of the table, as CSV.", and each column shows its description. The usability rating ticks File Description and Column Description.

Actual

  • "About this file" says "This file does not have a description yet", and no column has a description.
  • The usability rating still crosses File Description and Column Description.
  • Every command exits 0. metadata --update prints "successfully updated dataset metadata".
  • The other fields in the same update are saved: keywords, expectedUpdateFrequency, userSpecifiedSources and the cover image.

What else we tried

All of these were accepted with an empty errors list, and none changed the page:

  • resources with the documented types (string, number, datetime), and with integer.
  • Calling DatasetApiService.UpdateDatasetMetadata through kagglesdk directly, with settings.data set to DatasetSettingsFile entries in four forms: description only; description and totalBytes; those plus columns; and all three files of the dataset.
  • Three dataset versions uploaded with resources present (create once, then version twice).

GetDatasetMetadata returns info.data as an empty list after each update, as #410 also notes.

Example

A public dataset that shows the problem: https://www.kaggle.com/datasets/nationaldigitalau/au-humanitarian-program-grants. Version 3 was uploaded with resources for all three files and then sent the same settings through metadata --update.

Activity

  1. added
    bugSomething isn't working
    dependenciesPull requests that update a dependency file
    on Oct 1, 2026
  2. Hisernberg commented on Nov 10, 2026

    @Hisernberg

    I dug into this offline to check whether the client is really at fault, and on current main (fdc788f, kagglesdk 0.1.38) the wire payload is correct.

    Method: simulated the real client flow with the blob upload mocked (no network): built a dataset folder with dataset-metadata.json exactly as documented, ran KaggleApi.upload_files(...) from kaggle/api/kaggle_api_extended.py, then serialized the resulting ApiDatasetNewFile the same way the SDK does for the request body.

    Result — the wire JSON carries everything:

    {
      "token": "<token>",
      "description": "Rows of the table, as CSV.",
      "columns": [
        {"order": 0, "name": "year", "type": "string", "originalType": "string", "description": "Financial year."},
        {"order": 1, "name": "category", "type": "string", "originalType": "string", "description": "Visa category."},
        {"order": 2, "name": "grants", "type": "numeric", "originalType": "number", "description": "Visas granted."}
      ]
    }

    The chain is wired end-to-end on the client: dataset-metadata.json resources → _upload_file (matches file_name == item["path"], populates UploadFile.description and process_column() output) → _new_file (ApiDatasetNewFile.description + columns) → kagglesdk ApiDatasetNewFile._fields includes description, and ApiDatasetColumn._fields includes description (all verified in kagglesdk 0.1.38).

    So on the client side there is nothing left to fix for datasets create/version — this isolates the drop to the server's handling of the files[].description / files[].columns[].description fields on POST /api/v1/datasets/create/version (and the equivalent create path), consistent with the report. kaggle datasets metadata --update goes through a different endpoint (POST /datasets/metadata/... with DatasetSettings/DatasetSettingsFile), whose wire mapping I also verified as complete — so if descriptions are dropped there too, both drop sites are server-side.

    Happy to help test once the server-side fix lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdependenciesPull requests that update a dependency file

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions