Datasets

This guide showcases some features of the Dataset classes in the Rubrix client. The Dataset classes are lightweight containers for Rubrix records. These classes facilitate importing from and exporting to different formats (e.g., pandas.DataFrame, datasets.Dataset) as well as sharing and versioning Rubrix datasets using the Hugging Face Hub.

For each record type there’s a corresponding Dataset class called DatasetFor<RecordType>. You can look up their API in the reference section

Working with a Dataset

Under the hood the Dataset classes store the records in a simple Python list. Therefore, working with a Dataset class is not very different to working with a simple list of records:

[ ]:
import rubrix as rb

# Start with a list of Rubrix records
dataset_rb = rb.DatasetForTextClassification(my_records)

# Loop over the dataset
for record in dataset_rb:
    print(record)

# Index into the dataset
dataset_rb[0] = rb.TextClassificationRecord(inputs="replace record")

# log a dataset to the Rubrix web app
rb.log(dataset_rb, "my_dataset")

The Dataset classes do some extra checks for you, to make sure you do not mix record types when appending or indexing into a dataset.

Importing from other formats

When you have your data in a pandas DataFrame or a datasets Dataset, we provide some neat shortcuts to import this data into a Rubrix Dataset. You have to make sure that the data follows the record model of a specific task, otherwise you will get validation errors. Columns in your DataFrame/Dataset that are not supported or recognized, will simply be ignored.

The record models of the tasks are explained in the reference section.

Note

Due to it’s pyarrow nature, data in a datasets.Dataset has to follow a slightly different model, that you can look up in the examples of the Dataset*.from_datasets docstrings.

[ ]:
import rubrix as rb

# import data from a pandas DataFrame
dataset_rb = rb.read_pandas(my_dataframe, task="TextClassification")

# import data from a datasets Dataset
dataset_rb = rb.read_datasets(my_dataset, task="TextClassification")

Sharing via the Hugging Face Hub

You can easily share your Rubrix dataset with your community via the Hugging Face Hub. For this you just need to export your Rubrix Dataset to a datasets.Dataset and push it to the hub:

[ ]:
import rubrix as rb

# load your annotated dataset from the Rubrix web app
dataset_rb = rb.load("my_dataset", as_pandas=False)

# export your Rubrix Dataset to a datasets Dataset
dataset_ds = dataset_rb.to_datasets()

# push the dataset to the Hugging Face Hub
dataset_ds.push_to_hub("my_dataset")

Afterward, your community can easily access your annotated dataset and log it directly to the Rubrix web app:

[ ]:
from datasets import load_dataset

# download the dataset from the Hugging Face Hub
dataset_ds = load_dataset("user/my_dataset", split="train")

# read in dataset, assuming its a dataset for text classification
dataset_rb = rb.read_datasets(dataset_ds, task="TextClassification")

# log the dataset to the Rubrix web app
rb.log(dataset_rb, "dataset_by_user")