Migrate docstrings to doctests

[mathiasburger]

Changes

Migrate docstrings to doctest

• torchdata/dataloader2/adapter.py
• torchdata/datapipes/iter/load/*

Use PEP8 style in code examples, e.g. add newlines between defs:

  from torchdata.datapipes.iter import IterableWrapper


  def filepath_fn(name: str) -> str:
      return dir_path + name


  name_to_data = {"1.txt": b"DATA1", "2.txt": b"DATA2", "3.txt": b"DATA3"}
  source_dp = IterableWrapper(sorted(name_to_data.items()))
  fsspec_saver_dp = source_dp.save_by_fsspec(filepath_fn=filepath_fn, mode="wb")
  res_file_paths = list(fsspec_saver_dp)`

[mathiasburger]

Comparison of the documentation before and after the changes:

[ejguan]

Should them be os.remove(os.join(dir_path, "1.txt"))?

[mathiasburger]

filepath_fn constructs a filename and not a directory, so concatenation is correct here; I did not change the example itself and only added matching test cleanup

[ejguan]

Really appreciate this effort. Left a few comments above

[mathiasburger]

filepath_fn constructs a filename and not a directory, so concatenation is correct here; I did not change the example itself and only added matching test cleanup

[ejguan]

Makes sense to me. nit: Can we do a for loop here?

[ejguan]

Thank you. Left a few minor comments

[ejguan]

Makes sense to me. nit: Can we do a for loop here?

[ejguan]

This is too hacky. How about construct an empty list as s3_prefixes and add a comment about the pattern of each path.

[mathiasburger]

imho if we use an empty list of files, the example is not very descriptive. IterableWrapper(['s3://bucket-name/folder/', ...]) is necessary for the user to understand the example.

We could mock here, but we would basically do the same thing of emptying the list in order to make it work. But mocking would alter the example. So I find it difficult to achieve a good solution. Please note that the test setup is not shown in the example and that we don't really want to alter the example here.

[mathiasburger]

I'll try patching. As I cannot use a with block, I will use patch.start() in testsetup and patch.stop() in testcleanup

[ejguan]

Ditto

[mathiasburger]

changed to patching in testsetup and cleaning up the patch in testcleanup

[NivekT]

I think it will be great to add a section in the contribution guide (can be done in a separate PR) about doctests, briefly describing that it exists and is part of the docstring.

Perhaps somewhere in this section?

[SvenDS9]

I think it will be great to add a section in the contribution guide (can be done in a separate PR) about doctests, briefly describing that it exists and is part of the docstring.

Perhaps somewhere in this section?

I have added a section about doctests in 1aaa382 in #1069

[facebook-github-bot]

@NivekT has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[NivekT]

LGTM!

[facebook-github-bot]

@NivekT merged this pull request in 837ede1.