Release Page

Access data using native S3 clients

The data files are also available using S3-compatible clients, such as the MinIO CLI utility or the AWS CLI tools. There are many more clients and SDKs for various programming languages like Python's Boto3.

The example below uses the MinIO client mc mirror command to show how easy it is to download a data set.

First, create an alias called "des-data" with anonymous access:

$ mc alias set des-data https://ncsa.osn.xsede.org/ anonymous ""

Use the mc ls command to list files in a subfolder, in this case from the release called
y3-lt-widebinaries:

$ mc ls --recursive des-data/phy240006-bucket01/despublic/other_files/y3-lt-widebinaries/

Use the mc mirror command to robustly copy a data set, which acts similarly to rsync:

$ mc mirror \
    des-data/phy240006-bucket01/despublic/other_files/y3-lt-widebinaries/ \
    /tmp/y3-lt-widebinaries/

  ...nal_catalog.fits: 556.88 KiB / 556.88 KiB ━━━━━━━━━━━━━━━━ 2.37 MiB/s 0s

$ ls /tmp/y3-lt-widebinaries/

multiples_final_catalog.fits  old  pairs_gaia_ucd_des_ucd_final_catalog.fits  pairs_ucd_ucd_final_catalog.fits