Globus and Elm

Elm may be accessed through the Stanford Elm Storage (project: campus) collection.

Before continuing, you may need to do some one-time setup work. You will need to…

• Review the service limitations, to see if they will affect your usage. If you are still OK using Globus, your next step will be to…

• Create an Access Key, which will give you an access key (officially an “Access Key ID”) and a secret key (officially a “Secret Access Key”). Once your access key is created, you will need to…

• Load the credentials into Globus. This gives Globus access to Elm, with the credentials you provide.

Once the one-time setup work is complete, you should proceed to access your files on Elm.

Each step is described in detail below.

Service Limitations

Globus has a number of limitations when working with Elm. These limits might affect your use case—especially when dealing with research data or shared content—so you should review them before starting to use Globus with OneDrive.

Elm Limitations

Even though Elm is accessed through the S3 protocol, it has a number of limitations compared to Amazon S3. In particular…

• Big Data is Best: Elm works best with large files (hundreds of megabytes and larger). If you are thinking of uploading smaller files, you may wish to combine them into a larger .tar or .zip file before you copy them to Elm.

• Expect Slow Restores: Data on Elm are eventually sent to tape. Directory listings will be fast, but downloads from Elm will be slower than expected, and you may see timeouts.

• Workgroups Rule: Even if you give Globus full read, write, and delete access, your workgroup membership will limit what Globus can do. Specifically…

  • You must be a member of your bucket’s main workgroup to have any access.

  • You must be a member of your bucket’s “Writers” workgroup to have read/write access.

  • You must be a member of your bucket’s “Editors” workgroup to have full access.

• No Cross-Account Access: Elm does not support S3 cross-account bucket access. Instead, you should add users to the appropriate work, so they can create their own access keys.

Globus Limitations

The Globus File Manager only supports one access key per Mapped Collection, but it is possible to have a separate access key for each bucket. If you are someone who has multiple buckets on Elm, contact us for assistance.

Globus for Elm does not support the following S3 features:

• Custom Metadata / Tags: Custom Metadata and tags on existing objects are ignored when those objects are downloaded, and new objects do not have custom metadata or tags set.

• Versions: When downloading a file from an S3 bucket, Globus will always access the latest version.

• ACLs: ACLs on existing objects (and the bucket) will influence what you can download through Globus, but those ACLs are not copied out of S3. When uploading new objects, ACLs are not explicitly set, and so inherit any bucket-level ACL that is set.

• Additional Checksum Algorithms: At this time, if an object in Elm needs to be verified, Globus will re-download it in order to compute the checksum.

These limitations are present because Globus supports only a common set of features between storage platforms, to make file transfers as portable as possible.

If you are OK with the limitations above, you should move on to creating an Access Key, which Globus will need to interact with Elm.

Creating an Access Key

Once your Elm bucket is created, and you have been given access to the appropriate workgroup, you should log in to Elm. After going through Stanford Login, you will see your buckets.

Select the “Access Keys” option:

You will see the list of access keys. Click “Create access key”:

You’ll now be presented with the screen to configure the new access key.

To start, in the Name field, enter your name and the word “Globus”, so that others will know what you are using the Access Key for:

Next, turn on the “Restrict beyond user policy” setting:

You will now be presented with the current User Policy, allowing you to change it.

Setting a User Policy

The default User Policy allows more access than Globus needs. So, you should change it to a policy that restricts Globus to just the buckets, and actions, that you want to allow.

Below, fill in the names of the buckets you want to access through Globus, and choose if you want to allow Globus to upload and/or delete files.

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Effect": "Allow",
   "Action": [
    "s3:ListAllMyBuckets",
    "s3:GetBucketLocation"
   ],
   "Resource": [
    "arn:aws:s3:::*"
   ]
  },
  {
   "Effect": "Allow",
   "Action": [
    "s3:ListBucket",
    "s3:ListBucketMultipartUploads"
   ],
   "Resource": [
    "arn:aws:s3:::allcats" <------ ONE LINE PER BUCKET
   ]
  },
  {
   "Effect": "Allow",
   "Action": [
    "s3:DeleteObject",               <------- If allowing deletions
    "s3:PutObject",                  <------\
    "s3:ListMultipartUploadParts",   <------|- If allowing uploads
    "s3:AbortMultipartUpload",       <------/
    "s3:GetObject"
   ],
   "Resource": [
    "arn:aws:s3:::allcats/*" <------ ONE LINE PER BUCKET
   ]
  }
 ]
}

Replace the default Current User Policy with the custom policy above, and click “Create”:

Once created, you will be given your new Access Key and Secret Key. Make a note of these, to load into Globus.

Loading Credentials into Globus

With an Access Key configured, you can now upload your credentials to Globus.

Using the link at the top of the page, access the Elm collection. You might be asked to log in; if so, log in through Stanford University.

You will be taken to the main page for the collection. Click on the ‘Credentials’ tab.

If this is the first time you accessed this collection, you will be asked to give consent for Globus to store your Elm credentials. Click ‘Continue’.

Some institutions allow you to have multiple accounts. Stanford only allows one SUNetID per person, so click on your SUNetID.

Finally, click on ‘Allow’ to give Globus permission to store your Elm credentials.

Once consent is granted, you will be asked to enter your Elm credentials. Enter the Access Key and Secret Key from when you created your access key.

If you go to the Credentials tab after entering an Access Key, you will see the Access Key ID, along with an option to replace the credential (the gear icon) or delete the credential (the trashcan icon).

You should now proceed to accessing the collection!

Accessing Files on Elm

With Elm Access Key loaded and permissions granted, you may now proceed to access your data on Elm through Globus!

Using the link at the top of the page, access the Elm collection. You might be asked to log in; if so, log in through Stanford University.

Click on the “Open in File Manager” button. That will take you to the File Manager and connect to Elm.

First-Time Access

The first time you access the Elm collection, you will be asked for consent.

When you first loaded your credentials, you gave Globus consent to store those credentials for you. Now, you are giving Globus consent to actually use those credentials to talk to Elm. Click the “Continue” button.

Some institutions allow you to have multiple accounts. Stanford only allows one SUNetID per person, so click on your SUNetID.

Finally, click on ‘Allow’ to give Globus permission to use your AWS IAM User credentials to access Elm.

Subsequent Accesses

When you access the S3 collection—assuming you have previously provided consent—you should be greeted with a list of the buckets from Elm.

To access one of the buckets—assuming your Access Key has permissions—double-click on the bucket’s name.

If your Access Key does not have permissions, attempting to list the contents of the bucket will give an error.

Once you have access to a bucket, you can transfer files in and out like any other Globus collection.