Data Management

Tools are available to securely transfer data between NSLS-II servers and your home institute. Options include ssh, sftp, and Globus.

Proposal Directories

Each proposal will have its own folder where files are saved and processed. The folders for the proposals at the SRX beamline are found at /nsls2/data/srx/proposals. From there, the proposals are broken down into operating cycles, e.g. 2025-1. Finally, the prefix pass- with the proposal number is used to identify each proposal to make the folder name, e.g. pass-1234.

Inside each proposal folder, files will be written automatically. During data collection, raw detector data is written and stored in the assets/ folder. This folder is set as read-only. You are welcome to copy this data, but it will be difficult to correlate raw detector files to particular scans.

Additionally, at the completion of each scan, a process will automatically kick-off to export the scan data. If the scan was an XRF map, files will be generated and written to the root proposal directory with the filename autorun_scan2D_{scanid}_xs_sumch.h5. Spectroscopy files will also be written to the root of this proposal directory with a filename of scan_{scanid}_xanes.txt. Finally, a running logfile records the scan ID, UID, and some scan parameters for each scan that is run.

Transfer over Globus

Globus is an efficient system for transferring large data sets and folders. The Globus endpoint for NSLS-II systems is found at globus.nsls2.bnl.gov. The Globus origin ID is 819379a8-47db-439d-a5ba-a2387b79add9

Setup Globus Connect Personal

  1. Download and install Globus Connect Personal

  2. Run the application. If this is the first time, it should run a setup.

  3. Click to Login. This should bring up a website. Choose “Brookhaven National Laboratory” as the organization. Click Continue.

  4. Login to the BNL Single Sign On using your BNL account.

  5. Provide a label for this Endpoint and click Allow.

  6. Provide a collection name and description. Now you have an endpoint to transfer data to locally.

  7. Make sure to leave the Globus Personal Connect running.

Locate Proposal Data

  1. Go to the NSLS-II Globus endpoint. Make usre the organization login is set to “Brookhaven National Laboratory”. Click on Continue and login to the BNL Single Sign On using your BNL account and password.

  2. Once logged on, the Globus main page should appear. The collection at the top should say “NSLS2”

  3. If there is a message saying that “Authentication/Consent is required for Globus transfer service to manage data in this collection”, click Continue. Click on your email address. Allow Globus Web App to manage data using Globus Transfer.

  4. Once logged in, you may start off in your home directory, /nsls2/users/your_user_name

  5. In the Path textbox, the location of the proposals can be entered. It is convenient to start at /nsls2/data/srx/proposals/. From there the cycle, e.g. 2025-1, is selected followed by the folder that corresponds with the proposal number, e.g. pass-{proposal number}.

  6. At this point, the contents of the proposal directory are displayed. Data can be downloaded or transfered from here.

  7. Select your files for download. We recommend ommitting the assets/ folder. Click on Transfer of Sync to… Another panel will pop up where you can select your local Enpoint. Choose a location to download the data. Click start to start the transfer.