diff --git a/_episodes/01-introduction.md b/_episodes/01-introduction.md index 94e9a1f..de1b81e 100644 --- a/_episodes/01-introduction.md +++ b/_episodes/01-introduction.md @@ -12,6 +12,8 @@ keypoints: - This tutorial is brought to you by the DUNE Computing Consortium. - The goals are to give you the computing basis to work on DUNE. --- + +{% include 01-introduction.toc.md %} ## DUNE Computing Consortium The DUNE Computing Consortium works to establish a global computing network that will handle the massive data streams produced by distributing these across the computing grid. diff --git a/_episodes/02-storage-spaces.md b/_episodes/02-storage-spaces.md index d2c046c..29a252d 100644 --- a/_episodes/02-storage-spaces.md +++ b/_episodes/02-storage-spaces.md @@ -17,6 +17,7 @@ keypoints: - The tool suites idfh and XRootD allow for accessing data with appropriate transfer method and in a scalable way. --- +{% include 02-storage-spaces.toc.md %} ## This is an updated version of the 2023 training diff --git a/_includes/01-introduction.md.toc.md b/_includes/01-introduction.md.toc.md new file mode 100644 index 0000000..14f5f96 --- /dev/null +++ b/_includes/01-introduction.md.toc.md @@ -0,0 +1,7 @@ +## Table of Contents +- [DUNE Computing Consortium](#dune computing consortium) +- [Schedule](#schedule) + - [Workshop Introduction Video from December 2024](#workshop introduction video from december 2024) +- [Basic setup reminder](#basic setup reminder) +- [Instructional Crew](#instructional crew) +- [Support](#support) \ No newline at end of file diff --git a/_includes/01-introduction.toc.md b/_includes/01-introduction.toc.md new file mode 100644 index 0000000..79dde35 --- /dev/null +++ b/_includes/01-introduction.toc.md @@ -0,0 +1,7 @@ +## Table of Contents +- [DUNE Computing Consortium](#dune-computing-consortium) +- [Schedule](#schedule) + - [Workshop Introduction Video from December 2024](#workshop-introduction-video-from-december-2024) +- [Basic setup reminder](#basic-setup-reminder) +- [Instructional Crew](#instructional-crew) +- [Support](#support) \ No newline at end of file diff --git a/_includes/01.5-documentation.toc.md b/_includes/01.5-documentation.toc.md new file mode 100644 index 0000000..4ed8252 --- /dev/null +++ b/_includes/01.5-documentation.toc.md @@ -0,0 +1,9 @@ +## Table of Contents +- [Documentation access](#documentation-access) +- [DUNE tools list](#dune-tools-list) +- [Docdb (Requires FNAL SSO)](#docdb-(requires-fnal-sso)) +- [DUNE wiki (Requires FNAL SSO)](#dune-wiki-(requires-fnal-sso)) + - [Tutorials list on the wiki](#tutorials-list-on-the-wiki) +- [CERN EDMS](#cern-edms) +- [Github repositories](#github-repositories) +- [DUNE Computing FAQ](#dune-computing-faq) \ No newline at end of file diff --git a/_includes/02-storage-spaces.toc.md b/_includes/02-storage-spaces.toc.md new file mode 100644 index 0000000..b3fb83a --- /dev/null +++ b/_includes/02-storage-spaces.toc.md @@ -0,0 +1,35 @@ +## Table of Contents +- [This is an updated version of the 2023 training](#this-is-an-updated-version-of-the-2023-training) + - [Live Notes](#live-notes) + - [Workshop Storage Spaces Video from December 2024](#workshop-storage-spaces-video-from-december-2024) +- [Introduction](#introduction) +- [Vocabulary](#vocabulary) +- [Interactive storage volumes (mounted on dunegpvmXX.fnal.gov or lxplus.cern.ch)](#interactive-storage-volumes-(mounted-on-dunegpvmxx.fnal.gov-or-lxplus.cern.ch)) + - [Your home area](#your-home-area) + - [at Fermilab](#at-fermilab) + - [at CERN](#at-cern) + - [Locally mounted volumes](#locally-mounted-volumes) + - [Network Attached Storage (NAS)](#network-attached-storage-(nas)) + - [At Fermilab](#at-fermilab) + - [At CERN](#at-cern) + - [Grid-accessible storage volumes](#grid-accessible-storage-volumes) + - [Persistent dCache](#persistent-dcache) + - [Scratch dCache](#scratch-dcache) + - [Tape-backed dCache](#tape-backed-dcache) + - [Rucio Storage Elements](#rucio-storage-elements) + - [CVMFS](#cvmfs) + - [What is my quota?](#what-is-my-quota?) + - [Your home area at FNAL](#your-home-area-at-fnal) + - [Your home area at CERN](#your-home-area-at-cern) + - [The /app/ and /data/ areas at FNAL](#the-/app/-and-/data/-areas-at-fnal) + - [EOS at CERN](#eos-at-cern) + - [Fermilab dCache](#fermilab-dcache) +- [Summary on storage spaces](#summary-on-storage-spaces) +- [Monitoring and Usage](#monitoring-and-usage) +- [Commands and tools](#commands-and-tools) + - [ifdh](#ifdh) + - [xrootd](#xrootd) + - [What is the right xroot path for a file.](#what-is-the-right-xroot-path-for-a-file.) + - [The df command](#the-df-command) +- [Quiz](#quiz) +- [Useful links to bookmark](#useful-links-to-bookmark) \ No newline at end of file diff --git a/_includes/02.3-cvmfs.toc.md b/_includes/02.3-cvmfs.toc.md new file mode 100644 index 0000000..52c36e9 --- /dev/null +++ b/_includes/02.3-cvmfs.toc.md @@ -0,0 +1,4 @@ +## Table of Contents +- [CVMFS](#cvmfs) +- [Restrictions](#restrictions) +- [Useful links to bookmark](#useful-links-to-bookmark) \ No newline at end of file diff --git a/_includes/03-data-management.toc.md b/_includes/03-data-management.toc.md new file mode 100644 index 0000000..23ca7ea --- /dev/null +++ b/_includes/03-data-management.toc.md @@ -0,0 +1,39 @@ +## Table of Contents +- [Session Video](#session-video) + - [Live Notes](#live-notes) +- [Introduction](#introduction) + - [What we need to do to produce accurate physics results](#what-we-need-to-do-to-produce-accurate-physics-results) + - [How we do it ?](#how-we-do-it-?) + - [How do I use this.](#how-do-i-use-this.) +- [How to find and access official data](#how-to-find-and-access-official-data) +- [Official datasets](#official-datasets) + - [Fast web catalog queries](#fast-web-catalog-queries) + - [Command line tools and advanced queries](#command-line-tools-and-advanced-queries) + - [metacat web interface](#metacat-web-interface) + - [Example of finding reconstructed Monte Carlo](#example-of-finding-reconstructed-monte-carlo) + - [you can use the web data catalog to do advanced searches](#you-can-use-the-web-data-catalog-to-do-advanced-searches) +- [What is metacat?](#what-is-metacat?) + - [Find a file in metacat](#find-a-file-in-metacat) + - [Example of doing a metacat search](#example-of-doing-a-metacat-search) + - [then do queries to find particular groups of files](#then-do-queries-to-find-particular-groups-of-files) + - [What do those fields mean?](#what-do-those-fields-mean?) + - [find out how much raw data there is in a run using the summary option](#find-out-how-much-raw-data-there-is-in-a-run-using-the-summary-option) + - [Fast web catalog queries](#fast-web-catalog-queries) + - [Command line tools and advanced queries](#command-line-tools-and-advanced-queries) + - [metacat web interface](#metacat-web-interface) + - [Example of finding reconstructed Monte Carlo](#example-of-finding-reconstructed-monte-carlo) + - [you can use the web data catalog to do advanced searches](#you-can-use-the-web-data-catalog-to-do-advanced-searches) + - [find out how much data there is in a dataset](#find-out-how-much-data-there-is-in-a-dataset) + - [What describes a dataset?](#what-describes-a-dataset?) + - [What files are in that dataset and how do I use them?](#what-files-are-in-that-dataset-and-how-do-i-use-them?) +- [Finding those files on disk](#finding-those-files-on-disk) +- [Getting file locations using Rucio](#getting-file-locations-using-rucio) + - [What is Rucio?](#what-is-rucio?) + - [You will need to authenticate to use read files](#you-will-need-to-authenticate-to-use-read-files) +- [Accessing `rucio` and `justin` require a bit more](#accessing-`rucio`-and-`justin`-require-a-bit-more) + - [getting a token for xroot access in AL9](#getting-a-token-for-xroot-access-in-al9) + - [finding a file](#finding-a-file) +- [More finding files by characteristics using metacat](#more-finding-files-by-characteristics-using-metacat) +- [Accessing data for use in your analysis](#accessing-data-for-use-in-your-analysis) +- [Quiz](#quiz) +- [Useful links to bookmark](#useful-links-to-bookmark) \ No newline at end of file diff --git a/_includes/03.2-UPS.toc.md b/_includes/03.2-UPS.toc.md new file mode 100644 index 0000000..b395d93 --- /dev/null +++ b/_includes/03.2-UPS.toc.md @@ -0,0 +1,3 @@ +## Table of Contents +- [What is UPS and why do we need it?](#what-is-ups-and-why-do-we-need-it?) + - [UPS basic commands](#ups-basic-commands) \ No newline at end of file diff --git a/_includes/04-Spack.toc.md b/_includes/04-Spack.toc.md new file mode 100644 index 0000000..bab7abd --- /dev/null +++ b/_includes/04-Spack.toc.md @@ -0,0 +1,5 @@ +## Table of Contents +- [What is Spack and why do we need it?](#what-is-spack-and-why-do-we-need-it?) +- [Minimal spack for root analysis and file access](#minimal-spack-for-root-analysis-and-file-access) +- [A more flexible environment with more packages but you have to make choices of versions](#a-more-flexible-environment-with-more-packages-but-you-have-to-make-choices-of-versions) + - [Spack basic commands](#spack-basic-commands) \ No newline at end of file diff --git a/_includes/05-end-of-basics.toc.md b/_includes/05-end-of-basics.toc.md new file mode 100644 index 0000000..f1e3b06 --- /dev/null +++ b/_includes/05-end-of-basics.toc.md @@ -0,0 +1,3 @@ +## Table of Contents +- [You can ask questions here:](#you-can-ask-questions-here:) +- [You can continue on with these additional modules.](#you-can-continue-on-with-these-additional-modules.) \ No newline at end of file diff --git a/_includes/05.1-improve-code-efficiency.toc.md b/_includes/05.1-improve-code-efficiency.toc.md new file mode 100644 index 0000000..5bedb17 --- /dev/null +++ b/_includes/05.1-improve-code-efficiency.toc.md @@ -0,0 +1,11 @@ +## Table of Contents + - [Session Video](#session-video) + - [Live Notes](#live-notes) + - [Code Make-over](#code-make-over) + - [CPU optimization:](#cpu-optimization:) +- [Memory optimization:](#memory-optimization:) +- [I/O optimization:](#i/o-optimization:) +- [Build time optimization:](#build-time-optimization:) +- [Workflow optimization:](#workflow-optimization:) +- [Software readability and maintainability:](#software-readability-and-maintainability:) +- [Coding for Thread Safety](#coding-for-thread-safety) \ No newline at end of file diff --git a/_includes/10-closing-remarks.toc.md b/_includes/10-closing-remarks.toc.md new file mode 100644 index 0000000..ea1d5ab --- /dev/null +++ b/_includes/10-closing-remarks.toc.md @@ -0,0 +1,6 @@ +## Table of Contents +- [Video Session](#video-session) +- [Two Days of Training](#two-days-of-training) +- [Survey time!](#survey-time!) +- [Next Steps](#next-steps) +- [Long Term Support](#long-term-support) \ No newline at end of file diff --git a/_includes/setup.toc.md b/_includes/setup.toc.md new file mode 100644 index 0000000..310d2aa --- /dev/null +++ b/_includes/setup.toc.md @@ -0,0 +1,33 @@ +## Table of Contents +- [Objectives](#objectives) +- [Requirements](#requirements) +- [Step 1: DUNE membership](#step 1: dune membership) +- [Step 2: Getting accounts](#step 2: getting accounts) + - [With FNAL](#with fnal) + - [With CERN](#with cern) +- [Step 3: Mission setup (rest of this page)](#step 3: mission setup (rest of this page)) +- [0. Basic setup on your computer.](#0. basic setup on your computer.) +- [1. Kerberos business](#1. kerberos business) +- [2. ssh-in](#2. ssh-in) +- [3. Get a clean shell](#3. get a clean shell) +- [Software setup ](#software setup ) + - [4.1 Setting up DUNE software - Scientific Linux 7 version ](#4.1 setting up dune software - scientific linux 7 version ) + - [Caveats for later](#caveats for later) + - [4.2 Setting up DUNE software - Alma9 version ](#4.2 setting up dune software - alma9 version ) + - [Caveats for later](#caveats for later) +- [4.2 Setting up DUNE software - Alma9 version](#4.2 setting up dune software - alma9 version) + - [Caveats](#caveats) +- [5. Exercise! (For SL7 - it's easy)](#5. exercise! (for sl7 - it's easy)) +- [5. Exercise! (For AL9 - it's easy)](#5. exercise! (for al9 - it's easy)) +- [6. Getting setup for streaming and grid access](#6. getting setup for streaming and grid access) + - [Tokens method ](#tokens method ) + - [1. Get and store your token](#1. get and store your token) + - [2. Tell the system where your token is](#2. tell the system where your token is) +- [Set up on CERN machines ](#set up on cern machines ) + - [1. Setup in Alma9](#1. setup in alma9) + - [2. For SL7](#2. for sl7) + - [Source the DUNE environment SL7 setup script](#source the dune environment sl7 setup script) + - [3. Getting authentication for data access](#3. getting authentication for data access) + - [4. Access tutorial datasets](#4. access tutorial datasets) + - [5. Notify us](#5. notify us) + - [Useful Links](#useful links) \ No newline at end of file diff --git a/code/tocgen.py b/code/tocgen.py new file mode 100644 index 0000000..8c50517 --- /dev/null +++ b/code/tocgen.py @@ -0,0 +1,46 @@ +import os,sys,string + +def dealwithline(input,toclines): + for line in input: + if line.startswith('{% include'): + print ("dealing with include",line) + includename = line.split('%')[1].strip().split(' ')[1] + newfile = os.path.join("_includes/",includename) + newlines = open(newfile,'r').readlines() + dealwithline(newlines,toclines) + continue + # remove any labels + line = line.split(' ## If you run into problems now or later, check out the [Common Error Messages]({{ site.baseurl }}/ErrorMessages) page and the [FAQ page](https://github.com/orgs/DUNE/projects/19/) > if that doesn't help, use [DUNE Slack](https://dunescience.slack.com/archives/C02TJDHUQPR) channel `#computing-training-basics` to ask us about the problem - there is always a new one cropping up. {: .challenge} +{%include setup.toc.md%} + ## Requirements @@ -750,7 +751,7 @@ If you are experiencing issues, please contact us as soon as possible. Be sure t > If you have issues here, please go to the [#computing-training-basics](https://dunescience.slack.com/archives/C02TJDHUQPR)Slack channel to get support. Please note that you are on a CERN machine in your message. Thanks! {: .discussion} -### Useful Links +## Useful Links The [DUNE FAQ][DUNE FAQ] on GitHub. diff --git a/setup.toc.md b/setup.toc.md new file mode 100644 index 0000000..74d5e5f --- /dev/null +++ b/setup.toc.md @@ -0,0 +1,32 @@ +- [Objectives](#objectives) +- [Requirements](#requirements) +- [Step 1: DUNE membership](#step 1: dune membership) +- [Step 2: Getting accounts](#step 2: getting accounts) + - [With FNAL](#with fnal) + - [With CERN](#with cern) +- [Step 3: Mission setup (rest of this page)](#step 3: mission setup (rest of this page)) +- [0. Basic setup on your computer.](#0. basic setup on your computer.) +- [1. Kerberos business](#1. kerberos business) +- [2. ssh-in](#2. ssh-in) +- [3. Get a clean shell](#3. get a clean shell) +- [Software setup ](#software setup ) + - [4.1 Setting up DUNE software - Scientific Linux 7 version ](#4.1 setting up dune software - scientific linux 7 version ) + - [Caveats for later](#caveats for later) + - [4.2 Setting up DUNE software - Alma9 version ](#4.2 setting up dune software - alma9 version ) + - [Caveats for later](#caveats for later) +- [4.2 Setting up DUNE software - Alma9 version](#4.2 setting up dune software - alma9 version) + - [Caveats](#caveats) +- [5. Exercise! (For SL7 - it's easy)](#5. exercise! (for sl7 - it's easy)) +- [5. Exercise! (For AL9 - it's easy)](#5. exercise! (for al9 - it's easy)) +- [6. Getting setup for streaming and grid access](#6. getting setup for streaming and grid access) + - [Tokens method ](#tokens method ) + - [1. Get and store your token](#1. get and store your token) + - [2. Tell the system where your token is](#2. tell the system where your token is) +- [Set up on CERN machines ](#set up on cern machines ) + - [1. Setup in Alma9](#1. setup in alma9) + - [2. For SL7](#2. for sl7) + - [Source the DUNE environment SL7 setup script](#source the dune environment sl7 setup script) + - [3. Getting authentication for data access](#3. getting authentication for data access) + - [4. Access tutorial datasets](#4. access tutorial datasets) + - [5. Notify us](#5. notify us) + - [Useful Links](#useful links) \ No newline at end of file