ASimpleNanoporeTutorial/rsync_command_list.html at master · alexiswl/ASimpleNanoporeTutorial · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
<!DOCTYPE html>
<html>
  <head>
    <title>Rsync nanopore data onto a HPC</title>
    <meta charset="utf-8">
    <meta name="author" content="Alexis Lucattini" />
    <link href="libs/remark-css-0.0.1/default.css" rel="stylesheet" />
    <link href="libs/remark-css-0.0.1/default-fonts.css" rel="stylesheet" />
  </head>
  <body>
    <textarea id="source">
class: center, middle, inverse, title-slide

# Rsync nanopore data onto a HPC
### Alexis Lucattini
### 2017/08/17

---


# Sending data onto a server
The most common way to send data from a MinION laptop to a cluster is through the rsync command.

Rsync is installed by default on Ubuntu and MacOS systems.

Unforunately for Windows users, life is less straight forward.
You can either use:
1. Cygwin
2. A docker container such as andthensome/docker-node-rsync
  + This will require you to mount the appropriate directory and re-perform the ssh key setup in the previous step from within the docker container.

---

# Setting up a ssh-key
Make sure you can ssh into the rsync server prior to running the command.
See [tutorial 1](./basic_shell_logging.html) for assistance.

---

# The actual rsync command

This is the command to use. Do not be alarmed. We will go through it in next slides

```bash
# Setup variables
local_read_dir=/path/to/reads
username=&lt;server_username&gt;
servername=&lt;name_of_server
remote_read_dir=/path/to/server
# Rsync command
rsync --recursive --times --checksum \
--prune-empty-dirs --remove-source-files --stats \
--include '*/' --include '*.fast5.tar.gz' --include '*.tsv' \
--exclude '*' \
${local_read_dir} ${username}@${servername}:${remote_read_dir}
```

---

# Rsync options
* --recusive: Copy files within folder

* --times: Preserve time stamps

* --prune-empty-dirs: This ensures that any directories with no matching files are not placed on the server

* --remove-source-files: Deletes files from the source folder.

* --stats: Print verbose output of the transfer
* --include / --exclude: Select which files to send across.
  + Order of commands important.
  + When using this option, we must first use --include '*/' to search recursively.
  + Then we include all files ending with '\*.fast5.tar.gz' and '\*.tsv'
  + We then exclude all other files.
* Local path

* Remote path

---

# Debugging

Prior to running the command, add the --dry-run parameter.
This will print a summary of the files that would have been moved/modified without actually doing the moving.

Alternatively, you can remove the final parameter `${username}@${servername}:${remote_read_dir}` first. This will just print the list of files that will be transferred.
    </textarea>
<script src="https://remarkjs.com/downloads/remark-latest.min.js"></script>
<script>var slideshow = remark.create({
"highlightStyle": "github",
"highlightLines": true,
"countIncrementalSlides": false
});
if (window.HTMLWidgets) slideshow.on('afterShowSlide', function (slide) {
  window.dispatchEvent(new Event('resize'));
});
(function() {
  var d = document, s = d.createElement("style"), r = d.querySelector(".remark-slide-scaler");
  if (!r) return;
  s.type = "text/css"; s.innerHTML = "@page {size: " + r.style.width + " " + r.style.height +"; }";
  d.head.appendChild(s);
})();</script>

<script type="text/x-mathjax-config">
MathJax.Hub.Config({
  tex2jax: {
    skipTags: ['script', 'noscript', 'style', 'textarea', 'pre']
  }
});
</script>
<!-- dynamically load mathjax for compatibility with self-contained -->
<script>
(function () {
  var script = document.createElement('script');
  script.type = 'text/javascript';
  script.src  = 'https://cdn.bootcss.com/mathjax/2.7.1/MathJax.js?config=TeX-MML-AM_CHTML';
  if (location.protocol !== 'file:' && /^https?:/.test(script.src))
    script.src  = script.src.replace(/^https?:/, '');
  document.getElementsByTagName('head')[0].appendChild(script);
})();
</script>
  </body>
</html>