-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathexample-run.html
302 lines (276 loc) · 19.2 KB
/
example-run.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
<!DOCTYPE html>
<html lang="en" data-content_root="./">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Example pipeline execution — liger_iris_pipeline v0.5.dev</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="_static/bootstrap-astropy.css?v=9d21690f" />
<link rel="stylesheet" type="text/css" href="_static/graphviz.css?v=fd3f3429" />
<link rel="stylesheet" type="text/css" href="_static/plot_directive.css" />
<link rel="stylesheet" type="text/css" href="_static/nbsphinx-code-cells.css?v=2aa19091" />
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=3b3f3827"></script>
<script src="_static/doctools.js?v=9a2dae69"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script crossorigin="anonymous" integrity="sha256-Ae2Vz/4ePdIu6ZyI/5ZGsYnb+m0JlOmKPjt6XZ9JJkA=" src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.4/require.min.js"></script>
<script>window.MathJax = {"tex": {"inlineMath": [["$", "$"], ["\\(", "\\)"]], "processEscapes": true}, "options": {"ignoreHtmlClass": "tex2jax_ignore|mathjax_ignore|document", "processHtmlClass": "tex2jax_process|mathjax_process|math|output_area"}}</script>
<script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<script type="text/javascript" src="_static/sidebar.js"></script>
<script type="text/javascript" src="_static/copybutton.js"></script>
<link rel="icon" href="_static/astropy_logo.ico"/>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="IRIS Data Reduction System design" href="design.html" />
<link rel="prev" title="Getting Started" href="getting-started.html" />
<link href='https://fonts.googleapis.com/css?family=Source+Sans+Pro:200,600' rel='stylesheet' type='text/css'/>
</head><body>
<div class="topbar">
<a class="brand" title="Documentation Home" href="index.html"><span id="logotext1">liger_iris_pipeline</span><span id="logotext2"></span><span id="logotext3">:docs</span></a>
<ul>
<li><a class="homelink" title="Astropy Homepage" href="http://www.astropy.org"></a></li>
<li><a title="General Index" href="genindex.html">Index</a></li>
<li><a title="Module Index" href="py-modindex.html">Modules</a></li>
<li>
<form action="search.html" method="get">
<input type="text" name="q" placeholder="Search" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</li>
</ul>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right">
<a href="design.html" title="IRIS Data Reduction System design">
next »
</a>
</li>
<li class="right">
<a href="getting-started.html" title="Getting Started">
« previous
</a>
|
</li>
<li>
<a href="index.html">liger_iris_pipeline v0.5.dev</a>
»
</li>
<li>Example pipeline execution</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="Example-pipeline-execution">
<h1>Example pipeline execution<a class="headerlink" href="#Example-pipeline-execution" title="Link to this heading">¶</a></h1>
<p>Here is an example of what it takes to configure and run a pipeline with flat-fielding and background subtraction.</p>
<section id="Setup-CRDS">
<h2>Setup CRDS<a class="headerlink" href="#Setup-CRDS" title="Link to this heading">¶</a></h2>
<p>Make sure you have a local checkout of the CRDS cache as explained in the <a class="reference internal" href="getting-started.html"><span class="doc">Getting started page</span></a>. Either run <code class="docutils literal notranslate"><span class="pre">source</span> <span class="pre">setup_local_crds.sh</span></code> to setup the enviroment variables in the shell or execute <code class="docutils literal notranslate"><span class="pre">setup_local_crds.py</span></code> for Python. This is needed to point the CRDS software to the CRDS cache. Optionally source this in your shell configuration to automatically set this up.</p>
<div class="nbinput nblast docutils container">
<div class="prompt highlight-none notranslate"><div class="highlight"><pre><span></span>[ ]:
</pre></div>
</div>
<div class="input_area highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">%</span><span class="k">run</span> ../setup_local_crds.py
</pre></div>
</div>
</div>
</section>
<section id="Get-input-simulations">
<h2>Get input simulations<a class="headerlink" href="#Get-input-simulations" title="Link to this heading">¶</a></h2>
<p>Download simulated input FITS files for the IRIS imager from <a class="reference external" href="https://figshare.com/articles/TMT_IRIS_test_simulations/9941939">Figshare</a>. It contains a raw science frame, a raw flat frame and a raw background frame.</p>
<div class="nbinput nblast docutils container">
<div class="prompt highlight-none notranslate"><div class="highlight"><pre><span></span>[ ]:
</pre></div>
</div>
<div class="input_area highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">urllib.request</span>
<span class="kn">import</span> <span class="nn">zipfile</span>
<span class="kn">import</span> <span class="nn">os.path</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="s2">"iris_example_data/raw_flat_frame_cal.fits"</span><span class="p">):</span>
<span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlretrieve</span><span class="p">(</span>
<span class="s2">"https://ndownloader.figshare.com/articles/9941939/versions/1"</span><span class="p">,</span>
<span class="s2">"iris_example_data.zip"</span>
<span class="p">)</span>
<span class="k">with</span> <span class="n">zipfile</span><span class="o">.</span><span class="n">ZipFile</span><span class="p">(</span><span class="s2">"iris_example_data.zip"</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> <span class="k">as</span> <span class="n">zip_ref</span><span class="p">:</span>
<span class="n">zip_ref</span><span class="o">.</span><span class="n">extractall</span><span class="p">(</span><span class="s2">"iris_example_data"</span><span class="p">)</span>
<span class="c1"># this could take a few minutes</span>
</pre></div>
</div>
</div>
<div class="nbinput nblast docutils container">
<div class="prompt highlight-none notranslate"><div class="highlight"><pre><span></span>[ ]:
</pre></div>
</div>
<div class="input_area highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">%</span><span class="k">ls</span> iris_example_data/
</pre></div>
</div>
</div>
</section>
<section id="Preprocess-the-flat-frame">
<h2>Preprocess the flat frame<a class="headerlink" href="#Preprocess-the-flat-frame" title="Link to this heading">¶</a></h2>
<p>First we need to remove the dark frame from the flat frame and normalize it. A dark frame is already available in the CRDS and the pipeline knows how to retrieve it based on the metadata in the FITS file headers.</p>
<p>We can check in the package documentation what are the available pipelines and check the configuration options of the pipeline.ProcessFlatfieldL2 class.</p>
<p>We do not need to customize it so we can directly call it from tmtrun and pass the input FITS file:</p>
<p>This will pickup the relevant dark frame from the CRDS and process the file:</p>
<div class="nbinput nblast docutils container">
<div class="prompt highlight-none notranslate"><div class="highlight"><pre><span></span>[ ]:
</pre></div>
</div>
<div class="input_area highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">!</span>tmtrun<span class="w"> </span>iris_pipeline.pipeline.ProcessFlatfieldL2<span class="w"> </span>iris_example_data/raw_flat_frame_cal.fits
</pre></div>
</div>
</div>
</section>
<section id="Configure-the-image-processing-pipeline">
<h2>Configure the image processing pipeline<a class="headerlink" href="#Configure-the-image-processing-pipeline" title="Link to this heading">¶</a></h2>
<p>The <code class="docutils literal notranslate"><span class="pre">`ProcessImagerL2Pipeline</span></code> <api/iris_pipeline.pipeline.ProcessImagerL2Pipeline.rst#iris_pipeline.ProcessImagerL2Pipeline>`__ can be configured using a INI-style configuration file:</p>
<div class="nbinput nblast docutils container">
<div class="prompt highlight-none notranslate"><div class="highlight"><pre><span></span>[ ]:
</pre></div>
</div>
<div class="input_area highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">%%file</span> image2_iris.cfg
<span class="n">name</span> <span class="o">=</span> <span class="s2">"ProcessImagerL2Pipeline"</span>
<span class="k">class</span> <span class="err">= "</span><span class="nc">iris_pipeline</span><span class="o">.</span><span class="n">pipeline</span><span class="o">.</span><span class="n">ProcessImagerL2Pipeline</span><span class="s2">"</span>
<span class="n">save_results</span> <span class="o">=</span> <span class="kc">True</span>
<span class="p">[</span><span class="n">steps</span><span class="p">]</span>
<span class="p">[[</span><span class="n">bkg_subtract</span><span class="p">]]</span>
<span class="p">[[</span><span class="n">assign_wcs</span><span class="p">]]</span>
<span class="n">skip</span> <span class="o">=</span> <span class="kc">True</span>
<span class="p">[[</span><span class="n">flat_field</span><span class="p">]]</span>
<span class="n">config_file</span> <span class="o">=</span> <span class="n">flat_field</span><span class="o">.</span><span class="n">cfg</span>
<span class="p">[[</span><span class="n">photom</span><span class="p">]]</span>
<span class="n">skip</span> <span class="o">=</span> <span class="kc">True</span>
<span class="p">[[</span><span class="n">resample</span><span class="p">]]</span>
<span class="n">skip</span> <span class="o">=</span> <span class="kc">True</span>
</pre></div>
</div>
</div>
<p>first we specify that we want to execute the pipeline defined in the pipeline, then we can configure each of the steps, for example skip some of them. Also we can import the configuration of a step from another file, in this case flat_field.cfg:</p>
<div class="nbinput nblast docutils container">
<div class="prompt highlight-none notranslate"><div class="highlight"><pre><span></span>[ ]:
</pre></div>
</div>
<div class="input_area highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">%%file</span> flat_field.cfg
<span class="n">name</span> <span class="o">=</span> <span class="s2">"flat_field"</span>
<span class="k">class</span> <span class="err">= "</span><span class="nc">jwst</span><span class="o">.</span><span class="n">flatfield</span><span class="o">.</span><span class="n">FlatFieldStep</span><span class="s2">"</span>
<span class="n">override_flat</span> <span class="o">=</span> <span class="s1">'raw_flat_frame_flat.fits'</span>
</pre></div>
</div>
</div>
</section>
<section id="Define-the-input-data">
<h2>Define the input data<a class="headerlink" href="#Define-the-input-data" title="Link to this heading">¶</a></h2>
<p>JWST created a specification for defining how input files should be used by a pipeline, it is a JSON file named an association, see the <a class="reference external" href="https://jwst-pipeline.readthedocs.io/en/latest/jwst/associations/overview.html">JWST documentation</a>.</p>
<p>In our example we need to specify a input raw science frame ad a background to be subtracted:</p>
<div class="nbinput nblast docutils container">
<div class="prompt highlight-none notranslate"><div class="highlight"><pre><span></span>[ ]:
</pre></div>
</div>
<div class="input_area highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">%%file</span> association.json
<span class="p">{</span>
<span class="s2">"asn_rule"</span><span class="p">:</span> <span class="s2">"Asn_Lv2Image"</span><span class="p">,</span>
<span class="s2">"asn_pool"</span><span class="p">:</span> <span class="s2">"pool"</span><span class="p">,</span>
<span class="s2">"asn_type"</span><span class="p">:</span> <span class="s2">"image2"</span><span class="p">,</span>
<span class="s2">"products"</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">"name"</span><span class="p">:</span> <span class="s2">"test_iris_subtract_bg_flat"</span><span class="p">,</span>
<span class="s2">"members"</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">"expname"</span><span class="p">:</span> <span class="s2">"iris_example_data/raw_science_frame_sci.fits"</span><span class="p">,</span>
<span class="s2">"exptype"</span><span class="p">:</span> <span class="s2">"science"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">"expname"</span><span class="p">:</span> <span class="s2">"iris_example_data/raw_background_frame_cal.fits"</span><span class="p">,</span>
<span class="s2">"exptype"</span><span class="p">:</span> <span class="s2">"background"</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">}</span>
</pre></div>
</div>
</div>
</section>
<section id="Execute-the-pipeline-from-the-command-line">
<h2>Execute the pipeline from the command line<a class="headerlink" href="#Execute-the-pipeline-from-the-command-line" title="Link to this heading">¶</a></h2>
<p>Execute the pipeline from the command line We can use tmtrun from a terminal to execute the pipeline:</p>
<div class="nbinput nblast docutils container">
<div class="prompt highlight-none notranslate"><div class="highlight"><pre><span></span>[ ]:
</pre></div>
</div>
<div class="input_area highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">!</span>tmtrun<span class="w"> </span>image2_iris.cfg<span class="w"> </span>association.json
</pre></div>
</div>
</div>
</section>
<section id="Plot-the-reduced-frame">
<h2>Plot the reduced frame<a class="headerlink" href="#Plot-the-reduced-frame" title="Link to this heading">¶</a></h2>
<p>Finally we can use <code class="docutils literal notranslate"><span class="pre">iris_pipeline</span></code> to load the data and plot them:</p>
<div class="nbinput nblast docutils container">
<div class="prompt highlight-none notranslate"><div class="highlight"><pre><span></span>[ ]:
</pre></div>
</div>
<div class="input_area highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">iris_pipeline</span>
<span class="n">reduced_science_frame</span> <span class="o">=</span> <span class="n">iris_pipeline</span><span class="o">.</span><span class="n">datamodels</span><span class="o">.</span><span class="n">IRISImageModel</span><span class="p">(</span><span class="s2">"test_iris_subtract_bg_flat_cal.fits"</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="o">%</span><span class="k">matplotlib</span> inline
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">reduced_science_frame</span><span class="o">.</span><span class="n">data</span><span class="p">,</span> <span class="n">vmin</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">vmax</span><span class="o">=</span><span class="mi">1500</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s2">"Processed frame"</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
<span class="k">pass</span>
</pre></div>
</div>
</div>
<div class="nbinput nblast docutils container">
<div class="prompt highlight-none notranslate"><div class="highlight"><pre><span></span>[ ]:
</pre></div>
</div>
<div class="input_area highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">!</span>rm<span class="w"> </span>image2_iris.cfg<span class="w"> </span>association.json<span class="w"> </span>flat_field.cfg
</pre></div>
</div>
</div>
</section>
</section>
<div class="clearer"></div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="Main">
<div class="sphinxsidebarwrapper"><h3>Page Contents</h3>
<ul>
<li><a class="reference internal" href="#">Example pipeline execution</a><ul>
<li><a class="reference internal" href="#Setup-CRDS">Setup CRDS</a></li>
<li><a class="reference internal" href="#Get-input-simulations">Get input simulations</a></li>
<li><a class="reference internal" href="#Preprocess-the-flat-frame">Preprocess the flat frame</a></li>
<li><a class="reference internal" href="#Configure-the-image-processing-pipeline">Configure the image processing pipeline</a></li>
<li><a class="reference internal" href="#Define-the-input-data">Define the input data</a></li>
<li><a class="reference internal" href="#Execute-the-pipeline-from-the-command-line">Execute the pipeline from the command line</a></li>
<li><a class="reference internal" href="#Plot-the-reduced-frame">Plot the reduced frame</a></li>
</ul>
</li>
</ul>
</div>
</div>
<div class="clearer"></div>
</div>
<footer class="footer">
<p class="pull-right">
<a href="https://github.com/oirlab/liger_iris_pipeline/tree/master/docs/example-run.ipynb">Edit This Page on Github</a>
<a href="_sources/example-run.ipynb.txt"
rel="nofollow">Page Source</a>
<a href="#">Back to Top</a></p>
<p>
© Copyright 2024, Andrea Zonca, Arun Surya, Bryson Cale.<br/>
Created using <a href="http://www.sphinx-doc.org/en/stable/">Sphinx</a> 8.0.2.
Last built 02 Sep 2024. <br/>
</p>
</footer>
</body>
</html>