-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.xml
454 lines (454 loc) · 62.7 KB
/
index.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Home on GetLatestVersion – Europe</title><link>http://www.getlatestversion.eu/</link><description>Recent content in Home on GetLatestVersion – Europe</description><generator>Hugo -- gohugo.io</generator><language>en-us</language><lastBuildDate>Wed, 03 Nov 2021 03:00:00 +0200</lastBuildDate><atom:link href="http://www.getlatestversion.eu/index.xml" rel="self" type="application/rss+xml"/><item><title>The limit is your own imagination, do it with Azure DevOps Services REST API</title><link>http://www.getlatestversion.eu/2021/11/the-limit-is-your-own-imagination-do-it-with-azure-devops-services-rest-api/</link><pubDate>Wed, 03 Nov 2021 03:00:00 +0200</pubDate><author>Bartolomeo Lombardi</author><author/><guid>http://www.getlatestversion.eu/2021/11/the-limit-is-your-own-imagination-do-it-with-azure-devops-services-rest-api/</guid><description><p>Azure DevOps is the platform signed by Microsoft that helps software factories to improve the development process, automatic tests and frequent releases. Their aim is to have additional value for the end customer in terms of product quality.</p>
<p>With Azure DevOps Release you can separate the delivery environments into groups, obtaining integration, acceptance, production environments and so on. Most of time, the pipelines are configured to release the artifacts related to the main branches. However, Azure DevOps doesn&rsquo;t allow to modify easily a specific release for a single target within a Deployment Group (DG) starting from different artifacts.</p>
<h1 id="automated-through-the-azure-devops-services-rest-api">Automated through the Azure DevOps Services REST API</h1>
<p>Consider a scenario in which we have a DG called &ldquo;integration&rdquo; that collects 20 virtual machines (VMs), one for each developer. Through this DG we can use a pipeline that deploys the latest version of the product to all the VMs at the same time. Anyway, more often a developer wants to test a release with his own changes before merging on the main branch. He needs to deploy his build to only his VM, without involving the others. It&rsquo;s onerous for him unless he&rsquo;s an Azure DevOps user, capable of modifying his deployment group, the branch where to download the artifacts, etc. It&rsquo;s possible to automate all the previous steps needed to launch a release only on your target environment, starting from a particular artifact.</p>
<p>In the following the steps to achive it:</p>
<ol>
<li>set a new tag to the VM in the DG</li>
<li>put the tag into release pipeline</li>
<li>launch the release pipeline</li>
<li>remove the tag from DG and release pipeline</li>
</ol>
<p>We have taken advantage of the <a href="https://docs.microsoft.com/en-us/rest/api/azure/devops/?view=azure-devops-rest-6.1">Azure DevOps Services REST API</a> made available by Azure DevOps through a script in PowerShell.
Here there are the essential steps:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-powershell" data-lang="powershell">Write-Host <span style="color:#e6db74">&#34;Adding tag for your VM in the Deployment Group: $deploymentGroup&#34;</span>
$UriOrga = <span style="color:#e6db74">&#34;https://dev.azure.com/&lt;organization&gt;/&lt;project&gt;/_apis/distributedtask/deploymentgroups/&#34;</span> + $DeploymentGroupId + <span style="color:#e6db74">&#34;/targets?api-version=6.0-preview.1&#34;</span>
$output = Invoke-RestMethod -Uri $UriOrga -Method get -Headers $AzureDevOpsAuthenicationHeader
$UriOrga = <span style="color:#e6db74">&#34;https://dev.azure.com/&lt;organization&gt;/&lt;project&gt;/_apis/distributedtask/deploymentgroups/&#34;</span> + $DeploymentGroupId + <span style="color:#e6db74">&#34;/targets?api-version=6.0-preview.1&#34;</span>
Invoke-RestMethod -Method Patch -Uri $UriOrga -Headers $AzureDevOpsAuthenicationHeader -Body $TargetVMs -ContentType application/json
Write-Host <span style="color:#e6db74">&#34;Adding tag to Deployment Group stage in the Realese Pipeline&#34;</span>
$UriOrga = <span style="color:#e6db74">&#34;https://vsrm.dev.azure.com/&lt;organization&gt;/&lt;project&gt;/_apis/release/definitions/&#34;</span> + $ReleaseDefinitionId + <span style="color:#e6db74">&#34;?api-version=6.0&#34;</span>
$UpdateEnvironmentsBody = Invoke-RestMethod -Uri $UriOrga -Method Get -Headers $AzureDevOpsAuthenicationHeader -ContentType application/json
$UpdateEnvironmentsBody.environments[0].deployPhases[0].deploymentInput.tags = @( $Tag )
$UpdateEnvironmentsBody = ConvertTo-Json $UpdateEnvironmentsBody -Depth 10
$UriOrga = <span style="color:#e6db74">&#34;https://vsrm.dev.azure.com/&lt;organization&gt;/&lt;project&gt;/_apis/release/definitions?api-version=6.0&#34;</span>
Invoke-RestMethod -Uri $UriOrga -Method Put -Headers $AzureDevOpsAuthenicationHeader -Body $UpdateEnvironmentsBody -ContentType application/json
Write-Host <span style="color:#e6db74">&#34;Launching Release Pipeline for your VM&#34;</span>
$UriOrga = <span style="color:#e6db74">&#34;https://vsrm.dev.azure.com/&lt;organization&gt;/&lt;project&gt;/_apis/release/releases?api-version=6.0&#34;</span>
$CreateReleaseBody = ConvertTo-Json $CreateReleaseBody -Depth 10
Invoke-RestMethod -Uri $UriOrga -Method Post -Headers $AzureDevOpsAuthenicationHeader -Body $CreateReleaseBody -ContentType application/json
Write-Host <span style="color:#e6db74">&#34;Removing all tags from Deployment Group stage in the Realese Pipeline&#34;</span>
$UriOrga = <span style="color:#e6db74">&#34;https://vsrm.dev.azure.com/&lt;organization&gt;/&lt;project&gt;/_apis/release/definitions/&#34;</span> + $ReleaseDefinitionId + <span style="color:#e6db74">&#34;?api-version=6.0&#34;</span>
$UpdateEnvironmentsBody = Invoke-RestMethod -Uri $UriOrga -Method Get -Headers $AzureDevOpsAuthenicationHeader -ContentType application/json
$UpdateEnvironmentsBody.environments[0].deployPhases[0].deploymentInput.tags = @()
$UpdateEnvironmentsBody = ConvertTo-Json $UpdateEnvironmentsBody -Depth 10
$UriOrga = <span style="color:#e6db74">&#34;https://vsrm.dev.azure.com/&lt;organization&gt;/&lt;project&gt;/_apis/release/definitions?api-version=6.0&#34;</span>
Invoke-RestMethod -Uri $UriOrga -Method Put -Headers $AzureDevOpsAuthenicationHeader -Body $UpdateEnvironmentsBody -ContentType application/json
</code></pre></div><h1 id="conclusion">Conclusion</h1>
<p>Like any Azure service, Microsoft&rsquo;s DevOps counterpart can also be managed by a set of APIs that replicate everything that can be done through the portal. First you need to get a <a href="https://docs.microsoft.com/it-it/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&amp;tabs=preview-page">PAT (Personal Access Token)</a> through which you can authenticate inside your organization.
Then you can take advantage of any tool that invokes API Rest to do whatever you want, i.e. launching Release, creating or modifying Deployment Groups, starting specific Tests and so on. This can be a hint to create a user-friendly interface for developers not accustomed to DevOps. It could automate set of actions that would be otherwise onerous.</p>
<p>In these cases the limit is your own imagination.</p></description></item><item><title>Migrating work items from Jira to Azure DevOps</title><link>http://www.getlatestversion.eu/2020/08/migrating-work-items-from-jira-to-azure-devops/</link><pubDate>Tue, 18 Aug 2020 17:30:00 +0200</pubDate><author>Alessandro Alpi</author><guid>http://www.getlatestversion.eu/2020/08/migrating-work-items-from-jira-to-azure-devops/</guid><description><h1 id="intro">Intro</h1>
<p>Migrations are hard tasks to deal with. Not just for IT. Working with the culture of many companies, I&rsquo;ve got confirmation that the tool should be considered at the end of the migration process. After setting up the ceremonies and switching the methodologies from legacy to lean/iterative, it comes finally to choose from the available tools (<em>enterprise awareness</em>) and including new ones. The goal is to get all the stuff which fits the real scenario.</p>
<p>This post is a quick step by step guide to migrate work items from <a href="https://www.atlassian.com/software/jira">Jira cloud</a> to <a href="https://azure.microsoft.com/it-it/services/devops/">Azure DevOps Services</a>. I&rsquo;m going to describe the last step of one of my customers&rsquo; migration.</p>
<h2 id="table-of-content">Table of content</h2>
<ul>
<li><a href="#Getting-started">Getting started</a></li>
<li><a href="#The-software-selection">The software selection</a></li>
<li><a href="#Executing-the-tool">Executing the tool</a>
<ul>
<li><a href="#How-to-export-Jira-issues">How to export Jira issues</a></li>
<li><a href="#How-to-import-to-Azure-DevOps">How to import to Azure DevOps</a></li>
</ul>
</li>
<li><a href="#Conclusions">Conclusions</a></li>
</ul>
<h2 id="getting-started">Getting started</h2>
<p>Before going in-depth with technical details, I would like to share some tips. As we have already said, the migrations are complex tasks. Mandatory requirements should be a strong knowledge in terms of business and team management, enterprise awareness and years of experience on different real scenarios.</p>
<p>We must avoid any decision if our ideas and targets are not clear. Another important requirement is to understand in depth the workflows you will work on, both the legacy one and the new one you&rsquo;re figuring out. Some of the question we should ask ourselves are:</p>
<ul>
<li>Do we require these steps? And what about these work items?</li>
<li>Is this state workflow good for us? Should we change it? How?</li>
<li>Do we require to preserve relationships and items&rsquo; history?</li>
<li>Can we keep something which is working very well now? If so, what tools we&rsquo;re thinking about?</li>
</ul>
<h2 id="the-software-selection">The software selection</h2>
<p>The software selection has ended up on a tool made by <a href="https://solidify.se/">Solidify</a> (Thanks to the experienced members of our <a href="https://www.getlatestversion.eu/">getlatestversion.eu</a> community). Anyways, you can find more of them. For example:</p>
<ul>
<li><a href="https://ilclubdellesei.blog/2018/05/21/import-from-jira-to-vsts-in-5-steps/">TFS4JIRA</a></li>
<li><a href="http://www.codewrecks.com/blog/index.php/2019/01/19/import-work-item-from-external-system-to-azure-devops/">Importing work items to Azure DevOps by Gian Maria Ricci</a></li>
<li><a href="https://ilclubdellesei.blog/2018/05/21/import-from-jira-to-vsts-in-5-steps/">JiraToVsts </a> (via <a href="https://www.getlatestversion.eu/it/authors/phenix/">Michele Ferracin</a>)</li>
</ul>
<p>When exporting from Jira, the CLI implemented by <a href="https://solidify.se/">Solidify</a> connects to the Jira server (cloud by default and on-premises when configured), executes a <a href="https://www.atlassian.com/software/jira/guides/expand-jira/jql">JQL</a> query for extracting the Jira so-called &ldquo;Issues&rdquo;, evaluates and applies the mapping between users, work items, relationships and states, and exports a JSON file for each item.</p>
<p>When importing to Azure DevOps, the CLI imports the output JSON files using the same mapping configured into the configuration file in the exporting phase.</p>
<p>Why this tool? Because it has a couple of simple command lines and it consumes a JSON configuration which is clear. Additionally, it has many default behaviours, like the built-in configuration for <em>Scrum</em>, <em>agile</em> and <em>basic</em> process templates, which allows us to forget about the complexity of both the source and target software.</p>
<p><a href="#Intro">Back to top</a></p>
<h2 id="executing-the-tool">Executing the tool</h2>
<p>The scenario (Jira) has been configured with many states, sometimes with overlapping meaning (due to the legacy setup and different team&rsquo;s approach) and with custom workflows/links. On the other hand, Azure DevOps side, I&rsquo;ve created a customized Scrum template, with just two new work item types, which should support some of the customized source behaviours, and a couple of new states. So the tool has been configured as depicted in the following JSON (just a subset of maps):</p>
<p><img src="./01-config.png" alt="config"></p>
<p>Just that. Notice that we can configure project names, the JQL query for gathering issues, working folder names and the file for the user mappings.</p>
<p>First, download <a href="https://github.com/solidify/jira-azuredevops-migrator/releases">the latest release of the CLI from GitHub</a>. Then follow these steps-</p>
<h3 id="how-to-export-jira-issues">How to export Jira issues</h3>
<ol>
<li>
<p>create a folder called C:/Temp/JiraExport (you can configure this editing the JSON config file)</p>
</li>
<li>
<p>create a file called &ldquo;users.txt&rdquo; within that folder and put into it a list of <code>[email protected][email protected]</code> records</p>
<ul>
<li>please note that the Jira user can be represented without the domain, depending on its configuration</li>
</ul>
</li>
<li>
<p>copy the JSON configuration file (based on the template we&rsquo;re going to apply when importing) into the JiraExport folder</p>
<ul>
<li>modify the file in the maps section: link-map, type-map, field-map, and so on.</li>
</ul>
</li>
<li>
<p>get the credentials (as admin) and the Jira server URL</p>
</li>
<li>
<p>make your command line, which should look like the following:
<code>jira-export -u [email protected] -p userPwd --url https://jiraurl.ext --config config-scrum.json --force</code></p>
</li>
<li>
<p>run the command and look for the JSON files into the JiraExport folder</p>
<ul>
<li>look for any warning/error like the following and correct them (this will help you to import without any breaking change)</li>
</ul>
<p><img src="./02-errors.png" alt="errors"></p>
</li>
</ol>
<p><a href="#Intro">Back to top</a></p>
<h3 id="how-to-import-to-azure-devops">How to import to Azure DevOps</h3>
<p>It&rsquo;s time to execute the second command line, <strong>wi-import</strong>. As we can see, we have to get a personal access token (PAT) from Azure DevOps, as described <a href="https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&amp;tabs=preview-page">in the documentation</a>, with the full access.</p>
<p>Coming back to the configuration, we should pay attention to <code>base-area-path</code> and <code>base-iteration-path</code>. With these, we can choose the target of our migration, in terms of area and iteration. This means that we can manage our items after the import has been completed. With a query, for example, we can remove everything and start over with the migration. Cool. The command should like the following:</p>
<p><code>wi-import --token PAT --url https://dev.azure.com/org --config config-scrum.json --force</code></p>
<p><img src="./03-import.png" alt="import"></p>
<p>After ten minutes we&rsquo;ve got new areas and iterations:</p>
<p><img src="./04-iterations.png" alt="iterations"> <img src="./05-areas.png" alt="areas"></p>
<p>The Azure DevOps hierarchy of items (except for the Features, which I&rsquo;ve not mapped) has been preserved and also the history of any item:</p>
<p><img src="./06-history.png" alt="history"></p>
<p><a href="#Intro">Back to top</a></p>
<h2 id="conclusions">Conclusions</h2>
<p>In a couple of days, we&rsquo;ve migrated everything related to work items from Jira to Azure DevOps. Most of the time should be invested in configuring the JSON mappings and, for sure, to check for the errors caught during exporting items. Finally, after importing into AzDOs, you can start over and over, removing all the items under the pre-configured area/iteration, until everything looks good.</p>
<p><a href="#Intro">Back to top</a></p></description></item><item><title>Git Cherry-Pick for a strategic harvest</title><link>http://www.getlatestversion.eu/2020/07/git-cherry-pick-for-a-strategic-harvest/</link><pubDate>Mon, 13 Jul 2020 03:00:00 +0200</pubDate><author>Bartolomeo Lombardi</author><guid>http://www.getlatestversion.eu/2020/07/git-cherry-pick-for-a-strategic-harvest/</guid><description><h1 id="introduction">Introduction</h1>
<p>IT companies which develop and sell software product are always focused on delivering new features to propose new versions of them to the final client, while still considering the maintenance of the previous release.
Generally, we are talking about projects of millions of lines of code that requires multiple developing teams located around the world with strong coordination and collaboration among them, in which it is necessary to handle multiple versions of the same product with a significant number of release branches. Fixing a bug on multiple branches is a complex task because the developer needs to backport the same fix in the various release branches possible with Git and it is called &ldquo;cherry-picking&rdquo;.</p>
<h1 id="git-cherry-pick">Git cherry-pick</h1>
<p>One of the most powerful Git commands is Cherry-pick. It takes one or more commits as input parameters and shifts them to a different branch, by creating a new commit in the process.</p>
<p>Cherry-pick is commonly used in many Git workflows such as the Azure DevOps team’s, described in the following <a href="https://devblogs.microsoft.com/devops/improving-azure-devops-cherry-picking/">article</a>, see the image reported below for a visual representation of the workflow.
<img src="cherry-pick-workflow.jpg" alt="Cherry-Pick: way of working"></p>
<p>Every time a developer works with many version of the same product it is essential to ensure all the reported bugs have been fixed in as many version as possible. Cherry-picking happens on the main branch to avoid deploying a new release with the same bug.</p>
<h2 id="azure-devops-repos">Azure DevOps Repos</h2>
<p>Azure DevOps provides cherry-picking of a completed Pull Request (PR) or of a single commit by clicking a dedicated button. The process will create a new PR with the same fix.
<img src="azdo-cp.jpg" alt="Cherry-Pick Azure DevOps"></p>
<p>Moreover, a <a href="https://github.com/microsoft/azure-repos-pr-multi-cherry-pick">PR Multi-Cherry-Pick</a> is possible by means of an open source extension available on Azure DevOps Marketplace.</p>
<p>If Azure DevOps displays a warning about the cherry-picking not being performed automatically (see image below) this is related to conflicts generated during the merge, therefore it has to be performed locally.
<img src="azdo-cp-error.jpg" alt="Azure DevOps conflict errors"></p>
<p>Many development environments integrated with Git can perform this local operation by graphic user interface. Best candidates are Visual Studio or VS Code.</p>
<h1 id="git-cherry-pick---continue">Git cherry-pick &ndash;continue</h1>
<p>A few steps are needed to execute the following command <a href="https://git-scm.com/docs/git-cherry-pick">git cherry-pick</a>.</p>
<p>The first step is noting of the hash commit value to cherry-pick. If within a PR, the list of commits is available in the <strong>Commits</strong> tab in Azure Repos:
<img src="azdo-commits-tab.jpg" alt="Commit table on Azure DevOps"></p>
<p>Once you are on the branch (<code>git checkout &lt;branch-name&gt;</code>) that you want to commit, the following command must be executed <code>git cherry-pick &lt;commit&gt;</code>.</p>
<p>Knowing that the cherry-pick command shown in the previous example can generate another conflict, once the code is merged manually you need to run the <code>git cherry-pick --continue</code> to proceed. If you want to abort the process simply run git <code>cherry-pick --abort</code>.</p>
<p>All you need to do now is to perform <code>git push</code> to align the remote.</p></description></item><item><title>Flaky builds</title><link>http://www.getlatestversion.eu/2020/06/flaky-builds/</link><pubDate>Sun, 07 Jun 2020 20:00:00 +0100</pubDate><author>Giulio Vian</author><guid>http://www.getlatestversion.eu/2020/06/flaky-builds/</guid><description><p>The topic is huge and I do not have the room to go through all the details in one post, so I stick to a cursory view. I will sprinkle references to additional material so you can deep dive later.</p>
<h2 id="what-is-a-flaky-build">What is a &lsquo;flaky&rsquo; build?</h2>
<p>A &lsquo;flaky&rsquo; build is a build whose outcome is, in some ways, unpredictable. This is generally bad because you expect your CI to be algorithmic, deterministic, with no randomness. The worse consequence of an unpredictable build lies in the future: that day when you run the same build, maybe in the hurry of delivering a patch, and it fails unexpectedly. Some may think that is is just another piece of evidence for Murphy’s Law<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>, which applies to all time and space.
The unpredictability may be subtle; for example, the build always succeeds but its results are never quite the same.
In the following, I will review the most common flakiness scenarios.</p>
<h2 id="same-source-different-binaries">Same source, different binaries</h2>
<p>Have you tried comparing binaries built on two different machines? More often than not, they will be different! Even if you use the exact same source code version. Worse, even on the same machine, rebuilding the same code at a later time will produce different binaries.</p>
<p>My first example is a .NET 4.x assembly: the next picture shows the binary difference between files built from the same sources a few minutes apart.
<img src="dotnet4.8-hexdiff.png" alt="Binary diff of Exes">
Traditionally Microsoft tools embedded some kind of build timestamp in binaries for a number of reason, especially to help Windows&rsquo; Loader<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>.</p>
<p>I obtained a similar result with Maven building a JAR package, I used Beyond Compare to compare more easily.
<img src="mvn-diff1.png" alt="JAR content difference">
The difference is in the <code>pom.properties</code> manifest, which contains a time-stamp marking when the package was built.
<img src="mvn-diff2.png" alt="Maven manifest difference"></p>
<p>There are ways to fix these behaviors and join the camp of repeatable builds, like myself.</p>
<h3 id="why-should-i-care">Why should I care?</h3>
<p>The ability to compare binaries helps or is even required in many circumstances:</p>
<ul>
<li>troubleshooting, you can quickly compare a binary file generated on your machine with another from unknown source;</li>
<li>deployment, you can skip copying/deploying binary identical files;</li>
<li>patching, generating binary patches is easier;</li>
<li>auditing, you can demonstrate the version of code that is running in production matches source code;</li>
<li>security, the binary hasn&rsquo;t been tampered by an attacker.</li>
</ul>
<p>Hope this convince you that these goals are worth your time and effort. Auditing and security are becoming of greater importance these days.</p>
<h3 id="reproducible-builds">Reproducible builds</h3>
<p>Let&rsquo;s state a clear goal, we want reproducible builds that generate identical <em>main</em> artifacts when using the same version of source code.
What do I mean with <em>main</em> artifacts? That we allow for ancillary artifacts to change at every run; a trivial example is a build log or release note file. We forbid differences in executable binaries, data or configuration files, scripts and tools.
With this definition, a release note file displaying a build id or timestamp is fine.</p>
<p>To achieve this goal, you have to look at the tools&rsquo; manuals. I will detail a few common scenarios.
In addition to the following suggestions, you have to consider a proper archival and versioning of the toolchain in use to build the software. A newer version of compiler, SDK, build may produce different results.</p>
<h3 id="net-reproducible-builds">.NET Reproducible builds</h3>
<p>The Roslyn C# compiler offers the <a href="https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/compiler-options/deterministic-compiler-option"><code>-deterministic</code></a> flag a few years now, that is, from Visual Studio 2015. You turn on the option setting the <code>Deterministic</code> MSBuild property to <code>true</code> in the project file.</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"> <span style="color:#f92672">&lt;PropertyGroup&gt;</span>
<span style="color:#f92672">&lt;Deterministic&gt;</span>true<span style="color:#f92672">&lt;/Deterministic&gt;</span>
<span style="color:#f92672">&lt;PathMap&gt;</span>$(MSBuildThisFileDirectory)=\src<span style="color:#f92672">&lt;/PathMap&gt;</span>
<span style="color:#f92672">&lt;/PropertyGroup&gt;</span>
</code></pre></div><p>By the way, you do not need to rush editing you project files to set these properties. You can add a <code>Directory.Build.props</code> file<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup> at the root of your project and the setting will be applied to all projects under the same directory.
I have a good news: .NET Core SDK projects use the <code>deterministic</code> flag by default. The bad news is that it is not enough in some cases, but keep on reading.</p>
<p>In the above example, you have surely noted another property, <code>PathMap</code>, which matches the <a href="https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/compiler-options/pathmap-compiler-option"><code>-pathmap</code></a> compiler option. This option alters the paths embedded in executables and PDBs<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup> and is especially useful when building on a server. Typically each agent on the server has a different base directory, and the generated files embed the agent specific path. With <code>PathMap</code> you define a conventional directory (<code>\src</code> in the example) independent from the effective directory used for the build.
A useful resource is Jared Parsons&rsquo; post <a href="https://blog.paranoidcoding.com/2016/04/05/deterministic-builds-in-roslyn.html"><em>Deterministic builds in Roslyn</em></a>.</p>
<h3 id="maven-java-reproducible-builds">Maven (Java) Reproducible builds</h3>
<p>It is very easy to replace the timestamp in <code>pom.properties</code>, just add a <code>project.build.outputTimestamp</code> property</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"> <span style="color:#f92672">&lt;properties&gt;</span>
<span style="color:#f92672">&lt;project.build.outputTimestamp&gt;</span>2020-06-06T22:12:34Z<span style="color:#f92672">&lt;/project.build.outputTimestamp&gt;</span>
<span style="color:#f92672">&lt;/properties&gt;</span>
</code></pre></div><p>to the <code>pom.xml</code> file. This requires that you use a recent version (3.2 or later) of Maven plugins. You can find all the details in <a href="https://maven.apache.org/guides/mini/guide-reproducible-builds.html"><em>Configuring for Reproducible Builds</em></a>.</p>
<p>The <code>pom.properties</code> solves the simple issue but there are additional data, used during build, that can produce different outcomes. A more through approach use the <a href="https://zlika.github.io/reproducible-build-maven-plugin/">Reproducible Build Maven Plugin</a> which guarantees identical binaries given unchanged sources.</p>
<h3 id="other-languages">Other languages</h3>
<p>C/C++ developers can profitably study <a href="https://blog.conan.io/2019/09/02/Deterministic-builds-with-C-C++.html">An introduction to deterministic builds with C/C++</a>; this article explores the main issue on each major platform (Windows, Mac, and Linux).
I recommend the <a href="https://reproducible-builds.org/">https://reproducible-builds.org/</a> site for Linux and C/C++ information about Reproducible builds.
C/C++ reproducible can be complex to implement, caring for lots of detail, so I do not even dare to start in a short article like this one.</p>
<p>Go language has its quirks too. You may want to avoid the C compiler and linker, using the <code>CGO_ENABLED=0</code> setting, or embrace complexity. The other main issue is path strings embedded in binaries. The <a href="https://golang.org/cmd/go/#hdr-Compile_packages_and_dependencies"><code>-trimpath</code></a> flag, available with Go 1.13 and later, resolves using a conventional path (similar to C# <code>-pathmap</code>).</p>
<h2 id="same-definition-different-set-of-files">Same definition, different set of files</h2>
<p>Dependencies are the next source of troubles for build predictability. I see this as a separate topic from reproducible builds because the focus is not the individual file you coded, but the general outcome and files outside your direct control.</p>
<p>The issue caused by dependencies is simple to describe. Your project depends on, at least one, external file. The first time you build, you get a version of this external file, say v1.0. The next time you build, the version for the external file is different: it can be v1.0.1, v1.1, v2.0 or else.</p>
<p>We can set apart four reasons for non-predictable outcomes when it comes to dependencies:</p>
<ul>
<li>Loose specification &amp; newer packages versions available;</li>
<li>Newer versions available for indirect dependencies;</li>
<li>Loss of package source (direct or indirect);</li>
<li>Change in package manager and its algorithm.</li>
</ul>
<h3 id="loose-dependencies-specifications">Loose dependencies specifications</h3>
<p>Every modern library manager permits to specify a strict or a loose rule for a dependent file. The strict rule states a unique version for the library package. A loose rule defines a range of acceptable versions.
When the library author publishes a new version, what happens to your build depends on the rule. A strict rule will always retrieve the same version, a loose rule may force the package manager to download a different version. Let&rsquo;s see a practical example.</p>
<table>
<thead>
<tr>
<th align="center">Date</th>
<th align="center">Dependency Rule</th>
<th align="center">Available versions</th>
<th align="center">Version selected</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center">1 Jan</td>
<td align="center">= 1.0</td>
<td align="center">1.0</td>
<td align="center">1.0</td>
</tr>
<tr>
<td align="center">1 Jan</td>
<td align="center">1.0&lt;= and &lt;2.0</td>
<td align="center">1.0</td>
<td align="center">1.0</td>
</tr>
<tr>
<td align="center">1 Feb</td>
<td align="center">= 1.0</td>
<td align="center">1.0, 1.1</td>
<td align="center">1.0</td>
</tr>
<tr>
<td align="center">1 Feb</td>
<td align="center">1.0&lt;= and &lt;2.0</td>
<td align="center">1.0, 1.1</td>
<td align="center">1.1</td>
</tr>
<tr>
<td align="center">1 Mar</td>
<td align="center">= 1.0</td>
<td align="center">1.0, 1.1, 1.2, 2.0</td>
<td align="center">1.0</td>
</tr>
<tr>
<td align="center">1 Mar</td>
<td align="center">1.0&lt;= and &lt;2.0</td>
<td align="center">1.0, 1.1, 1.2, 2.0</td>
<td align="center">1.2</td>
</tr>
<tr>
<td align="center">1 Apr</td>
<td align="center">= 1.0</td>
<td align="center">1.2, 2.0, 2.1</td>
<td align="center">error</td>
</tr>
<tr>
<td align="center">1 Apr</td>
<td align="center">1.0&lt;= and &lt;2.0</td>
<td align="center">1.2, 2.0, 2.1</td>
<td align="center">1.2</td>
</tr>
</tbody>
</table>
<p>In this example, after the author publishes a new minor version, the build (<em>rectius</em> the package manager) use the new minor version for the looser rule, while the strict rule stick to version 1.0.
You may have noticed the 7th line, the build using the strict rule fails because the required version of the package is no more available. This is not the scenario we will discuss in <a href="#lost-source"><em>Lost source</em></a>, but the solution is the same.
Note that some package managers do not select the most recent version of a package (Maven is an example).</p>
<h3 id="indirect-dependencies">Indirect dependencies</h3>
<p>The <a href="#loose-dependencies-specifications">above</a> scenario is very simple because it analyses a direct dependency, i.e. the code you are building directly depends on the specified library version.
This is rarely the case in practice: the libraries you use depends on other libraries and so on. On average a project has a couple of hundred dependencies, direct or indirect <sup id="fnref:5"><a href="#fn:5" class="footnote-ref" role="doc-noteref">5</a></sup>.</p>
<p>While you can set very strict rules for your direct dependencies, you have no control on indirect dependencies. Only the library author can define the rules for the latter.
Let&rsquo;s see a simple example.</p>
<p><img src="diamond-dependency-graph.png" alt="Diamond dependency graph"></p>
<p>We say that <strong>Library C</strong> is a transitive dependency for App.
<strong>Library A</strong> requires at least version 2.0 of <strong>Library C</strong> while <strong>Library C</strong> requires a minimum 3.0 version. The package manager will pick version 3, which may have unexpected effects.</p>
<p>There is a solution to this and the previous issue and is generally referred as <em>locking</em>.</p>
<h3 id="fixing-loose-specifications-and-indirect-dependencies">Fixing loose specifications and indirect dependencies</h3>
<p>The most used library package managers offer a <em>locking</em> mechanism: the tool evaluates the whole dependency graph, determining the set of version that satisfy all dependency rules, save the result in a <em>lock</em> file, and use such result from now on without evaluating again. This mechanism is sometimes called <em>pinning</em> or <em>fixing</em> library versions, as you are taking a snapshot of dependency graph at a point in time.</p>
<p><strong>NuGet</strong> uses <a href="https://github.com/NuGet/Home/wiki/Enable-repeatable-package-restore-using-lock-file"><code>packages.lock.json</code> file</a> to control locking. The presence of the file, even empty, triggers the mechanism. You can have this file at the project level (e.g. same directory as <code>.csproj</code> project file) or at the root of the Git repository, depending how you prefer managing dependencies.</p>
<p><strong>Maven</strong> requires that you use the <a href="http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management"><code>&lt;dependencyManagement&gt;</code> stanza</a> to define the versions for direct and indirect dependencies. This mechanism is not 100% robust, so the community has devise some plugins to help manage the list and detect violations like <a href="https://github.com/vandmo/dependency-lock-maven-plugin">dependency-lock-maven-plugin</a>.</p>
<p><strong>Gradle</strong> offers a very sophisticated mechanism based on a set of <a href="https://docs.gradle.org/current/userguide/dependency_locking.html">lock files</a>. It is an opt-in mechanism that requires modification to the <code>build.gradle</code> script.</p>
<p><strong>npm</strong> uses <a href="https://docs.npmjs.com/configuring-npm/package-locks.html"><code>package-lock.json</code></a> file.</p>
<p>Python has a few tools to solve this problem. Notable is <a href="https://github.com/pypa/pipenv">pipenv</a> which uses <code>Pipfile.lock</code>. In a build script, you use the <code>pipenv sync</code> command which uses the <code>Pipfile.lock</code> exclusively.</p>
<h3 id="lock-files-and-pipelines">Lock files and pipelines</h3>
<p>Once the lock file is generated, commit it to source control. This guarantees that others developers, and build pipelines use the same lock file as you.</p>
<p>Be aware that the content of lock files is not written on stone. If you change the main dependency file (<code>.csproj</code>, <code>packages.config</code>, <code>packages.json</code>, etc.), the library package manager updates the lock at the first run, unless you block this behaviour. In a build must <strong>always</strong> force the use of a lock file, that is:</p>
<ul>
<li>set MSBuild <code>RestoreLockedMode</code> property to <code>true</code>;</li>
<li>use <code>dotnet restore</code> command with the <code>--locked-mode</code> option;</li>
<li>use <code>npm ci</code> <strong>not</strong> <code>npm install</code>;</li>
<li>use <code>pipenv sync</code> <strong>not</strong> <code>pipenv install</code>;</li>
</ul>
<h3 id="lost-source">Lost source</h3>
<p>We still have one scenario depicted in the table: how to cope with library versions that are no more available.
If you think that it never happens, please read the <a href="https://www.theregister.com/2016/03/23/npm_left_pad_chaos/">story of left-pad</a>.</p>
<p>The solution is to add some kind of buffer between your build and the external world, a storage for input artifacts. There quite a number of products, generally labelled as <em>Artifact Store</em> or <em>Artifact Manager</em>: <a href="https://jfrog.com/artifactory/">Artifactory</a>, <a href="https://www.myget.org/">MyGet</a>, <a href="https://www.sonatype.com/product-nexus-repository">Nexus</a>, <a href="https://inedo.com/proget">ProGet</a>, <a href="https://docs.microsoft.com/en-us/azure/devops/artifacts">Azure Artifacts</a> and so on.</p>
<p>These products require some tweaks on the build configuration so that the library package manager asks the Artifact Manager before going to the original source. NuGet uses the <a href=""><code>NuGet.Config</code> file</a> at machine or user level to search for sources; npm leverages the <a href="https://docs.npmjs.com/misc/config#registry"><code>registry</code></a> key in its configuration and so on.
The Artifact Manager returns the library package, if available locally, or goes to the original source and cache it locally.</p>
<p>Artifact Managers offer support to the most common library package formats (Maven, npm, NuGet, etc.) and a caching mechanism that shields you from a sudden disappearance of a library.</p>
<h3 id="tool-chain-issues">Tool chain issues</h3>
<p>The final piece is the library package manager tool itself. Matching algorithms, syntax, all might change on newer version. It is important to have copies of the executable, its configuration.
The golden rule is being able to recreate the exact build environment that was used at any point in the past. I discussed the value of Docker in details in <a href="http://blog.casavian.eu/2019/08/19/meta-pipelines-introduction/">this article</a>.</p>
<h2 id="same-tests-different-results">Same tests, different results</h2>
<p>Flaky tests are the third major cause of uncertainty in builds. Flaky tests randomly pass or fail despite the build being identical.</p>
<p>What kind of test can be so unpredictable? Tests that depends on external resources. For example, you deploy a web application and run a smoke test that checks if the site is up; if the web application host is slow to start, the remote call may time-out and the test fails. Unit tests, by definition, do not depends on external resources, so your focus should be on integration tests. Design integration tests for predictability; it is better that a whole suite of tests fails. e.g. by checking connectivity at entering the suite, than having a random test of the suite failing. Another scenario might be that tests have an implicit order, and the runner executes them in parallel, thus the random outcome.
Martin Fowler wrote a beautiful <a href="https://martinfowler.com/articles/nonDeterminism.html">essay on flaky tests</a> and I recommend studying it to gain a deep understanding why this happens and some preventions.</p>
<h3 id="how-to-harness">How to harness</h3>
<p>Practitioner have to face the reality of unforeseen problems. Luckily, modern build tools offer some relief.
You can automatically re-run failed tests and report which tests pass on the second (or more) attempt.</p>
<ul>
<li>in Azure DevOps you turn on this feature at the project level (<a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/test/flaky-test-management">https://docs.microsoft.com/en-us/azure/devops/pipelines/test/flaky-test-management</a>);</li>
<li>Jenkins has a plugin named <a href="https://plugins.jenkins.io/flaky-test-handler/"><em>Flaky Test Handler</em></a> somewhat limited;</li>
<li>GitLab suggests a <a href="https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#flaky-test">sophisticated process</a> that accounts for a number of failure scenarios, allowing tagging tests so the build pipeline knows about flakiness or other issues.</li>
</ul>
<h2 id="in-summary">In summary</h2>
<p>We explored cursory the topic of non-predictable builds. The root of randomness may lie in the compiler, the linker, the library manager, the dependency manager, in the test suites.
There are options, tools and techniques that help in minimise or even completely get rid of variation and have fully reproducible builds.
You should aim at repeatable, even reproducible builds: they are a defense weapon for the developer in the modern world.
A final word for the tool-chain. If you need to go back and rebuild years old versions, you must archive the exact set of tools you use for building and keep it safe. A different version for a compiler or a library manager can have a big impact on your ability to reproduce an old version.</p>
<hr>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>&ldquo;Anything that can go wrong will go wrong&rdquo;. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>Raymond Chen&rsquo;s post <a href="https://devblogs.microsoft.com/oldnewthing/20100318-00/?p=14563">What is DLL import binding?</a> and also <a href="https://devblogs.microsoft.com/oldnewthing/20180103-00/?p=97705">Why are the module timestamps in Windows 10 so nonsensical?</a>. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>MSBuild documentation <a href="https://docs.microsoft.com/en-us/visualstudio/msbuild/customize-your-build">Customize your build</a>. <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>Program database (PDB) is Microsoft&rsquo;s file format for storing debugging information. <a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>Source <a href="https://octoverse.github.com/#dependencies-overview">GitHub</a>: &ldquo;<em>203 package dependencies, on average, support every public and private repository with an enabled dependency graph</em>&rdquo; <a href="#fnref:5" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section></description></item><item><title>Top 10 Pipeline mistakes</title><link>http://www.getlatestversion.eu/2020/05/top-10-pipeline-mistakes/</link><pubDate>Sat, 23 May 2020 14:00:00 +0100</pubDate><author>Giulio Vian</author><guid>http://www.getlatestversion.eu/2020/05/top-10-pipeline-mistakes/</guid><description><p>Today I am going to start a series of posts detailing common issues or mistakes in a DevOps context.
I will try to refer to my experience and add some practical suggestion to identify and solve these issues.</p>
<p>Let&rsquo;s start with my list of top 10 CI/CD pipeline issues.</p>
<h2 id="the-list">The list</h2>
<blockquote>
<p><strong>NOTE</strong>:<br>
Published posts are in <strong>bold</strong>, the link for items in <em>italic</em> does not work.<br>
Last update: 10 June 2020</p>
</blockquote>
<ol>
<li><a href="../sloppy-secrets-handling"><em>Sloppy handling of Secrets</em></a> &ndash; leaking or hard-coding passwords, tokens or similar sensitive data;</li>
<li><a href="../untraceable-artifacts"><em>Untraceable artifacts</em></a> &ndash; when builds produce (or worse: deploy!) binaries of unknown source and version; this is a major red flag because it is cheap and easy to fix, but it is usually overlooked causing a major technical debt pile-up;</li>
<li><a href="../too-specific"><em>Too specific</em></a> &ndash; if your artifacts are not scrubbed from environment-specific dependencies, so they cannot be deployed to all environments;</li>
<li><a href="../what-quality"><em>What, quality?</em></a> &ndash; when your pipeline does not contain any check on quality, what do you expect as a result?;</li>
<li><a href="../bleeding-edge"><em>Bleeding edge</em></a> &ndash; using the latest and greatest technology is not always a wise choice;</li>
<li><a href="../galactic-builds"><em>Galactic Builds</em></a> &ndash; far-reaching builds that slow teams down instead of helping them;</li>
<li><a href="../flaky-builds"><strong>Flaky builds</strong></a> &ndash; builds generating unreproducible behaviours or random artefacts;</li>
<li><a href="../../06/flaky-builds"><strong>Flaky builds</strong></a> &ndash; builds generating unreproducible behaviours or random artefacts;</li>
<li><a href="../too-much-of-a-good-thing"><em>Too much of a good thing</em></a> &ndash; when you go too far to avoid the above mistakes, causing the fix to backfire;</li>
<li><a href="../implicit-assumption"><em>Implicit assumption</em></a> &ndash; any build that breaks when some undocumented environmental condition change;</li>
<li><a href="../untamed-plugins"><em>Untamed plugins</em></a> &ndash; similar to the previous one, it is the nightmare of people that manage your build environments, when the build software uses too many, or even conflicting plugins.</li>
</ol>
<p>The list is not really complete: there is one more to add.</p>
<h2 id="the-unforgivable-sin-having-no-pipeline">The unforgivable sin: having no pipeline</h2>
<p>This is the ultimate sin of any developer: having no automated process of any kind.<br>
At a minimum you can have a simple bash or PowerShell script to compile and publish your project.
With that it is going to be easy to integrate it in any of the most popular Continuous Integration tools: Jenkins, Azure DevOps, GitHub Actions, GitLab, TeamCity, etc.<br>
That script should check for dependencies and label the produced artifacts with a version number. This will be discussed in detail in future posts.<br>
If you do not need to automate the process for other people, at least, automate it for your future self!</p>
<p>See you soon with the next episode.</p></description></item><item><title>Code dependencies: Binary Composition is not only a mathematics calculation</title><link>http://www.getlatestversion.eu/2020/05/code-dependencies-binary-composition-is-not-only-a-mathematics-calculation/</link><pubDate>Sun, 17 May 2020 16:00:00 +0200</pubDate><author>Bartolomeo Lombardi</author><guid>http://www.getlatestversion.eu/2020/05/code-dependencies-binary-composition-is-not-only-a-mathematics-calculation/</guid><description><h1 id="introduction">Introduction</h1>
<p>Over the years, the growing code base of application software with multiple teams working on the same product, has lead to break up the solution into multiple solutions. This has been done trying to reduce the time required for the build and for their integration, to ease Integrated Development Environment in opening hundreds of projects, and other.
The main consequence of having multiple solutions is binary composition.</p>
<h1 id="what-is-binary-composition">What is binary composition?</h1>
<p>Binary composition occurs when one or more solutions reference the compiled binaries of another solution. Let suppose it is needed to make the binaries of Solution B available to Solution A before Solution A can build successfully.</p>
<p><img src="visual-studio-references.jpg" alt="image.png"></p>
<p>If you are forced to commit the code together with the binaries produced by build in the repository&hellip;Keep on reading :)</p>
<h1 id="how-can-we-distribute-binaries">How can we distribute binaries?</h1>
<p>We can make those binaries available in several ways.</p>
<h2 id="git-repository">Git repository</h2>
<p>A first possibility consists in committing them into a repository anytime a merge into the development/master branch by means of the Continuous Integration build pipeline is requested by a Pull Request. Of course, this increases the size of the repository introducing significant slowdown checkout times and performances. Imagine what could happen if teams work on different branches ending up using different versions of the same binaries creating merge conflicts.</p>
<h2 id="file-share">File share</h2>
<p>Another option consists in putting the binaries onto a file share. In this case, however, there is no index to find binaries quickly and there is no protection against overriding a version.</p>
<h2 id="package-management-with-azure-artifacts">Package Management with Azure Artifacts</h2>
<p>This should definitely be the most suitable solution because it allows putting binaries into NuGet (and other as npm, Maven, Python, and universal) packages making it possible for Solution A to reference these packages. Among the several advantages introduced by this methodology, in Continuous Integration Azure pipeline a NuGet published task can be added in order to make the update versioning procedure automatic and distributing it in a reliable way also.</p></description></item><item><title>Reducing the gap between operations and development using Azure DevOps</title><link>http://www.getlatestversion.eu/2020/05/reducing-the-gap-between-operations-and-development-using-azure-devops/</link><pubDate>Fri, 15 May 2020 11:30:00 +0200</pubDate><author>Alessandro Alpi</author><guid>http://www.getlatestversion.eu/2020/05/reducing-the-gap-between-operations-and-development-using-azure-devops/</guid><description><h1 id="intro">Intro</h1>
<p>As we all know, DevOps is culture. In a company that is going to adopt its practices and principles, everyone should be &ldquo;on the same side&rdquo;. Continuous improvement cannot be part of our work in IT if we wouldn&rsquo;t reduce the gap between development and operations. People like me, that worked in the 90s, know that dev and ops were always isolated in silos and this was &ldquo;the only&rdquo; way that everyone followed as a suggestion taken from the market leaders. Ticketing systems and extreme bureaucracy acted as a man-in-the-middle between those silos, transforming each organization in two people-unaware endpoints.</p>
<p>Speaking from a DevOps perspective, in such circumstances, a developer couldn&rsquo;t know anything about how to deploy, where and how is an environment configured. On the other hand, an operation guy couldn&rsquo;t get anything from a package in terms of what the application has been made for. Due to this gap we often see performance issues, underestimated hardware stuff and delayed releases. Last but not least, what about the lack of commitment and accountability of the employees working on the solution? In short, reducing such a gap is possible using a combination of DevOps culture and the right tools. We will see hereafter how my organization tries to do so using Azure DevOps.</p>
<h2 id="scenario">Scenario</h2>
<p>Let&rsquo;s get started with our team (DevTeam hereafter), which is working with agile methodologies, composed of ten developers and a PO. A quick note, we are using a process decision framework called <a href="https://www.disciplinedagileconsortium.org">Disciplined Agile</a>. Then, we have three operation professionals (OpsTeam from now on). Build and deploy pipelines already exist. Builds are hosted by <a href="https://azure.microsoft.com/it-it/services/devops/">Azure DevOps (cloud)</a> and deploys are managed by <a href="https://octopus.com/">Octopus Deploy</a>. Getting this, has been a difficult mission.</p>
<p>Everything is related to infrastructure in terms of servers, operative systems, virtual hosts, DNS, firewalls, security and so on, is the responsibility of our OpsTeam. As a result, they do many tasks for managing the environments. Here comes the problem. DevTeams used to create tasks in a dedicated backlog, but OpsTeam didn&rsquo;t. It&rsquo;s not just a matter of end-of-pipeline tasks, but also tasks for their daily basis work.</p>
<h2 id="our-solution">Our solution</h2>
<p>Modify the tool, adapting its shape in order to fit in with that real scenario. A piece of cake, when you&rsquo;re DevOps &ldquo;inside&rdquo;. How did we change Azure DevOps? Let&rsquo;s describe what we did in three parts:</p>
<h3 id="teams-in-azure-devops">Teams in Azure DevOps</h3>
<p>To create a team in Azure DevOps is really a piece of cake (according to the latest releases). Just navigate to the options and select <em>Teams</em>:</p>
<p><img src="post01-01-team-on-azuredevops.png" alt="image.png"></p>
<p>We can add many teams clicking on <em>New team</em>:</p>
<p><img src="post01-02-new-team.png" alt="image.png"></p>
<p>We can set the team&rsquo;s administrators, the permission set and an area under which every work item will be created. This area will be one of our best friends, especially when we will make our queries for gathering and analyzing the team&rsquo;s related data. Additionally, also the other teams&rsquo; members could create items with this area in order to make the OpsTeam aware of them.</p>
<p>Team&rsquo;s backlog
Now let&rsquo;s navigate to Backlogs:</p>
<p><img src="post01-03-backlogs.png" alt="image.png"></p>
<p>Good! The new backlog has been created. Going on it, we will see the team&rsquo;s drop-down as well as the one for iterations. Great feature!</p>
<p><img src="post01-04-sprints.png" alt="image.png"></p>
<p>Once created, we will see the teams&rsquo; drop-down:</p>
<p><img src="post01-05-team-switch.png" alt="image.png"></p>
<h3 id="work-items">Work items</h3>
<p>Now, let&rsquo;s create a new work item type. We call it Ops item. Navigate to Process customization page:</p>
<p><img src="post01-06-process.png" alt="image.png"></p>
<p>Before adding the new work item, we must ensure that the process is already a custom process, otherwise, all the edits will be blocked as shown in the following picture:</p>
<p><img src="post01-07-process-custom.png" alt="image.png"></p>
<p>We&rsquo;ve already created a SimplifiedScrum process. let&rsquo;s add our item now:</p>
<p><img src="post01-07-workitemtype.png" alt="image.png"></p>
<p>Now we are going to modify the fields of the new type. Each team should be able to understand all the item&rsquo;s properties. We will leave the item as is in this example. Then, we have to map the type to the backlog item list, since only the default work item types are shown initially. To do this, navigate to the Process customization page, Backlog Levels tab:</p>
<p><img src="post01-08-process-add-workitemtype-1.png" alt="image.png">
<img src="post01-09-process-add-workitemtype-2.png" alt="image.png"></p>
<p>As we can see, we can also set the default item for our requirements backlog. Moreover, every Sprint backlog, based on iterations, will enable us to add the new Ops item:</p>
<p><img src="post01-10-new-workitem.png" alt="image.png"></p>
<h3 id="wrapping-up">Wrapping up</h3>
<p>So, we&rsquo;ve got a backlog for the IT Operations team&rsquo;s members. Then, we&rsquo;ve related it to their Azure DevOps team. Additionally, we&rsquo;ve got a particular work item type (not mandatory, but really useful for querying it or adding it into dashboards) target of IT Operations&rsquo; job and a dedicated Area Path. We can make many relationships between items of both our backlogs. Here is an example of how an activity can be managed and organized (extension: <a href="https://marketplace.visualstudio.com/items?itemName=ms-devlabs.WorkItemVisualization">Work Item Visualize</a>):</p>
<p><img src="post01-11-workitem-visualize.png" alt="image.png"></p>
<p>As you can see, the Ops items are <em>Successor</em> of the &ldquo;development&rdquo; Product backlog items. Indeed, the Ops Items will be finished after the PBI, typically. Think about creating s DNS or a network path to let the production app work in production.</p>
<h2 id="conclusions">Conclusions</h2>
<p>With this solution, we&rsquo;re decoupling backlogs. Moreover, we&rsquo;re isolating the management maintaining the relationships between work items that reside on different boards. At the same time, we&rsquo;re making a strong synergy between Development and Operations.
Then, in a couple of clicks, we can switch teams and backlogs, using Azure DevOps Boards. We can track changes in both the departments, also for audit requirements. In my opinion, this helps the enterprise awareness and facilitates the continuous improvement of all the teams and any member&rsquo;s skill.</p></description></item><item><title>About</title><link>http://www.getlatestversion.eu/about/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>http://www.getlatestversion.eu/about/</guid><description><p><img src="http://www.getlatestversion.eu/images/320px-Flag_of_Europe.svg.png" alt="Europe flag"></p>
<h2 id="about-us">About Us</h2>
<p>GetLatestVersion.eu stems from GetLatestVersion.it, the successful Italian community specialised on DevOps, Application Lifecycle Management (ALM) and Software Development Life-Cycle (SDLC).</p>
<h2 id="what-we-do">What we do</h2>
<p>We publish posts and articles on DevOps, Agile, tools, design, implementation techniques.
Our <a href="https://www.youtube.com/GetLatestVersion">YouTube channel</a> hosts useful video of different kind: short introductions to a topic or tool, longer presentation on experiences and conference sessions.
In addition, we organise events in person and online on DevOps, Agile, ALM and SDLC topics.</p></description></item><item><title>Privacy</title><link>http://www.getlatestversion.eu/page/privacy/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>http://www.getlatestversion.eu/page/privacy/</guid><description><h2 id="list-of-cookies-used-in-this-site">List of Cookies used in this site</h2>
<p><strong>Google Analytics cookies</strong>: __utma, __utmb, __utmc, __utmv, __utmz, used to gather site statistic usage. You can read more info on Google Policies on: How Google uses data when you use our partners’ sites or apps and Safeguarding your Data.</p></description></item><item><title>Search</title><link>http://www.getlatestversion.eu/search/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>http://www.getlatestversion.eu/search/</guid><description><p class="error message js-hidden">You must have Javascript enabled to use this function.</p>
<p class="search-loading status message hidden">Loading search index…</p>
<div class="search-input hidden">
<form id="search-form" class="search-form" action="#" method="post" accept-charset="UTF-8" role="search">
<label for="query" class="visually-hidden">Search</label>
<input type="search" id="query" name="query" class="search-text" placeholder="Enter the terms you wish to search for." maxlength="128">
<button type="submit" name="submit" class="form-submit" >Search</button>
</form>
</div>
<div class="search-results"></div>
<template>
<article class="search-result list-view">
<header>
<h2 class="title title-submitted"><a href="#">Title here</a></h2>
<div class="submitted"><time class="created-date">Date here</time></div>
</header>
<div class="content">Summary here</div>
</article>
</template></description></item></channel></rss>