Sharing bash functions between scripts

I’ve spent most of the past week writing bash functions to help with automating build and deployment at work, and something I’ve never really paid much attention to is how to keep these .sh files structured. Trying to keep functions small and generic helps with reuse and saves me from having to write more code, and once I’ve tested something is working hopefully I won’t need to touch it again.

To quote the first item of the summarised Unix philosophy:

Write programs that do one thing and do it well.

It’s also made me think about another horrible mess of code that I’ve come to depend on, my .bash_aliases file.

This has truly become a dumping ground over the years to save me from having to remember almost everything I have done at one point or another. At what I feel is a whopping 1474 lines, my little black book of functions is something that I lean on a lot, but have never really taken any time to trim or polish, apart from some headings in the form of comments.

Creating some structure

So the first step in organising this mess is trying to group similar functions together. Near the very top of my .bash_aliases file I find this:

##########  Git aliases ##################
alias gs='git status'
alias ga='git add'
alias gd='git diff'
alias gc='git commit'
alias gca='git commit -a'
alias gb='git branch'
alias gbr='git branch --remote'
#########################################

So there are some aliases that I use because I’m lazy, and shaving off those extra keystrokes should save some wear and tear on my little digits. I’ve already noted in the file what this group applies to GIT so that is a great starting point.

Let’s create a new directory to store my newly structured files:
mkdir ~/scripts
There’s a new directory in my home directory called scripts. Next is to copy and paste the contents from .bash_aliases to a new file called git.sh within this folder.

Setting up the files

I could just copy and paste the contents in Emacs, but where’s the fun in that. I can use sed to grab the lines that I want and output them to this new file without having to leave the terminal.

sed -n '19,27p' ~/.bash_aliases > ~/scripts/git.sh

So here we are using sed to get the line between 19 and 27 from the file ~/.bash_aliases and outputting them to the file ~/scripts/git.sh easy. Bash files should have the following at the top ‘#!/bin/bash’, to tell the OS which interpreter should be used for the file. This way seems slightly hacky, but again, means I don’t need to leave the terminal.

sed -i '1s:^:#\!/bin/bash\n:' ~/scripts/git.sh

This sed command inserts the required text on the first command, followed by a new line, giving us what we want.

Next we need to delete the text from .bash_aliases. Again, let’s stay in the terminal, we’ve almost written the command we need already.

sed -n '19,27d' ~/.bash_aliases

So, sed lessons are over (almost), our files are set up and I’ll try and get this post back on track.

If we were to open a terminal at this point, the commands we’d receive a command not found message. This can easily be fixed, and I’m going to use sed again because why not?

sed -i '2s:^:\n\. ~/scripts/git\.sh\n\n:' ~/.bash_aliases

Here we are inserting a new line to our .bash_aliases file using the . command to source the contents of our git.sh file. Because .bash_aliases is run every time we open a bash shell, we now have contents of the ~/scripts/git.sh available at all times, whilst also keeping our git alias code separate.

I’ve got lots of functions that can be grouped together, so I envisage creating a azure-cli.sh file, docker.sh, code-generation.sh file and many others in the future.

Repeating this over the coming months should really help whip my .bash_aliases file into shape.

Conclusion

So this post devolved into a post about the Power of sed rather than what it was initially supposed to be, but the TD;DR is that:

1) massive files are bad
2) the . command will source or import code from another file
3) grouping similar functions into separate files helps make navigation of code easier
4) with the . command we can share code between files without repeating ourselves
5) we can use sed to manipulate text files succinctly, precisely with a couple of keystrokes

If you’ve made it this far, thanks for reading.

Running MSBuild on Windows in Git Bash

So I’m in the process of automating the build and publish of a .Net WinForms application with Squirrel in Windows.

The first step in achieving this is to get the project building programmatically, outside of Visual Studio.

Coming from a Linux background I prefer working with Bash over Powershell or Batch and Git Bash is my terminal of choice within Windows.

Discovering the tools we need

The application uses framework 4.6.1 and I wasn’t sure what version of the tools were needed. To find out what tools were already installed on my system I ran:

dir HKLM:\SOFTWARE\Microsoft\MSBuild\ToolsVersions\

This listed a couple of versions, 2.0, 3.5 and 4.0.30319. I decided to try to build the project with the latest installed tools using:

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe C:\path\to\solution.sln

Which caused the following error:
Project file contains ToolsVersion="15.0".

So now I know I need version 15.0 of the build tools.

One mystery solved.

Downloading and Installing the BuildTools

A quick search led me to this download page and after downloading the Build Tools for Visual Studio 2017, I ran the executable.

As I’m only currently trying to build a particular framework, under the Windows section I checked .NET desktop build tools and only checked the optional installation of Testing tools core features - Build Tools, as .NET Framework 4.6.1 SDK and targeting pack is included by default.

Now we’re cooking with gas.

So now we’ve got the correct build tools, let’s check we can build the project successfully. From a Powershell terminal, type:

C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe "C:\path\to\project\solution.sln"

Build Successful!

Working with Bash

Now we’ve proven that the tools work, we just need to make them accessible from Git Bash.

In order to be able to access MSBuild.exe from Git Bash we’ll need to add the path to MSBuild.exe used above to the $PATH environment variable. This is quick and easy.

PATH="$PATH:/c/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/MSBuild/15.0/Bin"

Some things to note from the above command:
– Forward slashes are used instead of backslashes for paths in Bash
– We can access the C:\ drive using just the drive letter in Git Bash

Now we can run the following command from our Git Bash window:

MSBuild.exe "/c/path/to/solution/solution.sln"

We should get identical output to when we ran the command initially in Powershell.

Adding switches to MSBuild

The final thing we will want to do is add some switches to the build command so we can Clean/Build/Rebuild and set the Configuration to Debug or Release.

In Powershell we could run the following:

MSBuild.exe "/c/path/to/solution/solution.sln" /t:Rebuild /p:Configuration=Release

Due to Bash using / for file paths, we need to escape any / characters with another /. So in Git Bash the above command becomes:

MSBuild.exe "/c/path/to/solution/solution.sln" //t:Rebuild //p:Configuration=Release

And there we have it, building a .NET solution with Git Bash and it only took a few minutes. By putting this in a script, we can now automate building of our WinForms solutions, ready for packaging and deployment with Squirrel.

Resources

In discovering the above, the below answers on StackOverflow helped steer me in the right direction.

  1. https://stackoverflow.com/questions/328017/path-to-msbuild
  2. https://stackoverflow.com/questions/17904199/automate-git-bisect-for-msbuild-nunit-net-windows-batch-commands-in-msysgit

Bonus Points

As described in the answer for this question, we could also script the install of the build tools, rather than using the GUI, because who in their right mind wants to use the GUI???

https://stackoverflow.com/questions/42696948/how-can-i-install-the-vs2017-version-of-msbuild-on-a-build-server-without-instal

That will be the next step so we can script the install of our build tools, making it effortless to configure a new build server.

Unable to access Puppet Learn from VM in VirtualBox

Puppet is an open-source automation platform designed to help automate software deployments and management of IT Infrastructure.

Helpfully, they have a VM which acts as a “a self-contained learning environment” so you can get to grips with what is possible.

After Downloading the VM from here, extracting the contents of the zip file and starting it in VirtualBox I was presented with the following screen.

Trying to access the IP listed here didn’t work.

The fix was to shutdown the VM, open the Settings in VirtualBox and update the Network > Adapter > “Attached To” value from “NAT” to “Bridged”.

After starting the VM again, the IP address had been updated to 192.168.1.119, which was accessible through a browser and I could continue with the guide.

UPDATE

Funnily enough, slightly further down the Puppet documentation it lists the above as a solution.

Puppet Adapter Bridged Configuration

At least if the documentation wasn’t so thorough, I’d have been able to easily sort this issue.

Using variables in strings in C#

Strings are great. Letters, numbers and symbols, all at the same time?

Awesome

Soon enough hard coded strings aren’t going to cut it and you’ll want them to be more dynamic.

Below are 3 methods of mixing variables with your strings:

The Concat Operator +

The + operator when used on string variables will allow you to join them together.

So given the variables firstName and lastName, we could join them together like so:

string firstName = "Frank";
string lastName = "Castle";
string fullName = firstName + lastName;
Console.WriteLine(fullName);

If you run this example your output will be “FrankCastle”.

Unfortunately this doesn’t give us exactly what we want.

To add a space to separate the names, update your code to the following:

string firstName = "Frank";
string lastName = "Castle";
string fullName = firstName + " " + lastName;
Console.WriteLine(fullName);

This will output “Frank Castle”. Exactly what we want.

As you can see, we’ve added an extra string in between the 2 variables with no characters, just a single space.

For those unfamiliar, Frank Castle has an alias, let’s add that in between his firstName and lastName for full effect.

string firstName = "Frank";
string lastName = "Castle";
string fullName = firstName + " 'The Punisher' " + lastName;
Console.WriteLine(fullName);

The output will be: “Frank ‘The Punisher’ Castle”.

Something to note is we’ve used (single quotes) rather than (double quotes) purposefully, so we don’t need to worry about “escaping” the double quotes. I’ll cover that in a future post.

Just to reiterate, we could have used the following to get our fullName variable, but we lose the flexibility and context that our variable names provide:

string fullName = "Frank" + " 'The Punisher' " + "Castle";

String.Format()

.Net comes with lots of handy functionality built in. The String.Format() function helps simplify string concatenation by allowing us to define how we want the output string to be structured, and then pass in the variables to use as argument after.

In order to achieve the output from our previous example we can use the following code:

string firstName = "Frank";
string lastName = "Castle";
string fullName = String.Format("{0} {1}", firstName, lastName);
Console.WriteLine(fullName);

The output again will be: “Frank Castle”.

To explain what is happening, the Format function is taking 3 arguments, the first is a string, the second and third are our variables. In the format string we are defining where we want our variables to be in the output string. {0} is our firstName argument, and {1} is our lastName argument. The reason we access them as 0 and 1 and not as 1 and 2 as you might expect is because in C# we use zero-based numbering. You will come across this when you use loops, arrays and lists. What it means to us is that our first item is located at position 0 and our second is at position 1.

Something nice is that we can now easily move our variables. If we wanted the output to read “Castle Frank”, we can do the following:

string firstName = "Frank";
string lastName = "Castle";
string fullName = String.Format("{1} {0}", firstName, lastName);
Console.WriteLine(fullName);

By swapping the index values our output string displays as expected.

We can also duplicate values by repeating their index. To get the output string “Frank Frank Castle” we do the following:

string firstName = "Frank";
string lastName = "Castle";
string fullName = String.Format("{0} {0} {1}", firstName, lastName);
Console.WriteLine(fullName);

Something else to be aware of is we can add in extra text as we please. To give Frank his full title, we just need:

string firstName = "Frank";
string lastName = "Castle";
string fullName = String.Format("{0} 'The Punisher' {1}", firstName, lastName);
Console.WriteLine(fullName);

We are still using single quotes so we don’t have to do any “escaping”.

String Interpolation

Saving the best ’til last, string interpolation. We get the simplicity offered by String.Format() and the clarity from the concatenation operator.

By simply adding a $ in front of our first double quote, we can use our variables by name. Our 3 example from above are reflected below:

string firstName = "Frank";
string lastName = "Castle";
string fullName = "{firstName} {lastName}";
Console.WriteLine(fullName);

Outputs: “Frank Castle”

string firstName = "Frank";
string lastName = "Castle";
string fullName = $"{firstName} {firstName} {lastName}";
Console.WriteLine(fullName);

Outputs: “Frank Frank Castle”

string firstName = "Frank";
string lastName = "Castle";
string fullName = $"{firstName} 'The Punisher' {lastName}";
Console.WriteLine(fullName);

Outputs: “Frank ‘The Punisher’ Castle”

Conclusion

By now this code should make sense and you should understand each method and be able to find one which suits your situation.

In practice I only use string interpolation now, but it is good to know what options are available, and forget the ones you don’t need.

An honourable mention is the StringBuilder class, which offers increased performance when dealing with lots of concatenations, and is probably worth another post in itself.

That wraps up the first post in my series for beginners, hopefully it has been useful.

More Reading

If you’d like some more information on how the + operator works see this StackOverflow answer for more details.