← Home

dotnet Core on Linux - Restoring Private NuGet Packages

Over the last few weeks, I’ve spent a lot of time starting up some new projects using dotnet Core hosted on Linux Containers.

It’s been a combination of fun and immense frustration.

Stuff that’s worked well:

  • Microsoft’s dotnet Docker images (https://hub.docker.com/r/microsoft/dotnet/) are really easy to use and work well
  • Visual Studio Code with OmniSharp - pretty good, but no autocomplete for Razor views
  • CircleCI’s ability to SSH into failing builds

Stuff that doesn’t work well:

  • dotnet compilation is really slow
  • dotnet Core itself project is moving so fast that information you read is either bang up-to-date, old, or you don’t know
  • XUnit - Lack of code coverage, such a huge omission from dotnet core
  • Nuget - Doesn’t run on Linux or OSX for a start, not easy to use
  • MSBuild - It’s back (dotnet had a simpler project.json structure for a while), and while it’s had a facelift, the full horror is still there underneath, just enable detailed verbosity

NuGet is the way to share redistributable software packages (.dlls) between dotnet projects, like NPM for Node.js or Maven for Java world.

Most .Net teams seem to use TeamCity as a build server which handily includes a built-in private NuGet server for your closed source packages. This works well inside a corporate network, but is a bit of a pain when you want third-party access or when the build server fails and you have to fix it.

Since we’re targeting Linux containers, we’ve been using CircleCI, which makes it easy to set up dotnet Core builds. Unfortunately, it doesn’t have NuGet support built-in.

AppVeyor (another cloud-based CI tool) doesn’t support Linux at all, but it does provide a private NuGet repository, so I was able to put our shared assemblies into that and access them from CircleCI.

Unfortunately, then using them on Linux isn’t as easy as you’d hope. It starts out well with public repositories, because the dotnet command ships with NuGet support. dotnet add package newtonsoft.json will get you a package added to your .csproj file. Great!

Unfortunately, there’s currently no support for adding packages from private repositories in the dotnet command. So you have to use the NuGet tool to add extra NuGet sources and store the credentials - see issue [0]

So, no problem, just run it… Not so fast, there’s no native OSX / Linux nuget so you have to install Mono. As you’d expect, the Microsoft dotnet Docker containers don’t have mono installed, so I needed some way to sort this out.

My first attempt to workaround was to simply install Mono on the build container to then be able to run nuget, but it took ages.

So I wrote a bash script which uses CURL to download the packages directly and put them in the local filesystem cache, but it didn’t fool NuGet, which stubbornly insisted on attempting to connect to the Web (a problem in itself)…

docker-restore:
	bash ./scripts/download-package.sh my.nuget.package 1.0.26 ${NUGET_USERNAME} ${NUGET_PASSWORD} https://ci.appveyor.com/nuget/xxx
	docker run -v `pwd`:/sln -v /home/ubuntu/.nuget/packages/my.nuget.package:/root/.nuget/packages/my.nuget.package --workdir /sln microsoft/aspnetcore-build:latest bash scripts/docker-restore.sh

After a bit of playing around, I realised that all that NuGet.exe was doing was creating a NuGet.config file in the solution directory, so I wrote a Python script to create the config file, based on environment variables which contain credentials and this did actually work. Phew, disaster averted.

from xml.sax.saxutils import quoteattr
import os

f = open('NuGet.config', 'w')
f.write('<?xml version="1.0" encoding="utf-8"?>\n')
f.write('<configuration>\n')
f.write('  <packageSources>\n')
f.write('    <add key="sourcename" value="https://ci.appveyor.com/nuget/xxx" />\n')
f.write('  </packageSources>\n')
f.write('  <packageSourceCredentials>\n')
f.write('    <sourcename>\n')
f.write('      <add key="Username" value=' + quoteattr(os.environ['APPVEYOR_NUGET_USERNAME']) + ' />\n')
f.write('      <add key="ClearTextPassword" value=' + quoteattr(os.environ['APPVEYOR_NUGET_PASSWORD']) + ' />\n')
f.write('    </sourcename>\n')
f.write('  </packageSourceCredentials>\n')
f.write('</configuration>\n')
f.close()

This got my private packages restored without the time cost of installing Mono.

Since this wasted a full day of time, I thought about getting rid of the private NuGet server complexity and just uploading my built DLLs into an S3 bucket, or publishing them as Github releases, which is when I found that there is no way to add a reference to a DLL in a dotnet core project - see [1]

To make things worse, by this point, something else had broken.

"/sln/xxx.sln" (Publish target) (1) ->
"/sln/src/xxx.Web/xxx.Web.csproj" (Publish target) (6) ->
  (CoreCompile target) ->
     Controllers/AccountController.cs(40,34): error CS1705: Assembly 'xxx.Api.ClientLibrary' with identity 'xxx.Api.ClientLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' uses 'System.Net.Http, Version=4.1.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' which has a higher version than referenced assembly 'System.Net.Http' with identity 'System.Net.Http, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' [/sln/src/xxx.Web/xxx.Web.csproj]

It worked locally on OSX, and when I ran the build inside a fresh container, it also worked fine.

Also, the Api.ClientLibrary had one dependency on Newtonsoft.Json and these issues usually only arise when:

  • A project is referencing version x of an external assembly, and one of the project’s dependencies is itself referencing version y.
  • Dependency a of a project is referencing version x of external assembly q, and dependency b is referencing version y.

The solution is often just to update all the packages to the latest version, but I just couldn’t find a way to do it in dotnet core, so I raised an issue on that one [2]

I ended up updating the .csproj files manually.

Even after I spent time doing that, it wasn’t the issue. What had happened is that the build server was configured to use the :latest docker image and Microsoft had updated the image. I couldn’t see it locally until I did a docker pull.

The solution to the problem was to migrate the library from targeting netcoreapp1.1 to netstandard1.4, since I read that executables should target a framework and libraries target a standard.

I hate spending the time my clients are paying for on unproductive stuff like this. I thought that dotnet core experience would be slicker given that:

  • It’s v1.1, not a beta
  • Microsoft have the standard library in C# and a rich source of copy / paste material from the Mono project (12 years old!)
  • The Rosylyn compiler has been available to the community from 2011
  • There’s a huge development community and tools ecosystem - e.g. MonoDevelop / Xamarin Studio, JetBrains and the Visual Studio team

I wonder how the teams are formed. Microsoft seem to have built teams up around particular areas like NuGet, MSBuild (why, why, why) and CoreCLR but I’m not sure if anyone is really concentrating on the end-to-end the whole experience.

I can’t help but compare this experience to my experience of using Go, which has been really positive. The build process is simple and compilation is so fast that there’s no need to store ready-made output binaries somewhere. All of the complexity of accessing remotely authenticated repositories to pull code is the responsibility of the source control system, outside the platform, which keeps it really simple.

C# is a fine language and the MVC stuff is well-featured, but I think the tooling is spoiling the experience.