Need help to deserialise yaml

[MichaelJCompton]

Hi,

Sorry this is such a basic question. I've been using the other dotnet k8s library and would like to switch to yours, but I'm failing at the first hurdle.

I can't work out how to read in my .yaml files (was doing with await Yaml.LoadFromFileAsync<V1Service>(...) in KubernetesClient, which was working fine).

I'm trying things like

Deserializer deserializer = new DeserializerBuilder()
  .Build();

... deserializer.Deserialize<ServiceV1>(File.ReadAllText(...)); 

but always get exceptions like: "Property 'selector' not found on type 'KubeClient.Models.ServiceSpecV1'."

What's the magic salt I need to read in a .yaml?

[tintoy]

Hi - I’m pretty sure this should work (PSkubectl does it). I’ll have a look in an hour or so when I’m back in front of my computer :)

[tintoy]

Try this:

Deserializer deserializer = new DeserializerBuilder()
    .IgnoreUnmatchedProperties() // If not specified, expects all properties on the model to be present in the YAML
    .Build();

[MichaelJCompton]

Thanks for the quick response!

That certainly doesn't throw an exception, but it also doesn't read the whole model.

I tried with the example service here

Deserializer deserializer = new DeserializerBuilder()
  .IgnoreUnmatchedProperties()
  .Build();

var example = deserializer.Deserialize<ServiceV1>(@"
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: MyApp
  ports:
  - protocol: TCP
    port: 80
    targetPort: 9376");

And it read in, but selector and ports both came out empty.

I've got this in csproj

    <PackageReference Include="KubeClient" Version="2.2.6"/>
    <PackageReference Include="KubeClient.Extensions.DependencyInjection" Version="2.2.6"/>

[MichaelJCompton]

I expect it's something like https://stackoverflow.com/questions/48677081/in-yamldotnet-how-can-i-deserialize-a-getter-only-property

If I deserialise like

var example = deserializer.Deserialize(new StringReader("..."));

it deserialises to key-value pairs just fine. If I deserialise to a dynamic, it works fine. If I deserialise as per ^^, turn it into JSON, and then JsonConvert.DeserializeObject<ServiceV1>(json); with json.net, it works fine.

Looks like PSkubectl does deserializer.Deserialize<Dictionary<string, object>>(yaml);.

I'm going to assume it ain't gunna work on those get-only ones and I'll just go via JSON. I'm happy for you to close this unless you want to dig deeper.

[tintoy]

I suspect those properties may be read-only. JSON.NET can handle that, but YamlDotNet can’t.

[tintoy]

Will take a look first thing tomorrow :)

[tintoy]

Oops - reading comprehension fail :) serves me right for reading on my phone. Will sit down with my laptop shortly and figure this out...

[tintoy]

I'll add a static helper for deserialising models from YAML (it'll go from YAML to JSON to models). Slightly inefficient, but it's not like it'll be used for HUGE YAML documents :)

[MichaelJCompton]

Don't know if this is best way, but I had just thrown this in to get me moving

        private T Deserialize<T>(string yaml) =>
            JsonConvert.DeserializeObject<T>(
                JsonConvert.SerializeObject(
                    yamlDeserialiser.Deserialize(new StringReader(yaml))));

[tintoy]

Thanks!

[tintoy]

Ok, will publish this change once I've had time to test the ServiceV1 changes (which are more extensive).

If you need it in the meanwhile, this is what I'll be adding:

using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System;
using System.IO;
using System.Text;
using YamlDotNet.Serialization;

using KubeResourceClient = KubeClient.ResourceClients.KubeResourceClient;

namespace KubeClient.Models
{
    /// <summary>
    ///     Helper methods for YAML serialisation / deserialisation of models.
    /// </summary>
    public static class Yaml
    {
        /// <summary>
        ///     The buffer size used when reading from / writing to streams.
        /// </summary>
        const int StreamBufferSize = 1024;

        /// <summary>
        ///     The singleton YAML <see cref="Deserializer"/> used by static methods on <see cref="Yaml"/>.
        /// </summary>
        static readonly Deserializer YamlDeserialiser = new DeserializerBuilder().IgnoreUnmatchedProperties().Build();

        /// <summary>
        ///     The singleton (JSON-compatible) YAML <see cref="Serializer"/> used by static methods on <see cref="Yaml"/>.
        /// </summary>
        static readonly Serializer YamlJsonSerialiser = new SerializerBuilder().JsonCompatible().Build();
        
        /// <summary>
        ///     The singleton <see cref="JsonSerializer"/> used by static methods on <see cref="Yaml"/>.
        /// </summary>
        static readonly JsonSerializer JsonSerializer = Newtonsoft.Json.JsonSerializer.Create(KubeResourceClient.SerializerSettings);

        /// <summary>
        ///     Convert YAML to JSON.
        /// </summary>
        /// <param name="yaml">
        ///     A <see cref="TextReader"/> containing the YAML to convert.
        /// </param>
        /// <returns>
        ///     A <see cref="JToken"/> representing the converted JSON.
        /// </returns>
        public static JToken ToJson(TextReader yaml)
        {
            if (yaml == null)
                throw new ArgumentNullException(nameof(yaml));
            
            object deserialisedYaml = YamlDeserialiser.Deserialize(yaml);
            
            using (MemoryStream buffer = new MemoryStream())
            {
                using (TextWriter jsonWriter = CreateTextWriter(buffer))
                {
                    YamlJsonSerialiser.Serialize(jsonWriter, deserialisedYaml);
                    jsonWriter.Flush();
                }

                buffer.Seek(0, SeekOrigin.Begin);

                using (JsonReader jsonReader = CreateJsonReader(buffer))
                {
                    return JToken.Load(jsonReader);
                }
            }
        }

        /// <summary>
        ///     Convert YAML to JSON.
        /// </summary>
        /// <param name="yaml">
        ///     A string containing the YAML to convert.
        /// </param>
        /// <param name="formatting">
        ///     A <see cref="Formatting"/> value indicating whether the converted JSON should be formatted (i.e. indented).
        /// </param>
        /// <returns>
        ///     A string containing the converted JSON.
        /// </returns>
        public static string ToJson(string yaml, Formatting formatting = Formatting.None)
        {
            if (yaml == null)
                throw new ArgumentNullException(nameof(yaml));
            
            object deserialisedYaml = YamlDeserialiser.Deserialize(
                new StringReader(yaml)
            );
            
            using (MemoryStream buffer = new MemoryStream())
            {
                using (TextWriter jsonWriter = CreateTextWriter(buffer))
                {
                    YamlJsonSerialiser.Serialize(jsonWriter, deserialisedYaml);
                    jsonWriter.Flush();
                }

                buffer.Seek(0, SeekOrigin.Begin);

                using (JsonReader jsonReader = CreateJsonReader(buffer))
                {
                    return JToken.Load(jsonReader).ToString(formatting);
                }
            }
        }

        /// <summary>
        ///     Deserialise a <typeparamref name="TModel"/> from YAML.
        /// </summary>
        /// <typeparam name="TModel">
        ///     The type of model to deserialise.
        /// </typeparam>
        /// <param name="yaml">
        ///     A <see cref="TextReader"/> containing the YAML.
        /// </param>
        /// <returns>
        ///     The deserialised <typeparamref name="TModel"/>.
        /// </returns>
        /// <remarks>
        ///     Delegates the actual deserialisation to JSON.NET, after converting the YAML to JSON.
        /// 
        ///     Not particularly efficient, but safe and reliable.
        /// </remarks>
        public static TModel Deserialize<TModel>(TextReader yaml)
        {
            if (yaml == null)
                throw new ArgumentNullException(nameof(yaml));

            object deserialisedYaml = YamlDeserialiser.Deserialize(yaml);

            using (MemoryStream buffer = new MemoryStream())
            {
                using (JsonWriter jsonWriter = CreateJsonWriter(buffer))
                {
                    JsonSerializer.Serialize(jsonWriter, deserialisedYaml);
                    jsonWriter.Flush();
                }

                buffer.Seek(0, SeekOrigin.Begin);

                using (JsonReader jsonReader = CreateJsonReader(buffer))
                {
                    return JsonSerializer.Deserialize<TModel>(jsonReader);
                }
            }
        }

        /// <summary>
        ///     Create a <see cref="JsonReader"/> that reads from the specified stream.
        /// </summary>
        /// <param name="stream">
        ///     The target stream.
        /// </param>
        /// <returns>
        ///     The new <see cref="JsonReader"/>.
        /// </returns>
        static JsonReader CreateJsonReader(Stream stream)
        {
            if (stream == null)
                throw new ArgumentNullException(nameof(stream));
            
            TextReader textReader = null;
            JsonReader jsonReader = null;

            try
            {
                textReader = CreateTextReader(stream);
                jsonReader = new JsonTextReader(textReader);
            }
            catch (Exception)
            {
                using (jsonReader)
                using (textReader)
                {
                    throw;
                }
            }

            return jsonReader;
        }

        /// <summary>
        ///     Create a <see cref="JsonWriter"/> that writes to the specified stream.
        /// </summary>
        /// <param name="stream">
        ///     The target stream.
        /// </param>
        /// <returns>
        ///     The new <see cref="JsonWriter"/>.
        /// </returns>
        static JsonWriter CreateJsonWriter(Stream stream)
        {
            if (stream == null)
                throw new ArgumentNullException(nameof(stream));
            
            TextWriter textWriter = null;
            JsonWriter jsonWriter = null;

            try
            {
                textWriter = CreateTextWriter(stream);
                jsonWriter = new JsonTextWriter(textWriter);
            }
            catch (Exception)
            {
                using (jsonWriter)
                using (textWriter)
                {
                    throw;
                }
            }

            return jsonWriter;
        }

        /// <summary>
        ///     Create a <see cref="TextReader"/> that reads from the specified stream.
        /// </summary>
        /// <param name="stream">
        ///     The target stream.
        /// </param>
        /// <returns>
        ///     The new <see cref="TextReader"/>.
        /// </returns>
        static TextReader CreateTextReader(Stream stream)
        {
            if (stream == null)
                throw new ArgumentNullException(nameof(stream));
            
            return new StreamReader(stream, Encoding.UTF8, detectEncodingFromByteOrderMarks: false, bufferSize: StreamBufferSize, leaveOpen: true);
        }

        /// <summary>
        ///     Create a <see cref="TextWriter"/> that writes to the specified stream.
        /// </summary>
        /// <param name="stream">
        ///     The target stream.
        /// </param>
        /// <returns>
        ///     The new <see cref="TextWriter"/>.
        /// </returns>
        static TextWriter CreateTextWriter(Stream stream)
        {
            if (stream == null)
                throw new ArgumentNullException(nameof(stream));
            
            return new StreamWriter(stream, Encoding.UTF8, bufferSize: StreamBufferSize, leaveOpen: true);
        }
    }
}

[tintoy]

Hi - sorry for taking so long to update this issue (I've been stuck on other things and this kept dropping off my radar until now).

The currently published version of the KubeClient package has the static Yaml helper class. Hope that helps :)

[MichaelJCompton]

Hi, no probs. I've actually moved on, so I don't rely on the fix anymore.

I really like the lib while I was using it.