C# - XML - Compression

Question

I have a situation where I am generating a XML file to be submitted to a webservice, sometimes due to the amount of data it exceeds 30mb or 50mb.

I need to compress the file, using c#, .net framework 4.0, rather one of the nodes which has most of the data.. I have no idea how i am going to do it .. is it possible if someone can give me a example of how I can get this done please.

the xml file looks like this

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<HeaderTalk xmlns="http://www.w3schools.com/xml">
<EnvelopeVersion>2.0</EnvelopeVersion>
<Header>
<MessageDetails>
  <Class>CHAR-CLM</Class>      
</MessageDetails>
<SenderDetails>
  <IDAuthentication>
    <SenderID>aaaaaa</SenderID>
    <Authentication>
      <Method>MD5</Method>
      <Role>principal</Role>
      <Value>a3MweCsv60kkAgzEpXeCqQ==</Value>
    </Authentication>
  </IDAuthentication>
  <EmailAddress>[email protected]</EmailAddress>
</SenderDetails>
</Header>
<TalkDetails>
  <ChannelRouting>
   <Channel>
     <URI>1953</URI>
     <Product>My product</Product>
     <Version>2.0</Version>
    </Channel>
</ChannelRouting>
</TalkDetails>
<Body>
   <envelope xmlns="http://www.w3schools.com/xml/">       
     <PeriodEnd>2013-08-13</PeriodEnd>
     <IRmark Type="generic">zZrxvJ7JmMNaOyrMs9ZOaRuihkg=</IRmark>
     <Sender>Individual</Sender>
     <Report>
       <AuthOfficial>
          <OffName>
            <Fore>B</Fore>
            <Sur>M</Sur>
          </OffName>
          <Phone>0123412345</Phone>
        </AuthOfficial>
    <DefaultCurrency>GBP</DefaultCurrency>
    <Claim>
      <OrgName>B</OrgName>
      <ref>AB12345</ref>
      <Repayment>
        <Account>
          <Donor>
            <Fore>Barry</Fore>
           </Donor>
            <Total>7.00</Total>              
        </Account>           
        <Account>
          <Donor>
            <Fore>Anthony</Fore>               
          </Donor>             
          <Total>20.00</Total>
        </Account>                  
      </Repayment>
      </Claim>
      </Report>
   </envelope>
 </Body>
</HeaderTalk>

The CLAIM node is what I want to Compress , as it can be Millions of records that get included in the XML.

I am a novice in coding, it has taken a long time for me to get this XML generated, and been searching to find a way to compress the node but I just cant get it to work.. the Result needs to be exactly same till the DefaultCurrency node.. and then

 </AuthOfficial>
 <DefaultCurrency>GBP</DefaultCurrency>
 <CompressedPart Type="zip">UEsDBBQAAAAIAFt690K1</CompressedPart>
 </Report>
 </envelope>
 </Body>
 </HeaderTalk>

or

 </AuthOfficial>
 <DefaultCurrency>GBP</DefaultCurrency>
 <CompressedPart Type="gzip">UEsDBBQAAAAIAFt690K1</CompressedPart>
 </Report>
 </envelope>
 </Body>
 </HeaderTalk>

Thank you everyone in advance please. Or if someone can suggest where I can look and get some idea, on what I want to do.

to create the file , I am simple iterating through a Dataset and Writing the nodes using XmlElements and setting innertexts to my values ..

The Code I have used to write is .. //claim

XmlElement GovtSenderClaim = xmldoc.CreateElement("Claim");
XmlElement GovtSenderOrgname = xmldoc.CreateElement("OrgName");
GovtSenderOrgname.InnerText = Charity_name;
GovtSenderClaim.AppendChild(GovtSenderOrgname);

 XmlElement GovtSenderHMRCref = xmldoc.CreateElement("ref");
 GovtSenderHMRCref.InnerText = strref ;
 GovtSenderClaim.AppendChild(GovtSenderref);

 XmlElement GovtSenderRepayments = xmldoc.CreateElement("Repayment");
 while (reader.Read())
 {
  XmlElement GovtSenderAccount = xmldoc.CreateElement("Account");
  XmlElement GovtSenderDonor = xmldoc.CreateElement("Donor");

   XmlElement GovtSenderfore = xmldoc.CreateElement("Fore");
   GovtSenderfore.InnerText = reader["EmployeeName_first_name"].ToString();
   GovtSenderDonor.AppendChild(GovtSenderfore);

   GovtSenderAccount .AppendChild(GovtSenderDonor);

   XmlElement GovtSenderTotal = xmldoc.CreateElement("Total");
   GovtSenderTotal.InnerText = reader["Total"].ToString();

   GovtSenderAccount .AppendChild(GovtSenderTotal);

   GovtSenderRepayments.AppendChild(GovtSenderAccount );
 }
  GovtSenderClaim.AppendChild(GovtSenderRepayments);


   GovtSenderReport.AppendChild(GovtSenderClaim);

and the rest of the nodes to close..

xanatos · Accepted Answer

You can try this: it will compress only the nodes you select. It's a little different from what you asked, because it will replace the content of the element, leaving the element + its attributes as they were.

{
    // You are using a namespace! 
    XNamespace ns = "http://www.w3schools.com/xml/";

    var xml2 = XDocument.Parse(xml);

    // Compress
    {
        // Will compress all the XElement that are called Claim
        // You should probably select the XElement in a better way
        var nodes = from p in xml2.Descendants(ns + "Claim") select p;

        foreach (XElement el in nodes)
        {
            CompressElementContent(el);
        }
    }

    // Decompress
    {
        // Will decompress all the XElement that are called Claim
        // You should probably select the XElement in a better way
        var nodes = from p in xml2.Descendants(ns + "Claim") select p;

        foreach (XElement el in nodes)
        {
            DecompressElementContent(el);
        }
    }
}

public static void CompressElementContent(XElement el)
{
    string content;

    using (var reader = el.CreateReader())
    {
        reader.MoveToContent();
        content = reader.ReadInnerXml();
    }

    using (var ms = new MemoryStream())
    {
        using (DeflateStream defl = new DeflateStream(ms, CompressionMode.Compress))
        {
            // So that the BOM isn't written we use build manually the encoder.
            // See for example http://stackoverflow.com/a/2437780/613130
            // But note that false is implicit in the parameterless constructor
            using (StreamWriter sw = new StreamWriter(defl, new UTF8Encoding()))
            {
                sw.Write(content);
            }
        }

        string base64 = Convert.ToBase64String(ms.ToArray());

        el.ReplaceAll(new XText(base64));
    }
}

public static void DecompressElementContent(XElement el)
{
    var reader = el.CreateReader();
    reader.MoveToContent();
    var content = reader.ReadInnerXml();

    var bytes = Convert.FromBase64String(content);

    using (var ms = new MemoryStream(bytes))
    {
        using (DeflateStream defl = new DeflateStream(ms, CompressionMode.Decompress))
        {
            using (StreamReader sr = new StreamReader(defl, Encoding.UTF8))
            {
                el.ReplaceAll(ParseXmlFragment(sr));
            }
        }
    }
}

public static IEnumerable<XNode> ParseXmlFragment(StreamReader sr)
{
    var settings = new XmlReaderSettings
    {
        ConformanceLevel = ConformanceLevel.Fragment
    };

    using (var xmlReader = XmlReader.Create(sr, settings))
    {
        xmlReader.MoveToContent();

        while (xmlReader.ReadState != ReadState.EndOfFile)
        {
            yield return XNode.ReadFrom(xmlReader);
        }
    }
}

The decompress is quite complex, because it's difficult to replace the content of an Xml. In the end I split the content XNode by Xnode in ParseXmlFragment and ReplaceAll in DecompressElementContent.

As a sidenote, you have two similar-but-different namespaces in you XML: http://www.w3schools.com/xml and http://www.w3schools.com/xml/

This other variant will do exactly what you asked (so it will create a CompressedPart node) minus the attribute with the type of compression.

{
    XNamespace ns = "http://www.w3schools.com/xml/";

    var xml2 = XDocument.Parse(xml);

    // Compress
    {
        // Here the ToList() is necessary, because we will replace the selected elements
        var nodes = (from p in xml2.Descendants(ns + "Claim") select p).ToList();

        foreach (XElement el in nodes)
        {
            CompressElementContent(el);
        }
    }

    // Decompress
    {
        // Here the ToList() is necessary, because we will replace the selected elements
        var nodes = (from p in xml2.Descendants("CompressedPart") select p).ToList();

        foreach (XElement el in nodes)
        {
            DecompressElementContent(el);
        }
    }
}

public static void CompressElementContent(XElement el)
{
    string content = el.ToString();

    using (var ms = new MemoryStream())
    {
        using (DeflateStream defl = new DeflateStream(ms, CompressionMode.Compress))
        {
            // So that the BOM isn't written we use build manually the encoder.
            using (StreamWriter sw = new StreamWriter(defl, new UTF8Encoding()))
            {
                sw.Write(content);
            }
        }

        string base64 = Convert.ToBase64String(ms.ToArray());

        var newEl = new XElement("CompressedPart", new XText(base64));
        el.ReplaceWith(newEl);
    }
}

public static void DecompressElementContent(XElement el)
{
    var reader = el.CreateReader();
    reader.MoveToContent();
    var content = reader.ReadInnerXml();

    var bytes = Convert.FromBase64String(content);

    using (var ms = new MemoryStream(bytes))
    {
        using (DeflateStream defl = new DeflateStream(ms, CompressionMode.Decompress))
        {
            using (StreamReader sr = new StreamReader(defl, Encoding.UTF8))
            {
                var newEl = XElement.Parse(sr.ReadToEnd());
                el.ReplaceWith(newEl);
            }
        }
    }
}

C# - XML - Compression

Tags:

c#

.net

xml

compression

nodes

user2664502

1 Answers

xanatos

Recent Activity

Donate For Us

C# - XML - Compression

Tags:

c#

.net

xml

compression

nodes

user2664502

1 Answers

xanatos

Related questions

Recent Activity

Donate For Us