Read Process StandardOutput before New Line Received

Question

I'm trying to do something that appears to be out of scope for the System.Diagnostics.Process object. Acceptable answers can propose a different approach as long as it uses .net 4.5/c#5.

My program is calling gdalwarp.exe to perform a long running processes on large tiff files. Galwarp.exe outputs in this format.

Creating output file that is 6014P x 4988L.  
Processing input file [FileName].tiff. 
Using band 4 of source image as alpha. 
Using band 4 of destination image as alpha.
0...10...20...30...40...50...60...70...80...90...100 - done.

The last line streams in slowly to indicate progress. I would like to read that line whenever it changes so I can move a progress bar keeping the user informed.

First I attempted to read Process.StandardOutput but it delivers no data until the entire process is complete. Second I tried to call Process.BeginOutputReadLine() and connect the event Process.OutputDataReceived but it only fires when a line is complete.

Here is the call to Execute GDalWarp.exe.

    public static void ResizeTiff(string SourceFile, string DestinationFile, float ResolutionWidth, float ResolutionHeight, Guid ProcessId)
    {
        var directory = GDalBin;
        var exe = Path.Combine(directory, "gdalwarp.exe");
        var args = " -ts " + ResolutionWidth + " " + ResolutionHeight + " -r cubic -co \"TFW=YES\" \"" + SourceFile + "\" \"" + DestinationFile + "\"";
        ExecuteProcess(exe, args, null, directory, 0, null, true, true, 0);
    }

Here is my working code in a static function that only reads the output after the process exits.

public static string ExecuteProcess(string FilePath, string Args, string Input, string WorkingDir, int WaitTime = 0, Dictionary<string, string> EnviroVariables = null, bool Trace = false, bool ThrowError = true, int ValidExitCode = 0)
{
    var processInfo =
        "FilePath: " + FilePath + "\n" +
        (WaitTime > 0 ? "WaitTime: " + WaitTime.ToString() + " ms\n" : "") +
        (!string.IsNullOrEmpty(Args) ? "Args: " + Args + "\n" : "") +
        (!string.IsNullOrEmpty(Input) ? "Input: " + Input + "\n" : "") +
        (!string.IsNullOrEmpty(WorkingDir) ? "WorkingDir: " + WorkingDir + "\n" : "") +
        (EnviroVariables != null && EnviroVariables.Count > 0 ? "Environment Variables: " + string.Join(", ", EnviroVariables.Select(a => a.Key + "=" + a.Value)) + "\n" : "");

    if(Trace)
        Log.Debug("Running external process with the following parameters:\n" + processInfo);

    var startInfo = (string.IsNullOrEmpty(Args))
        ? new ProcessStartInfo(FilePath)
        : new ProcessStartInfo(FilePath, Args);

    if (!string.IsNullOrEmpty(WorkingDir))
        startInfo.WorkingDirectory = WorkingDir;

    startInfo.UseShellExecute = false;
    startInfo.RedirectStandardOutput = true;
    startInfo.RedirectStandardError = true;
    startInfo.CreateNoWindow = true;

    if (!string.IsNullOrEmpty(Input))
        startInfo.RedirectStandardInput = true;

    if (EnviroVariables != null)
        foreach (KeyValuePair<String, String> entry in EnviroVariables)
            startInfo.EnvironmentVariables.Add(entry.Key, entry.Value);

    var process = new Process();
    process.StartInfo = startInfo;
    if (process.Start())
    {
        if (Input != null && Input != "")
        {
            process.StandardInput.Write(Input);
            process.StandardInput.Close();
        }
        var standardError = "";
        var standardOutput = "";
        int exitCode = 0;

        var errorReadThread = new Thread(new ThreadStart(() => { standardError = process.StandardError.ReadToEnd(); }));
        var outputReadTread = new Thread(new ThreadStart(() => { standardOutput = process.StandardOutput.ReadToEnd(); }));
        errorReadThread.Start();
        outputReadTread.Start();
        var sw = Stopwatch.StartNew();
        bool timedOut = false;
        try
        {
            while (errorReadThread.IsAlive || outputReadTread.IsAlive)
            {
                Thread.Sleep(50);
                if (WaitTime > 0 && sw.ElapsedMilliseconds > WaitTime)
                {
                    if (errorReadThread.IsAlive) errorReadThread.Abort();
                    if (outputReadTread.IsAlive) outputReadTread.Abort();
                    timedOut = true;
                    break;
                }
            }

            if (!process.HasExited)
                process.Kill();

            if (timedOut)
                throw new TimeoutException("Timeout occurred during execution of an external process.\n" + processInfo + "Standard Output: " + standardOutput + "\nStandard Error: " + standardError);

            exitCode = process.ExitCode;
        }
        finally
        {
            sw.Stop();
            process.Close();
            process.Dispose();
        }

        if (ThrowError && exitCode != ValidExitCode)
            throw new Exception("An error was returned from the execution of an external process.\n" + processInfo + "Exit Code: " + exitCode + "\nStandard Output: " + standardOutput + "\nStandard Error: " + standardError);

        if (Trace)
            Log.Debug("Process Exited with the following values:\nExit Code: {0}\nStandard Output: {1}\nStandard Error: {2}", exitCode, standardOutput, standardError);

        return standardOutput;
    }
    else return null;
}

Can anyone help me read this output in realtime?

Answer 1

Here is a solution of your problem but a little bit tricky, because gdalwarp.exe is blocking standart output, you can redirect its output to a file and read changes on it. It was possible to use FileSystemWatcher to detect changes in file but it is not reliable enough sometimes. A simple polling method of file size changes is used below in the outputReadThread if OutputCallback is not null.

Here is a call to ExecuteProcess with callback to receive process output immediately.

    public static void ResizeTiff(string SourceFile, string DestinationFile, float ResolutionWidth, float ResolutionHeight, Guid ProcessId)
    {
        var directory = GDalBin;
        var exe = Path.Combine(directory, "gdalwarp.exe");
        var args = " -ts " + ResolutionWidth + " " + ResolutionHeight + " -r cubic -co \"TFW=YES\" \"" + SourceFile + "\" \"" + DestinationFile + "\"";
        float progress = 0;
        Action<string, string> callback = delegate(string fullOutput, string newOutput)
        {
            float value;
            if (float.TryParse(newOutput, out value))
                progress = value;
            else if (newOutput == ".")
                progress += 2.5f;
            else if (newOutput.StartsWith("100"))
                progress = 100;
        };
        ExecuteProcess(exe, args, null, directory, 0, null, true, true, 0, callback);
    }

Here is a function to call any process and receive the results as they happen.

    public static string ExecuteProcess(string FilePath, string Args, string Input, string WorkingDir, int WaitTime = 0, Dictionary<string, string> EnviroVariables = null, bool Trace = false, bool ThrowError = true, int ValidExitCode = 0, Action<string, string> OutputChangedCallback = null)
    {
        var processInfo =
            "FilePath: " + FilePath + "\n" +
            (WaitTime > 0 ? "WaitTime: " + WaitTime.ToString() + " ms\n" : "") +
            (!string.IsNullOrEmpty(Args) ? "Args: " + Args + "\n" : "") +
            (!string.IsNullOrEmpty(Input) ? "Input: " + Input + "\n" : "") +
            (!string.IsNullOrEmpty(WorkingDir) ? "WorkingDir: " + WorkingDir + "\n" : "") +
            (EnviroVariables != null && EnviroVariables.Count > 0 ? "Environment Variables: " + string.Join(", ", EnviroVariables.Select(a => a.Key + "=" + a.Value)) + "\n" : "");

        string outputFile = "";
        if (OutputChangedCallback != null)
        {
            outputFile = Path.GetTempFileName();
            Args = "/C \"\"" + FilePath + "\" " + Args + "\" >" + outputFile;
            FilePath = "cmd.exe";
        }

        var startInfo = (string.IsNullOrEmpty(Args))
            ? new ProcessStartInfo(FilePath)
            : new ProcessStartInfo(FilePath, Args);

        if (!string.IsNullOrEmpty(WorkingDir))
            startInfo.WorkingDirectory = WorkingDir;

        startInfo.UseShellExecute = false;
        startInfo.CreateNoWindow = true;

        if (OutputChangedCallback == null)
        {
            startInfo.RedirectStandardOutput = true;
            startInfo.RedirectStandardError = true;
        }
        else
        {
            startInfo.RedirectStandardOutput = false;
            startInfo.RedirectStandardError = false;
        }

        if (!string.IsNullOrEmpty(Input))
            startInfo.RedirectStandardInput = true;

        if (EnviroVariables != null)
            foreach (KeyValuePair<String, String> entry in EnviroVariables)
                startInfo.EnvironmentVariables.Add(entry.Key, entry.Value);

        var process = new Process();
        process.StartInfo = startInfo;
        if (process.Start())
        {
            if (Trace)
                Log.Debug("Running external process with the following parameters:\n" + processInfo);

            try
            {
                if (!string.IsNullOrEmpty(Input))
                {
                    process.StandardInput.Write(Input);
                    process.StandardInput.Close();
                }
                var standardError = "";
                var standardOutput = "";
                int exitCode = 0;

                Thread errorReadThread;
                Thread outputReadThread;

                if (OutputChangedCallback == null)
                {
                    errorReadThread = new Thread(new ThreadStart(() => { standardError = process.StandardError.ReadToEnd(); }));
                    outputReadThread = new Thread(new ThreadStart(() => { standardOutput = process.StandardOutput.ReadToEnd(); }));
                }
                else
                {
                    errorReadThread = new Thread(new ThreadStart(() => { }));
                    outputReadThread = new Thread(new ThreadStart(() =>
                    {
                        long len = 0;
                        while (!process.HasExited)
                        {
                            if (File.Exists(outputFile))
                            {
                                var info = new FileInfo(outputFile);
                                if (info.Length != len)
                                {
                                    var content = new StreamReader(File.Open(outputFile, FileMode.Open, FileAccess.Read, FileShare.ReadWrite)).ReadToEnd();
                                    var newContent = content.Substring((int)len, (int)(info.Length - len));
                                    len = info.Length;
                                    OutputChangedCallback.Invoke(content, newContent);
                                }
                            }
                            Thread.Sleep(10);
                        }
                    }));
                }

                errorReadThread.Start();
                outputReadThread.Start();

                var sw = Stopwatch.StartNew();
                bool timedOut = false;
                try
                {
                    while (errorReadThread.IsAlive || outputReadThread.IsAlive)
                    {
                        Thread.Sleep(50);
                        if (WaitTime > 0 && sw.ElapsedMilliseconds > WaitTime)
                        {
                            if (errorReadThread.IsAlive) errorReadThread.Abort();
                            if (outputReadThread.IsAlive) outputReadThread.Abort();
                            timedOut = true;
                            break;
                        }
                    }

                    if (!process.HasExited)
                        process.Kill();

                    if (timedOut)
                        throw new TimeoutException("Timeout occurred during execution of an external process.\n" + processInfo + "Standard Output: " + standardOutput + "\nStandard Error: " + standardError);

                    exitCode = process.ExitCode;
                }
                finally
                {
                    sw.Stop();
                    process.Close();
                    process.Dispose();
                }

                if (ThrowError && exitCode != ValidExitCode)
                    throw new Exception("An error was returned from the execution of an external process.\n" + processInfo + "Exit Code: " + exitCode + "\nStandard Output: " + standardOutput + "\nStandard Error: " + standardError);

                if (Trace)
                    Log.Debug("Process Exited with the following values:\nExit Code: {0}\nStandard Output: {1}\nStandard Error: {2}", exitCode, standardOutput, standardError);

                return standardOutput;
            }
            finally
            {
                FileUtilities.AttemptToDeleteFiles(new string[] { outputFile });
            }
        }
        else
            throw new Exception("The process failed to start.\n" + processInfo);
    }