stdOut and stdErr handling ... can't find on StackOverflow...

Posted by: tfabris

stdOut and stdErr handling ... can't find on StackOverflow... - 06/10/2015 01:41

I need my C# program to interact with the stdOut and stdErr of a DOS console application. Reading its text output asynchronously and poking text back into it to answer its prompts.

It works, but I'm having a problem with it on one machine. Not certain why yet because I can't debug the machine that is having the problem. But I have a suspicion I'm wondering about. It worked fine until they fixed a problem with the machine being too slow. Now that the machine is fast enough, it fails.

I think it might be missing the very tip, the very start of the very first prompt of the app's output. Not certain, but I think that might be it.

Pseudo-code of what my program is doing:
Code:
           // Create a process object which will allow us to run the process and control it
            this.process = new Process
            {
                StartInfo = new ProcessStartInfo
                {
                    FileName = exeFileName,
                    WindowStyle = ProcessWindowStyle.Normal,
                    RedirectStandardOutput = true, // So that we can redirect and interpret the console output
                    RedirectStandardInput = true,
                    RedirectStandardError = true,
                    UseShellExecute = false,
                    ErrorDialog = false,
                }
            };

            // Special event handlers to allow the system to gather the text output from the running exe so that we can check its contents
            this.process.OutputDataReceived += new DataReceivedEventHandler(ConsoleOutputReceived);
            this.process.ErrorDataReceived += new DataReceivedEventHandler(ErrorOutputReceived);

            // Initialize the value of the text string that will be updated by the event handlers
            this.OutputText = string.Empty;

            // Launch the process and start its output event handlers
            this.process.Start();
			
			// ***************************** NOTE: CANNOT PLACE THESE NEXT TWO LINES BEFORE PROCESS.START.
			// ***************************** Runtime won't let you. Gets a runtime error that the process
			// ***************************** isn't started yet.
			// ***************************** I wonder if this is the problem: the process starts too quickly
			// ***************************** and the first burst of output is lost and never caught by
			// ***************************** these event handlers on that system?
            this.process.BeginOutputReadLine();
            this.process.BeginErrorReadLine();

			
			// ********** Then there is a loop here which waits for certain strings to be received and writes to the standard input
			// ********** The problem occurs because, on one system, no text ever appears. Well actually one blank line appears.
			// ********** Which is interesting because a blank line is the last line of the output from the program before it waits
			// ********** for user input.
			

      	// *********************************************************************
        // Farther down, these are the separate functions to handle the outputs:
        // *********************************************************************
			
        private void ConsoleOutputReceived(object sender, DataReceivedEventArgs e)
        {
            Console.WriteLine("StdOut: " + e.Data.SafeToStringNeutral());    // So I can see what's happening
            this.OutputText += e.Data.SafeToStringNeutral();
        }

        private void ErrorOutputReceived(object sender, DataReceivedEventArgs e)
        {
            Console.WriteLine("StdErr: " + e.Data.SafeToStringNeutral());    // So I can see what's happening
            this.OutputText += e.Data.SafeToStringNeutral();
        }



What I can't figure out is, in an architecture like the one above, how do I capture the output of those first few characters, the ones before it can get its redirection handlers running?

All the other options I'm seeing use process.StandardOutput.ReadToEnd() or process.StandardOutput.ReadLine(), but those sit there and hang until the process exits, and I can't have that because I need to interact with it.

I've searched the internet, but they always show variants of the Synchronous and Asynchronous examples cited here http://stackoverflow.com/questions/4291912/process-start-how-to-get-the-output ... I need a hybrid approach where I synchronously read the beginning lines without hanging, and then switch to asynchronous after those are all in.

Any ideas?
Posted by: peter

Re: stdOut and stdErr handling ... can't find on StackOverflow... - 06/10/2015 08:25

This chap agrees with you: http://alabaxblog.info/2013/06/redirectstandardoutput-beginoutputreadline-pattern-broken/ sounds like a paper-bag bug in BeginOutputReadLine (as opposed to in the kernel). It's not totally clear to me how his Task Parallel-based solution makes the synchronous calls act like asynchronous ones, but it might work for you to put your "expect-like" behaviour in a version of his ReadStream callback.

Peter
Posted by: Roger

Re: stdOut and stdErr handling ... can't find on StackOverflow... - 06/10/2015 10:24

Cunning plan: start the process suspended; then issue the BeginOutputReadLine; then resume the process...
Posted by: tfabris

Re: stdOut and stdErr handling ... can't find on StackOverflow... - 06/10/2015 15:45

Spectacular, you guys.

Peter, yes, that is exactly the thing I'm talking about. He seems to have a very detailed explanation for his solution not involving deadlocks. I'll see if I can do something similar in my code.

Roger,

That is also a great idea. Do you know how to start the process suspended in C#?
Posted by: canuckInOR

Re: stdOut and stdErr handling ... can't find on StackOverflow... - 06/10/2015 17:57

Originally Posted By: tfabris
Do you know how to start the process suspended in C#?

In C++, it's by providing the CREATE_SUSPENDED flag to CreateProcess(). Looks like C#'s Process() doesn't provide that same access, and you'll have to use PInvoke to call CreateProcess() yourself.
Posted by: tfabris

Re: stdOut and stdErr handling ... can't find on StackOverflow... - 06/10/2015 18:03

Originally Posted By: canuckInOR
In C++, it's by providing the CREATE_SUSPENDED flag to CreateProcess(). Looks like C#'s Process() doesn't provide that same access, and you'll have to use PInvoke to call CreateProcess() yourself.


Yep, I found that, too. That's a bit messier than I wanted it to be, but I understand it. If the other method doesn't work for me, then I'll try that.
Posted by: tfabris

Re: stdOut and stdErr handling ... can't find on StackOverflow... - 06/10/2015 20:22

Originally Posted By: peter
This chap agrees with you: http://alabaxblog.info/2013/06/redirectstandardoutput-beginoutputreadline-pattern-broken/ sounds like a paper-bag bug in BeginOutputReadLine (as opposed to in the kernel). It's not totally clear to me how his Task Parallel-based solution makes the synchronous calls act like asynchronous ones, but it might work for you to put your "expect-like" behaviour in a version of his ReadStream callback.


A variant of this ended up being the correct solution. His code didn't work for me because it still got deadlocks where I didn't want them, so I had to modify it heavily. But basically I took his idea of using Task.Factory.StartNew to create a separate thread to use the synchronous Process.StandardOutput.Read() to do my own reader which did not lose any of the starting text from the program.

Thank you!
Posted by: tfabris

Re: stdOut and stdErr handling ... can't find on StackOverflow... - 06/10/2015 20:27

My final code ended up being something like this.

Code:
            // Create a process object which will allow us to run the Process and control it for the duration of the test
            this.Process = new Process
            {
                StartInfo = new ProcessStartInfo
                {
                    FileName = ExeFile,
                    WindowStyle = ProcessWindowStyle.Normal,
                    RedirectStandardOutput = true, // So that we can redirect and interpret the console output
                    RedirectStandardInput = true,
                    RedirectStandardError = true,
                    UseShellExecute = false,
                    ErrorDialog = false,
                }
            };

            // Initialize the value of the text string that collects the 
            // console output which will be updated by the asynchronous tasks below
            this.OutputText = string.Empty;

            // Launch the process
            this.Process.Start();
            Console.WriteLine("Launched Process at: " + ExeFile);

            // Note: Cannot use this traditional method of asynchronous reads for this exe,
            // because it misses the first burst of output from the exe, causing test to fail:
            //     this.Process.OutputDataReceived += new DataReceivedEventHandler(ConsoleOutputReceived);
            // See this web page for details why:  http://alabaxblog.info/2013/06/redirectstandardoutput-beginoutputreadline-pattern-broken/

            // Can read from stderr text asynchronously though, since we are not waiting and depending upon its contents
            this.Process.ErrorDataReceived += new DataReceivedEventHandler(ErrorOutputReceived);
            this.Process.BeginErrorReadLine();
            Console.WriteLine("Console stderr output has been redirected.");

            // Create a cancelable, asynchronous task to read from the normal stdout stream using
            // the synchronous call, to work around the issue of missing the first burst of output
            // when using the traditional method of redirecting .OutputDataReceived
            var taskCanceler = new CancellationTokenSource();
            CancellationToken cancellationToken = taskCanceler.Token;
            Task outputReader = Task.Factory.StartNew
            (() =>
                {
                    // Keep track of the current line of output so that we can echo it to the console line-by-line
                    string currentLine = string.Empty;

                    // Infinitely loop while inside this special outputReader task
                    while (true)
                    {
                        // Drop out of the task if the task has been canceled
                        cancellationToken.ThrowIfCancellationRequested();

                        // Sleep a tiny amount while inside the loop, to prevent this task from churning the CPU
                        Thread.Sleep(1);

                        // Read from the stdout of the retdirected process output, in synchronous mode,
                        // to work around the issue where using the .OutputDataReceived feature misses
                        // the start of the output of fast-starting tasks; note that we cannot use
                        // "ReadToEnd()" or some such, because that would hang this thread, so we want
                        // to keep reading without stopping
                        int oneChar = this.Process.StandardOutput.Read();

                        // Process the character we read 
                        if (-1 != oneChar)
                        {
                            currentLine += string.Format("{0}", (char)oneChar);
                            this.OutputText += string.Format("{0}", (char)oneChar);
                            // If we get a linefeed, echo the buffered line to the console
                            if (10 == oneChar)
                            {
                                Console.Write("StdOut: " + currentLine);
                                currentLine = string.Empty;
                            }
                        }
                    }
                }
                , cancellationToken
            );
			
			
			// ************* Now down here you can do a nice leisurely loop where you look at the contents of 
			// this.OutputText and act upon it as needed.



            // Now we're done with the output reading thread, we can cancel it now
            taskCanceler.Cancel();


Thanks so much for your help!