SaveFile / LoadFile fails if file contains ANSI chars > x'7F'

Hi Mike,

Our SaveFile and LoadFile methods are very simple and pretty much follow the methods that .NET makes available to us for working with files.

The basic SaveFile just does "StreamWriter writer = File.CreateText(path)" and then outputs the text to that writer.  The other overload calls "File.Create(path)" to create the file and then makes a new StreamWriter that specifies the Encoding you pass, forcing it to use that.

Similarly the basic LoadFile uses File.OpenText and reads the string to the end.  The other overload uses File.OpenRead(path) and creates a StreamReader with the Encoding you pass, forcing it to use that.

If you are looking for more complicated logic, you could create it yourself since our logic is just about 3 lines long for each of those methods.  We don't have anything that scans text for adhering to particular encodings since we are only calling the core I/O methods described above for input/output.  I would be interested in seeing whatever logic you come up with for that, as I'm sure other customers could use it.

In that case you should change your documentation for the LoadFile() method since it doesn't really open a UTF8 file (which is what SaveFile creates) it actually EXPECTS to load a UTF8 file - but if it doesn't find a UTF8 BOM it will load the file as though it were an ANSI file. (effectively corrupting the file if it was not saved with ANSI encoding.)

The way to handle this is to use the following:

        If IsInCodePage() Then
            Document.SaveFile(fileName, Encoding.Default, LineTerminator.CarriageReturnNewline)
        Else
            Document.SaveFile(fileName, Encoding.UTF8, LineTerminator.CarriageReturnNewline)
        End If

    Private Function IsInCodePage() As Boolean
        'Check to see if the text is all within the current code page
        Dim encoder As Encoding = Encoding.GetEncoding(Encoding.Default.WindowsCodePage,
                                                       New EncoderExceptionFallback(),
                                                       New DecoderExceptionFallback())
        Try
            encoder.GetBytes(Document.Text)
            Return True
        Catch
            Return False
        End Try

    End Function

This is rather ineficient though if the document is large.

GetBytes() actually performs the encoding so if the text is entirely within the code page this encoding gets performed twice. Once in my code and a second time in your SaveFile method.
That is why this logic would normally be performed by the controls 'Save' method itself.

I realize I could simply open the output file myself and then write the byte array that was returned, but that would mean that the logic used to save the file in this specific case would look very different from when I save it with a user specified encoding. 

Hi Mike,

The MSDN documentation on File.CreateText(path), which is what we use in the one overload, says:  "Creates or opens a file for writing UTF-8 encoded text."  Based on that, I would have assumed that it would write no UTF8 BOM if the chars 0-127 are only used, but would use a BOM if any other characters were used.

Similarly, the MSDN docs say for File.OpenText(path):  "Opens an existing UTF-8 encoded text file for reading." 

From doing more searching on the web and looking at the StreamWriter source in Reflector, it does appear that the File.CreateText(path) documentation is misleading as it only seems to output UTF8 without a BOM.

As for your custom logic above, why not just wrap it up as an extension method for the Document class?  Then it wouldn't look any different to the callers.

You are correct. The MSDN info is also misleading.

As you say CreateText() writes UTF8 - but without a BOM
and OpenText() reads UTF8  - but only if it starts with a UTF8 BOM.

That means that OpenText() can not [always] correctly open a file that was saved using CreateText().
Which I think is a bug ... but getting Microsoft to fix something is like banging your head against a brick wall.
(I know this from trying for years to get them to fix some of the many bugs in their Data Provider for ODBC )

It would be helpful if you could change your docs to describe what really happens rather than what Microsoft implies in their own docs.

I do use an extension class already so I guess I could look at adding my own method to take care of the issue.

But if you sell your app to a lot of folks in Canada or Europe I suspect many of them will hit this issue also, so it would be very useful if you changed the way that specific SaveFile() method works so that it will work the way people expect a Save/Load pair to work.