Exception after clearing LexicalPatternGroups - SyntaxEditor for Windows Forms Forum

Posted 18 years ago by Matt Whitfield

Version: 4.0.0238

Platform: .NET 2.0

Environment: Windows XP (32-bit)

Ok, I have some code which makes my application fall over with a NullReferenceException

I have replicated this in the Sample application - I replaced the code for the Exit menu item in MainForm with the following:

foreach (object o in editor.Document.Language.LexicalStates)
{
if (o is DynamicLexicalState)
{
DynamicLexicalState dls = (DynamicLexicalState)o;
foreach (LexicalPatternGroup lpg in dls.LexicalPatternGroups)
{
lpg.Clear();
}
}
}

If you select a normal language like C# (not the .NET Add-on version), then cause that code to run, you get a NullReferenceException raised on the .ShowDialog call in the LauncherForm.

I get exactly the same in my application. In both cases, the stack trace is as follows:

--------------------
at a.e()
at ActiproSoftware.SyntaxEditor.MergableToken.get_Language()
at ActiproSoftware.SyntaxEditor.SyntaxLanguage.a(IToken A_0)
at ActiproSoftware.SyntaxEditor.EditorView.GetCursor(Point point)
at ActiproSoftware.WinUICore.UIControl.OnMouseMove(MouseEventArgs e)
at ActiproSoftware.SyntaxEditor.SyntaxEditor.OnMouseMove(MouseEventArgs e)
at System.Windows.Forms.Control.WmMouseMove(Message& m)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
at ActiproSoftware.SyntaxEditor.SyntaxEditor.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.DebuggableCallback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
at System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32 dwComponentID, Int32 reason, Int32 pvLoopData)
at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)
at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)
at System.Windows.Forms.Form.ShowDialog(IWin32Window owner)
at TestApplication.LauncherForm.launchSampleButton_Click(Object sender, EventArgs e) in C:\Program Files\Actipro Software\SyntaxEditor\v4.0.0238\TestApplication-CSharp.Net20\LauncherForm.cs:line 260
at System.Windows.Forms.Control.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnClick(EventArgs e)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.DebuggableCallback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
at System.Windows.Forms.UnsafeNativeMethods.SendMessage(HandleRef hWnd, Int32 msg, IntPtr wParam, IntPtr lParam)
at System.Windows.Forms.Control.SendMessage(Int32 msg, IntPtr wparam, IntPtr lparam)
at System.Windows.Forms.Control.ReflectMessageInternal(IntPtr hWnd, Message& m)
at System.Windows.Forms.Control.WmCommand(Message& m)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
at System.Windows.Forms.ContainerControl.WndProc(Message& m)
at System.Windows.Forms.Form.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.DebuggableCallback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
at System.Windows.Forms.UnsafeNativeMethods.CallWindowProc(IntPtr wndProc, IntPtr hWnd, Int32 msg, IntPtr wParam, IntPtr lParam)
at System.Windows.Forms.NativeWindow.DefWndProc(Message& m)
at System.Windows.Forms.Control.DefWndProc(Message& m)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ButtonBase.WndProc(Message& m)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.DebuggableCallback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
at System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32 dwComponentID, Int32 reason, Int32 pvLoopData)
at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)
at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)
at System.Windows.Forms.Application.Run(Form mainForm)
at TestApplication.Program.Main() in C:\Program Files\Actipro Software\SyntaxEditor\v4.0.0238\TestApplication-CSharp.Net20\Program.cs:line 22
at System.AppDomain.nExecuteAssembly(Assembly assembly, String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
--------------------

What am I doing wrong?

Comments (13)

Posted 18 years ago by Actipro Software Support - Cleveland, OH, USA

Matt,

First, any time you make a change to a language you need to wrap the change code with "language.IsUpdating = true;" and "language.IsUpdating = false;" calls. First add those and see if it helps. If not we can look into it further.

Actipro Software Support

Posted 18 years ago by Matt Whitfield

Ok thanks, that clears it up in the sample application...

However I had tried the same thing in my test application, where it didn't seem to work.

So I will look into it further and post back if I can find any more information about it.

However, is it by design that the control throws a NullReferenceException under these circumstances, or will this be fixed in a future revision?

Thanks again

Posted 18 years ago by Actipro Software Support - Cleveland, OH, USA

Hi Matt,

If you don't wrap changes with the IsUpdating, the language won't know to refresh itself and the tokens and could lead to random problems like this. If you are doing IsUpdating and still have problems then that would be a bug. So if you do end up running into that scenario, please give us repro steps or email over a project.

Actipro Software Support

Posted 18 years ago by Matt Whitfield

Ok

Well I think there may be a bug then - but it's worth noting that there is some separate thread activity between the sets - here is the relevant portion of the code:


        public void SetConnection(DataBaseConnectionDetails dbcd)
        {
            shtb.Document.Language.AutomaticOutliningBehavior = AutomaticOutliningBehavior.PostSemanticParse;
                
            dbcd_ = dbcd;

            fke_.Clear();

            shtb.Document.LexicalParsingEnabled = false;
            shtb.Document.SemanticParsingEnabled = false;
            shtb.Document.Language.IsUpdating = true;

            // create worker thread instance
            m_ConnectionWorkerThread =
                new Thread(new ThreadStart(SetConnectionThread));

            m_ConnectionWorkerThread.Name = "Set Connection Thread";
            m_ConnectionWorkerThread.Start();
        }

        private void SetConnectionThread()
        {
            List<ObjectCacheEntry> oceList = new List<ObjectCacheEntry>();

            if (dbcd_ != null)
            {
                using (SqlConnection connection = new SqlConnection(dbcd_.ConnectionString))
                {
                    connection.Open();

                    string strSQL = Atlantis.SQLEditor.Properties.Resources.ObjectCacheQuery;
                    using (SqlCommand cmd = new SqlCommand(strSQL, connection))
                    {
                        cmd.CommandType = CommandType.Text;
                        cmd.CommandTimeout = 1800;

                        using (SqlDataReader reader = cmd.ExecuteReader())
                        {
                            string Owner = "";
                            string ObjectName = "";
                            string ColName = "";
                            string ColType = "";
                            SQLWordType Type = SQLWordType.Word;

                            while (reader.Read())
                            {
                                // read owner too
                                Owner = reader.GetString(0);
                                ObjectName = reader.GetString(1);
                                Type = (SQLWordType)reader.GetInt32(2);
                                ColName = reader.GetString(3);
                                ColType = reader.GetString(4);

                                oceList.Add(new ObjectCacheEntry(Owner, ObjectName, ColName, ColType, Type));
                            }
                        }
                    }
                }
            }
            fke_.PrepareList();

            SetColourScheme(dsl_);

            ObjectReadCompleteDelegate d = new ObjectReadCompleteDelegate(ObjectReadComplete);
            try
            {
                this.Invoke(d, new object[] { oceList });
            }
            catch
            { }
            
        }

        void ObjectReadComplete(List<ObjectCacheEntry> oceList)
        {
            if (ObjectReadStatus != null)
            {
                ObjectReadStatus(this, new ObjectReadResults(false));
            }

            DynamicLexicalState dls = (DynamicLexicalState)dsl_.LexicalStates["DefaultState"];
            int SysTableTokenIndex = -1;
            int SysProcTokenIndex = -1;
            int UserSysProcTokenIndex = -1;
            int ProgrammableObjectIndex = -1;
            int StorageObjectIndex = -1;
            int UserSchemaIndex = -1;
            int DatabaseIndex = -1;

            foreach (LexicalPatternGroup lpx in dls.LexicalPatternGroups)
            {
                if (lpx.TokenKey == "SystemTableToken")
                {
                    SysTableTokenIndex = dls.LexicalPatternGroups.IndexOf(lpx);
                    //lpx.Clear();
                }
                if (lpx.TokenKey == "SystemProcedureToken")
                {
                    SysProcTokenIndex = dls.LexicalPatternGroups.IndexOf(lpx);
                    //lpx.Clear();
                }
                if (lpx.TokenKey == "UserSystemProcedureToken")
                {
                    UserSysProcTokenIndex = dls.LexicalPatternGroups.IndexOf(lpx);
                    //lpx.Clear();
                }
                if (lpx.TokenKey == "ProgrammableObjectToken")
                {
                    ProgrammableObjectIndex = dls.LexicalPatternGroups.IndexOf(lpx);
                    //lpx.Clear();
                }
                if (lpx.TokenKey == "StorageObjectToken")
                {
                    StorageObjectIndex = dls.LexicalPatternGroups.IndexOf(lpx);
                    //lpx.Clear();
                }
                if (lpx.TokenKey == "DatabaseToken")
                {
                    DatabaseIndex = dls.LexicalPatternGroups.IndexOf(lpx);
                    //lpx.Clear();
                }
                if (lpx.TokenKey == "UserSchemaToken")
                {
                    UserSchemaIndex = dls.LexicalPatternGroups.IndexOf(lpx);
                    //lpx.Clear();
                }
            }

            InitialiseDefaults();

            string Owner = "";
            string ObjectName = "";
            string ColName = "";
            string ColType = "";
            SQLWordType Type = SQLWordType.Word;

            Keyword kw = null;
            string lastOwner = "";
            string lastObjectName = "";

            foreach (ObjectCacheEntry oce in oceList)
            {
                // read owner too
                Owner = oce.Owner;
                ObjectName = oce.ObjectName;
                Type = oce.Type;
                ColName = oce.ColName;
                ColType = oce.ColType;

                Color HighlightColour = Color.Black;

                if ((lastOwner != Owner) ||
                    (lastObjectName != ObjectName))
                {
                    if (kw != null)
                    {
                        switch (kw.WordType)
                        {
                            case SQLWordType.SystemTable:
                            {
                                if (SysTableTokenIndex >= 0)
                                {
                                    LexicalPatternGroup lpg = (LexicalPatternGroup)dls.LexicalPatternGroups[SysTableTokenIndex];
                                    lpg.Add(new LexicalPattern(kw.Word));
                                }
                            } break;
                            case SQLWordType.SystemProgrammableObject:
                            {
                                if (SysProcTokenIndex >= 0)
                                {
                                    LexicalPatternGroup lpg = (LexicalPatternGroup)dls.LexicalPatternGroups[SysProcTokenIndex];
                                    lpg.Add(new LexicalPattern(kw.Word));
                                }
                            } break;
                            case SQLWordType.UserSystemProgrammableObject:
                            {
                                if (UserSysProcTokenIndex >= 0)
                                {
                                    LexicalPatternGroup lpg = (LexicalPatternGroup)dls.LexicalPatternGroups[UserSysProcTokenIndex];
                                    lpg.Add(new LexicalPattern(kw.Word));
                                }
                            } break;
                            case SQLWordType.StoredProcedure:
                            case SQLWordType.UserDefinedFunction:
                            case SQLWordType.UserDefinedTableFunction:
                            {
                                if (ProgrammableObjectIndex >= 0)
                                {
                                    LexicalPatternGroup lpg = (LexicalPatternGroup)dls.LexicalPatternGroups[ProgrammableObjectIndex];
                                    lpg.Add(new LexicalPattern(kw.Word));
                                }
                            } break;
                            case SQLWordType.Database:
                            {
                                if (DatabaseIndex >= 0)
                                {
                                    LexicalPatternGroup lpg = (LexicalPatternGroup)dls.LexicalPatternGroups[DatabaseIndex];
                                    lpg.Add(new LexicalPattern(kw.Word));
                                }
                            } break;
                            case SQLWordType.Table:
                            case SQLWordType.View:
                            {
                                if (StorageObjectIndex >= 0)
                                {
                                    LexicalPatternGroup lpg = (LexicalPatternGroup)dls.LexicalPatternGroups[StorageObjectIndex];
                                    lpg.Add(new LexicalPattern(kw.Word));
                                }
                            } break;
                        }
                        kw.CreateNameAndDescription();
                        fke_.AddKeyword(kw);
                    }
                    kw = new Keyword(Owner, ObjectName, Type);
                    if (Owner != "")
                    {
                        if (fke_.FindKeyword(Owner) == null)
                        {
                            fke_.AddKeyword(new Keyword(Owner, SQLWordType.UserSchema));
                            if (UserSchemaIndex >= 0)
                            {
                                LexicalPatternGroup lpg = (LexicalPatternGroup)dls.LexicalPatternGroups[UserSchemaIndex];
                                lpg.Add(new LexicalPattern(Owner));
                            }
                        }
                    }
                    lastObjectName = ObjectName;
                    lastOwner = Owner;
                }
                if (ColName.StartsWith("@"))
                {
                    kw.TableObject.Parameters.Add(new StorageColumn(ColName, ColType));
                }
                else
                {
                    kw.TableObject.Columns.Add(new StorageColumn(ColName, ColType));
                }
            }
            if (kw != null)
            {
                switch (kw.WordType)
                {
                    case SQLWordType.SystemTable:
                    {
                        if (SysTableTokenIndex >= 0)
                        {
                            LexicalPatternGroup lpg = (LexicalPatternGroup)dls.LexicalPatternGroups[SysTableTokenIndex];
                            lpg.Add(new LexicalPattern(kw.Word));
                        }
                    } break;
                    case SQLWordType.SystemProgrammableObject:
                    {
                        if (SysProcTokenIndex >= 0)
                        {
                            LexicalPatternGroup lpg = (LexicalPatternGroup)dls.LexicalPatternGroups[SysProcTokenIndex];
                            lpg.Add(new LexicalPattern(kw.Word));
                        }
                    } break;
                    case SQLWordType.UserSystemProgrammableObject:
                    {
                        if (UserSysProcTokenIndex >= 0)
                        {
                            LexicalPatternGroup lpg = (LexicalPatternGroup)dls.LexicalPatternGroups[UserSysProcTokenIndex];
                            lpg.Add(new LexicalPattern(kw.Word));
                        }
                    } break;
                    case SQLWordType.StoredProcedure:
                    case SQLWordType.UserDefinedFunction:
                    case SQLWordType.UserDefinedTableFunction:
                    {
                        if (ProgrammableObjectIndex >= 0)
                        {
                            LexicalPatternGroup lpg = (LexicalPatternGroup)dls.LexicalPatternGroups[ProgrammableObjectIndex];
                            lpg.Add(new LexicalPattern(kw.Word));
                        }
                    } break;
                    case SQLWordType.Database:
                    {
                        if (DatabaseIndex >= 0)
                        {
                            LexicalPatternGroup lpg = (LexicalPatternGroup)dls.LexicalPatternGroups[DatabaseIndex];
                            lpg.Add(new LexicalPattern(kw.Word));
                        }
                    } break;
                    case SQLWordType.Table:
                    case SQLWordType.View:
                    {
                        if (StorageObjectIndex >= 0)
                        {
                            LexicalPatternGroup lpg = (LexicalPatternGroup)dls.LexicalPatternGroups[StorageObjectIndex];
                            lpg.Add(new LexicalPattern(kw.Word));
                        }
                    } break;
                    case SQLWordType.UserSchema:
                    {
                        if (UserSchemaIndex >= 0)
                        {
                            LexicalPatternGroup lpg = (LexicalPatternGroup)dls.LexicalPatternGroups[UserSchemaIndex];
                            lpg.Add(new LexicalPattern(kw.Word));
                        }
                    } break;
                }
                kw.CreateNameAndDescription();
                fke_.AddKeyword(kw);
            }
            fke_.PrepareList();

            SetColourScheme(dsl_);

            shtb.Document.LexicalParsingEnabled = true;
            shtb.Document.SemanticParsingEnabled = true;
            shtb.Document.Language.IsUpdating = false;
            
            if (ObjectReadStatus != null)
            {
                ObjectReadStatus(this, new ObjectReadResults(true));
            }
        }

Please let me know if you'd still like the whole project. I did get around the bug by just re-loading the DSL from the XML and then adding the tokens I needed again - this didn't seem to have much of a negative performance impact so i figured i would stick with it.

Thanks for your help.

Posted 18 years ago by Actipro Software Support - Cleveland, OH, USA

Yes the multi-threading might be causing the issues because I'm not aware
of any problems with run-time language manipulation when in a single-thread mode.
I would recommend perhaps building up new instances of LexicalPatternGroups in your
worker code and then once complete, doing simple pseudocode like:

Create new pattern groups with all your new patterns... don't attach them to the language yet.
language.IsUpdating = true;
Remove existing lexical pattern groups that will be replaced
Add in new pre-built pattern groups with your updated patterns
language.IsUpdating = false;

This way you keep the actual time between IsUpdating to a minimum.
Also, make sure you do it in the main UI thread.

Actipro Software Support

Posted 18 years ago by Adam Dickinson

I've gotten the exact same assert in version 4.0.239, but unfortunately, I don't know how to reproduce it. I was just typing away in the editor window when suddenly it went bank, then was replaced by a big red box with a red X through it before Microsoft's crash handler dialog box came up. We're using the SemanticParsingService with our own custom language (aka non-dynamic). We're using the SemanticParseDataChanged event to do some work with the AST nodes that were created. Could we be in a similar situation where this event handler is taking too long and ends up causing the crash?

Posted 18 years ago by Actipro Software Support - Cleveland, OH, USA

Can you describe how your custom token is and are you able to break at the point this happens (like maybe turn on break on exceptions in VS) to be able to look at the token that is blowing up? I'd be interested in what the various properties of it are at the time it occurs. Perhaps e-mail us this info and we can discuss more.

Actipro Software Support

Posted 18 years ago by Adam Dickinson

Because of the seemingly-randomness of the bug, I'm not sure how long it will take me to capture it in the debugger. Let's hope I get lucky sooner rather than later.

My custom token is derived from MergableToken. I've been meaning to change it to be derived from NonMergableToken, but haven't had the time. Anyways, the special items I've added are as follows:


        private byte m_overrideID = (byte)(MyTokenID.Invalid | 0x80);

        /// <summary>
        /// Gets the ID assigned to the token.
        /// </summary>
        /// <value>
        /// The ID assigned to the token.
        /// </value>
        public override int ID
        {
            get
            {
                byte enabled = (byte)(m_overrideID & 0x80);
                byte overrideID = (byte)(m_overrideID & 0x3f);
                if ( enabled == 0x80 )
                {
                    if ( (int)overrideID != MyTokenID.Invalid )
                    {
                        return (int)overrideID;
                    }

                    return base.ID;
                }
                else
                {
                    return MyTokenID.Disabled;
                }
            }
        }

        /// <summary>
        /// Overrides the original ID of this token.  Setting back to MyTokenID.Invalid will cancel the override.
        /// </summary>
        /// <param name="tokenID">The new ID of the token.</param>
        public void SetOverrideID( int tokenID )
        {
            m_overrideID = (byte)((byte)tokenID | (m_overrideID & 0x80));

            this.SetFlag( LexicalParseFlags.ScopeStart, this.IsPairedStart );
            this.SetFlag( LexicalParseFlags.ScopeEnd, this.IsPairedEnd );
        }

        /// <summary>
        /// Overrides the original and the override ID to make this token Disabled
        /// </summary>
        /// <param name="enable"></param>
        public void SetEnabled( bool enable )
        {
            m_overrideID = (byte)((m_overrideID & 0x3f) | (enable ? 0x80 : 0x00));
        }

The purpose of the override and enabled is to 1) gray out tokens that are in a #ifdef block that evaluates to false 2) or otherwise, change the color of certain important keywords in the language. These special keywords are defined by the user, and I collect them after the Semantic Parse. I have a step where I go back through the Document's Token collection, calling SetOverrideID and SetEnabled depending on the situation.


        public void editorDocument_SemanticParseDataChanged( object sender, EventArgs e )
        {
            Document doc = sender as Document;
            if ( (doc != null) && (doc.SemanticParseData != null) )
            {
                // Add identifiers to the cache
                UpdateIdentifierCache( doc.SemanticParseData as CompilationUnit, doc.LanguageData as ISemanticParseDataTarget );
             
                // Fixup the identifier tokens so they get colorized properly
                FixupIdentifierTokenIDs( doc );

                // This line will force a refresh so the new colors appear.
                doc.InvalidatePaint();
            }
        }

Inside FixupIdentifierTokenIDs, I have a try-catch because, as oftentimes occurs, the Document Token collection gets modified by the main thread because of edits made.

Posted 18 years ago by Actipro Software Support - Cleveland, OH, USA

Ok try this for me... around your code that occurs in a separate thread,
try doing a lock on the Document.SyncRoot. But do a lot of testing
after trying that to ensure that a deadlock situation doesn't arise
because of it.

Actipro Software Support

Posted 18 years ago by Matt Whitfield

erm, it's just a suggestion - but have you tried using something similar to the code I pasted above? I have several pre-defined groups in my SQL language to which I add tokens later. Maybe you could do that, and load the tokens at language load time, rather than having to mess about with the semantic parsing?

sorry if it's not helpful! :)

Posted 18 years ago by Adam Dickinson

Thanks, but no that won't work for us. The Identifier token IDs that get reassigned can only be done so after a Semantic Parse. For example, we differentiate between a local variable Identifier and a global variable Identifier, assigning them different token IDs, in part so we can colorize them differently. During a Lexical Parse where you would normally assign the token IDs, I don't always know what kind of Identifier I've found. I need that extra contextual information.

Posted 18 years ago by Matt Whitfield

yeah i figured you were probably in that sort of situation

Just thought i'd chip in in case it was one of those 'can't see the wood for the trees' situations :)

Posted 18 years ago by Adam Dickinson

I added the lock, and so far have not experienced any adverse affects. But truth be told, I was never able to get the original Exception a second time even before adding the lock. So, thanks, I think =D

The latest build of this product (v24.1.1) was released 6 days ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.