That is correct, nested categories are supported, but cannot be used out of the box when populating the PropertyGrid via SelectedObject(s)/Properties. Properties generated from these items utilize the CategoryAttribute, which takes a single string. You can achieve nested categories by populating PropertyGrid.Items/ItemsSource directly (as you show above), or by implementing a custom DataFactory. The latter of which is responsible for converting SelectedObject(s), Properties, and various other settings into a list of "data accessors".
We do have a TODO item down for supporting nested categories using the CategoryAttribute (using a special syntax like "One\Two"), but it is currently a lower priority item. I've added this post to the notes, so that you will be notified when it's completed.
The reason the validation errors are not presented has to do with your setup. You effectively have three object TextBox, PropertyGridPropertyItem, and I (your data object). TextBox.Text is bound to PropertyGridPropertyItem.Value, which is in turn bound to I.A.
By throwing an exception in the setter for I.A, the binding between PropertyGridPropertyItem.Value and I.A will be marked as invalid. The binding between the TextBox.Text and PropertyGridPropertyItem.Value is still considered valid, and this is the binding that WPF uses to determine if it should render the "error template" or the red box.
You might be able to use binding groups to link the two bindings, so the red box is shown. More info on that can be found here