::before:nth-child(1) is currently invalid. This is defined in a non-normative note for pseudo-classing in Selectors 4:

Note that, unless otherwise specified in a future specification, pseudo-classes other than the user action pseudo-classes are not valid when compounded to a pseudo-element; so, for example, ::before:first-child is an invalid selector.

There is a resolution to allow more pseudo-classes following a pseudo-element, but :nth-*-child() are not included.

I think pseudo-elements that are not associated to an originating pseudo-element should be invalid before/in :nth-*-child(), since :nth-child(1 of ::before) is equivalent to ::before:nth-child(1).

EDIT: actually no, I think :nth-child(1 of ::before) is equivalent to *:nth-child(1 of *::before), which can match nothing.

:nth-*-child() has been deferred to Selectors 5 in #3760 with the following resolution: defer complex selectors for all of these selectors and have a note in the current level mentioning this is an enhancement we'll get to in the next level. I do not see any mention of this in Selectors 4, maybe because it is already implemented in some browsers? I am not sure there are good use cases for using a pseudo-element as the subject or an argument of :nth-*-child() at the moment though.

I am not sure the concept of a pseudo-element defined with an internal structure will remain in Selectors 5, but I think a pseudo-element that can appear in such pseudo-element would also be valid in :nth-*-child().