Despite their many similarities, Brandom flags one point of disagreement with McDowell:
I do not see that we need – either in epistemology or, more important, in semantics – to appeal to any intermediaries between perceptual facts and reports of them that are noninferentially elicited by the exercise of reliable differential responsive dispositions. There are, of course, many causal intermediaries, since the noninferential observation report is a propositionally contentful commitment the acknowledgment of which stands at the end of a whole causal chain of reliably covarying events, including a cascade of neurophysiological ones. But I do not see that any of these has any particular conceptual or (therefore) cognitive or semantic significance. (Articulating Reason, 24n7)
Brandom mentions McDowell as opposed to this view (ibid.). Not that McDowell would deny that neurophysiology is operative in perception, but rather that, for him, neurophysiology too resides within the bounds of the conceptual (as Berkeley puts it, the brain too is a “sensible thing,” and since “sensible things are all immediately perceivable; and those things which are immediately perceivable are ideas; and these exist only in the mind” (2.12), it won’t do to “you have been all this while account for ideas by certain motions or impressions of the brain; that is, by some alterations in an idea, whether sensible or imaginable it matters not”; which is absurd (2.20).
Brandom perhaps objects to this thesis as overly baroque, but McDowell would suggest that the alternative leaves the door open to skepticism — we should just maintain that what we know in no way comes short of the fact itself (no part of world lies “between” the knower and the known, not even the causal intermediaries of neurophysiology). But this does point up an important question that idealists (I’m inclined to read McDowell against himself as at his strongest when if he endorses a metaphysical as well as epistemological idealism) need to answer, namely, What’s the purpose of a “scientific” description of reality (i.e., at the biological, molecular, or quantum levels, as opposed to the world of ordinary human experience)?
A thoroughgoing naturalist has an easy answer to this question: the most fundamental scientific explanation is the one that ought to replace all the others, which are just so much “folk psychology.” (If you’re at all inclined to or made anxious by that line of thought, read the chapter on consciousness in David B. Hart’s The Experience of God.) But if you think that we have direct, conceptual knowledge of things (if there’s no gap between mind and world, so that there’s no epistemically causal role for the various intermediaries that compose us), then it’s less clear what all of our cerebral hardware is for (or for that matter, what atoms are, since we don’t know them in the same way as we know the table they compose).
Berkeley’s principal answer to this question (and, perhaps in a different way, Leibniz’s) is “order”: imagined ideas are 1) “faint and indistinct,” 2) dependent on the will, and 3) “not…connected, and of a piece with the preceding and subsequent transactions of our lives” (Three Dialogues 3.44). And Robert Adams argues that, for Leibniz, “Phenomena are real, in a weak sense, if and only if they fit into a single scientifically adequate system of harmonious phenomena of all perceivers. Those phenomena, and only those, that are real in this weaker sense are also real in a fuller sense to the extent that there exist real monads that are appropriately expressed by organic bodies belonging to the system of phenomena that is at least weakly real” (Leibniz: Determinist, Theist, Idealist, 261). Reality and the ordered interconnection of events go together, and neurophysiology and atoms belong to the deep structure (the “hidden necessities,” as James Ross puts it), that render the world mappable in terms of natural laws.