Well, since bean flours also have some starch in them, and pretty much every vegetable I know does too, I'm guessing you can't handle large concentrations of starch, like corn, tapioca, or potato starch as is often added to bread recipes.
Anyway, there are certainly flours which are not from grains, which work well for breads. One is buckwheat, which is not actually a grain (no relation to wheat either). Other flours that work are from all sorts of different things, from coconut, to sweet potatoes, and even peas.
Here's a list of some flours:
http://www.barryfarm.com/flours.htmAs
Takala points out, you will certainly need a binder of some sort. I use guar gum, which is a fiber from the guar bean. So that seems to fit your criteria.
In place of egg yokes, try lecithin. It does not have to be from soy, though I've read posts from members who can eat lecithin from soy, but not soy itself. If you do a Google search for sunflower lecithin, you will find a number of sources for it.
I don't use yeast either, and while the texture isn't the same as a yeast bread, it's good enough for me. I grew up on whole grain breads, so I never liked the garbage white bread. That alone makes it easier to accept the kind of breads I've been able to make.