Set aside time. Building Xvfb is likely to take all day.
Much of what I know about Xvfb, including the reasons you might want to use it, appears in our article for SunWorld Online.
While I'd found glitches in generation of Xvfb under Solaris, FreeBSD, Linux, and Tru64 Unix during fall of 1999, I had a couple of "smooth sailing" reports in March 2000: RedHat 6.1 customer Earl Stutes says it's "a standard part of the build"; and Steve Christensen cleanly generated and tested a version for Solaris 2.5 that he'll soon make available on his Freeware for Solaris site.
It's said that backports to XllR5 are available. I haven't worked with them.