Soldiers used their haversack to carry all their worldly goods: toothbrush, comb, harmonica, writing materials, wallet, matches, photos of loved ones, eating utensils, extra food, bible, etc. Reenactors use it to store the same kinds of things, as well as to hide their 20th century articles: car keys, sunblock, kleenex, pills, etc. It's really the soldier's "purse" and soldiers get a new appreciation for ladies' ability to find anything in their purses!
Haversacks come in all shapes, sizes and colors. Usually, though, they are either natural linen (shown above) or linen painted black ("tarred"). If you are planning to occasionally defect to the Federal side, get a tarred haversack so you will fit in there. It has the added benefit of being (sort of) waterproof. Get one with an inner bag - it keeps your things clean and, when emptied out and stuffed with a spare shirt, makes a swell pillow!
Don't let your haversack hang down too low off your shoulder. Research seems to indicate that soldiers wore them relatively high - perhaps centered on the belt line. This facilitated marching for long distances since the sack was less likely to bang against the leg. Shorten the strap by cutting it and resewing it. Knotting it up on the shoulder is considered "farbie."
C & D Jarnigan & Co. (2000 prices)
#306 CS White Haversack with Inner Bag $18.95 #304 US Regulation Haversack with Inner Bag (Tarred) $29.95
If you would like to get something even more authentic, Don Smith makes excellent hand-sewn haversacks:
Trans Mississippi Depot Co.
718 E. Alvarado Ln.
Cottonwood, AZ, 86326