Core Quests for Garrison levels and buildings.
A level 1 garrison gives one small and one large plot.
pick up Establish Your Garrison
A level 2 garrison gives one small, one medium, one large plot, plus unlocks the mine, herb and fishing plots.
Go to Wor'gol.
The idea is to do just these quests to gain access to level 2 buildings.
I plan on skipping the side quests with all my alts. This should be do-able by level 92.
At this stage you can have 2 crafting plots.
My choices would be the barracks (large), 2 primary crafts (2 small), a lumber mill (medium).
Options are salvage yard (small), trading post (medium).
I suggest waiting until level 100 before trying the other building. Level up the followers, go look for followers to add to your list.
Once you are ready, only then try the Warmill, Inn, Stables. ( I wasted a lot of resources and gold chopping and changing!)
This post is really for my own reference, to help with my numerous alts.