This is a game of resources, so buildings that give you resources are important.
Farms give you Food and Mines give you Iron. Farms and Mines aren't the only way to get resources, though. The Watchtower also generates Iron and Food, as does winning Raids, completing Stages and most Quests, and for completing Wanted Missions.
Still, at early levels, Farms should be the first upgraded to match your Stronghold, then Mines. This is true until your Stronghold is about level 9 and then use your own discretion. If you feel like you're constantly full, then don't upgrade anymore.