Legally? Probably, but that didn’t stop Hitler from invading anyplace else.
There’s a number of reasons why Germany didn’t invade Switzerland. The romanticized version of this is that Switzerland can’t be invaded. While the country may not have been as easy to conquer as, say, Belgium, it could have been taken.

Germany wrote a battleplan for taking Switzerland during the Battle of France. Had the French not surrendered, Germany would have gone through Switzerland to make one long front shared by them and Italy. France surrendered the day after this plan was approved and it was never authorized.
Hitler did not like the Swiss. He called them a “pimple on face of Europe.” The biggest problem Hitler had with them was that it was a largely Germanic country that favored French culture.
The main reason that Germany did not invade Switzerland is due to Swiss collaboration. The Swiss turned thousands of refugees away from their borders and into the hands of the Germans.
During the course of the war, they officially accepted 50,000 refugees and unofficially another 60,000. Yet their early rejection of these refugees helped their relations with Germany. The Swiss helped design German passports to better identify Jews as they tried to cross the border.
Switzerland produced a great number of supplies for the Germans. What’s more, these supplies were made without the constant threat of Allied bombers.
The Swiss were also the bankers for Germany. They bought most of the gold that was captured, much of it from the Jews. Germany could then turn the money into weapons.
An additional factor in leaving Switzerland alone was that there was no strategic importance in capturing the country. Like Sweden, the Swiss posed no threat to Germany, nor could these countries be used as any sort of staging ground for an invasion.
Contrary to public belief, Hitler was not a madman before mid 1943. He knew that sending troops into Switzerland and Sweden would result in a German victory there, but for what purpose? On top of losses during the conflicts, he has to garrison the countries. These are troops that could be used elsewhere.
One might suggest that the Swiss collaborated out of fear, and that might have a lot to due with their actions. On the other hand, a lot of Swiss people, especially in the government, were making a lot of money collaborating with the Germans.
I want to address something that’s been mentioned numerous times in the comments:
Switzerland is not a mighty fortress surrounded by mountains that are impossible to cross. All of the “victory points” for the country are on relatively flat, easily accessible ground to the north.
The Swiss army might flee to the mountains. If so, the Germans would be more than happy to just let them starve there while occasionally dropping bombs on them.
However the legal status existed Hitler continued his military attacks against other countries.

The reasons for Germany not attacking Switzerland were numerous. Despite popular belief Switzerland remained an unconquerable fortress despite the fact that it was possible for Germany to take it. The military operation required greater effort than Belgium but Germany demonstrated enough capability to conquer it.
Germany already possessed a battle plan to occupy Switzerland throughout the Battle of France. Germany planned to enter Switzerland to extend their battle presence with Italy until France surrendered thus making an invasion unnecessary. Soon after Germany authorized the invasion plan France surrendered and thus the invasion became obsolete.
Hitler didn’t like the Swiss. In his eyes Switzerland served as a small blemish across the European continent. The German origins of Swiss society brought frustration to him because France remained influential in their cultural landscape.
Swiss collaboration was the key factor why Germany never launched an invasion. Thousands of refugees received a rejection from Swiss authorities due to which many ended up in German territory. The Swiss authorities admitted just 50,000 refugees yet unofficial entry of 60,000 more refugees occurred.
Switzerland initially rejected accepting numerous refugees during this stage therefore maintaining friendly ties with Germany. Alongside Switzerland’s official staff the country worked together with Germany to design passports because this arrangement served as a detection system against Jewish escape attempts.
Switzerland transported necessary raw materials to Germany because they operated free from Allied bombing attacks. Swiss banks received German funds that contained gold taken from Jewish population which Germany used to purchase weapons.
Germany did not commit invasion because Switzerland lacked vital military value for their objectives. Sweden shared similar characteristics with Switzerland because it posed no threat to any enemy force and had no purpose for launching military attacks.
Multiple people view Switzerland as an impenetrable stronghold yet this notion remains false. The majority of German key cities together with infrastructure existed in the northern parts of the country where direct assaults were straightforward. German forces would either attack the Swiss soldiers with aerial bombs or prevent their supply routes to force the army members to die of starvation.
Before mid-1943, Hitler was not reckless. He planned his moves carefully. Germany could possibly triumph in both Sweden and Switzerland yet making those conquests proved impractical. German military resources needed for other battles would have been diverted by an occupation of Switzerland along with its subsequent management.
The Swiss government worked with Germany through fear while a large number of Swiss government agents together with business leaders accumulated significant profits from their German collaboration.