Resources on "Physical Computing" and Arduino microcontrollers
"Physical Computing" is an engineering discipline and an artform where you build a computer equipped with sensors and capable of interacting with its environment, and program it to perform a well-defined task. It is used everywhere from factory floors[1] to art galleries[2], even in fashion[3].
Buying and installing an Arduino
The cheapest option would be to buy directly from a manufacturer. Just search for "Arduino" on direct-from-manufacturer websites like aliexpress.com (as low as $3 per board). For more expensive, but better supported hardware you can look at Sparkfun.com or Adafruit.com - both of those websites serve as online electronics shops and as great sources of learning materials. Their Arduino boards are more expensive (around $20), but also come with more professional support materials.
You will need the official free (open source) Arduino software to program your board.
If you choose the cheaper Arduinos, you might need to install additional drivers so that your computer can talk to the boards. Like the FTDI chip driver and the CH340 chip driver.
Learning the basics
Your most important resource after installing the software is the Examples folder. Go to File->Examples and you will find a very rich library of small, well commented sketches that you can use as the basis for more complicated projects. The second resource you should keep in mind, especially if you want to learn the Arduino language, is the Reference on the official website. After you are comfortable with those two, you can start downloading additional libraries that permit fancy new sensors to work with the Arduino. Or, if you want, you can delve deeper in the C++ programming language (the one that Arduino is based on) and start writing your own libraries.
The sky is the limit
You can make your LEDs blink and fade. You can measure voltage from sensors. You can control motors, pumps, and lights. You can even connect to the internet with a properly equipped Arduino and record your data online as it is being measured. You can even take it a step further and combine these simple building blocks together to build automatons and art of unimaginable complexity. Our resources page has Arduino projects you can try (each page comes with a list of materials and sources where you can purchase them cheaply). Alternatively, you can check out the "Arduino" tagged articles.